Helpdesk Automation Profiler

Analyze and evaluate automation potential for support tickets

Reset Application

Are you sure you want to reset the application? This will delete all test runs and results. This action cannot be undone.

Select Models

Loading models...

(( operator ))

Test Configuration

Evaluate support ticket resolution workflows, tool usage, and scoring.

Standard mode for testing exact matches or content validation.

Choose JSON test suite files... (( testSuiteFiles.length )) file(s) selected

Upload one or multiple JSON test suite files

(( file.name ))

(( file.testCount )) test(s)

System prompt: (( file.systemPrompt.length > 100 ? file.systemPrompt.substring(0, 100) + '...' : file.systemPrompt ))

Hint for the LLM about the primary language of the tickets.

tests

Runs only the first N tests specified in the test suite.

Runs a random sample of N tests from the test suite.

Controls randomness: Lower values (e.g., 0.2) make output more focused/deterministic, higher values (e.g., 0.8) make it more creative/diverse and potentially better at analyzing sparse details.

Add optional delay (ms) between requests to avoid overloading the API

Set the maximum number of tests to run in parallel.

Manual Test Editor

No test cases yet. Click "Add Test Case" to create one.

Test Case #(( index + 1 ))

Each test will be repeated this many times

View Previous Result File

Upload a previously downloaded JSON result file to view it below.

Choose result JSON file...

Active Runs

No active test runs
(( group.filename || 'Manual Test' )) (( formatDate(group.timestamp) )) ( (( group.runs.length )) models ) Iterations: (( group.iterations )) (( group.isImported ? 'Imported' : group.status )) Progress: (( group.completed_tests )) / (( group.total_tests )) Error: (( group.error )) (( group.retrying_tests_count )) retrying...
(( run.model_display_name )) Temperature: (( typeof run.temperature === 'number' ? run.temperature.toFixed(2) : '...' ))
Run ID: (( run.id )) (( run.status || 'initializing' )) (( run.completed_tests || 0 )) / (( run.total_tests )) tests (( run.successful_tests )) successful) Runtime: (( formatCompactRuntime(run.runtime_seconds) )) (retrying...) Rate: (( formatScoreAsPercentage(run.score_average) )) Error: (( run.error ))

Runtime

(( formatCompactRuntime(selectedRun.summary.runtime_seconds) ))

Total Tests

((selectedRun.summary.total_tests))

Fully Automatable

(( Math.round(selectedRun.summary.success_rate * 100) ))% ( ((selectedRun.summary.successful_tests)) )

Automation Rate

(( formatScoreAsPercentage(selectedRun.summary?.score_average) ))

Tools Usage Summary

Tool Name Usage Count Usage %
(( tool )) (( count )) (( calculateToolPercentage(count, selectedRun.summary.tests_completed) ))%

📊 Dataset Statistics

Total Tickets
(( getStatistics().basic_stats.total_tickets || 'N/A' ))
Unique Customers
(( getStatistics().basic_stats.unique_customers || 'N/A' ))
Unique Users
(( getStatistics().basic_stats.unique_users || 'N/A' ))
Unique Categories
(( getStatistics().basic_stats.unique_categories || 'N/A' ))

💡 Key Insights

  • (( insight ))

⏱️ Resolution Time (Minutes)

Average: (( getStatistics().processing_time_analysis.resolution_time_minutes.statistics.mean?.toFixed(2) )) min
Median: (( getStatistics().processing_time_analysis.resolution_time_minutes.statistics.p50?.toFixed(2) )) min
95th percentile: (( getStatistics().processing_time_analysis.resolution_time_minutes.statistics.p95?.toFixed(2) )) min
Max: (( getStatistics().processing_time_analysis.resolution_time_minutes.statistics.max?.toFixed(2) )) min

📈 Quick Stats

Under 1 hour: (( getStatistics().processing_time_analysis.resolution_time_minutes.tickets_resolved_under_1h )) tickets
Under 1 day: (( getStatistics().processing_time_analysis.resolution_time_minutes.tickets_resolved_under_1d )) tickets
Over 1 week: (( getStatistics().processing_time_analysis.resolution_time_minutes.tickets_over_1_week )) tickets

📂 Top Categories

(( category ))
(( count )) (( getStatistics().category_analysis.category_weights?.[category]?.toFixed(1) || '0' ))%

🔥 Priority Breakdown

Priority (( priority ))
(( count ))

🎫 Topic Analysis

(( formatTopicName(topic) ))
(( data.percentage?.toFixed(1) ))%
(( term ))

Test Results Summary

Test Name Category Status Expected Actual
((result.test_name || 'Unnamed Test')) (Iter: ((result.iteration || 1))) ((result.category || 'No Category')) (( result.success ? 'Pass' : 'Fail' )) 
(( result.validation ? JSON.stringify(result.validation.expected, null, 2) : 'N/A' ))
 
((result.output || 'No output'))

Error: (( result.error ))

Score Distribution

Test Cases

((result.test_name)) (Iter: ((result.iteration || 1)))

Tool: (( result.tools_assessed.join(', ') )) Tool: None Assessed

(( result.success ? 'Pass' : (parseAgentResponse(result.output)?.score >= result.validation?.expected?.min_score ? 'Partial' : 'Fail') )) (( Math.round(parseAgentResponse(result.output)?.score) ))/10

Click to view details

Error: (( result.error ))

Test Details: (( selectedRun.results[selectedTestIndex].test_name )) (Iter: ((selectedRun.results[selectedTestIndex].iteration || 1)))

Input Ticket

(( cleanInputText(selectedRun.results[selectedTestIndex].input) ))

Input prompt not available for this test case.

Response & Tools

Score:
(( Math.round(agentResponse.score) ))/10 N/A
Steps Taken:
  • (( step ))

(( agentResponse.steps && agentResponse.steps[0] ? agentResponse.steps[0] : 'None listed' ))

Tools Identified:
  • (( tool ))

None listed

Assessed Tools Needed:
  • (( tool ))

None Assessed

Raw Output (Parse Failed)
((selectedRun.results[selectedTestIndex].output))

Error: (( selectedRun.results[selectedTestIndex].error ))

No response output or error recorded for this test.

Required Steps Coverage

  • ((step))

No required steps defined for this test.

Score Distribution