Skip to main content
A validation example freezes one agent turn from a real call: the conversation leading up to it, the relevant caller audio, what the agent actually did, and the ground truth for what it should do. Replaying a set against the current draft catches regressions without placing another call. Use the organization-wide Validation sets page (/dashboard/validation) to build and review the dataset. Use an individual agent’s Validation tab to run the set and inspect run history.

​
Add examples from call history

1

Choose Add examples

Pick the agent whose behavior the example should validate. The workspace shows up to 25 recent non-simulation calls for that agent.
2

Select a call and agent turn

Read the transcript, hover the agent turn you want to preserve, and select the checklist action. The conversation through that turn and its caller audio spans are copied into the validation dataset.
3

Review the expectation

ThunderPhone drafts a description of correct behavior. Edit it so it is specific and testable, and optionally edit the accepted example response and expected tool calls.
4

Save or activate

Save draft leaves the example in Needs review. Save & activate includes it in future validation runs.
Promoting the same turn twice reuses the existing example rather than creating two copies.

​
Example kinds and statuses

Kinds describe why an example exists: Statuses control the lifecycle:

​
Browse and refine the dataset

Filter the organization dataset by agent, status, or kind. Selecting an example shows:
  • Frozen conversation context and playable caller audio spans.
  • The source call and original response.
  • Ground-truth expectation, accepted spoken response, and tool calls.
  • The latest replay beside the original and expected behavior.
  • The latest verdict and judge explanations, when available.
Choose Edit ground truth to revise the expectation or accepted response. Choose Replay against draft to test only that example without starting a full run. Replays use the agent’s current draft, not necessarily its deployed configuration.

​
Run a validation set

From the agent’s Validation tab, start a run over every active example. Each example receives one of these verdicts:
  • Pass — the response satisfies the expectation and caller audio.
  • Warn — the response is usable but one check found uncertainty.
  • Fail — it contradicts the audio or misses required behavior.
  • Skipped — required caller audio could not be loaded; it is excluded from the pass-rate denominator.
  • Error — replay or grading could not complete.
The run summary reports pass, warning, failure, and skipped counts, the pass rate, estimated cost, and any regression from pass to fail since the preceding completed run. Only one validation run per agent can be active at a time.

​
Deploy with validation evidence

When an agent has validation examples, the deploy dialog reports whether the latest completed run matches the current draft. You can:
  • Deploy with the displayed pass rate and regression count.
  • Run validation first if no matching run exists.
  • Deliberately deploy anyway when the result has been reviewed.
Validation is evidence and a deployment check, not an automatic production traffic split. For live comparisons, use Experiments (A/B testing).

​
API automation

The Validation Sets API covers promoting turns, managing examples, streaming pinned audio, individual replays, batch runs, run results, deploy-time status, and the organization overview.