> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thunderphone.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Validation sets: replay real call turns before deployment

> Turn real ThunderPhone call moments into reusable validation examples, review expected behavior, replay them against an agent draft, and catch regressions before deploy.

A **validation example** freezes one agent turn from a real call: the
conversation leading up to it, the relevant caller audio, what the agent
actually did, and the ground truth for what it should do. Replaying a set
against the current draft catches regressions without placing another call.

Use the organization-wide **Validation sets** page
(`/dashboard/validation`) to build and review the dataset. Use an individual
agent's **Validation** tab to run the set and inspect run history.

## Add examples from call history

<Steps>
  <Step title="Choose Add examples">
    Pick the agent whose behavior the example should validate. The workspace
    shows up to 25 recent non-simulation calls for that agent.
  </Step>

  <Step title="Select a call and agent turn">
    Read the transcript, hover the agent turn you want to preserve, and select
    the checklist action. The conversation through that turn and its caller
    audio spans are copied into the validation dataset.
  </Step>

  <Step title="Review the expectation">
    ThunderPhone drafts a description of correct behavior. Edit it so it is
    specific and testable, and optionally edit the accepted example response
    and expected tool calls.
  </Step>

  <Step title="Save or activate">
    **Save draft** leaves the example in **Needs review**.
    **Save & activate** includes it in future validation runs.
  </Step>
</Steps>

Promoting the same turn twice reuses the existing example rather than creating
two copies.

## Example kinds and statuses

Kinds describe why an example exists:

| Kind        | Use                                                        |
| ----------- | ---------------------------------------------------------- |
| **Failure** | A real response that exposed behavior you want to prevent. |
| **Golden**  | A known-good response worth preserving.                    |
| **Manual**  | A manually curated case that is neither category.          |

Statuses control the lifecycle:

| Status           | Meaning                                                                  |
| ---------------- | ------------------------------------------------------------------------ |
| **Needs review** | Ground truth is still being reviewed; excluded from runs.                |
| **Active**       | Included in validation runs.                                             |
| **Stale**        | The prompt changed since the example was frozen; review its expectation. |
| **Incompatible** | The example cannot be replayed with the current configuration.           |
| **Archived**     | Preserved for history but excluded from runs.                            |

## Browse and refine the dataset

Filter the organization dataset by **agent**, **status**, or **kind**. Selecting
an example shows:

* Frozen conversation context and playable caller audio spans.
* The source call and original response.
* Ground-truth expectation, accepted spoken response, and tool calls.
* The latest replay beside the original and expected behavior.
* The latest verdict and judge explanations, when available.

Choose **Edit ground truth** to revise the expectation or accepted response.
Choose **Replay against draft** to test only that example without starting a
full run. Replays use the agent's current draft, not necessarily its deployed
configuration.

## Run a validation set

From the agent's **Validation** tab, start a run over every active example. Each
example receives one of these verdicts:

* **Pass** — the response satisfies the expectation and caller audio.
* **Warn** — the response is usable but one check found uncertainty.
* **Fail** — it contradicts the audio or misses required behavior.
* **Skipped** — required caller audio could not be loaded; it is excluded from
  the pass-rate denominator.
* **Error** — replay or grading could not complete.

The run summary reports pass, warning, failure, and skipped counts, the pass
rate, estimated cost, and any regression from pass to fail since the preceding
completed run. Only one validation run per agent can be active at a time.

## Deploy with validation evidence

When an agent has validation examples, the deploy dialog reports whether the
latest completed run matches the current draft. You can:

* Deploy with the displayed pass rate and regression count.
* Run validation first if no matching run exists.
* Deliberately deploy anyway when the result has been reviewed.

Validation is evidence and a deployment check, not an automatic production
traffic split. For live comparisons, use
[Experiments (A/B testing)](/guides/experiments-ab-testing).

## API automation

The [Validation Sets API](/api-reference/validation-sets) covers promoting
turns, managing examples, streaming pinned audio, individual replays, batch
runs, run results, deploy-time status, and the organization overview.