> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thunderphone.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Validation Sets

> Frozen real-call turns replayed against your draft configuration, graded automatically before you deploy.

Where [Test Scenarios](/api-reference/test-scenarios) simulate whole
calls end-to-end, a **validation set** is a collection of frozen
single turns from real calls. Each **validation example** pins the
exact conversation history and caller audio of one agent turn, plus
an expectation of how the agent should respond. Running the set
replays every example against the agent's **current draft
configuration** and grades each response with two automated judges —
one listens to the pinned caller audio for contradictions, the other
checks the response against the agent's prompt and the example's
expectation. The result is a pass rate and a list of regressions you
can review before deploying.

## Endpoints

| Method                     | Path                                                      | Description                                   |
| -------------------------- | --------------------------------------------------------- | --------------------------------------------- |
| `POST`                     | `/v1/calls/{call_id}/validation-examples`                 | Promote a call turn into a validation example |
| `GET`                      | `/v1/agents/{agent_id}/validation-examples`               | List the agent's examples                     |
| `GET`                      | `/v1/validation-examples`                                 | List every example in the org, across agents  |
| `GET` / `PATCH` / `DELETE` | `/v1/validation-examples/{example_id}`                    | Manage one example                            |
| `GET`                      | `/v1/validation-examples/{example_id}/audio/{span_index}` | Stream a pinned caller-audio span             |
| `POST`                     | `/v1/validation-examples/{example_id}/replay`             | Replay one example against the draft config   |
| `GET` / `POST`             | `/v1/agents/{agent_id}/validation-runs`                   | List / start validation runs                  |
| `GET`                      | `/v1/validation-runs/{run_id}`                            | Run detail with per-example results           |
| `GET`                      | `/v1/agents/{agent_id}/validation-status`                 | Deploy-time summary for the agent             |
| `GET`                      | `/v1/validation-sets`                                     | Org-wide overview across agents               |

## Example object

```json theme={null}
{
  "id": 42,
  "agent_id": 12,
  "kind": "failure",
  "status": "pending_review",
  "labels": ["refunds"],
  "expectation": "Confirms the refund window and offers to escalate.",
  "accepted_responses": [{ "spoken_response": "…", "tool_calls": [] }],
  "original_response": { "spoken_response": "…", "tool_calls": [] },
  "target_turn_index": 7,
  "source_call_id": 903212880123,
  "source_issue_id": null,
  "prompt_hash": "d41d8…",
  "is_stale_prompt": false,
  "audio_span_indices": [3, 5, 7],
  "expectation_proposal_error": null,
  "last_verdict": "pass",
  "last_run_id": 9,
  "context": [ { "role": "user", "content": "…" } ],
  "created_at": "2026-07-22T18:24:10.113Z",
  "updated_at": "2026-07-22T18:24:10.113Z"
}
```

| Field                             | Type             | Description                                                                                                                                                                                          |
| --------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kind`                            | string           | `failure` (promoted from a bad turn), `golden` (known-good turn), or `manual`                                                                                                                        |
| `status`                          | string           | `pending_review` → `active` once the expectation is confirmed; `archived` examples are excluded from runs. Archived examples can be restored to `active`; pending examples can be archived directly. |
| `expectation`                     | string           | What the agent should do on this turn. New examples get an AI-proposed expectation to review (`expectation_proposal_error` is set if proposing failed).                                              |
| `accepted_responses`              | array            | Optional known-good responses (`spoken_response` + `tool_calls`)                                                                                                                                     |
| `original_response`               | object           | What the agent actually said/did on the source call                                                                                                                                                  |
| `prompt_hash` / `is_stale_prompt` | string / boolean | Hash of the prompt at freeze time; `is_stale_prompt` is true once the draft prompt has changed since                                                                                                 |
| `audio_span_indices`              | array of integer | Turn indices whose caller audio is pinned for this example                                                                                                                                           |
| `last_verdict` / `last_run_id`    | string \| null   | Latest completed-run outcome for this example                                                                                                                                                        |
| `context`                         | array            | Display transcript excerpt around the frozen turn (list endpoints omit it)                                                                                                                           |

### Promote a call turn

`POST /v1/calls/{call_id}/validation-examples` with `turn_index`
(the agent turn to freeze), `kind`, and optionally `source_issue_id`.
Returns the created example with `201`. The turn's history and caller
audio are snapshotted immediately, so the example survives the source
call being deleted. Promoting the same turn twice returns `409` with
`existing_id`.

### List examples

`GET /v1/agents/{agent_id}/validation-examples` returns
`{ "examples": [...] }` (without `context`).

`GET /v1/validation-examples` returns the same shape across every
agent in the organization, newest first, with `agent_name` added to
each row. Optional query filters: `agent_id`, `status`, `kind`.

### Manage one example

`GET` returns the full example. `PATCH` accepts any subset of
`expectation`, `accepted_responses`, `labels`, `kind`, and `status`
(allowed transitions: `pending_review → active|archived`,
`active → archived`, `archived → active`). `DELETE` removes the
example and its pinned audio.

### Caller audio

`GET /v1/validation-examples/{example_id}/audio/{span_index}` streams
the pinned WAV for one of the example's `audio_span_indices`.

### Replay one example

`POST /v1/validation-examples/{example_id}/replay` (empty body)
replays the frozen turn against the agent's current draft
configuration and returns:

```json theme={null}
{
  "replayed_output": { "spoken_response": "…", "tool_calls": [] },
  "replay_metadata": { "audio_spans_expected": 3, "audio_spans_loaded": 3, "skipped": false },
  "raw_prompt_hash": "ab12…",
  "cached": false
}
```

If any pinned audio span fails to load, `skipped` is true and the
result is never graded or counted. Identical replays (same draft
config) are served from cache with `cached: true`.

## Validation runs

`POST /v1/agents/{agent_id}/validation-runs` starts a batch run over
the agent's **active** examples (body is optional;
`config_source: "draft"` is the only supported value). Returns the
run with `201`. One run per agent may be in flight (`409` with
`existing_id` otherwise), and concurrent runs per organization are
capped (`429`). Starting a run with no active examples returns `400`.

```json theme={null}
{
  "id": 9,
  "agent_id": 12,
  "config_source": "draft",
  "status": "completed",
  "prompt_hash": "ab12…",
  "pass_count": 11,
  "warn_count": 1,
  "fail_count": 2,
  "skipped_count": 1,
  "total_count": 15,
  "pass_rate": 0.786,
  "regressions": [{ "example_id": 42, "previous": "pass", "current": "fail" }],
  "estimated_cost_cents": 8,
  "error_message": null,
  "created_at": "2026-07-22T18:24:10.113Z",
  "started_at": "2026-07-22T18:24:11.001Z",
  "completed_at": "2026-07-22T18:26:44.913Z",
  "triggered_by_name": "Alec Bell"
}
```

`status` is `pending`, `running`, `completed`, or `failed`.
`pass_rate` excludes skipped examples from the denominator.
`regressions` lists examples that passed in the previous completed
run but failed in this one.

### Run detail

`GET /v1/validation-runs/{run_id}` returns the run plus `results`,
one per graded example:

```json theme={null}
{
  "results": [
    {
      "id": 71,
      "example_id": 42,
      "example_expectation": "Confirms the refund window…",
      "example_kind": "failure",
      "verdict": "fail",
      "replayed_output": { "spoken_response": "…", "tool_calls": [] },
      "replay_metadata": { "audio_spans_expected": 3, "audio_spans_loaded": 3, "skipped": false },
      "audio_verdict": { "contradiction": true, "explanation": "…" },
      "instruction_verdict": { "followed": false, "explanation": "…" },
      "created_at": "2026-07-22T18:25:02.113Z"
    }
  ]
}
```

`verdict` combines both judges: `pass`, `warn`, `fail`, or `skipped`
(audio unavailable). Each judge verdict includes an explanation you
can surface to reviewers.

### List runs

`GET /v1/agents/{agent_id}/validation-runs` returns
`{ "runs": [...] }` — the 20 most recent runs, newest first.

## Validation status

`GET /v1/agents/{agent_id}/validation-status` powers the
deploy-confirmation check:

```json theme={null}
{
  "active_example_count": 15,
  "latest_run": { "…": "run object or null" },
  "latest_run_matches_draft": true
}
```

`latest_run_matches_draft` is true when the latest completed run was
executed against the current draft prompt — i.e. the pass rate on
screen reflects what you are about to deploy.

## Org overview

`GET /v1/validation-sets` returns one row per agent in the
organization that has validation examples, for a cross-agent
dashboard view:

```json theme={null}
{
  "agents": [
    {
      "agent_id": 12,
      "agent_name": "Support line",
      "active_count": 15,
      "pending_count": 2,
      "archived_count": 1,
      "latest_run": { "…": "run object or null" },
      "latest_run_matches_draft": true
    }
  ]
}
```

Agents without any validation examples are omitted. Rows are sorted
by agent name.