Validation Sets

Where Test Scenarios simulate whole calls end-to-end, a validation set is a collection of frozen single turns from real calls. Each validation example pins the exact conversation history and caller audio of one agent turn, plus an expectation of how the agent should respond. Running the set replays every example against the agent’s current draft configuration and grades each response with two automated judges — one listens to the pinned caller audio for contradictions, the other checks the response against the agent’s prompt and the example’s expectation. The result is a pass rate and a list of regressions you can review before deploying.

Endpoints

Method	Path	Description
`POST`	`/v1/calls/{call_id}/validation-examples`	Promote a call turn into a validation example
`GET`	`/v1/agents/{agent_id}/validation-examples`	List the agent’s examples
`GET`	`/v1/validation-examples`	List every example in the org, across agents
`GET` / `PATCH` / `DELETE`	`/v1/validation-examples/{example_id}`	Manage one example
`GET`	`/v1/validation-examples/{example_id}/audio/{span_index}`	Stream a pinned caller-audio span
`POST`	`/v1/validation-examples/{example_id}/replay`	Replay one example against the draft config
`GET` / `POST`	`/v1/agents/{agent_id}/validation-runs`	List / start validation runs
`GET`	`/v1/validation-runs/{run_id}`	Run detail with per-example results
`GET`	`/v1/agents/{agent_id}/validation-status`	Deploy-time summary for the agent
`GET`	`/v1/validation-sets`	Org-wide overview across agents

Example object

{
  "id": 42,
  "agent_id": 12,
  "kind": "failure",
  "status": "pending_review",
  "labels": ["refunds"],
  "expectation": "Confirms the refund window and offers to escalate.",
  "accepted_responses": [{ "spoken_response": "…", "tool_calls": [] }],
  "original_response": { "spoken_response": "…", "tool_calls": [] },
  "target_turn_index": 7,
  "source_call_id": 903212880123,
  "source_issue_id": null,
  "prompt_hash": "d41d8…",
  "is_stale_prompt": false,
  "audio_span_indices": [3, 5, 7],
  "expectation_proposal_error": null,
  "last_verdict": "pass",
  "last_run_id": 9,
  "context": [ { "role": "user", "content": "…" } ],
  "created_at": "2026-07-22T18:24:10.113Z",
  "updated_at": "2026-07-22T18:24:10.113Z"
}

Field	Type	Description
`kind`	string	`failure` (promoted from a bad turn), `golden` (known-good turn), or `manual`
`status`	string	`pending_review` → `active` once the expectation is confirmed; `archived` examples are excluded from runs. Archived examples can be restored to `active`; pending examples can be archived directly.
`expectation`	string	What the agent should do on this turn. New examples get an AI-proposed expectation to review (`expectation_proposal_error` is set if proposing failed).
`accepted_responses`	array	Optional known-good responses (`spoken_response` + `tool_calls`)
`original_response`	object	What the agent actually said/did on the source call
`prompt_hash` / `is_stale_prompt`	string / boolean	Hash of the prompt at freeze time; `is_stale_prompt` is true once the draft prompt has changed since
`audio_span_indices`	array of integer	Turn indices whose caller audio is pinned for this example
`last_verdict` / `last_run_id`	string \| null	Latest completed-run outcome for this example
`context`	array	Display transcript excerpt around the frozen turn (list endpoints omit it)

Promote a call turn

POST /v1/calls/{call_id}/validation-examples with turn_index (the agent turn to freeze), kind, and optionally source_issue_id. Returns the created example with 201. The turn’s history and caller audio are snapshotted immediately, so the example survives the source call being deleted. Promoting the same turn twice returns 409 with existing_id.

List examples

GET /v1/agents/{agent_id}/validation-examples returns { "examples": [...] } (without context). GET /v1/validation-examples returns the same shape across every agent in the organization, newest first, with agent_name added to each row. Optional query filters: agent_id, status, kind.

Manage one example

GET returns the full example. PATCH accepts any subset of expectation, accepted_responses, labels, kind, and status (allowed transitions: pending_review → active|archived, active → archived, archived → active). DELETE removes the example and its pinned audio.

Caller audio

GET /v1/validation-examples/{example_id}/audio/{span_index} streams the pinned WAV for one of the example’s audio_span_indices.

Replay one example

POST /v1/validation-examples/{example_id}/replay (empty body) replays the frozen turn against the agent’s current draft configuration and returns:

{
  "replayed_output": { "spoken_response": "…", "tool_calls": [] },
  "replay_metadata": { "audio_spans_expected": 3, "audio_spans_loaded": 3, "skipped": false },
  "raw_prompt_hash": "ab12…",
  "cached": false
}

If any pinned audio span fails to load, skipped is true and the result is never graded or counted. Identical replays (same draft config) are served from cache with cached: true.

Validation runs

POST /v1/agents/{agent_id}/validation-runs starts a batch run over the agent’s active examples (body is optional; config_source: "draft" is the only supported value). Returns the run with 201. One run per agent may be in flight (409 with existing_id otherwise), and concurrent runs per organization are capped (429). Starting a run with no active examples returns 400.

{
  "id": 9,
  "agent_id": 12,
  "config_source": "draft",
  "status": "completed",
  "prompt_hash": "ab12…",
  "pass_count": 11,
  "warn_count": 1,
  "fail_count": 2,
  "skipped_count": 1,
  "total_count": 15,
  "pass_rate": 0.786,
  "regressions": [{ "example_id": 42, "previous": "pass", "current": "fail" }],
  "estimated_cost_cents": 8,
  "error_message": null,
  "created_at": "2026-07-22T18:24:10.113Z",
  "started_at": "2026-07-22T18:24:11.001Z",
  "completed_at": "2026-07-22T18:26:44.913Z",
  "triggered_by_name": "Alec Bell"
}

status is pending, running, completed, or failed. pass_rate excludes skipped examples from the denominator. regressions lists examples that passed in the previous completed run but failed in this one.

Run detail

GET /v1/validation-runs/{run_id} returns the run plus results, one per graded example:

{
  "results": [
    {
      "id": 71,
      "example_id": 42,
      "example_expectation": "Confirms the refund window…",
      "example_kind": "failure",
      "verdict": "fail",
      "replayed_output": { "spoken_response": "…", "tool_calls": [] },
      "replay_metadata": { "audio_spans_expected": 3, "audio_spans_loaded": 3, "skipped": false },
      "audio_verdict": { "contradiction": true, "explanation": "…" },
      "instruction_verdict": { "followed": false, "explanation": "…" },
      "created_at": "2026-07-22T18:25:02.113Z"
    }
  ]
}

verdict combines both judges: pass, warn, fail, or skipped (audio unavailable). Each judge verdict includes an explanation you can surface to reviewers.

List runs

GET /v1/agents/{agent_id}/validation-runs returns { "runs": [...] } — the 20 most recent runs, newest first.

Validation status

GET /v1/agents/{agent_id}/validation-status powers the deploy-confirmation check:

{
  "active_example_count": 15,
  "latest_run": { "…": "run object or null" },
  "latest_run_matches_draft": true
}

latest_run_matches_draft is true when the latest completed run was executed against the current draft prompt — i.e. the pass rate on screen reflects what you are about to deploy.

Org overview

GET /v1/validation-sets returns one row per agent in the organization that has validation examples, for a cross-agent dashboard view:

{
  "agents": [
    {
      "agent_id": 12,
      "agent_name": "Support line",
      "active_count": 15,
      "pending_count": 2,
      "archived_count": 1,
      "latest_run": { "…": "run object or null" },
      "latest_run_matches_draft": true
    }
  ]
}

Agents without any validation examples are omitted. Rows are sorted by agent name.

Overview

Organizations

Agents

Phone Numbers

Calls

Billing

Developer

Endpoints

Example object

Promote a call turn

List examples

Manage one example

Caller audio

Replay one example

Validation runs

Run detail

List runs

Validation status

Org overview

​Endpoints

​Example object

​Promote a call turn

​List examples

​Manage one example

​Caller audio

​Replay one example

​Validation runs

​Run detail

​List runs

​Validation status

​Org overview

Endpoints

Example object

Promote a call turn

List examples

Manage one example

Caller audio

Replay one example

Validation runs

Run detail

List runs

Validation status

Org overview