> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thunderphone.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Test Scenarios & Suites
> Saved regression scenarios, AI scenario generation, condition-matrix runs, and release-gate suites.
Where [Simulations](/api-reference/test-calls) are ad-hoc, **test
scenarios** are saved, repeatable test cases for an agent. Scenarios
can be written by hand, generated by AI from the agent's prompt, or
**promoted from a real production call** (optionally anchored to an
issue report). Executing scenarios produces a **test run** (a
[simulation batch](/api-reference/test-calls#batch-object) under the
hood), optionally expanded across a matrix of audio/speech/behavior
conditions. **Suites** bundle scenarios into a pass/fail release gate
that can run on a schedule.
## Endpoints
| Method | Path | Description |
| -------------------------- | ---------------------------------------------------- | -------------------------------------------------- |
| `GET` / `POST` | `/v1/agents/{agent_id}/test-scenarios` | List / create scenarios |
| `GET` / `PATCH` / `DELETE` | `/v1/agents/{agent_id}/test-scenarios/{scenario_id}` | Manage one scenario |
| `POST` | `/v1/agents/{agent_id}/test-scenarios/generate` | AI-generate scenarios from the agent prompt |
| `POST` | `/v1/agents/{agent_id}/test-scenarios/refresh` | Re-generate stale scenarios after prompt changes |
| `POST` | `/v1/calls/{call_id}/promote-to-scenario` | Turn a real call into a scenario |
| `GET` | `/v1/agents/{agent_id}/test-runs` | List the agent's test runs (scenario batches) |
| `POST` | `/v1/agents/{agent_id}/test-runs/execute` | Execute scenarios (with optional condition matrix) |
| `GET` | `/v1/agents/{agent_id}/test-runs/{batch_id}` | Test-run detail with per-scenario results |
| `GET` / `POST` | `/v1/suites` | List / create suites |
| `GET` / `PATCH` / `DELETE` | `/v1/suites/{suite_id}` | Manage one suite (UUID id) |
## Scenario object
```json theme={null}
{
"id": 311,
"agent_id": 12,
"title": "Caller asks for a refund past the window",
"scenario_prompt": "You bought the product 45 days ago and demand a refund…",
"labels": ["refunds", "edge-case"],
"audio_conditions": { "noise_asset": "none" },
"speech_style": { "pace": "natural", "disfluencies": "none", "accent": "default" },
"caller_behavior": { "long_silence_seconds": 0, "dtmf_sequence": "", "topic_change": "" },
"telephony": { "voicemail_answer": false },
"direction": "inbound",
"source": "ai",
"prompt_hash": "d41d8…",
"source_call_id": null,
"source_issue_trace_id": null,
"source_turn_index": null,
"latest_verdict": "pass",
"is_stale": false,
"created_at": "2026-04-20T18:24:10.113Z",
"updated_at": "2026-04-20T18:24:10.113Z"
}
```
| Field | Type | Description |
| ---------------------------------------------------------------- | --------------- | --------------------------------------------------------------------------------------------------------------------- |
| `id` | integer | Scenario id |
| `title` | string | ≤ 200 chars |
| `scenario_prompt` | string | ≤ 4000 chars — the role-play brief for the test caller |
| `labels` | array of string | Up to 10 labels, ≤ 50 chars each |
| `audio_conditions` | object | `noise_asset` ∈ `none/cafe/street/car/office`; `snr_db` ∈ `0/5/10/20` (required when a noise asset is set) |
| `speech_style` | object | `pace` ∈ `natural/slow/fast`; `disfluencies` ∈ `none/light`; `accent` ∈ `default/en-US/en-GB/en-IN/es-MX/en-AU/fr-CA` |
| `caller_behavior` | object | `long_silence_seconds` 0–15; `dtmf_sequence` (≤ 12 of `0-9*#`); `topic_change` (≤ 500 chars) |
| `telephony` | object | `voicemail_answer` boolean — answer as a voicemail machine |
| `direction` | string | `inbound` or `outbound` |
| `source` | string | `manual`, `ai`, or `production` (promoted from a call) |
| `prompt_hash` | string | Hash of the agent prompt the scenario was generated against |
| `source_call_id` / `source_issue_trace_id` / `source_turn_index` | mixed \| null | Provenance for promoted scenarios |
| `latest_verdict` | string | `pass`, `fail`, or `unknown` — from the most recent run |
| `is_stale` | boolean | True for `ai`/`production` scenarios whose `prompt_hash` no longer matches the agent prompt |
### Create / update / delete
`POST` requires `title` + `scenario_prompt` (other fields optional,
defaults above) and returns `201`. `PATCH` accepts any subset.
`DELETE` returns `204`.
### AI-generate scenarios
```bash cURL theme={null}
curl -X POST https://api.thunderphone.com/v1/agents/12/test-scenarios/generate \
-H "Authorization: Bearer sk_live_YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"count": 6, "include_edge_cases": true, "locale": "en"}'
```
| Field | Type | Default | Description |
| -------------------- | ------- | ------- | -------------------------------- |
| `count` | integer | 6 | 1–12 scenarios |
| `include_edge_cases` | boolean | `true` | Mix in adversarial cases |
| `locale` | string | `en` | Language of the generated briefs |
Returns `201` with `{"scenarios": [ … ]}` (`source: "ai"`), or `502`
if generation failed.
### Refresh stale scenarios
After you change the agent's prompt, AI scenarios go `is_stale: true`.
`POST /test-scenarios/refresh` with `{"scenario_ids": [...]}` (≤ 50)
or `{"only_stale": true}` (or an empty body to refresh **all**)
re-generates their briefs against the current prompt and returns
`{"scenarios": [...]}`. `502` on generation failure.
### Promote a call to a scenario
Builds one regression scenario from a real call's transcript, grade,
and (optionally) a filed issue.
```bash cURL theme={null}
curl -X POST https://api.thunderphone.com/v1/calls/987654321/promote-to-scenario \
-H "Authorization: Bearer sk_live_YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"issue_trace_id": "5f9a…", "failing_turn_index": 7}'
```
| Field | Type | Required | Description |
| -------------------- | ----------- | -------- | ----------------------------------------------------------------------------------------------- |
| `issue_trace_id` | UUID | no | Anchor the scenario to an [issue report](/api-reference/issue-reports) on this call |
| `failing_turn_index` | integer ≥ 0 | no | Transcript index of the **agent** turn that went wrong — focuses generation on a ±3-turn window |
Returns `201` with the new scenario (`source: "production"`). If the
same call (+ same issue) was already promoted, returns `200` with the
existing scenario instead of creating a duplicate. `400` when the turn
index is out of range or not an agent turn.
***
## Test runs (execute scenarios)
```bash cURL theme={null}
curl -X POST https://api.thunderphone.com/v1/agents/12/test-runs/execute \
-H "Authorization: Bearer sk_live_YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"channel": "web",
"scenario_ids": [311, 312],
"matrix": {
"audio_conditions": [{ "noise_asset": "none" }, { "noise_asset": "cafe", "snr_db": 10 }]
},
"consent_to_charge": true
}'
```
| Field | Type | Required | Description |
| ------------------- | ---------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `channel` | string | yes | `web` (browser-style calls against the agent) or `telephony` (real calls through a phone number) |
| `phone_number_id` | integer | for `telephony` | Which of your numbers to exercise |
| `scenario_ids` | array of integer | no | ≤ 50 — omit to run **all** of the agent's scenarios |
| `matrix` | object | no | Condition axes to expand: `audio_conditions` (≤ 5), `speech_style` (≤ 7), `caller_behavior` (≤ 8), `telephony` (≤ 2) — each scenario runs once per combination. Total expanded run count is capped; oversize matrices return `400` |
| `stagger_seconds` | integer | no | 0–60, default 5 |
| `consent_to_charge` | boolean | yes | Must be `true` |
Returns `201` with the [test-run object](#test-run-object). `422` when
no scenarios match; `404` for a missing phone number.
### Test-run object
Test runs are returned by `GET /test-runs` (array, `limit` ≤ 200
default 50, `offset`) without per-run details, and by
`GET /test-runs/{batch_id}` with a `runs` array:
```json theme={null}
{
"id": 201,
"agent_id": 12,
"channel": "web",
"phone_number_id": null,
"phone_number_display": null,
"status": "in_progress",
"total": 4,
"completed": 1,
"failed": 0,
"pass_count": 1,
"fail_count": 0,
"matrix_dimensions": { "audio_conditions": [ … ] },
"estimated_minutes": 8,
"estimated_cost_cents": 64,
"agent_prompt_hash": "d41d8…",
"error_message": "",
"requested_by_id": 7,
"started_at": "2026-04-20T18:24:09.011Z",
"completed_at": null,
"created_at": "2026-04-20T18:24:09.011Z",
"updated_at": "2026-04-20T18:24:09.011Z",
"runs": [
{
"id": 7110,
"batch_id": 201,
"scenario_id": 311,
"scenario_title": "Caller asks for a refund past the window",
"scenario_prompt": "…",
"scenario_labels": ["refunds"],
"condition_dimensions": { "audio_conditions": { "noise_asset": "cafe", "snr_db": 10 } },
"test_call_run_id": 7110,
"call_id": 987654321,
"status": "completed",
"verdict": "pass",
"grade_score": 92,
"duration_seconds": 54,
"transcript_excerpt": "…",
"transcript": [ { "role": "user", "content": "…", "timestamp": null } ],
"grade": { /* grade object */ },
"recording_url": "https://…",
"error_message": "",
"started_at": "2026-04-20T18:24:10.113Z",
"completed_at": "2026-04-20T18:25:04.822Z"
}
]
}
```
Per-run `verdict` is `pass` / `fail` / `unknown` (derived from the
grade when the run completes). `call_id` is the public call id — link
it to [`GET /v1/calls/{call_id}`](/api-reference/calls).
***
## Suites (release gates)
A **suite** is a named set of scenarios for one agent plus pass
criteria. Suite runs are recorded against the suite; scheduled runs
fire automatically (hourly / daily / weekly), and the dashboard's
Simulations page can trigger them on demand.
### Suite object
```json theme={null}
{
"id": "b4e7c1a9-…",
"name": "Release gate — support agent",
"agent_id": 12,
"agent_name": "Customer Support Agent",
"scenario_ids": [311, 312],
"scenarios": [
{ "id": 311, "title": "Caller asks for a refund past the window", "labels": ["refunds"], "is_stale": false }
],
"criteria": { "min_pass_rate": 100, "zero_critical_failures": true },
"schedule": {
"enabled": true,
"frequency": "daily",
"timezone": "America/Los_Angeles",
"hour": 6,
"minute": 0,
"weekday": 0,
"next_run_at": "2026-04-21T13:00:00+00:00",
"last_run_at": null
},
"last_green_at": null,
"created_at": "2026-04-20T18:24:10.113Z",
"updated_at": "2026-04-20T18:24:10.113Z",
"runs": [
{
"id": "9d2f…",
"suite_id": "b4e7c1a9-…",
"batch_id": 201,
"baseline_run_id": null,
"trigger": "schedule",
"status": "running",
"verdict": "pending",
"pass_rate": null,
"critical_failure_count": 0,
"regression_count": 0,
"results": [],
"regressions": [],
"accepted_at": null,
"completed_at": null,
"created_at": "2026-04-20T18:24:10.113Z"
}
]
}
```
### Create / update a suite
```bash cURL theme={null}
curl -X POST https://api.thunderphone.com/v1/suites \
-H "Authorization: Bearer sk_live_YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Release gate — support agent",
"agent_id": 12,
"scenario_ids": [311, 312],
"criteria": { "min_pass_rate": 100, "zero_critical_failures": true },
"schedule": { "frequency": "daily", "timezone": "America/Los_Angeles", "hour": 6 }
}'
```
| Field | Type | Required on create | Description |
| --------------------------------- | ---------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | string | yes | ≤ 200 chars |
| `agent_id` | integer | yes | |
| `scenario_ids` | array of integer | yes | 1–50 scenarios; every one must belong to the suite's agent |
| `criteria.min_pass_rate` | integer | no | 0–100, default 100 |
| `criteria.zero_critical_failures` | boolean | no | Default `true` |
| `schedule` | object \| null | no | `frequency` ∈ `hourly/daily/weekly` (required inside the object), `timezone` (IANA, default UTC), `hour` 0–23, `minute` 0–59, `weekday` 0–6 (weekly). Pass `null` to remove the schedule |
`POST /v1/suites` returns `201`; `PATCH /v1/suites/{suite_id}` accepts
any subset; `DELETE` returns `204`. List and detail responses embed
the suite's most recent runs (up to 50). A run's `verdict` becomes
`pass` only when the pass rate meets `min_pass_rate` and (if enabled)
no critical-severity failures occurred; `regressions` lists scenarios
that passed in the baseline run but fail now.
***
## Related
The run/batch primitives test runs are built on.
Promote issues into regression scenarios and track `regression_guarded`.
Scenario staleness tracks the agent's prompt.
Every scenario run produces a call log.