Skip to main content
Prefer the dashboard? The same capability lives in Simulations (/dashboard/simulations), including AI scenario generation — see Simulate a call. This page covers the programmatic path.
Iterating on an AI agent means iterating on its prompt, its tools, and the way it handles edge cases. The test-calls API runs real (bot-to-bot or SIP loopback) calls against an agent using a scenario prompt you supply — every run produces a real call log with transcript, grading, and billing, so you see exactly how the agent behaves and what it costs. Use it for:
  • Pre-deploy smoke tests after every prompt edit
  • Regression suites wired into CI (hook test-call.completed webhook → fail the build if score drops)
  • Stress-testing concurrency limits

One-shot: single run

Fields: The response is a Test call run object in status="queued". Poll until status becomes completed or failed; once call_id is set, load the transcript via GET /v1/calls/{call_id}/transcript.

Batches: parallel scenarios

Run N scenarios concurrently — useful for regression suites that hit every known edge case in parallel:
Response carries a run_ids list of child run ids. Fetch batch status:
run_count is capped at 20; stagger_seconds spaces out the spawn to avoid hammering the agent (0–60 s).

Hook it into CI

Create a release gate suite on the Simulations page (/dashboard/simulations) — pick the agent, add scenarios by hand or click Generate scenarios with AI to draft them from the agent’s prompt (with an optional edge-case pass), and group them into a suite. A suite pins its scenarios and agent, plus a minimum pass rate and an optional zero-critical-failures rule. Passing runs become the accepted baseline; later pass→fail transitions are returned as regressions. Use an organization API key in CI. This script triggers the suite, polls until grading and comparison are complete, and exits nonzero unless the verdict is pass:
POST /v1/orgs/{org_id}/suites/{suite_id}/run returns 202 with the run id. GET /v1/orgs/{org_id}/suites/{suite_id}/runs/{run_id} returns status, verdict, pass_rate, critical_failure_count, and the baseline regressions list. Both endpoints bind the URL organization to the API key’s organization.

Patterns

Per-prompt regression corpus

Maintain a JSON file of {name, scenario_prompt, expected_outcome} tuples. On every prompt change, run the full set as a batch; diff the transcripts and grades against the previous run.

Per-release smoke test

A single batch of five happy-path scenarios you run after every deploy. Latency-sensitive, so keep stagger_seconds: 0.

Latency benchmarking

Run identical scenarios against different product tiers (spark, bolt, storm-base). Compare the call.graded scores and the duration_seconds from each resulting call log.

Next steps

Test calls reference

Every query parameter, status code, and batch shape.

AI grading

Auto-score every test run to track quality over time.

Issue reports

Flag specific tests for human review.

test-call.completed webhook

Stream results into your CI / Slack / PagerDuty.