Prefer the dashboard? The same capability lives in Simulations
(
/dashboard/simulations), including AI scenario generation — see
Simulate a call. This page covers the
programmatic path.- Pre-deploy smoke tests after every prompt edit
- Regression suites wired into CI (hook
test-call.completedwebhook → fail the build if score drops) - Stress-testing concurrency limits
One-shot: single run
The response is a Test call run object
in
status="queued". Poll until status becomes completed or
failed; once call_id is set, load the transcript via
GET /v1/calls/{call_id}/transcript.
Batches: parallel scenarios
Run N scenarios concurrently — useful for regression suites that hit every known edge case in parallel:run_ids list of child run ids. Fetch batch
status:
run_count is capped at 20; stagger_seconds spaces out the spawn
to avoid hammering the agent (0–60 s).
Hook it into CI
Create a release gate suite on the Simulations page (/dashboard/simulations) — pick the agent, add scenarios by hand or
click Generate scenarios with AI to draft them from the agent’s
prompt (with an optional edge-case pass), and group them into a suite.
A suite pins its scenarios and agent, plus a minimum pass rate and an
optional zero-critical-failures rule. Passing runs become the accepted
baseline; later pass→fail transitions are returned as regressions.
Use an organization API key in CI.
This script triggers the suite, polls until grading and comparison are
complete, and exits nonzero unless the verdict is pass:
POST /v1/orgs/{org_id}/suites/{suite_id}/run returns 202 with the
run id. GET /v1/orgs/{org_id}/suites/{suite_id}/runs/{run_id} returns
status, verdict, pass_rate, critical_failure_count, and the
baseline regressions list. Both endpoints bind the URL organization
to the API key’s organization.
Patterns
Per-prompt regression corpus
Maintain a JSON file of{name, scenario_prompt, expected_outcome}
tuples. On every prompt change, run the full set as a batch; diff the
transcripts and grades against the previous run.
Per-release smoke test
A single batch of five happy-path scenarios you run after every deploy. Latency-sensitive, so keepstagger_seconds: 0.
Latency benchmarking
Run identical scenarios against different product tiers (spark,
bolt, storm-base). Compare the call.graded scores and the
duration_seconds from each resulting call log.
Next steps
Test calls reference
Every query parameter, status code, and batch shape.
AI grading
Auto-score every test run to track quality over time.
Issue reports
Flag specific tests for human review.
test-call.completed webhook
Stream results into your CI / Slack / PagerDuty.