> ## Documentation Index > Fetch the complete documentation index at: https://docs.thunderphone.com/llms.txt > Use this file to discover all available pages before exploring further. # Realtime WebSocket > Exchange live call audio, transcripts, and call-control events over one WebSocket. The Realtime WebSocket API provides bidirectional audio over a single connection. It uses a focused realtime event subset for connecting ThunderPhone agents to your telephony stack. One WebSocket connection represents one call. Realtime calls appear in [call history](/api-reference/calls) and are billed like any other call at your product's per-minute rate. ## Create a managed realtime session `POST /v1/realtime/sessions` creates a LiveKit room and returns a scoped participant token. Authenticate with a secret API key and provide either a saved `agent_id` or an inline `config`, but not both. ```bash theme={null} curl -X POST https://api.thunderphone.com/v1/realtime/sessions \ -H "Authorization: Bearer sk_live_YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"agent_id": 12}' ``` Inline configuration accepts only `prompt`, `voice`, `product`, `tools`, `primary_language`, `additional_languages`, `background_track`, `telephony`, `placeholders`, `outbound`, and `call_events`. `prompt` is required. Unknown keys are rejected with a validation error rather than silently ignored. ```json theme={null} { "config": { "prompt": "Listen carefully and help complete the call.", "voice": "olivia", "product": "spark" } } ``` The operation returns `201 Created` with the room credentials: ```json theme={null} { "call_id": 123456789, "room_name": "realtime-123456789", "livekit_url": "wss://your-livekit-host.example.com", "token": "" } ``` ## Endpoint | Protocol | URL | Description | | --------- | ---------------------------------------- | ----------------------- | | WebSocket | `wss://api.thunderphone.com/v1/realtime` | Start one realtime call | The realtime endpoint shares its host with the REST API — the same `api.thunderphone.com` base URL and API key work for both. ## Connect and authenticate Authenticate the WebSocket upgrade with your secret API key: ```http theme={null} Authorization: Bearer sk_live_YOUR_API_KEY ``` Browser clients that cannot set an upgrade header can pass the key in the query string instead: ```text theme={null} wss://api.thunderphone.com/v1/realtime?api_key=sk_live_YOUR_API_KEY ``` A query-string key can appear in browser history and network logs. Use the `Authorization` header whenever your WebSocket client supports it, and never expose a secret API key in public client code. After the connection opens, the server sends `session.created`. Configure the call with `session.update`; the server confirms the accepted settings with `session.updated`. ## Configure the session Send the GA nested session shape before streaming audio. The first `session.update` starts the call. When `instructions` is omitted or empty, ThunderPhone uses `You are a helpful voice assistant.` ```json theme={null} { "type": "session.update", "session": { "type": "realtime", "instructions": "Listen to the caller and help them complete the call.", "audio": { "input": { "format": { "type": "audio/pcm", "rate": 16000 } }, "output": { "format": { "type": "audio/pcm", "rate": 16000 }, "voice": "olivia" } } } } ``` ### Session fields | Field | Type | Required | Description | | ------------------------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `session.type` | string | no | Use `"realtime"` | | `session.instructions` | string | yes | Instructions that define the agent's behavior for this call | | `session.audio.input.format.type` | string | no | `"audio/pcm"` (default), `"audio/pcmu"`, or `"audio/pcma"`; beta names `"pcm16"`, `"g711_ulaw"`, `"g711_alaw"` are also accepted | | `session.audio.input.format.rate` | integer | no | PCM input rate: `16000` (default) or `24000` Hz; G.711 is always `8000` | | `session.audio.input.noise_reduction` | object | no | `{"type":"near_field"}` selects noise cancellation tuned for wideband or browser-style audio; `{"type":"far_field"}` or omission uses the default telephony tuning | | `session.audio.output.format.type` | string | no | `"audio/pcm"` (default), `"audio/pcmu"`, or `"audio/pcma"`; beta names `"pcm16"`, `"g711_ulaw"`, `"g711_alaw"` are also accepted | | `session.audio.output.format.rate` | integer | no | PCM output rate: `16000` (default) or `24000` Hz; G.711 is always `8000` | | `session.audio.output.voice` | string | no | ThunderPhone voice name from the [voice gallery](/api-reference/voices); omitted (or an OpenAI voice name) falls back to the default voice | | `session.binary_output` | boolean | no | When `true`, return audio deltas as raw binary WebSocket frames instead of base64 JSON events | | `session.config.tools` | array | no | Inline function definitions delivered back to this client when the agent calls them | | `session.config.product` | string | no | Product tier for the call, such as `storm-base` (the default) | `session.config` accepts only `voice`, `product`, `tools`, `primary_language`, `additional_languages`, `background_track`, `telephony`, `placeholders`, `outbound`, and `call_events`. The call prompt comes from `session.instructions`; there is no separate `prompt` key over the WebSocket. Unknown keys are rejected with a validation error rather than silently ignored. Set `telephony` to `false` for the same wideband or browser-style input tuning selected by `session.audio.input.noise_reduction.type` set to `near_field`. All audio is mono. PCM audio is signed 16-bit little-endian at 16000 or 24000 Hz; G.711 (`audio/pcmu` μ-law, `audio/pcma` A-law) is 8000 Hz, matching telephony trunks — no client-side transcoding needed. Input and output formats are configured independently, but neither can change after the call starts. After the call starts, agent configuration is immutable: `instructions`, `voice`, `tools`, `product`, and other `session.config` fields cannot change. Bridge-local transport options such as `binary_output` remain changeable. For compatibility, flat `instructions`, `voice`, `input_audio_format`, and `output_audio_format` fields are also accepted on `session.update`. New integrations should use the nested shape above. Inline sessions can define custom functions under `session.config.tools`. The function schema is available to the agent; your WebSocket client remains responsible for executing the function and returning its output. ```json theme={null} { "type": "session.update", "session": { "instructions": "Help callers check an order.", "config": { "tools": [ { "type": "function", "function": { "name": "lookup_order", "description": "Look up an order by id.", "parameters": { "type": "object", "properties": { "order_id": { "type": "string" } }, "required": ["order_id"] } } } ] } } } ``` Custom tools on saved-agent sessions continue to use the routing configured on the saved agent; they are not delivered to the WebSocket client. ### Inline sessions are client-steered With inline configuration, ThunderPhone adds no automatic conversation behaviors. The agent never speaks first, there are no silence check-ins or timeouts, and by default nothing is spoken while a response is being prepared. If you want progress speech during slow responses — for example while the agent waits on one of your tool results — set `placeholders` in the session config to a list of short phrases in your own words (e.g. `{"placeholders": ["One moment.", "Still checking."]}`). They are spoken in the session voice, cycling only when a response takes long enough to need them. If the caller goes quiet, the session remains open until your client prompts the agent or hangs up. Silence-behavior config keys are not accepted and are rejected with a validation error. To have the agent open the call, configure the session, append a system or user message, and explicitly request one response: ```json theme={null} { "type": "session.update", "session": { "type": "realtime", "instructions": "Help the caller." } } ``` ```json theme={null} { "type": "conversation.item.create", "item": { "type": "message", "role": "system", "content": [ { "type": "input_text", "text": "Greet the caller and offer help." } ] } } ``` ```json theme={null} { "type": "response.create" } ``` `session.created` is sent as soon as the WebSocket connects. `session.updated` is sent only after the configuration has been accepted and the session fully provisioned, so it is the signal that the call is live; a rejected configuration produces an `error` event instead, and the session stays usable for a corrected retry. You do not need to wait, though: message items and `response.create` sent early are buffered and delivered in order once the agent is ready, so the opening sequence above is safe to send immediately after `session.update`. Sessions created with `agent_id` instead retain the behaviors configured on the saved agent, including its greeting order and silence check-ins. ### Use a saved agent To use an existing [agent](/api-reference/agents), connect with its id: ```text theme={null} wss://api.thunderphone.com/v1/realtime?agent_id=12 ``` The saved agent starts immediately, so a `session.update` with instructions is not required. Set its wire audio before minting with the optional `input_audio_format`, `output_audio_format`, `input_rate`, and `output_rate` query parameters. Formats accept `audio/pcmu`, `audio/pcma`, `g711_ulaw`, `g711_alaw`, or `pcm16`; PCM rates accept `16000` or `24000` (G.711 is always 8000 Hz). For example: ```text theme={null} wss://api.thunderphone.com/v1/realtime?agent_id=12&input_audio_format=g711_ulaw&output_audio_format=pcm16&output_rate=24000 ``` Agent instructions and voice come from the saved agent. ## Client events Send client events as JSON text frames unless otherwise noted. | Event | Fields | Behavior | | ---------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `session.update` | `session` | Configures the session using the nested shape above; flat compatibility fields are also accepted | | `input_audio_buffer.append` | `audio` | Appends a base64-encoded mono pcm16 audio chunk | | Raw binary frame | binary pcm16 | Appends audio without base64 encoding; this is a ThunderPhone extension | | `input_audio_buffer.commit` | — | Acknowledged with `input_audio_buffer.committed`; turn detection remains automatic | | `input_audio_buffer.clear` | — | Acknowledged with `input_audio_buffer.cleared`; turn detection remains automatic | | `conversation.item.create` | `item` | Sends a caller utterance (`user`) or silent guidance (`system`/`developer`), or acknowledges a function call; message items are acknowledged with `conversation.item.added` then `conversation.item.done` using your item id | | `conversation.item.truncate` | `item_id`, `content_index`, `audio_end_ms` | Acknowledged with `conversation.item.truncated` | | `conversation.item.delete` | `item_id` | Acknowledged with `conversation.item.deleted` | | `response.cancel` | `response_id` optional | Stops the in-progress response's remaining audio and closes it with status `cancelled`; errors with `response_cancel_not_active` when nothing is streaming | | `response.create` | — | Triggers exactly one model response using the conversation so far; if a response is active, emits an `error` with code `conversation_already_has_active_response` and does not start another | ### Append audio ```json theme={null} { "type": "input_audio_buffer.append", "audio": "" } ``` Send audio at its natural pace rather than uploading an entire recording at once. You can send each chunk as a raw binary frame instead; no session flag is required for binary input. ### Send text guidance Message items are appended to the conversation and never trigger a response by themselves. Use `user` for a completed caller utterance and `system`, with `developer` accepted as an alias, for silent guidance. A response begins either from server turn detection on audio or from an explicit `response.create`. Message items are acknowledged with `conversation.item.added` followed by `conversation.item.done`; both echo the client-provided `item.id`, or a generated id when it is absent. Text is not converted into caller audio. ```json theme={null} { "type": "conversation.item.create", "item": { "type": "message", "role": "user", "content": [ { "type": "input_text", "text": "Ask whether they need anything else." } ] } } ``` For silent guidance, set `"role": "system"` in the same message shape. Assistant-role message injection is unsupported. ### Request a response Send `response.create` to trigger exactly one response using the conversation so far. If a response is already in progress, the server emits an `error` event with code `conversation_already_has_active_response` and does not start another. After handling a custom function call, the client must send a `function_call_output` item with the received `call_id`. The `output` can be a JSON string or object. The default deadline is 30 seconds. If the deadline expires, the conversation continues with an error result so the agent can recover. Every function call names one of your declared tools and expects a `function_call_output`; platform actions arrive as `call.*` events instead (see below) and take no output. ```json theme={null} { "type": "conversation.item.create", "item": { "type": "function_call_output", "call_id": "call_abc123", "output": "{\"status\":\"ok\"}" } } ``` ## Server events Server events are JSON text frames. Each JSON event includes a unique `event_id`. | Event | Important fields | Description | | ------------------------------------------------------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `session.created` | `session` | Sent when the WebSocket opens with the session id and defaults | | `session.updated` | `session` | Confirms the current session configuration | | `conversation.item.added` | `item` | A conversation item (user or assistant message, or function call) was added | | `conversation.item.done` | `item` | A conversation item reached its final state | | `conversation.item.input_audio_transcription.delta` | `item_id`, `delta` | Caller transcript segment | | `conversation.item.input_audio_transcription.completed` | `item_id`, `transcript` | Final caller transcript for the utterance | | `conversation.item.truncated` | `item_id`, `audio_end_ms` | Acknowledges `conversation.item.truncate` | | `conversation.item.deleted` | `item_id` | Acknowledges `conversation.item.delete` | | `response.created` | `response` | Starts an agent response | | `rate_limits.updated` | `rate_limits` | Emitted after each `response.created` | | `response.output_item.added` | `response_id`, `item` | Adds an audio message or function call to the response | | `response.content_part.added` | `response_id`, `item_id`, `part` | Opens the audio content part of a message item | | `response.output_audio.delta` | `response_id`, `item_id`, `delta` | Base64-encoded output audio; replaced by raw binary frames when binary output is enabled | | `response.output_audio_transcript.delta` | `response_id`, `item_id`, `delta` | Agent transcript segment | | `response.output_audio.done` | `response_id`, `item_id` | Output audio for the item is complete | | `response.output_audio_transcript.done` | `response_id`, `item_id`, `transcript` | Final agent transcript for the response | | `response.content_part.done` | `response_id`, `item_id`, `part` | Closes the audio content part with the final transcript | | `response.function_call_arguments.delta` | `call_id`, `delta` | Incremental JSON arguments for a function call | | `response.function_call_arguments.done` | `call_id`, `arguments` | Complete JSON arguments for a function call | | `response.output_item.done` | `response_id`, `item` | Completes an audio message or function-call item | | `response.done` | `response` | Completes the response | | `input_audio_buffer.speech_started` | `audio_start_ms`, `item_id` | The server detected the start of caller speech; `item_id` is a stable advisory id for the speech interval and may not match the later history item id | | `input_audio_buffer.speech_stopped` | `audio_end_ms`, `item_id` | The server detected the end of caller speech; `item_id` is a stable advisory id for the speech interval and may not match the later history item id | | `input_audio_buffer.committed` | `item_id`, `previous_item_id` | The caller utterance was committed (also acknowledges a manual `commit`) | | `input_audio_buffer.cleared` | — | Acknowledges `input_audio_buffer.clear` | | `error` | `error.type`, `error.code`, `error.message` | Reports an authentication, validation, or server error | ### Conformance notes ThunderPhone supports PCM at 16000 Hz as an extension. The OpenAI GA schema pins `audio/pcm` to 24000 Hz, so strict validators of `session.created` and `session.updated` should also allow rate 16000. The normal audio response sequence is: ```text theme={null} response.created rate_limits.updated response.output_item.added conversation.item.added response.content_part.added response.output_audio.delta (repeated) response.output_audio_transcript.delta (repeated) response.output_audio.done response.output_audio_transcript.done response.content_part.done response.output_item.done conversation.item.done response.done ``` Audio and transcript deltas can be interleaved within the response. ## Function calls Function calls arrive as `function_call` output items and always name one of the tools your client declared. Read the function name and `call_id` from `response.output_item.added`, accumulate argument deltas, and act after `response.function_call_arguments.done`. Platform actions are never function calls — they arrive as [call events](#call-events). ### Call capabilities Every session's agent can end the call and can deliberately stay silent for a turn while the far side is still talking — fundamental phone-call behaviors, always on. Sessions with `outbound: true` additionally wait out hold queues, and — with `call_events` enabled — press keypad digits to navigate menus. Transfers also require `call_events`: the agent decides to transfer, and your client executes it on your telephony stack from the `call.transfer` instruction. Capabilities that instruct your client are only granted when the event channel exists to carry the instruction; there is nothing else to configure or declare. A set of function names is reserved for the platform and rejected in your `tools` with a validation error: `end_call`, `transfer_call`, `send_keypad_input`, `no_response`, `wait_on_hold`, `search_knowledge_base`, `play_sound`, `speak_uninterruptible`. ### Call events Platform call actions and state changes are delivered as dedicated events in the `call.*` namespace — a ThunderPhone extension to the realtime event set. They are an explicit opt-in: set `call_events: true` in the session config to receive them. They are notifications: nothing is sent in reply, and there is no `call_id` or output. Handle every type below in your event loop — clients ported from other realtime APIs often switch on known event types and would otherwise drop them. | Event | Payload | Meaning | | --------------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `call.keypad` | `{"digits":"2#"}` | Send these keypad digits out-of-band through your telephony stack now | | `call.transfer` | `{"phone_number":"+15551234567"}` | Transfer your telephony leg to this number now; a `call.ended` with reason `transfer` follows | | `call.ended` | `{"reason":"agent_hangup" \| "remote_hangup" \| "transfer" \| "error"}` | The call is over; the server closes the WebSocket shortly after this event, so drive your teardown from it | | `call.hold.started` / `call.hold.ended` | — | The agent is waiting out a hold queue / the hold ended and normal conversation resumed | | `call.speech_ignored` | `{"reason":"no_response" \| "hold" \| "superseded"}` | The far side said something and the agent deliberately produced no response — it judged the audio as an automated system still talking (`no_response`), hold content while waiting for a human (`hold`), or newer audio made its drafted response stale and it was dropped (`superseded`) | ```json theme={null} { "type": "call.keypad", "event_id": "event_1a2b", "digits": "2" } ``` ```json theme={null} { "type": "call.ended", "event_id": "event_3c4d", "reason": "agent_hangup" } ``` ### Outbound calls Set `outbound: true` in the session config when the session is an outbound call — the agent dialed out, so the far side may be an automated menu, a hold queue, voicemail, or a screener rather than a person. This enables the engine's outbound call handling: the agent recognizes and navigates automated systems reliably, responds immediately when a human picks up so they are not met with dead air, and applies cost-saving measures that keep the price down while the call is working through menus, waiting on hold, or handling voicemail. It is off by default; sessions without it treat the far side as a human caller. Menu navigation and hold handling need no further configuration — steer them through your instructions: ```json theme={null} { "type": "session.update", "session": { "type": "realtime", "instructions": "Call the pharmacy, navigate the phone menu, and ask whether the prescription is ready.", "audio": { "output": { "voice": "olivia" } }, "config": { "outbound": true, "call_events": true, "tools": [ { "type": "function", "function": { "name": "report_status", "description": "Report the prescription status back to the app.", "parameters": { "type": "object", "properties": { "status": { "type": "string" } }, "required": ["status"] } } } ] } } } ``` A standalone function call follows this sequence: ```text theme={null} response.created response.output_item.added response.function_call_arguments.delta response.function_call_arguments.done response.output_item.done response.done ``` ## Using the official OpenAI SDK Existing OpenAI Realtime integrations work by changing only the WebSocket base URL and the API key. Pass the base ending at `/v1` — the SDK appends `/realtime` itself, mirroring `wss://api.openai.com/v1`: ```python theme={null} from openai import AsyncOpenAI client = AsyncOpenAI( api_key="sk_live_YOUR_API_KEY", websocket_base_url="wss://api.thunderphone.com/v1", ) async with client.realtime.connect(model="thunderphone-realtime") as connection: await connection.session.update(session={ "type": "realtime", "instructions": "Listen to the caller and help them complete the call.", }) async for event in connection: ... ``` The `model` argument is echoed back but does not change behavior; the agent configuration comes from the session instructions or a saved agent. ## Minimal Python client Install `websockets`, set `THUNDERPHONE_API_KEY`, and provide an uncompressed mono pcm16 WAV file sampled at 16000 Hz: ```bash theme={null} pip install websockets export THUNDERPHONE_API_KEY=sk_live_YOUR_API_KEY python realtime_client.py caller.wav ``` ```python theme={null} import asyncio import base64 import json import os import sys import wave import websockets URL = "wss://api.thunderphone.com/v1/realtime" RATE = 16_000 CHUNK_MS = 100 def read_wav(path): with wave.open(path, "rb") as wav: if (wav.getnchannels(), wav.getsampwidth(), wav.getframerate()) != (1, 2, RATE): raise ValueError("WAV must be mono pcm16 at 16000 Hz") return wav.readframes(wav.getnframes()) async def send_audio(ws, pcm): chunk_size = RATE * 2 * CHUNK_MS // 1000 loop = asyncio.get_running_loop() deadline = loop.time() for offset in range(0, len(pcm), chunk_size): chunk = pcm[offset : offset + chunk_size] await ws.send(json.dumps({ "type": "input_audio_buffer.append", "audio": base64.b64encode(chunk).decode("ascii"), })) deadline += CHUNK_MS / 1000 await asyncio.sleep(max(0, deadline - loop.time())) async def receive(ws): async for message in ws: if isinstance(message, bytes): continue # output audio, when binary output is enabled event = json.loads(message) event_type = event["type"] if event_type == "conversation.item.input_audio_transcription.completed": print("caller:", event.get("transcript", "")) elif event_type == "response.output_audio_transcript.delta": print(event.get("delta", ""), end="", flush=True) elif event_type == "response.output_audio_transcript.done": print() elif event_type == "response.output_item.done": item = event.get("item", {}) if item.get("type") == "function_call": arguments = json.loads(item.get("arguments") or "{}") print("function:", item["name"], arguments) # Run your declared tool here and return its output. elif event_type == "error": print("error:", event["error"], file=sys.stderr) async def main(path): headers = {"Authorization": f"Bearer {os.environ['THUNDERPHONE_API_KEY']}"} async with websockets.connect(URL, additional_headers=headers) as ws: await ws.send(json.dumps({ "type": "session.update", "session": { "type": "realtime", "instructions": "Listen carefully and help complete the call.", "audio": { "input": {"format": {"type": "audio/pcm", "rate": RATE}}, "output": { "format": {"type": "audio/pcm", "rate": RATE}, "voice": "olivia", }, }, }, })) while json.loads(await ws.recv())["type"] != "session.updated": pass await asyncio.gather(send_audio(ws, read_wav(path)), receive(ws)) asyncio.run(main(sys.argv[1])) ``` ## Protocol notes * Turn detection uses server-side voice activity detection and is always on. Clients do not need to commit or clear the input audio buffer. * Transcript events are emitted as per-utterance segments, not token-by-token deltas. Use the `completed` and `done` events as final text. * Unknown or unsupported client event types produce an `error` event without closing the WebSocket. The client can correct the request and continue.