Field guide / 10 min
Design Realtime as a live system
Transport, turn detection, interruption, and tools shape the experience.
Realtime sessions are designed for low-latency live interactions. A production voice experience is more than streaming audio: it needs a transport chosen for the client, explicit session state, interruption behavior, tool latency budgets, and recovery from partial turns.
Working principles
Outcome selects architecture
Use a live session for interactive low-latency work; use request-based audio for files and bounded generation.
Interruption is normal
Users speak over systems, networks jitter, and tools finish late. Treat partial audio and cancellation as core states.
Keep credentials server-side
Clients should receive narrowly scoped session credentials or use a server-mediated path; never ship a long-lived platform key.
Field procedure
- 01
Define the conversation job
State whether the system responds, translates, or transcribes. These outcomes use different session lifecycles.
- 02
Choose the transport
Prefer browser-oriented realtime transport for direct media clients and server-oriented streaming for telephony or media pipelines.
- 03
Model turn states
Track listening, committing, thinking, speaking, interrupted, tool-waiting, reconnecting, and ended.
- 04
Budget tool latency
Set deadlines, acknowledgement cues, and fallback language for tools that cannot finish inside a conversational pause.
- 05
Test real conditions
Evaluate noisy rooms, accents, slow networks, device changes, double-talk, reconnects, and long sessions.
PASS / FAIL
Acceptance checklist
- Long-lived API keys never reach clients.
- The UI exposes listening and speaking state.
- Interruption cancels stale output and tool work safely.
- Session logs connect transport, turn, tool, and outcome events.
WATCH / REJECT
Failure patterns
- Treating a demo microphone loop as production architecture.
- Ignoring echo, reconnect, or overlapping speech.
- Allowing late tool results to mutate a newer conversational turn.