Abstract
A voice agent looks like a chatbot with a microphone. Architecturally, it's a real-time distributed system with extremely unforgiving constraints: less than 500ms of end-to-end response latency across a multi-model pipeline, intelligent and natural-sounding conversation, and reliability when scaling to thousands of concurrent calls.
This talk walks through the architecture of a production voice agent end to end: the streaming pipeline that chains speech-to-text, turn detection, an LLM, and text-to-speech into a single system, and the design decisions that make it work under real traffic. We'll dig into where the latency budget actually goes, why colocating and serving your own models can beat just calling inference APIs, and why autoscaling stateful, long-lived audio connections is harder than it looks.
