Abstract
A voice agent looks like a chatbot with a microphone. Architecturally, it's a real-time distributed system with extremely unforgiving constraints: less than 500ms of end-to-end response latency across a multi-model pipeline, intelligent and natural-sounding conversation, and reliability when scaling to thousands of concurrent calls.
This talk walks through the architecture of a production voice agent end to end: the streaming pipeline that chains speech-to-text, turn detection, an LLM, and text-to-speech into a single system, and the design decisions that make it work under real traffic. We'll dig into where the latency budget actually goes, why colocating and serving your own models can beat just calling inference APIs, and why autoscaling stateful, long-lived audio connections is harder than it looks.
Speaker
Rishabh Bhargava
Director of ML @Together AI
Rishabh Bhargava is Director of ML at Together AI, where he leads the Voice AI team building low-latency speech models and real-time inference for production voice agents. Previously, Rishabh was co-founder and CEO of Refuel.ai, which built purpose-built LLMs and a platform for enterprise-scale data workflows and was acquired by Together AI in 2025. Across 10+ years in AI/ML infrastructure, he has shipped ML systems at Primer.ai and Datacoral (acquired by Cloudera). He did his graduate work in CS at Stanford and is based in San Francisco.
