Abstract
Every LLM request is really two very different jobs. Reading your prompt (prefill) is compute-intensive, while generating the response token by token (decode) is dominated by memory bandwidth. For years, the industry ran both phases on the same hardware and accepted the compromise. Today, the largest AI providers are pulling inference apart, running each phase on the hardware best suited for it and streaming the model's working state between machines mid-request. The result can be up to 5x higher throughput, but at the cost of turning inference into a fundamentally distributed systems problem.
At Cerebras, we built one of the industry's first production disaggregated inference platforms spanning GPUs and the Cerebras Wafer-Scale Engine. In this talk, we'll break down the anatomy of that architecture: why prefill and decode compete for resources when co-located, what actually moves across the network when they're separated, and why the interconnect quickly becomes the most critical part of the system. We'll dive into the engineering challenges that emerge when a single request spans heterogeneous hardware: routing and scheduling work across multiple machines, balancing prefill and decode capacity across an entire fleet, debugging requests that hop between accelerators, handling failures without disrupting in-flight generations, and maximizing utilization under real datacenter power and capacity constraints.
No prior ML infrastructure experience is required. If you've built distributed systems, worked with queues, caches, schedulers, or networked services, you'll leave with a practical understanding of the architecture behind the next generation of large-scale AI inference, and the engineering tradeoffs required to make it work.
