Running LLMs requires significant computational power, which scales with model size and context length. We will discuss strategies for fitting models to various hardware configurations and share techniques for optimizing inference latency and throughput at Meta.

As we transition from stand-alone LLMs to production grade systems that support LLM at a global scale, we delve into our approach to constructing systems that accommodate dynamic user requests and widespread product adoption. This includes implementing caching strategies and addressing infra latency, efficiency and reliability issues within real data centers of a heterogeneous hardware fleet.

Finally, we will present case studies that demonstrate our methods for achieving a balance between model quality, latency, throughput, reliability, and cost in a complex and demanding environment.