One Platform to Serve Them All: Autoscaling Multi-Model LLM Serving

AI teams are moving to self-hosted inference away from hosted LLMs as fine-tuning drives model performance. The catch is scale, hundreds of variants create long-tail traffic, cold starts, and duplicated stacks. This talk shows how one platform can autoscale multi-model serving with shared base weights, hot-swapped adapters, dynamic loading, and smart eviction. You will see how to hold P95 and P99 while serving hundreds of fine-tuned models at near single-model cost. We focus on Kubernetes, dynamo, vLLM and SGLang as core technologies, and the metrics that matter, such as: tokens per second, model load latency, and cost per 1k tokens.