Speaker
Abstract
AI inference is expensive, but it doesn’t have to be. In this talk, we’ll break down how to systematically drive down the cost per token across different types of AI workloads. Using real-world examples from data transformation, offline agents, and aggregated insights, we’ll unpack how to measure, optimize, and ultimately produce the world’s cheapest tokens. The session will be hardware-agnostic, featuring analysis of both Nvidia and AMD GPUs, and will include advice which can be implemented by using open-source serving frameworks such as Dynamo, vLLM, and SGLang.
What you'll take away:
- Token Economics 101 - Understand what actually drives cost per token
- Inference Optimization Tactics that can be used to drive down unit economics depending on the AI workload type
- Right GPU, Right Job - Ho two choose hardware and deployment strategy for maximum cost performance
Topics
QCon San Francisco 2025 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Wednesday 19 November
10:35 Ballroom A Session AI/ML Producing the World's Cheapest Tokens: A How-to Guide Meryem Arik Co-Founder and CEO @Doubleword (Previously TitanML), Recognized as a Technology Leader in Forbes 30 Under 30, Recovering Physicist AI inference is expensive, but it doesn’t have to be. In this talk, we’ll break down how to systematically drive down the cost per token across different types of AI workloads. 11:45 Ballroom A Session Capacity Planning How Netflix Shapes our Fleet for Efficiency and Reliability Joseph Lynch, Argha C Netflix runs on a complex multi-layer cloud architecture made up of thousands of services, caches, and databases. As hardware options, workload patterns, cost dynamics and the Netflix products evolve, the cost-optimal hardware and configuration for running our services is constantly changing. 13:35 Ballroom A Session AI Architecture Realtime and Batch Processing of GPU Workloads Joseph Stein Principal Architect of Research & Development @SS&C Technologies, Previous Apache Kafka Committer and PMC Member SS&C Technologies runs 47 trillion dollars of assets on our global private cloud. We have the primitives for infrastructure as well as platforms as a service like Kubernetes, Kafka, NiFi, Databases, etc. 14:45 Ballroom A Session Architecture From ms to µs: OSS Valkey Architecture Patterns for Modern AI Dumanshu Goyal Uber Technical Lead @Airbnb Powering $11B Transactions, Formerly @Google and @AWS As AI applications demand faster and more intelligent data access, traditional caching strategies are hitting performance and reliability limits. 15:55 Ballroom A Session Platform Engineering Write-Ahead Intent Log: A Foundation for Efficient CDC at Scale Vinay Chella, Akshat Goel As companies grow, so does the complexity of keeping distributed systems in sync. At DoorDash, we tackled this challenge while building a high-throughput, domain-oriented data platform for capturing changes across hundreds of services.