Speaker
Abstract
Testing AI agents properly is surprisingly hard. Production data sources are moving targets, static datasets go stale, and pointing agents at real systems is risky and impossible to reproduce. As agents take on more complex, stateful work, the testing approaches that served traditional software start to break down.
In this talk, I’ll share what we’ve learned about testing agents reliably and at scale. Where does realistic test data come from, and how do you capture the tribal knowledge that never makes it into documentation? How do you run 100 agents concurrently, each with its own isolated, stateful view of the world, without the infrastructure falling over? And when the real tools your agents depend on change, how do your tests keep up?
Attendees will leave with a clearer picture of what it takes to test agents in
ways that are realistic, reproducible, and built to scale.
1 Why agent testing breaks traditional approaches: the specific ways live
data, stale datasets, and real-system dependencies undermine reliable,
reproducible agent tests.
2 How to build realistic test conditions: what goes into curating
believable test data and surfacing the tribal knowledge that agents need but
no one wrote down.
3 How to scale agent testing without sacrificing isolation: the
engineering challenges of running many concurrent, stateful agent sessions,
and keeping tests in sync as the underlying tools evolve.
Sponsored session
$3,250, Conference (3 days). Current pricing ends November 20th. All pass options.
QCon San Francisco 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Slides & video
Video Not available
Part of the track
Sponsored Solution Track VFrom the same track
Wednesday 18 November
10:35 Pacific LM Sponsored Simulating the Universe, 100 Times Over: Engineering Highly- Concurrent Agent Testing Andy Liang, Anvit Sinha Testing AI agents properly is surprisingly hard. Production data sources are moving targets, static datasets go stale, and pointing agents at real systems is risky and impossible to reproduce. 11:45 Pacific LM Sponsored Multi-Agent Coding with Omnigent Evelyn Hur, Denny Lee Most developers now use several coding agents at once: one for quick edits, another for long refactors, a third that a teammate recommended. Each lives in its own tool with its own setup, and none of them share what the others know. Switching means rebuilding context by hand. 13:35 Pacific LM Sponsored The Fast, The Furious and The Fatigued Christian Weber For years, achieving daily releases has been the gold standard of DevOps maturity. But what happens when you get there? According to new data from over 700 engineering leaders, high deployment frequency is increasingly masking a crisis of off-hours rework, instability, and engineer burnout.