Testing AI agents properly is surprisingly hard. Production data sources are moving targets, static datasets go stale, and pointing agents at real systems is risky and impossible to reproduce. As agents take on more complex, stateful work, the testing approaches that served traditional software start to break down.

In this talk, I’ll share what we’ve learned about testing agents reliably and at scale. Where does realistic test data come from, and how do you capture the tribal knowledge that never makes it into documentation? How do you run 100 agents concurrently, each with its own isolated, stateful view of the world, without the infrastructure falling over? And when the real tools your agents depend on change, how do your tests keep up?
Attendees will leave with a clearer picture of what it takes to test agents in
ways that are realistic, reproducible, and built to scale.
1 Why agent testing breaks traditional approaches: the specific ways live
data, stale datasets, and real-system dependencies undermine reliable,
reproducible agent tests.
2 How to build realistic test conditions: what goes into curating
believable test data and surfacing the tribal knowledge that agents need but
no one wrote down.
3 How to scale agent testing without sacrificing isolation: the
engineering challenges of running many concurrent, stateful agent sessions,
and keeping tests in sync as the underlying tools evolve.