Silent Agent Failures: Detect and Fix Them Before Your Customers Notice

QCon San Francisco 2026

Session

Silent Agent Failures: Detect and Fix Them Before Your Customers Notice

Monday Nov 16 / 01:35PM PST (50 minutes), Ballroom A at Hyatt Regency, San Francisco

Register

$3,250, Conference (3 days)
Current pricing ends November 20th

Abstract

An agent run can receive a success verdict, complete every tool call without an error, and still fail the customer. It skips a necessary step, cites the wrong evidence, or delivers a confident answer that doesn’t resolve the request. Conventional alerts report that everything worked, so these silent failures surface only when a customer complains.

Silent failures come in three kinds, and each needs its own way to detect and fix it.

  • LLM flip-flops, where the same input gets different verdicts on different runs, cluster in the gray zone near a decision boundary. Disagreement across reruns finds them, and memory grounds the agent: semantic memory for domain knowledge and policies, and episodic memory for similar past cases.
  • Other known failures, such as a sort column that comes back entirely empty or a claim its own cited evidence doesn’t support, can be caught by rules or a fast classifier. They can be corrected inline by validators in the agent harness that reject, repair, or flag a call before the customer sees the result.
  • Unknown failures call for anomaly detection on production traces: how agents plan, use tools, recover from uncertainty, and reach answers, grouped into failure modes and tied to user outcomes.

Detection is only half the work. Each failure you surface can become a known, cheaper check, closing a self-improvement loop that many teams want but struggle to make operational. In this session, you’ll get a map of where agents go wrong, organized by module and by lifecycle stage, and a practical playbook that combines rules, disagreement, anomaly detection, memory, and inline feedback.

You’ll learn how to:

  • Recognize silent failures that conventional alerts and one-shot tests miss.
  • Diagnose where an agent went wrong, using a taxonomy of failures by module and by lifecycle stage.
  • Use disagreement across reruns to find the gray zone, and ground the agent with semantic and episodic memory.
  • Place validators at the right hook points (before a tool call, after it, and at the end of a turn) to reject, repair, or flag.
  • Surface unknown failures from trace patterns, and turn each one into a cheaper known check.
  • Measure whether inline retries improve outcomes without adding unnecessary retries or new failure modes.
Register

$3,250, Conference (3 days). Current pricing ends November 20th. All pass options.

76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon San Francisco 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Slides & video

Slides Not available

Share

From the same track

Monday 16 November

10:35 Ballroom A Session Running Software Factory Efficiently at Uber Scale: Holding Agent Spend Steady While Requests Grew 9X Uday Kiran Medisetty VP, Distinguished Engineer @Uber 11:45 Ballroom A Session AI How Airbnb Guardrailed Its AI Customer Support Agent Weiping Peng Distinguished Engineer @Airbnb, Previously Principal Architect @Salesforce for AI platform 13:35 Ballroom A Session Silent Agent Failures: Detect and Fix Them Before Your Customers Notice Diane Lin Staff Engineer @Datadog 14:45 Seacliff D Unconference Unconference: Guardrails & Safety Nets 15:55 Ballroom A Session Agentic Trust: Securing AI Agents in Production Amanda Walker Sr. Director, Engineering - Privacy, Safety and Security @Google 17:05 Ballroom A Session Into the Labyrinth: What Distributed Systems Have Taught Us About Guardrails Akshay Shah Field CTO @Antithesis