Instrumentation at Scale: Having Your Performance Cake and Eating It Too

QCon San Francisco 2025

Session

Instrumentation at Scale: Having Your Performance Cake and Eating It Too

Wednesday Nov 19 / 11:45AM PST, Pacific DEKJ at Hyatt Regency, San Francisco

Abstract

In high-performance code, a single misplaced counter increment can cost more than the operation it’s measuring. That creates a paradox: instrument too much and you slow the system down; instrument too little and you miss the insights you need to continuously deliver.

This talk focuses on techniques for instrumenting latency-sensitive, high-throughput systems with minimal impact—approaches rooted in C and Rust, but with lessons that may apply more broadly. We’ll examine the true costs of metrics collection, the pitfalls of percentile reporting, and how to extend observability from application code down to the kernel using eBPF. Along the way, we’ll discuss cacheline-aware counter design, the trade-offs in struct and memory layout, and the value of a unified metrics framework for both application and infrastructure insights.

Attendees will gain practical, language-level strategies for building observability into performance-critical systems—without sacrificing the speed their users expect.

76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon San Francisco 2025 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Wednesday 19 November

10:35 Pacific DEKJ Session Rust The Rust High Performance Talk You Did Not Expect Ruth Linehan Software Engineer @Momento, Previously APIs/Webhooks @GitHub and @Puppet 11:45 Pacific DEKJ Session Instrumentation at Scale: Having Your Performance Cake and Eating It Too Brian Martin Co-founder and Software Engineer @IOP Systems, Focused on High-Performance Software and Systems, Previously @Twitter 13:35 Pacific DEKJ Session Performance When Every Bit Counts: How Valkey Rebuilt Its Hashtable for Modern Hardware Madelyn Olson Principal Engineer @AWS, Maintainer of the Open-Source Valkey Project 14:45 Pacific DEKJ Session Performance Accelerating Performance by Incrementally Integrating Rust Into Existing Codebase Lily Mara Staff Engineer @Discord, Author of "Refactoring to Rust", Previously Engineering Manager @OneSignal 15:55 Pacific DEKJ Session Python Python, Numba, and Algorithm Design: Building Efficient Models in Financial Services Chad Schuster Principal @Milliman Focusing on Risk Management, Modeling, and Technology Consulting Services