Instrumentation at Scale: Having Your Performance Cake and Eating It Too

Abstract

In high-performance code, a single misplaced counter increment can cost more than the operation it’s measuring. That creates a paradox: instrument too much and you slow the system down; instrument too little and you miss the insights you need to continuously deliver.

This talk focuses on techniques for instrumenting latency-sensitive, high-throughput systems with minimal impact—approaches rooted in C and Rust, but with lessons that may apply more broadly. We’ll examine the true costs of metrics collection, the pitfalls of percentile reporting, and how to extend observability from application code down to the kernel using eBPF. Along the way, we’ll discuss cacheline-aware counter design, the trade-offs in struct and memory layout, and the value of a unified metrics framework for both application and infrastructure insights.

Attendees will gain practical, language-level strategies for building observability into performance-critical systems—without sacrificing the speed their users expect.


Speaker

Brian Martin

Co-founder and Software Engineer @IOP Systems, Focused on High-Performance Software and Systems, Previously @Twitter

Brian is a software engineer who focuses on performance optimization and distributed systems. He worked at Twitter for 8 years, initially with the Cache Team and later as a member of the newly created Performance Team. After November 2022, Brian joined his teammates from Twitter as a co-founder of IOP Systems and continues to work on improving software and platform performance, efficiency, and reliability.

Read more

From the same track

Session

The High Performance Talk You Did Not Expect To Hear about Rust

Abstract: Rust makes things faster but it slows down engineers, right? This was our team’s assumption when we decided to rewrite our code from Kotlin into Rust. But, we were wrong in completely unexpected ways.

Speaker image - Ruth Linehan

Ruth Linehan

Software Engineer @Momento, Previously APIs/Webhooks @GitHub and @Puppet

Session

When Every Bit Counts: How Valkey Rebuilt Its Hashtable for Modern Hardware

Ever wondered what happens when a bunch of performance-obsessed developers decide their blazing-fast database isn't quite blazing-fast enough?

Session

Building High-Concurrency Systems for the Cloud

Details coming soon.

Session

Modern Systems Programming for the Next Decade

Details coming soon.