Sprinkling eBPF Onto Your Observability

Interview:

What is the focus of your work these days?

I work on continuous profiling software. People may be familiar with regular profiling where you look at a single process for a 10 secs period of time. We get to see, for example, where CPU time within that process is being spent. Continuous profiling means that we're always doing this to all processes and infrastructure and storing this data over time. That allows us to do some super interesting analysis on this data. For example, if we had an incident, we can go back and look at what our processes were doing. We can look at not just a 10 secs period of time, we can look at where the CPU time in our entire infrastructure is being spent. So it allows us to do a novel introspection of our running infrastructure that is incredibly useful for a variety of use cases.

What's the motivation behind your talk?

Some of my talk is related to what we do with continuous profiling, because in order to be able to collect this data at low overhead, we needed to look around at what technology was most suitable to do this. eBPF happened to be the right tool for the job. Actually, at the beginning of when we started looking into the continuous profiling space, we very intentionally did not want to concern ourselves with the collection of the data. We were thinking the storage and querying of this stuff is hard enough, but it became painfully obvious at some point that the overhead of collection was going to be vital for adoption of this technology. 

I think the formula is easy to understand, if the overhead outweighs the benefit of optimizations that we can do based on this data, then it's not really worth doing it. So it's a game of getting the overhead down as much as possible while getting the potential outcome as high as possible. And eBPF allows us to grab this data at a super, super low overhead.

How would you describe the persona and level of the target audience for your session?

This is interesting because eBPF is so versatile, it can be anyone from software engineers who just want to have meaningful insight into their applications. Like I said, with continuous profiling, for example, we find that sometimes even CFOs are interested in this because they can make calculations like, this is the cost of resources per customer or something like that. They want to be able to drive that down as much as possible. So it is really huge, but typically we find that SREs, infra people or software engineers have the most use out of this technology.

Is there anything that you'd like to highlight that you would like this persona to walk away with after watching your session?

It's a mix that eBPF is already, but will continue, to disrupt the observability space. The nature of how eBPF works is is just amazing because all of these trace points have been in the Linux kernel forever, but they were inaccessible, and eBPF all of the sudden all of these trace points that have been maintained by kernel developers over many, many years, now they're becoming accessible to the common folk. One point I want to warn people to walk away with is that there are incredibly exciting opportunities with eBPF. 

The second point is maybe a little bit more realistic where I want to show that while all of these things are really shiny and there is a lot of opportunity, there is also a lot of work that needs to go into this to actually make it happen. 

My last point is going to be, that there are already projects such as the open source project that I happen to work on that have put in this work so that some use cases using eBPF are already very viable to be used today. So this progression of eBPF is exciting, then it's still hard, but there are projects out there that are very successfully using it.