Twitter’s Observability team provides Twitter developers a monitoring infrastructure including real time dashboards and alerting for their services. Twitter has grown orders of magnitude since it debuted at SXSW. The monitoring infrastructure has followed suit, ingesting orders of magnitude more metrics, from millions to billions.

As the company grew, so did the challenges of providing an always available alerting system. Design decisions made when Twitter had only one datacenter and a monolithic architecture created unresolvable scale and reliability issues in the original alerting service. Monitoring 2.0—Twitter’s next-gen alerting system—solves these issues, improves upon the configuration experience, enhances configuration quality, and focuses on surfacing useful information quickly to the oncall to reduce time to resolution. In this talk we will discuss lessons learned during the evolution of our alerting service, both from a systems and a UX perspective.