You are viewing content from a past/completed QCon -

Presentation: Avoiding Alerts Overload From Microservices

Track: DevOps: You Build It, You Run It

Location: Ballroom BC

Day of week: Monday

Slides: Download Slides

Level: Intermediate

Persona: Architect, Chaos/Resiliency/SRE, CTO/CIO/Leadership, Developer

What You’ll Learn

  • Focus on the stuff that matters when it comes to developing Microservices
  • Understand how to build a system with support in mind
  • Understand practical approaches to optimizing alerts and the infrastructure to support them

Abstract

Microservices can be a great way to work: the services are simple, you can use the right technology for the job, and deployments become smaller and less risky. Unfortunately, other things become more complex. You probably took some time to design a deployment pipeline and set up self-service provisioning, for example. But did the rest of your thinking about what “done” means catch up? Are you still setting up alerts, run books, and monitoring for each microservice as though it was a monolith?

Two years ago, a team at the FT started out building a microservices-based system from scratch. Their initial naive approach to monitoring meant that an underlying network issue could mean 20 people each receiving 10,000 alert emails overnight. With that volume, you can’t pick out the important stuff. In fact, your inbox is unusable unless you have everything filtered away where you’ll never see it. Furthermore, you have information radiators all over the place, but there’s always something flashing or the wrong color. You can spend the whole day moving from one attention-grabbing screen to another.

That team now has over 150 microservices in production. So how they get themselves out of that mess and regain control of their inboxes and their time? First, you have to work out what’s important, and then you have to ruthlessly narrow down on that. You need to be able to see only the things you need to take action on in a way that tells you exactly what you need to do. Sarah shares how her team regained control and offers some tips and tricks.

Question: 

What is the focus of your work today?

Answer: 

I’m the tech lead for the Content platform at the Financial Times. The platform handles publication of content from multiple content management systems, annotating that content with metadata via concept extraction and editorial curation, and making all of that information available via a set of APIs, so any product within the FT or outside that delivers our content has a stable base to build on.

As part of this work, we are completely revamping our metadata from a taxonomy based metadata - essentially, lists of terms in a variety of categories such as authors, people, companies - with an ontology based one, i.e. based on real things, with an ability to navigate the relationships between those things. As an example, in our new metadata, an author is a person who is a writer. This means we don’t have the same name appearing as a term within people, authors and brands (this is highly confusing to deal with) and gives us a lot more flexibility to show content based on that metadata.

We have five development teams working on our system, which is made up of nearly 300 microservices. We were early adopters of docker, building a lot of the cluster management tools ourselves. My focus this year is making sure that all the work we do on our platform works towards our functional and architectural goals - no local optimizations - and that our production stack is as stable and easy to use as possible so we spend time on the things that matter to our business: for example, we are migrating to Kubernetes and replacing some of our hand written tools.

Question: 

What’s the motivation for your talk?

Answer: 

This was the first microservices architecture I’ve worked on, and one of the early things I learnt is that you just can’t operate microservices the way you did a monolith.

Microservices make the code easier to reason about and deploy frequently, but you have to do DevOps to make it work, and you have to keep a keen focus on building things for operability.

In the early days, my inbox was full of alerts every day, and it was very hard to work out what was something I needed to take action on. We’ve invested a lot of time in finding ways to solve that problem and I wanted to share that experience with others.

Question: 

How you you describe the persona of the target audience of this talk?

Answer: 

Tech Lead/Architect/Developer/Senior Management: anyone operating a microservices architecture or planning to.

Question: 

How would you rate the level of this talk?

Answer: 

It’s a technical talk. It assumes you know about microservices, that you have been responsible for supporting a system.

Question: 

QCon targets advanced architects and sr development leads, what do you feel will be the actionable that type of persona will walk away from your talk with?

Answer: 

There are lots of concrete suggestions of things to install or develop that will help with operating a microservices-based system, but I hope the main actionable will be that you have to care about this stuff and work on it constantly.

Question: 

What do you feel is the most disruptive tech in IT right now?

Answer: 

Serverless feels like it could completely change things. At the moment I’m seeing us use Lambda mostly for more ‘housekeeping’ tasks rather than for production-critical systems, but that’s going to change and I can see the attraction of not having a server to maintain.

I wonder how easy it will be to support a system made up hundreds of functions running on hardware you can’t see. It’ll be a whole new set of observability challenges!

Speaker: Sarah Wells

Principal Engineer @FT (Financial Times)

Sarah Wells is currently leading work at the FT on building a semantic publishing platform, making it easy to discover and access all the FT’s published content via APIs in a common and flexible format. Sarah has been a developer for 15 years, working across consultancy, financial services, and media. She is more dev than ops, but definitely shifting. Her recent focus has been on Go, microservices, containerisation, devops, and how to influence teams to do the right things.

Find Sarah Wells at

Last Year's Tracks

  • Monday, 16 November

  • Clientside: From WASM to Browser Applications

    Dive into some of the technologies that can be leveraged to ultimately deliver a more impactful interaction between the user and client.

  • Languages of Infra

    More than just Infrastructure as a Service, today we have libraries, languages, and platforms that help us define our infra. Languages of Infra explore languages and libraries being used today to build modern cloud native architectures.

  • Mechanical Sympathy: The Software/Hardware Divide

    Understanding the Hardware Makes You a Better Developer

  • Paths to Production: Deployment Pipelines as a Competitive Advantage

    Deployment pipelines allow us to push to production at ever increasing volume. Paths to production looks at how some of software's most well known shops continuous deliver code.

  • Java, The Platform

    Mobile, Micro, Modular: The platform continues to evolve and change. Discover how the platform continues to drive us forward.

  • Security for Engineers

    How to build secure, yet usable, systems from the engineer's perspective.

  • Tuesday, 17 November

  • Modern Data Engineering

    The innovations necessary to build towards a fully automated decentralized data warehouse.

  • Machine Learning for the Software Engineer

    AI and machine learning are more approachable than ever. Discover how ML, deep learning, and other modern approaches are being used in practice by Software Engineers.

  • Inclusion & Diversity in Tech

    The road map to an inclusive and diverse tech organization. *Diversity & Inclusion defined as the inclusion of all individuals in an within tech, regardless of gender, religion, ethnicity, race, age, sexual orientation, and physical or mental fitness.

  • Architectures You've Always Wondered About

    How do they do it? In QCon's marquee Architectures track, we learn what it takes to operate at large scale from well-known names in our industry. You will take away hard-earned architectural lessons on scalability, reliability, throughput, and performance.

  • Architecting for Confidence: Building Resilient Systems

    Your system will fail. Build systems with the confidence to know when they do and you won’t.

  • Remotely Productive: Remote Teams & Software

    More and more companies are moving to remote work. How do you build, work on, and lead teams remotely?

  • Wednesday, 18 November

  • Operating Microservices

    Building and operating distributed systems is hard, and microservices are no different. Learn strategies for not just building a service but operating them at scale.

  • Distributed Systems for Developers

    Computer science in practice. An applied track that fuses together the human side of computer science with the technical choices that are made along the way

  • The Future of APIs

    Web-based API continue to evolve. The track provides the what, how, and why of future APIs, including GraphQL, Backend for Frontend, gRPC, & ReST

  • Resurgence of Functional Programming

    What was once a paradigm shift in how we thought of programming languages is now main stream in nearly all modern languages. Hear how software shops are infusing concepts like pure functions and immutablity into their architectures and design choices.

  • Social Responsibility: Implications of Building Modern Software

    Software has an ever increasing impact on individuals and society. Understanding these implications helps build software that works for all users

  • Non-Technical Skills for Technical Folks

    To be an effective engineer, requires more than great coding skills. Learn the subtle arts of the tech lead, including empathy, communication, and organization.