Orchestrating Resilience: Building Modern Asynchronous Systems

Building asynchronous, event-driven systems can be daunting. Managing states, ensuring resilience, maintaining traceability, and handling a myriad of other challenges often require more effort than building the functionality itself. This talk will dive into my journey at Twilio, where I faced these complexities firsthand and turned to workflow orchestrators as a solution. I'll share practical examples from our projects, the challenges we faced, and how we overcame them.

We will explore the landscape of various workflow orchestrators, using our experience to highlight why we chose Temporal over other options like Apache Airflow and AWS Step Functions. Moreover, we'll examine strategies for handling partial and known failures, and how orchestrators can simplify and expedite these processes.

In the end, the audience will take away an in-depth understanding of workflow orchestrators' transformative role in asynchronous systems, empowering developers to build reliable, efficient, and easy-to-navigate systems.

Key Takeaways:

1. Gaining insights into the challenges encountered while developing asynchronous systems, including state management, resilience, traceability, observability, and maintainability.
2. Understanding the value of workflow orchestrators in building reliable and efficient asynchronous systems, and how they can significantly reduce the development overhead.
3. Understanding different  workflow orchestrators available today, and their unique features and suitability for various use cases.