This session explores the evolution of infrastructure strategy, focusing on the shift from traditional YAML configurations to pipelines-as-code. We explore how workflows defined in modern CI/CD systems like Github Actions, Gitlab, and CircleCI often start simple, but eventually transform into a labyrinth of unmanageable YAML code. Using Airbyte (an open source ETL sync tool)’s journey as a case study, we introduce Dagger, a tool that allows defining pipelines in your preferred language, simplifying CI/CD management. This session highlights the benefits of pipelines-as-code over traditional methods, providing practical strategies to enhance developer experience and build agile, reliable, and scalable CI/CD processes.
Interview:
What's the focus of your work these days?
My name is Conor and I work on the infrastructure powering Airbyte. I’m primarily concerned currently with building scalable systems that involve orchestration of large amounts of repeated work like moving data and CI/CD.
What's the motivation for your talk at QCon San Francisco 2023?
To share strategies around how to build scalable CI/CD systems and show others that there are people building good systems beyond YAML (there is a light at the end of the tunnel) and that it’s not an intractable problem
How would you describe your main persona and target audience for this session?
The ideal persona for this talk is an engineer or someone in the engineering organization that regularly feels the pain of a poorly maintained or scaled CI/CD system looking for actionable insights on how to think about how their own CI/CD system is structured. The target audience should have at least a high level knowledge of CI/CD systems and CLI software works in order to gain the most actionable insights from the talk.
Is there anything specific that you'd like people to walk away with after watching your session?
Ideally the consumer of the presentation has actionable approaches for designing and implement their own CI/CD system as code incrementally, and the benefits/cost savings that they can realize if they do so.
Speaker
Conor Barber
Senior Software Engineer, Infrastructure @Airbyte, Over a Decade of Experience in Data and Infrastructure Engineering, Previously @Apple
Conor Barber is a software engineer at Airbyte, bringing over a decade of experience in data and infrastructure engineering from leading tech companies. Previously at Apple, Conor develops scalable solutions for complex CI/CD processes involved in managing hundreds of connectors and the ELT platform that the connectors run on at Airbyte.
Find Conor Barber at:
Date
Monday Oct 2 / 03:55PM PDT ( 50 minutes )
Location
Pacific DEKJ
