Speaker
Abstract
This session explores the evolution of infrastructure strategy, focusing on the shift from traditional YAML configurations to pipelines-as-code. We explore how workflows defined in modern CI/CD systems like Github Actions, Gitlab, and CircleCI often start simple, but eventually transform into a labyrinth of unmanageable YAML code. Using Airbyte (an open source ETL sync tool)’s journey as a case study, we introduce Dagger, a tool that allows defining pipelines in your preferred language, simplifying CI/CD management. This session highlights the benefits of pipelines-as-code over traditional methods, providing practical strategies to enhance developer experience and build agile, reliable, and scalable CI/CD processes.
Interview
My name is Conor and I work on the infrastructure powering Airbyte. I’m primarily concerned currently with building scalable systems that involve orchestration of large amounts of repeated work like moving data and CI/CD.
To share strategies around how to build scalable CI/CD systems and show others that there are people building good systems beyond YAML (there is a light at the end of the tunnel) and that it’s not an intractable problem
The ideal persona for this talk is an engineer or someone in the engineering organization that regularly feels the pain of a poorly maintained or scaled CI/CD system looking for actionable insights on how to think about how their own CI/CD system is structured. The target audience should have at least a high level knowledge of CI/CD systems and CLI software works in order to gain the most actionable insights from the talk.
Ideally the consumer of the presentation has actionable approaches for designing and implement their own CI/CD system as code incrementally, and the benefits/cost savings that they can realize if they do so.
Topics
QCon San Francisco 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Languages of Infra: Beyond YAML Hosted by Justin Cormack CTO @DockerFrom the same track
Monday 2 October
10:35 Pacific DEKJ Session Cloud Engineering Pulumi Adventures: How Python Empowered My Infrastructure Beyond YAML Adora Nwodo Community Engineer In recent times and as technology keeps evolving, the boundary between software engineering and DevOps continues to blur, presenting unique challenges and opportunities for software engineers. 11:45 Pacific DEKJ Session Frontend Framework Defined Infrastructure (FdI) – an Evolution of Infrastructure as Code (IaC) Malte Ubl CTO @Vercel Discover the future of infrastructure management with Framework-defined Infrastructure (FdI), an evolution of the industry-standard Infrastructure as Code (IaC). 13:35 Pacific DEKJ Session Infrastructure as Code What if Infrastructure as Code Never Existed? Adam Jacob CEO @System Initiative, CTO @Chef; Systems Administrator who loves building products We rewind the clock and ask ourselves: what if Infrastructure as Code never existed? What different solutions might we have come up with to the problems we faced? We’ll leave you full of possibilities about the future. 14:45 Seacliff D Unconference Unconference: Languages of Infra: Beyond YAML What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions. 15:55 Pacific DEKJ Session CI/CD CI/CD Beyond YAML Conor Barber Senior Software Engineer, Infrastructure @Airbyte, Over a Decade of Experience in Data and Infrastructure Engineering, Previously @Apple This session explores the evolution of infrastructure strategy, focusing on the shift from traditional YAML configurations to pipelines-as-code. 17:05 Pacific DEKJ Session Kubernetes Kubernetes without YAML David Flanagan Kubernetes Whisperer Kubernetes ONLY understands JSON, yet we all write a ton of YAML. This is because it's meant to be easier for us to write ... but writing our manifests in YAML can be complex, error-prone, and damn right opaque as we adopt more custom resources.