CI/CD Beyond YAML

This session explores the evolution of infrastructure strategy, focusing on the shift from traditional YAML configurations to pipelines-as-code. We explore how workflows defined in modern CI/CD systems like  Github Actions, Gitlab, and CircleCI often start simple, but eventually transform into a labyrinth of unmanageable YAML code. Using Airbyte (an open source ETL sync tool)’s journey as a case study, we introduce Dagger, a tool that allows defining pipelines in  your preferred language, simplifying CI/CD management. This session highlights the benefits of pipelines-as-code over traditional methods, providing practical strategies to enhance developer experience and build  agile, reliable, and scalable CI/CD processes.

Interview:

What's the focus of your work these days?

My name is Conor and I work on the infrastructure powering Airbyte. I’m primarily concerned currently with building scalable systems that involve orchestration of large amounts of repeated work like moving data and CI/CD.

What's the motivation for your talk at QCon San Francisco 2023?

To share strategies around how to build scalable CI/CD systems and show others that there are people building good systems beyond YAML (there is a light at the end of the tunnel) and that it’s not an intractable problem

How would you describe your main persona and target audience for this session?

The ideal persona for this talk is an engineer or someone in the engineering organization that regularly feels the pain of a poorly maintained or scaled CI/CD system looking for actionable insights on how to think about how their own CI/CD system is structured. The target audience should have at least a high level knowledge of CI/CD systems and CLI software works in order to gain the most actionable insights from the talk.

Is there anything specific that you'd like people to walk away with after watching your session?

Ideally the consumer of the presentation has actionable approaches for designing and implement their own CI/CD system as code incrementally, and the benefits/cost savings that they can realize if they do so.


Speaker

Conor Barber

Senior Software Engineer, Infrastructure @Airbyte, Over a Decade of Experience in Data and Infrastructure Engineering, Previously @Apple

Conor Barber is a software engineer at Airbyte, bringing over a decade of experience in data and infrastructure engineering from leading tech companies. Previously at Apple, Conor develops scalable solutions for complex CI/CD processes involved in managing hundreds of connectors and the ELT platform that the connectors run on at Airbyte.

Read more
Find Conor Barber at:

Date

Monday Oct 2 / 03:55PM PDT ( 50 minutes )

Location

Pacific DEKJ

Topics

CI/CD Infrastructure Platform Engineering

Share

From the same track

Session Infrastructure as Code

What if Infrastructure as Code Never Existed?

Monday Oct 2 / 01:35PM PDT

We rewind the clock and ask ourselves: what if Infrastructure as Code never existed? What different solutions might we have come up with to the problems we faced? We’ll leave you full of possibilities about the future.

Speaker image - Adam Jacob

Adam Jacob

CEO @System Initiative, CTO @Chef; Systems Administrator who loves building products

Session Frontend

Framework Defined Infrastructure (FdI) – an Evolution of Infrastructure as Code (IaC)

Monday Oct 2 / 11:45AM PDT

Discover the future of infrastructure management with Framework-defined Infrastructure (FdI), an evolution of the industry-standard Infrastructure as Code (IaC).

Speaker image - Malte Ubl

Malte Ubl

CTO @Vercel

Session Cloud Engineering

Pulumi Adventures: How Python Empowered My Infrastructure Beyond YAML

Monday Oct 2 / 10:35AM PDT

In recent times and as technology keeps evolving, the boundary between software engineering and DevOps continues to blur, presenting unique challenges and opportunities for software engineers.

Speaker image - Adora Nwodo

Adora Nwodo

Community Engineer

Session Kubernetes

Kubernetes without YAML

Monday Oct 2 / 05:05PM PDT

Kubernetes ONLY understands JSON, yet we all write a ton of YAML. This is because it's meant to be easier for us to write ... but writing our manifests in YAML can be complex, error-prone, and damn right opaque as we adopt more custom resources.

Speaker image - David Flanagan

David Flanagan

Kubernetes Whisperer

Session

Unconference: Languages of Infra: Beyond YAML

Monday Oct 2 / 02:45PM PDT

What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.