Presentation: Future of Data Engineering

Track: Modern Data Architectures

Day of week: Tuesday

Share this on:

Abstract

The current generation of data engineering has left us with data pipelines, data warehouses, and machine learning platforms that are largely batch-based and centrally managed. They're often largely manually operated, and integrating new systems can be cumbersome. Over the next few years, a number of trends are going to require us to rethink how and what we build. Data is now realtime, companies are running many database technologies, teams are demanding more control of their data, and regulatory policy has begun dictating how and when we store data. This talk will present a vision of what it will take for data engineers deliver a next generation data ecosystem.

Speaker: Chris Riccomini

Distinguished Engineer @WePay

Qcon

Software engineer. WePay, LinkedIn, PayPal. Co-author Apache Samza. Committer/PMC Apache Airflow.

Find Chris Riccomini at

Tracks

Monday, 11 November

  • Building & Scaling High-Performing Teams

    To have a high-performing team, everybody on it has to feel and act like an owner. Organizational health and psychological safety are foundational underpinnings to support ownership.

  • Bare Knuckle Performance

    Killing latency and getting the most out of your hardware

  • Ethics, Regulation, Risk, and Compliance

    With so much uncertainty, how do you bulkhead your organization and technology choices? Learn strategies for dealing with uncertainty.

  • Software Supply Chain

    Life of a software artifact from commit to deployment. Security, observability and provenance of the software supply chain.

  • Architectures You've Always Wondered About

    Next-gen architectures from the most admired companies in software, such as Netflix, Google, Facebook, Twitter, & more

  • Languages of Infrastructure

    This track explores languages being used to code the infrastructure. Expect practices on toolkits and languages like Cloudformation, Terraform, Python, Go, Rust, Erlang.

Tuesday, 12 November

Wednesday, 13 November