Embracing Shift-Left in Data Architecture

QCon San Francisco 2024

Track

Embracing Shift-Left in Data Architecture

Tuesday 19 November · 6 sessions, 50 minutes each

About the track

In the rapidly evolving digital landscape, the way we approach data architecture is undergoing a transformative shift. This shift is not just about adopting new technologies but about fundamentally rethinking our approach to data management, governance and architecture design. Welcome to the concept of "Shift-Left Data Architecture" – a methodology that promises to set the foundation for future-ready data ecosystems.

As data's role in decision-making, operations and machine learning has become increasingly critical, the need for a more proactive approach has become evident. We need to reconsider traditional methods where data considerations, and supporting ML often came later in the development process, which led to inefficiencies, increased costs, and data quality and outcomes issues. By shifting left, organizations can avoid costly revisions, enhance data security, and ensure that their data architecture is robust and scalable.

Join us to learn more about this new era in data architectures, the building blocks of a shift-left architecture, the tools and technologies that enable it, and gain insights on how to implement these principles effectively within your organization.

Sessions in this track

Tuesday 19 November. 6 sessions per track, chosen and introduced by the Track Host.

10:35 Ballroom A Session Platform Engineering Beyond Durability: Enhancing Database Resilience and Reducing the Entropy Using Write-Ahead Logging at Netflix Prudhviraj Karumanchi, Vidhya Arvind 11:45 Pacific DEKJ Session Architecture OpenSearch Cluster Topologies for Cost-Saving Autoscaling Amitai Stern Engineering Manager @Logz.io, Managing Observability Data Storage of Petabyte Scale, OpenSearch Leadership Committee Member and Contributor 13:35 Ballroom A Session Stream All the Things — Patterns of Effective Data Stream Processing Adi Polak Director, Advocacy and Developer Experience Engineering @Confluent, Author of "Scaling Machine Learning with Spark" and "High Performance Spark 2nd Edition" 14:45 Ballroom A Session Stream and Batch Processing Convergence in Apache Flink Jiangjie (Becket) Qin Principal Staff Software Engineer @LinkedIn, Data Infra Engineer, PMC Member of Apache Kafka & Apache Flink, Previously @Alibaba and @IBM 15:55 Ballroom A Session Data Pipelines Efficient Incremental Processing with Netflix Maestro and Apache Iceberg Jun He Staff Software Engineer @Netflix, Managing and Automating Large-Scale Data/ML Workflows, Previously @Airbnb and @Hulu 17:05 Seacliff D Unconference Unconference: Shift-Left Data Architecture
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon San Francisco 2024 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share