One Does Not Simply Add Synchronous Writes - Bridging Batch and Real Time with Apache Iceberg

QCon San Francisco 2026

Session

One Does Not Simply Add Synchronous Writes - Bridging Batch and Real Time with Apache Iceberg

Tuesday Nov 17 / 01:35PM PST (50 minutes), Seacliff ABC at Hyatt Regency, San Francisco

Register

$3,250, Conference (3 days)
Current pricing ends November 20th

Abstract

What happens when a batch ingestion platform needs to behave like an operational database? Datadog’s Reference Tables product was designed for asynchronous uploads of large datasets, but customers increasingly needed synchronous row-level edits, without sacrificing batch scale. To support both models, we redesigned the platform with Apache Iceberg at the core. The system then uses Temporal workflows to propagate committed changes to low-latency serving systems.

This talk examines how a seemingly local feature requirement exposed deeper mismatches between our system guarantees and the architecture supporting them. We’ll look at how making Iceberg the source of truth simplified correctness and coordination, and how its snapshot model can support operational workflows, not just analytical queries.

Attendees will leave with practical patterns for using Iceberg in operational systems, designing around explicit correctness boundaries, and migrating distributed production systems incrementally without a big-bang rewrite.

Interview

This session is about what happens when a new requirement exposes that a system’s abstractions no longer match the guarantees the product needs. It describes how we used Apache Iceberg’s atomic snapshots in an operational workload, redesigned around stronger invariants, and migrated the system incrementally.

Systems routinely evolve beyond their original workloads, and local feature fixes can quietly create architectural debt. Leaders need to recognize when to revisit the system contract instead of adding more coordination around a mismatched design.

A significant challenge is balancing short-term feature delivery with long-term architectural health. The fastest way to ship a new capability can create duplicate paths, tighter coupling, and harder-to-reason-about guarantees that make future changes more expensive.

Look for places where your system relies on lots of custom logic to maintain correctness, and ask whether the underlying framework could provide that guarantee directly instead.

QCon stands out for the depth of its technical content and the wide array of expert speakers. It’s a rare opportunity to hear experienced practitioners go deep on real systems, tradeoffs, and lessons learned across a wide range of software engineering domains.

Register

$3,250, Conference (3 days). Current pricing ends November 20th. All pass options.

76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon San Francisco 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Tuesday 17 November

10:35 Seacliff ABC Session Breaking Down the Walls to Python & Big Data + AI Performance with Transpilation and LLMs Holden Karau Data Engineer @Snowflake & Co-Founder of Fight Health Insurance, Previously @Netfilx, @Apple, and @Google 11:45 Seacliff ABC Session Data Infrastructure Building a Data Platform with AI Mouli Mukherjee Engineering Manager @OpenAI 13:35 Seacliff ABC Session One Does Not Simply Add Synchronous Writes - Bridging Batch and Real Time with Apache Iceberg Paige Elinson Staff Software Engineer @Datadog 14:45 Seacliff ABC Session Building a StreamHouse: Turning Kafka Streams into a Millisecond Query Layer RJ Atwal Internal Database Engineer, Principal Engineer @Confluent, Previously Founding Engineer @MotherDuck and @AWS 15:55 Seacliff ABC Session From Datasets to Data Products: Powering Trusted AI for Social Impact Shannon McKenzie Sr Principal Engineer @Blackbaud - Building the Data Platform for Social Impact 17:05 Seacliff ABC Session Iceberg V4: The Next Chapter in Table Formats Daniel Weeks MTS @Databricks, Co-Founder of Tabular and Co-Creator of Apache Iceberg, 20+ Years Designing & Developing Large-Scale Distributed Systems