Abstract
What happens when a batch ingestion platform needs to behave like an operational database? Datadog’s Reference Tables product was designed for asynchronous uploads of large datasets, but customers increasingly needed synchronous row-level edits, without sacrificing batch scale. To support both models, we redesigned the platform with Apache Iceberg at the core. The system then uses Temporal workflows to propagate committed changes to low-latency serving systems.
This talk will examine the tradeoffs between synchronous guarantees and asynchronous processing and how Iceberg can support operational as well as analytical use cases. Attendees will leave with practical patterns for ensuring data integrity, migrating distributed production systems incrementally, and supporting workloads with a range of latency and scale requirements.
$2,955, Conference (3 days). Current pricing ends October 13th. All pass options.
QCon San Francisco 2026 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
From the same track
Tuesday 17 November
10:35 Seacliff ABC Session Breaking Down the Walls to Python & Big Data + AI Performance with Transpilation and LLMs Holden Karau Data Engineer @Snowflake & Co-Founder of Fight Health Insurance, Previously @Netfilx, @Apple, and @Google Many of us have a love hate relationship with Apache Spark, especially when it comes to PySpark. Especially in Python, the UDF performance can be a dumpster fire. 11:45 Seacliff ABC Session Data Infrastructure Building a Data Platform with AI Mouli Mukherjee Engineering Manager @OpenAI Data platform sits at the intersection of infrastructure, governance, operations, and user experience. 13:35 Seacliff D Unconference Unconference: Data Platforms Reimagined 14:45 Seacliff ABC Session One Does Not Simply Add Synchronous Writes - Bridging Batch and Real Time with Apache Iceberg What happens when a batch ingestion platform needs to behave like an operational database? Datadog’s Reference Tables product was designed for asynchronous uploads of large datasets, but customers increasingly needed synchronous row-level edits, without sacrificing batch scale. 15:55 Seacliff ABC Session Controlling Costs in the Era of Infinite Compute Details coming soon. 17:05 Seacliff ABC Session Handling High-Velocity Ingestion Outside the Cloud Details coming soon.