Apache Spark Structured Streaming is a distributed stream processing engine built on Spark SQL and is widely used for ETL-style workloads, where seconds- to minutes-level latency is acceptable. But millisecond-latency use cases have historically been out of reach.
Real-time Mode (RTM) introduces a new low-latency processing capability to Structured Streaming, enabling millisecond-latency processing and allowing Spark to power these workloads as well. This helps organizations consolidate their data platform: instead of using Spark for batch/ETL and a separate system for low-latency streaming, they can now use Spark across workloads with different latency requirements.
In this talk, we’ll explain how Real-time Mode works, share insights into how we extended the Structured Streaming architecture to enable low-latency processing, and highlight how users are using it.