Proponents of the Lambda Architecture argue that streaming systems are unreliable. That they can’t be used to provide consistent results. That they’re difficult to backfill when data changes. That the only way to get low latency and have precise results and be able to respond to changes in upstream data is by maintaining two separate systems: one streaming and one batch. That no matter how hard we try, we are forever bound to the Lambda Architecture as we strive for fast, reliable, and flexible results.

We believe it is possible to build a streaming system you can rely on, making the Lambda Architecture unnecessary. I’ll cover in depth the APIs we’ve developed for manipulating data within axes of time and the strong consistency semantics we provide that bring powerful, yet flexible, streaming data processing within easy reach.

Recommended pre-reading:

• How to Beat the CAP Theorem by Nathan Marz
• Questioning the Lambda Architecture by Jay Kreps
• MillWheel: Fault Tolerant Stream Processing at Internet Scale published at VLDB 2013