Presentation: High throughput stream processing with ACID guarantees
Distributed realtime event processing, also called stream processing, gained a lot of traction in the Big Data community recently. Frameworks like Apache Storm and Apache Samza tackle various challenges in stream processing including performance, scalability, reliability and fault tolerance. However, very limited support on transactions are available in these frameworks. The lack of complete transaction support puts burden on developers to maintain data integrity by implementing rollback partial writes in case of failures and designing for read isolation of uncommitted writes.
We designed and implemented a new distributed stream processing framework that has built-in transaction support with full ACID properties while not compromising on scalability, reliability and fault tolerance of Storm and Samza. We use the Apache Twill library on Hadoop YARN as the execution environment to provide elastic scalability, fault tolerance, and logs & metrics collection. The transaction support guarantees stream events to have exactly-once processing semantics, together with any data operations performed with full ACID properties. This new framework allows developers to focus more on the application logic rather than dealing with complicated distributed systems and distributed transactions.
Tracks
Covering innovative topics
Monday, 3 November
-
Architectures You've Always Wondered about
The newest and biggest Internet architectures
-
Real World Functional
Putting functional programming concepts to work in the real world.
-
The Future of Mobile
The future of mobile and performance improvements
-
Continuous Delivery: From Heroics to Becoming Invisible
Continuous Delivery philosophies, cultures, hiccups, and best practices.
-
Unleashing the Power of Streaming Data
This track explores a variety of use-cases, platforms, and techniques for processing and analyzing stream data from the companies deploying them at scale!
-
Sponsored Solutions Track I
Tuesday, 4 November
-
Engineering for Product Success
Architectures that make products more successful
-
Reactive Service Architecture
Reactive, Responsive, Fault Tolerant and More.
-
Modern CS In the Real World
How modern CS tackles problems in the real world.
-
Applied Machine Learning and Data Science
Understand your big big data!
-
Deploying at Scale
Containerizing Applications, Discovering Services, and Deploying to the Grid.
-
Sponsored Solutions Track II
Wednesday, 5 November
-
Beyond Hadoop
Emerging Big Data Frameworks and Technology
-
Scalable Microservice Architectures
This track addresses the ways companies with hundreds of fine-grained web-services (e.g. Netflix, LinkedIn) manage complexity!
-
Java at the Cutting Edge
The latest and greatest in the Java ecosystem
-
Engineering culture
Successes and failures in creating an engineering culture.
-
Next gen HTML5 and JS
How Web Components, the Future of CSS, and more are changing the web.
-
Sponsored Solutions Track III