Batch processing based on the Map-Reduce framework has been the dominant paradigm for analyzing a large amount of data in the past few years. The growing popularity of this paradigm can be attributed to a combination of factors, including the increased ease with which companies can produce (e.g. via web-based click telemetry), collect (e.g. via ETL technology), and glean insights from (e.g. via ML platforms) a large amount of data at rest. As popular as this paradigm is, it is inherently “batch oriented” — i.e. it requires processing of the entire data set to get an answer and the entire data set under review must be at rest for the duration of the processing. In many cases, the business requires an answer within a few minutes and does not require the entire data set to be processed. In these cases, a new paradigm has emerged : Real-time (a.k.a. Streaming) data processing.
For example, many e-commerce sites (e.g. ITunes) and credit card issuers use a form of online-analytics to detect and mitigate possibly-fraudulent financial transactions before they occur — this is because the cost of repairing fraud is too high! In another example, most web scale companies rely on real-time or near-real-time system metrics to detect when a portion of a web site has become unhealthy. In most cases, some level of automated fail-over is possible. Where automation does not bring about recovery, real-time alerting is key to getting the right people to take evasive action to avoid a site meltdown! In yet another example, companies like LinkedIn generate real-time recommendations based on recent user actions (e.g. social gestures such as shares and likes).
What do you need to know when you eventually build and deploy one of these systems? This track explores a variety of use-cases, platforms, and techniques for processing and analyzing stream data from the companies deploying them at scale.