Conference: Nov 13-15, 2017
Workshops: Nov 16-17, 2017
Presentation: Unified Integration for Data-Driven Applications
Abstract
This talk proposes and presents a modern, unified architecture for the data stored in and applications running on Hadoop and other open source big data infrastructure.
Building and running production applications in the Hadoop ecosystem is often a long and challenging process filled with slow development cycles and painful production operations. Traditional development tools and techniques are missing from the Hadoop ecosystem. Now, as the Hadoop ecosystem continues to grow and use cases mature, these challenges are increasing as SLAs intensify and services become customer-facing and revenue-generating.
Based on my experiences as an early committer on Apache HBase, building real-time systems on Hadoop at Facebook, as well as working with customers at Cask, I would like to introduce the notion of a Data Application Platform (DAP), a modern, unified integration platform for data and applications. The DAP aims to address the challenges associated with building and operating production, real-time applications on Hadoop. Born out of the idea of traditional App Servers like Weblogic or JBoss, a DAP is an application-oriented middleware layer for big data that presents a single environment for design, operations, data science and governance. It provides higher-level APIs, reusable libraries, tools for testing, an in-memory environment for CI, a local environment for development and a distributed environment for production.
This talk will outline the requirements of building and running real-time production applications on Hadoop, the challenges, the proposal and architecture of a DAP, and addressing some of the remaining gaps in Hadoop.
Similar Talks
.
Tracks
Monday Nov 7
-
Architectures You've Always Wondered About
You know the names. Now learn lessons from their architectures
-
Distributed Systems War Stories
“A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.” - Lamport.
-
Containers Everywhere
State of the art in Container deployment, management, scheduling
-
Art of Relevancy and Recommendations
Lessons on the adoption of practical, real-world machine learning practices. AI & Deep learning explored.
-
Next Generation Web Standards, Frameworks, and Techniques
JavaScript, HTML5, WASM, and more... innovations targetting the browser
-
Optimize You
Keeping life in balance is a challenge. Learn lifehacks, tips, & techniques for success.
Tuesday Nov 8
-
Next Generation Microservices
What will microservices look like in 3 years? What if we could start over?
-
Java: Are You Ready for This?
Real world lessons & prepping for JDK9. Reactive code in Java today, Performance/Optimization, Where Unsafe is heading, & JVM compile interface.
-
Big Data Meets the Cloud
Overviews and lessons learned from companies that have implemented their Big Data use-cases in the Cloud
-
Evolving DevOps
Lessons/stories on optimizing the deployment pipeline
-
Software Engineering Softskills
Great engineers do more than code. Learn their secrets and level up.
-
Modern CS in the Real World
Applied, practical, & real-world dive into industry adoption of modern CS ideas
Wednesday Nov 9
-
Architecting for Failure
Your system will fail. Take control before it takes you with it.
-
Stream Processing
Stream Processing, Near-Real Time Processing
-
Bare Metal Performance
Native languages, kernel bypass, tooling - make the most of your hardware
-
Culture as a Differentiator
The why and how for building successful engineering cultures
-
//TODO: Security <-- fix this
Building security from the start. Stories, lessons, and innovations advancing the field of software security.
-
UX Reimagined
Bots, virtual reality, voice, and new thought processes around design. The track explores the current art of the possible in UX and lessons from early adoption.