Data Pipeline AMA w/ Sid Anand & Allen Wang

Sid currently serves as the Chief Architect and Head of Engineering for Datazoom, where he and his team build high-fidelity, low-latency data streaming systems. Prior to joining Datazoom, Sid served as PayPal's Chief Data Engineer, where he helped build systems, platforms, teams, and processes, all with the aim of building access to the hundreds of petabytes of data under PayPal's management. Prior to joining PayPal, Sid held senior technical positions at Netflix, LinkedIn, eBay, & Etsy to name a few. He earned my BS and MS degrees in CS from Cornell University, focusing on Distributed Systems.

Outside of work, Sid advises early-stage companies and several conferences. Once an active committer on Apache Airflow, he is now mostly a fan.

Sid's body of work includes but is not limited to :

• The world's first cloud-based streaming video service -- I was the first engineer to work on the cloud at Netflix
• LinkedIn's Federated Search Typeahead (a.k.a. auto-complete)
• LinkedIn's (Big Data) Self-service Marketing Analytics tool
• PayPal's DBaaS - an internal self-service system to provision & manage heterogenous databases
• PayPal's CDC - an internal self-service CDC system to stream DB updates to nearline applications
• eBay-over-Skype : Following the Skype-acquisition, I built a P2P version of eBay offers
• eBay's Best Match Search Ranking Engine powered by an In-Memory Database
• eBay's Fuzzy-match name/email Search
• Agari's Data Platform : Batch & Streaming Predictive Data Platform as a Service
• Datazoom's Platform : High-fidelity, Low-latency Streaming Data Platform as a Service

Allen Wang is currently a tech lead at data platform at DoorDash. He is the architect for the Iguazu event processing system and a founding member of the real time streaming platform team. 

Prior to joining DoorDash, he was a lead in the real time data infrastructure team at Netflix where he created the Kafka infrastructure for Netflix’s Keystone data pipeline and was a main contributor to shape the Kafka ecosystem at Netflix.

He is a contributor to Apache Kafka and NetflixOSS, and a two-time QCon speaker.