Doug: I am the Director of Engineering at Datadog. I am responsible for a few different teams that span our online data business and our offline data work. So data science, data engineering, and the monitoring team (that deals with ensuring that our customers get alerted whenever there is an issue).

Doug: Yes, pretty much every major cloud environment and also in the datacenter as well. It works wherever folks have hosts or containers.

Doug: In this context, it means that the compute resources we are using, the hosts, are entirely scaled up and down for the jobs that we need to do in the cloud. In addition (at least for the high-latency big data portion of the platform), they are entirely using spot instances. We’re doing ephemeral instances and dynamic pricing. We are balancing all that together for the workload and using different types of hardware based on the given workload. That’s the sort of thing that you can only do in the cloud. The cloud offers a lot of advantages for cost and for flexibility, but it also presents challenges for managing that level of dynamism.

Doug: For this framework, I am talking mostly about batch data, so Hadoop and Spark mostly. We use Luigi for stream workflow management. In the talk, I want to consider other options for all of those things. For people in the audience that want to build something like this, I particularly want to focus on what the options are at every layer of the stack. Things like: what you are looking at, what’s out there for you, and what you are going to have to build.  Some of this stuff we’ve custom built ourselves, but others are freely available as open source.