We have a couple of main supported archetypes. We've got the frontend stack which is node based, and the backend side that is primarily in Python right now. We have a shared library for all of those services to use, and that brings with it things like automatic tracing and metrics collection, logging etc. Underlying all of that is a set of Puppet modules that's shared across all of this which does log shipping, metrics collection and so on. It reduces the amount of work that each person has to do when launching a new service.

An early step was the jump from data center to the cloud. There's some interesting stuff there in terms of how the network latency changed drastically, how the architecture had to work and so on. One of the major components of Reddit is listings on the site, like a sub-Reddit or commentaries. They have gone through a ton of iterations in how those are stored and eventually pre-computed and fetched on the fly. Also dealing with the huge amounts of data we have.

That it's never done. Don't let the perfect be the enemy of the good. And know that lot of the trade offs involved are actually human, not technical.

Some things are really obvious in retrospect, like coordination for example becomes really important, but I think it's also that there's certain things such getting designs more thought out upfront becomes way more important because they become harder to change when you've got different teams on other sides of boundaries. Communication is a big part.