Keith: I think people see Slack as a site that serves a lot of people, has a lot of customers, and also as something that does real-time messaging. They feel like they have seen those things before. Real-time messaging is pretty crowded. There has been WhatsApp, Facebook, and Google serving a lot more users than us. All that is true, but what is not true is how  those messages are served to those users. While Facebook and Google have real-time messaging products, the actual number of people in a given conversation (in a given virtual space) is usually quite small. 

Slack’s challenges are around maintaining low latency and high reliability. Low latency, for example, is important to making the product actually feel good. This is mostly below the threshold of conscious perception, but, if you mess this up, the product feels bad. It’s hard to say why it feels bad. It just does. In essence, these are textual equivalents to stops and starts in bad video conferencing.

In a nutshell, the challenge for Slack is synchronous group communication with large numbers of participants.

Keith: There is this piece of Slack that resembles the previous generation LAMP stack apps. The part of Slack that remembers your email address, your avatar, the name of your team,  custom images and so forth. That is recognizable. It is the LAMP stack app of people who have been doing this since the early days. It has had a lot of features, growth, and change. So it’s had to adapt to a more complicated infrastructure than your usual LAMP stack app does, but it’s recognizable as LAMP stack app. 

That’s no longer a single machine doing this function. We are busy diversifying that into pieces of the infrastructure that are oriented towards presence. For instance, as groups get larger you start spending more and more communication bandwidth on just communicating who has a green dot next to their name and who doesn’t. 

A quick side note (I am going to try and be honest about this in the talk), Slack’s architecture is a moving target right now. We know that the architecture we are using today is going to change over the next couple of years. I’m going to be really open about these things. 

Keith: MySQL is worth a mention, in a similar vain as PHP. It’s an old sword that has killed a lot of orcs. Other businesses might be more directly impacted by the returns of trying something new. But for us, this is a relative commodity thing. The win from being on Cassandra or something like that would be an incremental small constant factor over what MySQL can do right now. 

For example, we use a master-master replication in MySQL. If you ask a lot of people what master-master replication is, they will look at you like you have two heads. 

There is some interesting stuff there because which one do you back up? There are simple tasks that can be written, getting the same row and I read it twice in a row byte-wise that actually might be subtly broken by this. We do a little bit of hygiene on this. Imagine you have got the master-master pair in there. They are basically writing back and forth to each other. The way that we rotate these things in and out of their lifecycle is that we decide, ‘Okay, you are a month old. We’re worried you have gotten stale somehow.’ So we will attach something that will eventually be a new master to the other side of the master-master pair and then set it up as a slave and replicate to it. Once it has caught up to the master it is attached to, we cut off the old master, and it is like a little snaking tail that moves around. It’s an interesting little process that gives us write availability in the presence of one failure. We know how we’d go to three if we had to get to three. We are not buying some fancy product that costs money. Our customers would have to pay for that.