Does it bother you when all your microservice logs show up out of order in your log analysis tool because your system clocks aren't synchronized? Did you ever wonder what time it was when the dotted version vector on a piece of data became 2.4.5? If so, then this is the talk for you!

Time is a notoriously difficult concept in distributed systems. Perfect clock synchronization is impossible, and most logical timestamp schemes (like Lamport clocks) don't bear a relationship to "wall clock time" that is meaningful to human operators.

In this talk, we'll learn about distributed monotonic clocks (DMC) whose timestamps can reflect causality but which have a component that stays close to wall clock time. This scheme builds on previous hybrid logical clock proposals by adding important operational hooks and by building in mechanisms to prevent a single runaway system clock from dragging a whole cluster's logical clocks forward into the future.

DMCs can be implemented as a "piggyback protocol" suitable for transport in existing application messages--for example, as additional HTTP headers. I'll describe a novel coordination protocol in this unusual piggyback style--a style that may also be useful in ad-hoc clusters formed by Internet-attached "things".