Distributed systems · time
S6 said it: across machines there is no global "now". So how does a backend handle time at all? Three families do the work: physical clocks (what time is it), logical clocks (what happened first), and hybrids (both at once). This sheet is the whole menu, plus the one bug everyone hits. Companion to S6 · Distributed Systems Patterns.
"What time is it" and "what happened first" are two different questions. A timestamp answers the first well and the second badly. Across machines, clocks drift, so a timestamp is fine for "roughly when" and useless for "in what order." Almost every time-related outage is someone using one tool for the other job.
Before any tool, diagnose what you actually need. A wall clock tells you it is about 3pm. A numbered deli ticket tells you who is next. They are not the same tool, and you cannot order the queue by glancing at everyone's watch.
A timestamp to log, store, or show a human. Approximate is fine. Use a physical clock kept honest by NTP. For "how long did it take," use a monotonic clock.
Which event happened before which, across machines. Wall time cannot answer this. Use a logical clock (Lamport or vector), or a hybrid if you also want real-ish time.
Sorting events from different machines by their raw timestamps. The clocks disagree by milliseconds to seconds, so "latest write wins" silently picks the wrong winner. Order is something you impose (sequence numbers, a leader, a logical clock), not something you read off a clock.
Every time mechanism in a backend falls into one of these. Know which question each answers.
| Family | Answers | Examples | Gives you | Cannot |
|---|---|---|---|---|
| Physical | "what time is it?" | NTP, chrony, PTP, cloud time | human-meaningful, approximate time | prove cross-machine order |
| Logical | "what happened first?" | Lamport, vector, version vectors | exact causal order | tell you the wall-clock time |
| Hybrid | both at once | HLC, Google TrueTime | real-ish time that is also ordered | escape a small accuracy / wait cost |
Wall clock tells you approximately when. Monotonic clock tells you how long. Logical clocks tell you definitely in what order. Hybrids give you when and order together. You almost never want the wall clock deciding order.
Two clocks live on every machine and they are not interchangeable. Mixing them is the single most common time bug in application code.
| Clock | Use it for | Reads as | Can it jump? | Call in code |
|---|---|---|---|---|
| Wall | a timestamp to store or show | real calendar time | yes, even backwards on NTP correction | time.time() · System.currentTimeMillis() · Date.now() |
| Monotonic | durations, timeouts, rate limits | seconds since some arbitrary point | never, only moves forward | time.monotonic() · System.nanoTime() · performance.now() |
Never measure "how long" with the wall clock. When NTP corrects the clock it can step, sometimes backwards. If you timed something with the wall clock and it stepped mid-measurement, your duration goes negative or wildly wrong. Use the calendar to say "the meeting was at 3pm," use a stopwatch to say "it lasted 40 minutes." Durations always come from the stopwatch.
The same bug, in code
# WRONG — wall clock can step back, latency goes negative
start = time.time()
do_work()
latency = time.time() - start # may be < 0 after an NTP step
# RIGHT — monotonic only ever moves forward
start = time.monotonic()
do_work()
latency = time.monotonic() - start # always sane
per language: Python time.monotonic() · Java System.nanoTime()
Go time.Since() (monotonic since 1.9) · JS performance.now()
Rust Instant::now() · C clock_gettime(CLOCK_MONOTONIC)
Rule of thumb: anything you subtract (latency, timeouts, retry backoff, rate-limit windows, cache TTL deadlines, leader leases) uses the monotonic clock. Anything you store or display (created_at, log lines, an expiry shown to a user) uses the wall clock.
Clocks drift because each machine counts on a cheap crystal that runs slightly fast or slow. A sync daemon talks to a more accurate reference and continuously nudges the local clock back. You run this layer, you rarely write it, but you must know it exists. Each card: accuracy typical accuracy, use when to reach for it, watch the catch.
chrony is the modern daemon: settles faster, copes with laptops and flaky links.Pattern: run chrony and forget it for almost everything. Reach down the list only when "a few ms of skew" genuinely is not good enough, and even then, physical clocks never give you order.
The breakthrough: forget wall time for ordering. Track causality with counters that travel inside your messages. If A could have caused B, A's counter is lower. No synchronised clocks required, just arithmetic. Think numbered deli tickets: ticket 7 was clearly here before ticket 12, no clock needed.
max(mine, theirs) + 1.Worked example: vector clocks spot a conflict Lamport would hide
flowchart TB S["start: A:0 B:0 C:0"] S --> AW["A writes: A:1 B:0 C:0
A has heard nothing from B"] S --> BW["B writes: A:0 B:1 C:0
B has heard nothing from A"] AW --> CMP["compare A:1 B:0 vs A:0 B:1
neither is greater-or-equal in every slot"] BW --> CMP CMP --> CONC["CONCURRENT: a real conflict you must merge"] CONC --> LAM["a Lamport clock would give them 1 and 1,
so you silently pick a winner and lose one write"] class CONC bad class LAM accent classDef bad fill:#FEE4E2,stroke:#C8102E,color:#09244B classDef accent fill:#DCEBFE,stroke:#4A90D9,color:#09244B
A version vector applies vector-clock logic to object versions, so replicas can tell whether one copy strictly supersedes another or whether they diverged and need reconciling. It is how eventually-consistent stores avoid blindly overwriting.
Pure logical clocks lose the human-friendly "it is 3pm." Pure physical clocks lie about order. Hybrids glue them together, and this is what modern distributed databases run on.
physical : logical.earliest and latest, a few ms wide.latest has passed before acknowledging, so nobody later can claim an earlier time.Rather than claim "it is exactly 3:00:00," the clock says "it is between 3:00:00.000 and 3:00:00.004." Before calling a commit done, wait those 4ms out. Now no later transaction can honestly claim an earlier moment. You buy correctness with a tiny pause.
Where clocks meet everyday backend work. You need primary keys that are unique without coordination across many machines, and ideally sortable by creation time so they cluster well in a B-tree index. This is one of the most common system-design interview questions ("design a unique ID generator").
| Scheme | Shape | Sortable by time? | Coordination-free? | Catch |
|---|---|---|---|---|
| Auto-increment | DB sequence | yes | no, single writer | hard to shard; leaks row counts |
| UUIDv4 | 122 random bits | no | yes | random inserts fragment the index (page splits) |
| Snowflake | 41b time + 10b node + 12b seq | yes | yes (per-node) | needs a unique worker id; clock-skew sensitive |
| ULID | 48b time + 80b random | yes | yes | 128-bit, base32 text |
| UUIDv7 | 48b unix-ms + 74b random | yes | yes | the modern default, drop-in UUID |
Snowflake: a 64-bit ID is mostly a timestamp
flowchart LR Z["1 bit: 0"] T["41 bits timestamp
ms since epoch
sorts chronologically"] N["10 bits node
machine"] Q["12 bits seq
per-ms counter
4096 ids per node per ms"] Z --> T --> N --> Q Q --> R["sortable by time, unique across nodes,
no central coordinator"] class R accent classDef accent fill:#DCEBFE,stroke:#4A90D9,color:#09244B
Because v4 is random, every insert lands in a random spot in the index B-tree, causing page splits and write amplification and killing cache locality. A time-ordered ID (UUIDv7, ULID, Snowflake) inserts at the "end" of the index, so writes stay sequential. Reach for UUIDv7 as the modern default; Snowflake when you want a compact 64-bit integer.
Snowflake and ULID assume the wall clock only moves forward. If NTP steps the clock backwards, a naive generator can mint a duplicate or out-of-order ID. Real implementations detect this and either wait until the clock catches up or refuse to issue. This is exactly the wall-clock trap from section 3, showing up in production ID generation.
The day-to-day decisions where clocks bite real services. Each of these is a common interview follow-up and a common production incident.
timestamptz, never naive local times.created_at and ordering timestamps at the server edge, not from the client payload.exp / nbf with a small clock-skew leeway (typically 30 to 60s) between issuer and verifier.23:59:60.Pick by the job, not by habit. Start at the top question every time: do I need order, or just a time?
flowchart TB
Q{"do you need a TIME, or an ORDER?"}
Q --> T["just a TIME to log or show:
WALL clock + NTP / chrony"]
Q --> M["how LONG it took, a timeout, a rate:
MONOTONIC clock"]
Q --> O["the ORDER of events"]
O --> O1["one stream is fine:
LAMPORT timestamps"]
O --> O2["must detect conflicts:
VECTOR / version vectors"]
Q --> B["real TIME and ORDER together"]
B --> B1["one distributed store:
HYBRID LOGICAL CLOCK, HLC"]
B --> B2["cross-datacentre transactions:
TRUETIME / ClockBound + commit-wait"]
class T,M,O1,O2,B1,B2 accent
classDef accent fill:#DCEBFE,stroke:#4A90D9,color:#09244B
| You need… | Reach for | Seen in |
|---|---|---|
| Roughly-right timestamps for logs & records | Wall clock + NTP / chrony | everywhere |
| Correct durations, timeouts, rate limits | Monotonic clock | every language stdlib |
| Sub-microsecond accuracy | PTP / GPS reference | finance, telco |
| Order events by causality | Lamport timestamps | messaging, logs |
| Detect concurrent / conflicting writes | Vector / version vectors | Dynamo, Riak, Cassandra |
| Real-ish time that is also ordered | Hybrid Logical Clock | CockroachDB, MongoDB, Yugabyte |
| Globally consistent transactions | TrueTime / ClockBound + commit-wait | Spanner, AWS |
The ones that come up, and the one or two sentences that show you actually understand the trade-off rather than reciting a term.
Q · Why can't you order events across servers by timestamp?
Each machine has its own drifting clock, so two timestamps a few ms apart prove nothing. Impose order instead: a sequence number, a single leader that stamps, or a logical clock.
Q · Wall vs monotonic, when each?
Wall clock for a timestamp to store or show. Monotonic for anything you subtract: latency, timeouts, backoff, leases. Wall clocks can step backwards; monotonic never does.
Q · Lamport vs vector clock?
Lamport gives a total order with one counter but can't tell concurrent from causal. A vector clock (one count per node) can detect genuine concurrency, so you can flag conflicts to merge.
Q · How does Spanner stay consistent without a perfect clock?
TrueTime returns a time interval with bounded error, then Spanner commit-waits out that uncertainty before acknowledging, so no later transaction can claim an earlier time.
Q · Design a unique ID generator. Why not UUIDv4?
v4 is random, so inserts fragment the index. Use a time-ordered ID, UUIDv7 / ULID / Snowflake, so keys are unique, coordination-free, and sort by creation time for sequential writes.
Q · What breaks a Snowflake generator?
The clock moving backwards (NTP step). It can mint duplicate or out-of-order IDs, so the generator must wait for the clock to catch up or refuse to issue.
Q · Why are TTL-based distributed locks unsafe?
A paused process can think it still holds an expired lock, so two writers act at once. Fix with a fencing token: an increasing number the resource checks, rejecting stale holders.
Q · How do you validate a JWT given clock skew?
Check exp / nbf with a small leeway (30 to 60s) for skew between issuer and verifier. Tight enough to limit replay, loose enough to not reject valid tokens.
The ways smart people get time wrong, and the one habit that catches all of them.
The traps
The repeatable move
1. a time, or an order?
2. if time: wall (when) or monotonic (how long)?
3. if order: do you need to spot conflicts?
4. need both? reach for a hybrid
The tell in any design review: a wall-clock timestamp being used to decide order, when it should only ever say when. Spot that phrase and push back.
Wall clock for "when," monotonic for "how long," logical clocks for "what order." Don't mix them up. This whole sheet is the S6 "no global clock" slide unpacked, and idempotency and sagas are how you cope with the order you can't read off a clock.
Companion to S6 · Distributed Systems Patterns. Further reading: Lamport, Time, Clocks & the Ordering of Events · Hybrid Logical Clocks · Google Spanner / TrueTime · RFC 9562 · UUIDv7 · ULID spec · Kleppmann · distributed locking & fencing · AWS Time Sync & ClockBound.