Last week we designed the contract between services. Tonight, the hard truth of having more than one machine: a call can succeed and you'll never hear back. These are the handful of patterns that tame that.
In one program, things either work or throw. The moment two machines must talk, a third outcome appears: it might have worked, and you'll never know. Every pattern tonight is an answer to one question, what do I do when I don't know what happened?
You post a letter and never hear back. Did it get lost? Did the reply get lost? You can't tell. Distributed systems are that, all day. The trick isn't to make the post perfect, it's to send things so that getting one twice is harmless, and to always have a plan for "no reply came."
Calling a function in your own process is instant, reliable, and has exactly one outcome. The day that call goes over a network to another machine, all three promises quietly break. Most distributed bugs are someone forgetting that.
flowchart LR F(["a function call
same process"]) --> FOK(["instant · reliable ·
one clear outcome"]) N(["a network call
another machine"]) --> N1["can be slow"] N --> N2["can be lost"] N --> N3["can succeed but
the reply never arrives"] classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class F,FOK good; class N io; class N1,N2,N3 bad;
In-process is talking to someone in the same room. A network call is shouting across a valley: maybe they didn't hear you, maybe they did and you missed the reply, maybe the echo takes a while. You cannot assume you were heard. Everything tonight builds on that one shift.
Here is the problem the whole field is built around. You call "charge the card." A clear success or a clear error is easy. But a timeout, silence, is genuinely ambiguous: it may have worked, it may not have. You have to act without knowing.
flowchart TB
C(["you call: charge the card"]) --> T{"what comes back?"}
T -- "200 OK" --> OK(["clearly worked"])
T -- "error response" --> ERR(["clearly failed,
safe to retry"])
T -- "nothing · timeout" --> U(["did it work??
you genuinely cannot tell"])
classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726;
classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
class C io;
class T ask;
class OK,ERR good;
class U bad;
You tap your card and the screen freezes. Did it charge? If you tap again you might pay twice; if you don't you might not pay at all. That frozen screen is partial failure. The grown-up answer isn't "make it never freeze", it's "make tapping twice safe." Hold that thought, it returns all night.
Every machine has its own clock, and they drift apart by milliseconds, sometimes more. So when two things happen on two machines, you cannot trust the timestamps to tell you which came first. Ordering is something you have to deliberately impose, not read off a clock.
flowchart LR A(["Machine A clock
10:00:00.500"]) --> E1["records: user clicked"] B(["Machine B clock
10:00:00.480"]) --> E2["records: user paid"] E1 --> Q(["which truly happened first?
the clocks disagree,
so you cannot say"]) E2 --> Q classDef io fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef ev fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class A,B io; class E1,E2 ev; class Q bad;
Two people in different time zones each write the time on a letter. You can't line the letters up by those times, their watches aren't in sync. So real systems impose order on purpose: version numbers, sequence counters, or a single leader who stamps everything. Never "just sort by timestamp" across machines.
Sometimes the network breaks between your machines, they're all alive but can't reach each other (a "partition"). A write arrives. You face a forced choice: refuse it to stay Consistent, or accept it to stay Available. You cannot have both during the split. That's the whole theorem.
flowchart TB P(["network partition:
nodes alive but cannot talk"]) --> Q{"a write arrives
during the split"} Q -- "choose Consistency" --> C["refuse the write
return an error until healed · CP"] Q -- "choose Availability" --> A["accept the write
reconcile the copies later · AP"] classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726; classDef cp fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef ap fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class P io; class Q ask; class C cp; class A ap;
Two shops share one stock list, then the phone line between them dies. Do you stop selling so you never oversell (Consistent), or keep selling and sort out any clash later (Available)? A bank picks the first; a shopping cart picks the second. The skill is saying which, and why, for your case.
People argue about consistency because they mean different points on a spectrum. From strongest and slowest to fastest and stalest. You don't need the strongest everywhere, you need to pick the right point per feature.
flowchart LR S["STRONG
everyone sees the latest,
always · slow, costly"] --> RW["READ-YOUR-WRITES
you always see
your own changes"] RW --> EV["EVENTUAL
fast, cheap ·
briefly out of date"] classDef strong fill:#FEE4E2,stroke:#1F2937,color:#0E1726; classDef mid fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef weak fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class S strong; class RW mid; class EV weak;
A bank balance needs strong, never show the wrong number. Your own profile edit needs read-your-writes, you must see your change even if others lag. A follower count is fine eventual, who cares if it's off by a few for a second. Match the dial to the cost of being wrong.
You keep N copies of the data. Instead of waiting for all of them, you wait for a majority. The magic rule is R + W > N: if your write touches W copies and your read checks R copies, and those two sets overlap, your read is guaranteed to see the latest write. No single leader required.
flowchart TB W(["write to W copies
N=3, W=2"]) --> WC["2 of 3 confirm the write"] R(["read from R copies
R=2"]) --> RC["2 of 3 answer the read"] WC --> O{"R + W > N ?
2 + 2 > 3 · yes"} RC --> O O --> F(["the read set always overlaps
the write set → you see the latest"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef step fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class W,R io; class WC,RC step; class O ask; class F good;
Three friends each keep a copy of the plan. You tell two of them the new time (write). Later you ask any two what the time is (read). Because two-plus-two is more than three, at least one person you ask must be someone you told. You can't get a stale answer. That's how Cassandra and DynamoDB stay correct without a boss.
Sometimes you need everyone to agree on a single fact, who the leader is, what order writes happened in. That's consensus, and algorithms like Raft do it by majority vote: nodes elect one leader, and the leader orders every change so all followers apply them identically.
flowchart TB
N1["node"] --> V{"elect a leader:
majority vote"}
N2["node"] --> V
N3["node"] --> V
V --> L["the Leader
stamps every write in order"]
L --> F1["follower
copies in the same order"]
L --> F2["follower
copies in the same order"]
classDef node fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726;
classDef lead fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
classDef fol fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
class N1,N2,N3 node;
class V ask;
class L lead;
class F1,F2 fol;
A committee that must never contradict itself: they vote in one chairperson, and only the chair writes in the minutes book, everyone else copies it word for word. As long as most of the committee is present and agrees, the book is consistent. Lose the majority and they stop writing rather than risk two versions.
Agreement by majority vote means lots of chatter and waiting, so it's slow, and it stalls if too many nodes are down. The senior move isn't writing Raft. It's recognising when you need it and reaching for a tested tool that already implements it.
"We needed agreement on the leader, so we used etcd rather than rolling our own Raft" is a staff sentence. Hand-writing consensus is how you get a subtle outage two years later.
A leader goes quiet, so the others elect a new one. Then the old leader wakes up, it was just frozen, not dead, and still believes it's the boss. Now two leaders accept writes and the data forks. The fix is a fencing token: an ever-increasing number, and everyone ignores the lower one.
flowchart TB OLD(["old leader · token 7
froze, assumed dead"]) --> NEW(["new leader elected · token 8"]) NEW --> X{"old leader wakes up,
still thinks it is boss"} X -- "no guard" --> SB(["two leaders writing →
split brain, corrupt data"]) X -- "fencing token" --> FIX(["storage rejects token 7,
accepts only 8 → old one is fenced out"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726; classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class OLD,NEW io; class X ask; class SB bad; class FIX good;
A manager goes on leave, the team appoints a stand-in. The old manager strolls back in and starts giving orders too. Chaos. The fix: every order carries a round number, and staff only obey the highest one. The old manager's orders are "round 7" in a "round 8" world, so they're simply ignored.
When one service sends a message to another, there are three possible guarantees. Understanding which one you actually have changes how you write every handler. Spoiler: the one everybody wants doesn't exist on the wire.
flowchart TB M(["send a message"]) --> AM["AT-MOST-ONCE
send and forget ·
might be lost, never duplicated"] M --> AL["AT-LEAST-ONCE
retry until acknowledged ·
never lost, might duplicate"] M --> EO["EXACTLY-ONCE
the dream ·
not achievable on the network alone"] classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef meh fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef dream fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class M io; class AM meh; class AL good; class EO dream;
At-most-once = "I texted you, didn't check you got it." At-least-once = "I'll keep texting until you reply, so you might get a few." Almost every real queue (Kafka, SQS, RabbitMQ) is at-least-once. So duplicates aren't a rare bug, they're the guaranteed normal. Plan for them.
You can't stop a message arriving twice, the network forces retries. But you can make the second arrival do nothing. Reliable delivery (retry until acked) plus a handler that ignores repeats gives you the effect of exactly-once. That combination is what "exactly-once" means in every real system.
flowchart LR A(["at-least-once delivery
may arrive twice"]) --> PLUS["+"] PLUS --> B(["idempotent handler
twice has same effect as once"]) B --> R(["effectively exactly-once
what production actually delivers"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef op fill:#ffffff,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class A,B io; class PLUS op; class R good;
You can't promise the postman delivers each letter exactly once, sometimes he redelivers. But if each letter says "ignore me if you've already actioned order 123", a second copy is harmless. Stop trying to perfect delivery. Make the handler not care. That single reframe removes most distributed-systems pain.
The mechanism is simple and it's the same one from last week's APIs. Each message carries a unique idempotency key. Before acting, the handler checks a small store: seen this key already? If yes, skip and acknowledge. If no, do the work and record the key, atomically.
flowchart TB M(["message arrives
key: abc-123"]) --> Q{"seen key abc-123
before?"} Q -- "no, first time" --> DO["do the work +
store the key together"] Q -- "yes, duplicate" --> SKIP["skip · already handled"] DO --> ACK(["acknowledge"]) SKIP --> ACK classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class M io; class Q ask; class DO,SKIP,ACK good;
A bouncer with a guest list and a marker. First time your name comes up, he serves you and ticks you off. Show up again with the same name, he sees the tick and waves you through without serving twice. The "do the work and tick the name in one step" is the bit people get wrong, more on that next.
Here's the bug at the heart of most microservice messes. Handling one action, you must do two writes: save to your database and publish an event for other services. They're separate systems, so they can't share one transaction. If you crash between them, they disagree, and nobody notices.
flowchart TB S(["handle one order"]) --> A["step 1 · save order to the database · OK"] A --> B["step 2 · publish order-placed event to the queue"] B --> CRASH(["crash before step 2 completes"]) CRASH --> R(["the database has the order,
but no event went out →
shipping + email never hear about it"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef step fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class S io; class A ok; class B step; class CRASH,R bad;
You write a cheque and text your friend "I paid you." If your phone dies between the two, the cheque exists but the friend never knows. You can't wrap a bank cheque and a text message in one undo-able action. That gap is the dual-write problem, and the next slide is the standard fix.
The fix is a trick. Don't publish to the queue directly. Instead, in the one database transaction that saves the order, also insert the event into an outbox table. Both commit together or neither does. A separate relay then reads the outbox and publishes, retrying safely until it succeeds.
flowchart TB T(["ONE database transaction"]) --> O1["save the order"] T --> O2["insert event row into outbox table"] O1 --> C(["commit · both or neither"]) O2 --> C C --> REL["a relay polls the outbox"] REL --> PUB["publishes the event,
retries until acked"] PUB --> MQ(["queue · downstream services"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef step fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class T io; class O1,O2,REL,PUB step; class C,MQ good;
Instead of "do the thing, then separately tell people", you write "the thing" and "the note to tell people" on the same page, so they save together. Then a helper comes by, reads the notes, and sends them out, ticking each off. Now the database and the event can never disagree. This one pattern prevents a huge class of bugs.
The obvious idea: a coordinator asks every service "ready?", and if all say yes, tells them all to commit. That's two-phase commit. It does give you all-or-nothing, but it's a trap at scale: while everyone waits for the verdict, they're locked, and if the coordinator dies mid-vote, they're stuck.
flowchart TB CO(["coordinator asks:
everyone ready?"]) --> A["service A: ready, now locked"] CO --> B["service B: ready, now locked"] CO --> C["service C: ready, now locked"] A --> W{"all said yes →
tell everyone to commit"} B --> W C --> W W --> BAD(["if the coordinator dies here,
A, B, C stay locked, waiting →
slow, fragile, avoided at scale"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef lock fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726; classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class CO io; class A,B,C lock; class W ask; class BAD bad;
Getting four friends to commit to a plan only if all agree: everyone has to hold their evening free while the organiser collects yeses. If the organiser's phone dies, everyone's stuck waiting, evening wasted. Fine for three databases in one room; miserable across services. So instead of locking everyone, we use a saga.
Instead of one giant locked transaction, a saga is a sequence of small local ones, each committing on its own. If a later step fails, you don't roll back, you can't, it's already committed. You run compensating actions that undo the earlier steps, in reverse.
flowchart LR S1["reserve stock"] --> S2["charge the card"] S2 --> S3["book the courier"] S3 -. "this step fails" .-> C2["compensate:
refund the card"] C2 --> C1["compensate:
release the stock"] classDef fwd fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef comp fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class S1,S2,S3 fwd; class C2,C1 comp;
Booking a holiday: flight, then hotel, then car. The car is sold out, you can't "undo" the trip atomically, you've already paid for flight and hotel. So you compensate: cancel the hotel, refund the flight. A saga is just "do the steps, and if one fails, deliberately walk the earlier ones back."
flowchart TB
subgraph ORC["Orchestration: a conductor tells each step what to do"]
direction TB
O(["orchestrator
owns the workflow"]) --> OA["service A"]
O --> OB["service B"]
O --> OC["service C"]
end
subgraph CHO["Choreography: services react to each other"]
direction LR
CA["A finishes,
emits an event"] --> CB["B hears it,
does its bit, emits"] --> CC["C hears it,
does its bit"]
end
classDef orc fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
classDef cho fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
class O,OA,OB,OC orc;
class CA,CB,CC cho;
Orchestration = an orchestra with a conductor: one place owns the order, easy to follow and change, but the conductor is a central piece. Choreography = dancers reacting to each other: no central owner, very decoupled, but no single place shows you "what's the whole flow?" Start with orchestration; it's far easier to debug.
Retries (with backoff and jitter, from last week) handle brief blips. But if a dependency is truly down, endless retries just pile on, you DDoS your own struggling service. A circuit breaker notices the flood of failures and "trips": it stops calling for a bit, fails fast, then carefully tests if it's back.
flowchart LR
R(["a call fails"]) --> RT["retry with backoff + jitter"]
RT --> CB{"failing a lot,
repeatedly?"}
CB -- "no, just a blip" --> OK(["carry on normally"])
CB -- "yes, it is down" --> OPEN["OPEN the breaker:
stop calling, fail fast"]
OPEN --> HALF["after a pause, let ONE
test call through"]
HALF --> CL(["recovered? close it.
still bad? stay open"])
classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
classDef step fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726;
classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
class R io;
class RT,HALF step;
class CB ask;
class OK,CL good;
class OPEN bad;
It's the trip switch in your house. When something's shorting, the switch cuts power instead of letting the wiring cook. A circuit breaker cuts calls to a dead service so your threads aren't all stuck waiting on it, and so the dying service gets room to recover instead of being buried.
A queue between two services smooths out bursts, the producer drops work in, the consumer takes it at its own pace. But a queue is not infinite. If the producer keeps outrunning the consumer, the queue grows without bound until memory runs out. Backpressure is the signal that says "slow down."
flowchart LR
P(["fast producer"]) --> Q[["queue buffers the burst"]]
Q --> C(["slower consumer"])
Q --> BP{"queue getting
dangerously full?"}
BP -- "push back" --> SLOW(["tell producer to slow down,
or shed lower-priority work"])
BP -- "ignore it" --> CRASH(["queue grows forever →
out of memory, everything topples"])
classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
classDef q fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726;
classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
class P,C io;
class Q q;
class BP ask;
class SLOW good;
class CRASH bad;
A sink with the tap on full and a slow drain. Either the sink overflows, or you turn the tap down. Backpressure is turning the tap down. A queue feels like it makes the problem disappear, it just hides it until the queue is full. Always ask: what happens when the consumer can't keep up?
When a single user action hops through five services, "check the logs" stops working, each service only sees its own slice. The fix is a correlation (trace) ID: stamp the request once at the edge and pass it along, so every log line for that journey shares one ID you can search.
flowchart LR U(["request arrives
assign trace-id x9"]) --> A["service A
logs with x9"] A --> B["service B
logs with x9"] B --> C["service C
logs with x9"] C --> T(["search x9 → the whole
journey, in order, across all three"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef svc fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class U io; class A,B,C svc; class T good;
A parcel with one tracking number that every depot scans. Without it, five depots each say "yeah, something passed through" and you can't stitch the story. With it, you type one number and see the whole route. This is the difference between debugging distributed systems in minutes versus days.
| The failure you face | Pattern | In one line |
|---|---|---|
| Read must see the latest write | Quorum (R + W > N) | overlap the read and write sets |
| Need one agreed leader or value | Consensus (Raft) | majority vote, use a tool |
| A message may arrive twice | Idempotency key | make a replay a no-op |
| DB write + event must both happen | Outbox | one commit, relay publishes after |
| Multi-service workflow, no 2PC | Saga | local steps + compensations |
| A dependency keeps failing | Circuit breaker | stop hammering, fail fast |
| Producer outruns consumer | Backpressure | push back or shed load |
Notice the left column is always a failure, never a technology. That's the senior habit: name the failure you're defending against first, then the pattern is almost obvious.
5 · The naked dual write, saving to the DB and publishing an event as two separate steps, with no outbox to make them atomic. The fix for all five is the same instinct: name the failure first, then pick the pattern.
Pick one flow from your own system that touches more than one service or a queue, "place an order", "process a payment", "send a notification". Don't open an editor. Answer these four.
Drop your flow + its idempotency key in the chat. We'll run a few live, the compensation question is where it bites.
Assume partial failure, no global clock, at-least-once delivery. Make handlers idempotent, reach for consensus rarely (and via a tool), and use the outbox + sagas for cross-service consistency instead of 2PC. Add breakers, backpressure and trace IDs so it stays up and stays debuggable.