Reliability · resilience
S9 said it: at scale something is always broken, so you contain failure, you don't prevent it. This sheet is the whole toolkit on one page: the four patterns that bound a call, how to degrade instead of dying, how to stay correct under retry, and how to choose a failure mode on purpose. Companion to S9 · Designing for Failure.
You can't stop a dependency from failing; you can stop its failure from spreading. Every pattern below does one of two jobs: it bounds a failure (in time, repetition, persistence, or concurrency) so it stays local, or it chooses what happens when a dependency is gone (degrade, fall back, fail open or closed). Reliability is containment plus a deliberate failure mode, not the absence of failure.
Before any specific pattern, two questions frame the work. What does this failure take down with it (the blast radius), and when a dependency is gone, what should this path do? Everything else is the tooling that answers those two.
Walk the dependency graph and label each edge hard (caller dies without it) or soft (caller degrades but survives). The blast radius is everything reachable through hard edges. The cheapest win is turning a hard edge soft.
For each path, decide in advance: does it fail open (stay available, permissive) or fail closed (refuse, safe)? Saying which, out loud, per path, is the highest-signal sentence in a design review.
Treating reliability as "make it not fail." At scale that is impossible, and chasing it hides the real question. The grown-up framing is "when this fails, and it will, what stays up and what do we do?" Patterns are how you answer; they are not a substitute for answering.
Each pattern bounds one axis or chooses one behaviour. Know which failure each one prevents.
| Pattern | Bounds / chooses | Prevents | One-line example |
|---|---|---|---|
| Timeout | time per call | a hang holding the thread forever | cap the DB query at 400ms |
| Retry + jitter | repetition | a transient blip becoming a failure | retry a 503, backoff, spread the herd |
| Circuit breaker | persistence | hammering a dependency that is down | fail fast for 30s, then test once |
| Bulkhead | concurrency | one slow dependency draining all threads | separate pool per dependency |
| Fallback | behaviour | a blank error when a default would do | serve a generic list if recs are down |
| Graceful degradation | behaviour | going fully dark under strain | read-only mode when the DB is hot |
| Load shedding | admission | overload collapsing the whole service | drop low-priority traffic at the edge |
Four patterns bound a failure: timeout (time), retry (repetition), breaker (persistence), bulkhead (concurrency). Three patterns choose what happens when something is gone: fallback, graceful degradation, load shedding. Reach for a bounding pattern to keep a failure local; reach for a behaviour pattern to keep the user served.
Each bounds a different axis of a call. Together they stop a single slow dependency from climbing the stack into a site-wide outage. Each card: bounds what it limits, use how to apply it, watch the catch.
Circuit breaker: the state machine
stateDiagram-v2
CLOSED: CLOSED
calls flow, failures counted
OPEN: OPEN
fail fast, no calls
HALF_OPEN: HALF-OPEN
one trial call
CLOSED --> OPEN: failure rate over threshold
OPEN --> HALF_OPEN: cool-off elapsed
HALF_OPEN --> CLOSED: trial call succeeds
HALF_OPEN --> OPEN: trial fails
classDef good fill:#D6F5E3,stroke:#1B7F46,color:#09244B
classDef bad fill:#FEE4E2,stroke:#C8102E,color:#09244B
class CLOSED good
class OPEN bad
When you can't serve the perfect response, serve a useful one. These are designed modes: decide in advance what you drop first and what you protect last, so the user notices less rather than everything at once.
One slow dependency fills retry queues, the retries exhaust a shared pool, callers time out and retry harder, and the slowness climbs until the whole site is down, often after the original fault healed. Bounding patterns are the firebreaks; degradation and shedding are how the parts still standing keep serving.
Retries and at-least-once delivery mean the same operation will arrive twice. The fix is not to prevent duplicates; it is to make duplicates harmless. This is the other half of resilience: staying correct while you retry.
| Concept | What it is | When to reach for it |
|---|---|---|
| Idempotency key | a client-supplied unique key the server dedupes on, so a repeated request applies once | any non-idempotent write that may be retried (payments, orders) |
| At-least-once | the realistic delivery guarantee: messages may arrive more than once, never zero | default for queues; pair with idempotent consumers |
| Exactly-once | at-least-once delivery plus idempotent processing; true end-to-end exactly-once is mostly a myth | when you say it, you mean dedupe on the consumer |
| Dead-letter queue | where a message goes after N failed retries, so a poison message stops blocking the queue | every consumer, so one bad message can't wedge the pipe |
Idempotency key: make "twice" safe
POST /charges
Idempotency-Key: a1b2-c3d4 (client generates one per logical action)
server:
seen = lookup(a1b2-c3d4)
if seen: return seen.response # duplicate: replay, do NOT charge again
else: result = charge_card()
store(a1b2-c3d4, result)
return result
=> the timeout-then-retry that would double-charge now applies once.
at-least-once delivery + idempotent handler = effectively exactly-once.
"How do you get exactly-once?" You usually don't, end to end. You take at-least-once delivery and make the consumer idempotent (idempotency keys, dedup tables, or naturally idempotent operations), so a duplicate is a no-op. That combination is what people mean when they say exactly-once.
When a dependency dies, a path either keeps working permissively (fail-open, favours availability) or refuses safely (fail-closed, favours safety). There is no global answer; it is a per-path call, and the trade is security versus availability.
| Path / dependency down | Choice | Why |
|---|---|---|
| Auth / permission check | fail closed | an open turnstile is a security hole; deny access rather than leak it |
| Fraud / risk scoring | fail closed | money is safety-critical; hold the charge rather than wave it through |
| Recommendations / "you may also like" | fail open | a generic list beats blocking the shop; uptime beats perfection |
| Personalisation / non-critical enrichment | fail open | serve the un-personalised version; the core path still works |
| Rate limiter backing store down | it depends | fail open risks abuse; fail closed risks an outage, decide by which hurts more |
Ask: if this check is unavailable, is it safer to let traffic through or to stop it? Security-critical paths fail closed; availability-driven, non-critical paths fail open. The senior move is naming the choice per path, in the design, before the incident decides for you.
The supporting practices that make the patterns work in production. Without these, a breaker has nothing to trip on and a blast radius is a guess.
Start from what is hurting and reach for the pattern that bounds or chooses around it.
flowchart TB
Q{"what is the failure mode?"}
Q --> A1["a call hangs, no response"]
A1 --> A2["TIMEOUT, budget per hop"]
Q --> B1["a dependency blips intermittently"]
B1 --> B2["RETRY plus backoff plus JITTER"]
B2 --> B3["and the write is not idempotent"]
B3 --> B4["plus IDEMPOTENCY KEY"]
Q --> C1["a dependency is hard-down"]
C1 --> C2["CIRCUIT BREAKER, fail fast"]
Q --> D1["one slow dep starves everything"]
D1 --> D2["BULKHEAD, isolate pools"]
Q --> E1["a dependency is gone but optional"]
E1 --> E2["FALLBACK or graceful DEGRADATION"]
Q --> F1["the service is overloaded"]
F1 --> F2["LOAD SHEDDING plus backpressure"]
Q --> G1["the check itself is unavailable"]
G1 --> G2["choose FAIL-OPEN vs FAIL-CLOSED"]
classDef accent fill:#DCEBFE,stroke:#4A90D9,color:#09244B
class A2,B2,B4,C2,D2,E2,F2,G2 accent
| You want to… | Reach for | Pairs with |
|---|---|---|
| Stop a hang holding a thread | Timeout (budgeted) | deadlines propagated per hop |
| Survive a transient blip | Retry + backoff + jitter | idempotency, circuit breaker |
| Stop hammering a dead dependency | Circuit breaker | fallback for the open state |
| Stop one dep starving the rest | Bulkhead | separate pools, async |
| Keep serving when a dep is gone | Fallback / degradation | serve stale, read-only |
| Make a retried write safe | Idempotency key | dedup table, at-least-once |
| Survive overload | Load shedding | backpressure, priority |
The resilience questions that come up, and the one or two sentences that show you understand the trade-off rather than reciting a term.
Q · Why is a hang worse than a crash?
A crash frees the thread and surfaces an error you can handle. A hang holds the thread, connection and user indefinitely, and a few hangs exhaust the pool and take everything down. Bound every call with a timeout.
Q · Why add jitter to retries?
Without it, every client retries on the same beat and the synchronised spike re-kills a recovering service (a retry storm). Jitter spreads retries over a window so the service can actually come back.
Q · Walk the circuit breaker states.
Closed: calls flow, failures counted. Over threshold it trips open: fail fast, no calls, for a cool-off. Then half-open: one trial call. Succeeds, close; fails, open again. Half-open is what lets it recover safely.
Q · Bulkhead vs circuit breaker?
A breaker stops calling a dependency that is down. A bulkhead isolates resources so one slow dependency can't starve the others. One bounds persistence, the other bounds concurrency; you usually want both.
Q · How do you make a retried payment safe?
An idempotency key per logical charge: the server dedupes on it and replays the stored response instead of charging again. That turns a timeout-then-retry from a double-charge into a no-op.
Q · Is exactly-once delivery real?
End to end, rarely. You take at-least-once delivery and make the consumer idempotent, so duplicates are harmless. That combination is what people mean by exactly-once.
Q · Fail-open or fail-closed for auth?
Closed. If the auth check is unavailable, deny rather than risk leaking access. Recommendations, by contrast, fail open. The choice is per path: security-critical closed, availability-driven open.
Q · What is a dead-letter queue for?
A place to park a message after N failed retries so a single poison message can't block the whole queue. You alert on the DLQ and process or discard it out of band.
The ways resilience goes wrong, and the habit that catches all of them.
The traps
The repeatable move
1. what is the blast radius? hard vs soft edges
2. bound every call: time, repetition, persistence, concurrency
3. retried? make it idempotent
4. dependency gone? fail open or closed, on purpose
The tell in a design review: a call with no timeout, a retry with no jitter or idempotency, or a failure mode nobody named. Spot the gap and push back.
Contain the blast, bound every call, make retries safe, and choose the failure mode out loud. This whole sheet is the S9 "you contain failure, you don't prevent it" slide unpacked, and observability (S10) is how you see the containment working.
Companion to S9 · Designing for Failure. Further reading: Azure · Circuit Breaker pattern · Azure · Bulkhead pattern · AWS · Timeouts, retries and backoff with jitter · Fowler · CircuitBreaker · Google SRE · Handling overload.