Designing for Failure.

Blast radius, fail-open vs fail-closed, and the resilience patterns that keep a failure local. Two live drills, four real outages.

Recommendations got 300ms slower. Nine minutes later, checkout was down.

The run sheet.

You'll be able to…

You don't prevent failure. You contain it.

At scale something is always broken. Reliability isn't the absence of failure, it's making sure a failure stays small and visible. The design question is never "will it fail" but "what does it take down with it?"

Blast radius: what one failure takes down with it.

Before a component ships, you should be able to say exactly what dies when it dies. Walk the dependency graph outward from the component: every caller that can't tolerate its absence is inside the blast radius. You compute this before the incident, not during.

flowchart TB
  AUTH(["Auth service"]) --> API(["API gateway"])
  CACHE(["Session cache"]) --> API
  API --> WEB(["Web app"])
  API --> MOB(["Mobile app"])
  REC(["Recommendations"]) --> WEB
  classDef hit fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
  classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  classDef opt fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
  class AUTH,API,WEB,MOB hit;
  class CACHE ok;
  class REC opt;
        

Compute it: critical vs optional, per edge.

For every dependency edge, label it hard (caller cannot function without it) or soft (caller degrades but survives). The blast radius of a node is everything reachable through hard edges. The cheapest reliability win in any review is turning a hard edge into a soft one.

Map the blast radius of your capstone.

Bound every call. Four patterns do it.

An unbounded call is how a single slow dependency becomes a site-wide outage. These four patterns each bound a different axis: time, repetition, persistence, and concurrency. We take one slide each, with the failure it prevents.

A hang is the worst failure. Bound every hop.

A crash is honest, it frees the thread and surfaces an error. A hang holds the thread, the connection, and the user, forever. Give every outbound call a timeout, and make the timeouts add up: each hop gets a slice of the budget left by the hop above it.

flowchart LR
  U(["User
budget 1000ms"]) --> A(["API
900ms left"])
  A --> B(["Service
600ms left"])
  B --> D(["DB query
400ms cap"])
  D -.->|"over budget?"| X(["fail fast,
don't hang the chain"])
  classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
  classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
  class U,A,B io;
  class D ok;
  class X bad;
        

Retry only the retryable. Add backoff and jitter.

Retries paper over a blip, but a naïve retry is a loaded gun. Retry only idempotent, transient failures (timeouts, 503s), never a 400. Back off exponentially, and add jitter so a thousand clients don't all retry on the same beat and hammer a recovering service flat again.

flowchart TB
  F(["dependency blips"]) --> S{"retry strategy?"}
  S -- "immediate, in lockstep" --> STORM(["retry storm:
synchronized spike
re-kills the service"])
  S -- "backoff + jitter" --> CALM(["spread-out retries:
service recovers"])
  classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
  classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726;
  classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
  class F io;
  class S ask;
  class CALM good;
  class STORM bad;
        

Retries multiply down the stack. Then they explode.

Retries feel local, but they compound at every hop. If each of four layers retries three times on failure, one user click doesn't make one backend call. It makes…

Stop calling the dead. Closed → open → half-open.

When a dependency is clearly down, retrying just wastes threads and slows everyone. A breaker watches the failure rate: trip open and fail fast for a cool-off, then go half-open to let one trial call test the water before you trust it again. Without half-open you either stay dark forever or slam it back open blind.

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: failure rate over threshold
  Open --> HalfOpen: cool-off elapsed
  HalfOpen --> Closed: trial call succeeds
  HalfOpen --> Open: trial call fails
  note right of Closed: calls flow, failures counted
  note right of Open: fail fast, no calls
        

Isolate the pools so one leak can't sink the ship.

Share one thread or connection pool across every dependency and the slowest one starves all the others: a stuck "recommendations" call holds threads that "checkout" needed. Give each dependency its own bounded pool. One floods, the rest sail on.

flowchart TB
  subgraph Shared["One shared pool · fragile"]
    R1(["slow dep eats
every thread"]) --> DOWN(["checkout starves too"])
  end
  subgraph Bulk["Bulkheaded · isolated"]
    P1(["pool: checkout"]) --> OKC(["unaffected"])
    P2(["pool: recommendations"]) --> SLOW(["only this degrades"])
  end
  classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
  classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  classDef opt fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
  class R1,DOWN,SLOW bad;
  class P1,OKC good;
  class P2 opt;
        

Fail-open vs fail-closed: decide per path, out loud.

When a dependency dies, the path either keeps working permissively (fail-open, favours availability) or refuses safely (fail-closed, favours safety). There is no global answer, it's per path. Saying which, and why, is the highest-signal sentence in a design review.

flowchart LR
  AUTH(["Auth check down"]) --> AC(["FAIL CLOSED:
deny access
safety beats uptime"])
  REC(["Recommendations down"]) --> RO(["FAIL OPEN:
show generic list
uptime beats perfection"])
  PAY(["Fraud score down"]) --> PC(["FAIL CLOSED:
hold the charge
money is safety-critical"])
  classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
  classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  classDef opt fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
  class AUTH,PAY bad;
  class AC,PC bad;
  class REC opt;
  class RO good;
        

Now say yours. Round the room.

Degrade on purpose. Partial service beats a blank page.

When you can't serve the perfect response, serve a useful one. Degradation is a designed mode, not an accident: decide in advance what you drop first and what you protect to the last. The user should notice less, not everything at once.

Cascading failure: how local becomes global.

Here is the cold open, drawn out. A single slow dependency fills retry queues, the retries exhaust the shared pool, callers time out and retry harder, and the slowness climbs the stack until the whole site is down, often well after the original fault healed. Every pattern tonight is a firebreak that stops the climb at one hop.

flowchart LR
  D(["one DB slows"]) --> Q(["retries pile up"])
  Q --> P(["shared pool exhausted"])
  P --> T(["callers time out,
retry harder"])
  T --> G(["site-wide outage"])
  P -. "bulkhead" .-> FB1(["pool stays local"])
  Q -. "breaker + jitter" .-> FB2(["storm never forms"])
  classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
  classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  class D,Q,P,T,G bad;
  class FB1,FB2 good;
        

Worked example: the toolkit on one real system.

A booking app (think the ClubCP case study): web → API → bookings DB, with an email provider and a recommendations service hanging off the side. The email provider goes slow. Walk the toolkit and the failure stays a footnote instead of a front-page outage.

flowchart TB
  WEB(["Web"]) --> API(["API"])
  API --> DB(["Bookings DB
hard edge · fail closed"])
  API --> EMAIL(["Email provider
soft · timeout 800ms"])
  API --> REC(["Recommendations
soft · fail open"])
  EMAIL -. "slow" .-> BRK(["breaker trips →
queue email, return 200"])
  REC -. "down" .-> DEG(["serve generic list"])
  classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
  classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  class WEB,API,DB ok;
  class EMAIL,REC io;
  class BRK,DEG good;
        

Four outages, four missing patterns.

You're reviewing this config. What's wrong?

# PaymentService · resilience.yaml (up for review)

http:
  connect_timeout: none
  read_timeout:    none

retry:
  attempts:  5
  backoff:   fixed 200ms
  retry_on:  [500, 503, 400, timeout]

breaker:
  trip_after: 20 failures
  reset:      manual

pools:
  mode:    shared
  size:    200 threads
  used_by: [fraud, ledger, email, recs]

The four you should have circled.

Contain the blast. Choose the failure mode.

Assume everything fails, map the blast radius, and bound every call with timeouts, retries with jitter, breakers and bulkheads. Then decide fail-open vs fail-closed per path, out loud, and design the degradation modes before you need them.