hub.gazar.dev · senior-to-staff · worked system design

Design Stripe: payments

The third common prompt, and the one where correctness beats everything. We walk it in the course order: requirements, numbers, the idempotent API, data, the high-level diagram, the deep dives that actually decide it, then reliability, observability, and the trade-offs. The lever here is the opposite of the other two: not availability, but not losing or duplicating a cent. Idempotency, a double-entry ledger, and reconciliation are how. Back to the course.

The prompt, scoped

"Design Stripe" is "process a card payment correctly, at scale, and be able to prove the books are right." We keep the spine: create a charge, confirm it asynchronously, record it in a ledger exactly once, and reconcile against the processor. Out of scope, said up front: the full PCI surface, fraud-model internals, payouts and treasury, and FX. The interesting part is correctness under retries and partial failure, not throughput.

Step 1

Requirements: functional and non-functional

The functional core:

Out of scope (say it): PCI vault internals, fraud ML, payouts, FX, disputes UI.

The non-functional shape is the inverse of a social feed:

correctness > availability strong consistency · ACID idempotent everywhere durable + auditable fail closed

The defining fact: a lost or duplicated charge is unacceptable, where a late tweet was fine. That single inversion flips every default toward consistency, durability, and refusing rather than guessing. Hold it.

Maps to S2 (requirements) and S1 (consistency vs availability, here firmly consistency).

Step 2

Estimation: the numbers that bound the design

QuantityAssumptionResult
Charges~5k charges/sec at peakmodest QPS by web standards
Ledger writesdouble-entry, plus state transitions~2 to 4 durable writes per charge
Webhooks ineach charge confirms asyncinbound network events ~ charge rate
Retentionyears, for audit and disputesevery charge + transition kept, an archive not a buffer
The real constraintnot QPSevery single write must be correct, durable, idempotent

The number that decides the architecture

It is not requests per second, which is small. It is that the acceptable error rate on money is effectively zero. That forces an ACID store, an idempotency key on every mutation, a double-entry ledger that must always balance, and a reconciliation job to catch the drift that still happens. Correctness is the budget, not latency.

Maps to S2 (capacity estimation, and knowing when QPS is not the constraint).

Step 3

API: idempotency is the headline feature

The signature of a payments API is the Idempotency-Key header. It is what makes a retry safe, and it is the first thing an interviewer wants to hear.

EndpointDoesNotes
POST /v1/payment_intentscreate + confirm a chargeIdempotency-Key header; same key returns the same result, never re-charges
POST /v1/refundsrefund a chargeidempotent; writes the reversing ledger entries
POST /v1/webhooks (in)receive network confirmationverify signature; dedupe on event id (at-least-once)
merchant webhook (out)notify the merchantat-least-once with retries; the merchant must be idempotent too

Maps to S5 (API design): idempotency keys, signed webhooks. See the API cheat sheet.

Step 4

Data model and store choices

PaymentIntent(id PK, amount, currency, status, idempotency_key UNIQUE)
Charge(id PK, payment_intent_id, network_ref, status)
LedgerEntry(id PK, account, direction=debit|credit, amount, charge_id, created_at)
IdempotencyKey(key PK, request_hash, response, status)
WebhookEvent(id PK = provider event id, processed_at)     dedupe inbound

invariant:  for every charge, SUM(debits) == SUM(credits)        the ledger must balance

stores
  Everything above .. needs ACID transactions + strong consistency  -> PostgreSQL
  (state transition + ledger entries + idempotency key are written in ONE transaction)
erDiagram
    IDEMPOTENCY_KEY ||--o| PAYMENT_INTENT : "guards creation of"
    PAYMENT_INTENT  ||--o{ CHARGE          : "has attempts"
    CHARGE          ||--|{ LEDGER_ENTRY     : "posts debit + credit"
    CHARGE          ||--o{ WEBHOOK_EVENT    : "confirmed by"
    PAYMENT_INTENT {
        uuid   id PK
        int    amount_minor
        string currency
        string status
        string idempotency_key UK
    }
    CHARGE {
        uuid   id PK
        uuid   payment_intent_id FK
        string network_ref
        string status
    }
    LEDGER_ENTRY {
        uuid   id PK
        string account
        string direction "debit | credit"
        int    amount_minor
        uuid   charge_id FK
    }
    IDEMPOTENCY_KEY {
        string key PK
        string request_hash
        json   response
        string status "pending | done"
    }
    WEBHOOK_EVENT {
        string id PK "provider event id"
        uuid   charge_id FK
        ts     processed_at
    }
The five tables and their relationships. Every CHARGE posts at least one matched pair of LEDGER_ENTRY rows (a debit and an equal credit), which is why the relationship is one-to-many-but-never-zero. IDEMPOTENCY_KEY guards intent creation; WEBHOOK_EVENT dedupes inbound confirmations. All of it lives in one Postgres so a charge's state change and its ledger entries commit in a single transaction.

Why one ACID store, not a fashionable distributed one

  • The charge state change, the two ledger entries, and the idempotency record must be atomic: all or nothing. That is a transaction, and a relational store gives it for free.
  • Double-entry means every movement writes a debit and an equal credit, so the books always balance and any single bad write shows up as an imbalance you can alert on.
  • QPS is low enough that one well-run Postgres (with read replicas and partitioning by time) handles it. Reach for sharding only when the numbers force it, and here they do not.

Maps to S3 (store from access patterns: ACID wins here) and S4 (scale only when forced). See the data-stores cheat sheet.

Step 5

High-level architecture

A charge is a state machine that survives retries and partial failure. The payment service checks the idempotency key, writes the intent and ledger entries to an ACID store in one transaction, and calls the card network. Confirmation comes back asynchronously as a webhook, which updates state and writes the settling ledger entries, again transactionally and idempotently. A reconciliation job compares our ledger against the processor's daily settlement file and flags any drift.

Client + Idempotency-Key API gateway L7 · auth · TLS Payment service idempotency + state machine Card network acquirer · processor · external Postgres ledger ACID · double-entry intents · charges · entries · keys Webhook handler async · signed · idempotent Reconciliation job daily · flags drift ① one txn: intent + 2 entries + key ② authorize ③ async result ④ settle (txn) ⑤ compare settlement file (daily)
Numbered by order of events. Solid navy = our synchronous path; orange dashed = asynchronous returns from the external card network. ① The payment service writes state, both ledger entries and the idempotency key in one ACID transaction, then ② authorizes. ③ Confirmation arrives later as a signed webhook, which ④ settles the ledger idempotently in its own transaction. ⑤ Reconciliation compares our ledger against the processor's daily settlement file and flags any drift.

Read as a timeline instead of a topology, the same flow is a sequence: the client is acked the moment the intent is durably recorded, long before the network has actually decided. Everything after the ack is asynchronous and idempotent.

sequenceDiagram
    autonumber
    actor C as Client
    participant P as Payment service
    participant DB as Postgres ledger
    participant N as Card network
    C->>P: POST /payment_intents + Idempotency-Key
    P->>DB: BEGIN, insert key + intent + ledger entries
    Note over P,DB: unique key picks one winner, a retry reuses the row
    DB-->>P: COMMIT (status = processing)
    P->>N: authorize
    P-->>C: 200, intent processing (durable, not yet settled)
    Note over N: later, the network decides
    N-->>P: webhook: authorized / captured (at-least-once)
    P->>DB: dedupe event id, then settle in ONE txn
    DB-->>P: status = succeeded, settling entries posted
    P-->>C: (out) merchant webhook: succeeded
The charge as a timeline. The client is acked at step 5, as soon as the intent is durably committed, so its latency never waits on the card network. Steps 6 to 9 happen asynchronously: the network's confirmation is deduped and applied exactly once, and only then is the merchant notified.

And because confirmation is asynchronous, the intent is a state machine the network drives. We never move ourselves to succeeded; only a verified webhook does. Every transition is idempotent, so a replayed webhook re-asserts a state without ever moving it backward.

stateDiagram-v2
    [*] --> RequiresConfirmation: created (amount, currency, method)
    RequiresConfirmation --> Processing: confirm, authorize sent
    Processing --> Succeeded: webhook, authorized + captured
    Processing --> Failed: webhook, declined / expired
    Processing --> Processing: duplicate webhook (dedupe by event id)
    Succeeded --> Refunded: POST /refunds, reversing entries
    Failed --> [*]
    Succeeded --> [*]
    Refunded --> [*]
    note right of Processing
        the network is the source of truth here,
        we never self-transition to Succeeded
    end note
The PaymentIntent lifecycle. Only a verified network webhook moves Processing forward to Succeeded or Failed; the self-loop shows a duplicate webhook re-asserting Processing as a no-op. A refund is a forward transition that posts reversing ledger entries, never an in-place edit of the original.

Maps to S7 (architectural styles) and S6 (idempotency, async confirmation, the transactional path).

Step 6

Deep dive A: idempotency keys

The mechanism that makes a retry safe. The client generates one key per logical charge; the server guarantees that key produces exactly one charge, no matter how many times the request arrives.

POST /payment_intents
Idempotency-Key: idem_a1b2

server, inside one transaction:
  row = INSERT IdempotencyKey(idem_a1b2) ... ON CONFLICT DO NOTHING
  if conflict (key already exists):
      if status = done  -> return the STORED response   (duplicate: no re-charge)
      if status = pending -> 409 / retry-after           (a concurrent attempt)
  else (we won the insert):
      create PaymentIntent, write ledger entries, mark key done
      return response

=> a timeout-then-retry, or two concurrent clicks, settle to ONE charge.
flowchart TD
    R["Request arrives
Idempotency-Key: idem_a1b2"] --> I["INSERT IdempotencyKey(idem_a1b2)
ON CONFLICT DO NOTHING"] I --> Q{"Did we win
the insert?"} Q -->|"yes, first attempt"| W["Create PaymentIntent
+ write ledger entries
+ mark key done
(one txn) · return response"] Q -->|"no, key already exists"| S{"Stored status?"} S -->|"done"| D["Return the STORED response
(no re-charge)"] S -->|"pending"| P["409 / Retry-After
(a concurrent attempt in flight)"]
The whole idempotency guarantee is one atomic insert with three outcomes. Winning the race does the work; losing it either replays the stored response (the duplicate case) or backs off (the concurrent case). Because the database, not the application, picks the single winner, two requests with the same key can never both charge.

The detail interviewers probe

Concurrency. Two requests with the same key arriving at once must not both charge. A unique constraint on the key (the database picks one winner) plus storing the in-progress state is what makes it correct, not just a "check then insert" that races. The key is also scoped and expired so it cannot be replayed forever.

Maps to S6 (idempotency) and S9 (safe retries). See the resilience cheat sheet.

Step 6 · continued

Deep dive B: the ledger and exactly-once accounting

Money is tracked in a double-entry ledger: every movement writes a matching debit and credit, so the books always balance. Exactly-once is achieved not by magic delivery but by idempotent application inside transactions.

a successful charge of 9.99 writes TWO entries, in one transaction:
   DEBIT   customer_receivable   9.99
   CREDIT  merchant_payable      9.99
   ───────────────────────────────────
   invariant: SUM(debits) == SUM(credits)   always

exactly-once accounting (over at-least-once webhooks)
   webhook arrives -> INSERT WebhookEvent(event_id) ON CONFLICT DO NOTHING
   if conflict -> already processed, drop it (idempotent)
   else        -> apply state change + ledger entries in ONE txn

=> the network re-sends a webhook (it will); we process it once.
customer_receivable DEBIT CREDIT 9.99 · merchant_payable DEBIT CREDIT · 9.99 Σ debits 9.99 == Σ credits 9.99 ✓ balanced
One charge, posted as two entries. The customer_receivable account is debited 9.99 and the merchant_payable account is credited 9.99, in the same transaction. The invariant is not that any one account is zero, but that across every account Σ debits == Σ credits. Any single mis-posted row breaks that sum and trips the alert.

The webhook that triggers settlement arrives at least once. Exactly-once accounting comes from deduping the event before applying it, all inside one transaction:

sequenceDiagram
    autonumber
    participant N as Card network
    participant W as Webhook handler
    participant DB as Postgres
    N->>W: POST charge.succeeded {event_id, sig}
    W->>W: verify signature (reject if bad)
    W->>DB: INSERT WebhookEvent(event_id) ON CONFLICT DO NOTHING
    alt first delivery (row inserted)
        W->>DB: apply state change + ledger entries, ONE txn
        DB-->>W: COMMIT
        W-->>N: 200 OK
    else duplicate (conflict)
        W-->>N: 200 OK (already processed, no-op)
    end
    Note over N,W: any non-2xx makes the network re-send, we still process once
Exactly-once over at-least-once delivery. The dedupe insert and the ledger write share one transaction, so a webhook can never be half-applied. A duplicate short-circuits to a 200 no-op, and a signature check keeps a forged event from ever reaching the ledger.

Why double-entry, not a balance column

A single mutable "balance" field hides errors: an off-by-one is invisible. A double-entry ledger is append-only and self-checking, every movement must balance, so a bug surfaces as debits != credits that you alert on immediately. The ledger is the source of truth; balances are derived.

Maps to S6 (exactly-once via idempotency) and S1 (consistency over availability).

Step 6 · continued

Deep dive C: reconciliation and fail-closed

Even with all of the above, our view and the processor's view drift: a webhook is missed, a settlement differs by a fee. Reconciliation is the backstop that catches it, and fail-closed is the default that keeps a doubtful charge from going through.

reconciliation (daily)
   pull the processor's settlement file
   for each settled txn: match against our ledger by network_ref
   flag: in-theirs-not-ours, in-ours-not-theirs, amount-mismatch
   => a human (or an automated rule) resolves the exceptions

fail-closed by default
   fraud/risk service down?     -> DECLINE or hold, do not approve blind
   cannot reach the network?    -> return pending, never a fake success
   idempotency store unavailable? -> refuse the write, do not risk a double-charge

Reconciliation is a daily join of two sources of truth, ours and the processor's, that sorts every transaction into matched or one of three exception buckets:

flowchart TD
    L["Our ledger"] --> M["Match by network_ref
(amount + status)"] F["Processor settlement file
(daily)"] --> M M --> Q{"Matched?"} Q -->|"yes, amounts equal"| OK["✓ reconciled"] Q -->|"in theirs, not ours"| E1["Missing webhook
→ replay / post entry"] Q -->|"in ours, not theirs"| E2["Not settled yet / dropped
→ investigate, maybe reverse"] Q -->|"amounts differ"| E3["Fee or FX difference
→ post an adjustment"] E1 --> H["Exception queue
human or automated rule resolves"] E2 --> H E3 --> H
Reconciliation as a set-difference. Everything that matches on network_ref and amount is done; everything else falls into exactly one of three buckets, each with a known resolution. Because it runs every cycle, any drift is bounded to a single day, never accumulating silently.

And the fail-closed posture is a single decision chain: at every point where the system is uncertain about money, the default branch is to refuse, never to guess in the customer's or the merchant's favour.

flowchart TD
    A["Charge request"] --> R{"Risk / fraud
service up?"} R -->|no| X1["DECLINE or hold
(never approve blind)"] R -->|yes| N{"Card network
reachable?"} N -->|no| X2["Return PENDING + retry
(never fake a success)"] N -->|yes| K{"Idempotency
store available?"} K -->|no| X3["Refuse the write
(never risk a double-charge)"] K -->|yes| GO["Proceed: authorize
+ post ledger (one txn)"]
Fail-closed as control flow. Three gates guard the write, and every "no" exits to a safe refusal rather than falling through. Only when risk, network, and the idempotency store are all healthy does the charge proceed. A feed would fail open at each of these gates to stay available; payments does the opposite to stay correct.

The posture that defines payments

Everywhere a feed would fail open to stay available, payments fail closed to stay correct. When in doubt, refuse. Reconciliation accepts that drift is inevitable and makes it visible and bounded, rather than pretending the happy path is the only path.

Maps to S9 (fail-closed, blast radius) and S10 (the ledger imbalance is the alert).

Step 7

Reliability & failure design

If this fails…ModeWhat happens
Fraud / risk servicefail closeddecline or hold; never approve without the check
Card network unreachablefail closedreturn pending and retry; never fake a success
Inbound webhook lostbackstopreconciliation catches it next cycle; the network also re-sends
Duplicate webhookidempotentdeduped on event id; processed exactly once
Un-processable eventDLQparked after N retries, alerted, resolved out of band

Every retry is safe because every mutation is idempotent. Calls to the network are bounded with timeouts and a circuit breaker, retried only where idempotent. The reconciliation job is the safety net under all of it: even if a message is lost, the books get corrected and the drift is bounded to one cycle, not forever.

Maps to S9 (designing for failure): fail closed, idempotent retries, a reconciliation backstop.

Step 7 · continued

Observability & operability

Maps to S10 (observability). See the observability cheat sheet.

Step 8

Trade-offs along the six core axes

Axis (S1)The call hereWhy it is least-wrong
Consistency vs availabilityconsistency, harda wrong balance is worse than a slow or refused charge
Latency vs correctnesscorrectnessdurable write + idempotency check before ack; a few ms is fine
Fail-open vs fail-closedfail closed everywherewhen in doubt about money, refuse, do not guess
Simplicity vs scalabilityone ACID storeQPS is low; a transaction is simpler and safer than a distributed dance
Exactly-once vs at-least-onceat-least-once + idempotentreal exactly-once over a network is a myth; idempotency delivers it in effect
Coupling vs cohesionasync webhooks + reconciliationdecouple from the slow network, then reconcile to catch what async drops

The whole design is one inversion held consistently: correctness outranks availability. Idempotency makes retries safe, a double-entry ledger makes errors visible, fail-closed makes doubt safe, and reconciliation makes the inevitable drift bounded. Say that and the boxes follow.

Maps to S1 (the six core trade-offs) and S12 (present and defend).

Use this in the cohort