Last week we picked a database. Tonight, the harder one: what do you do when one machine can't cope any more? Every answer gives you room, and hands you a new bill.
More machines, copies, caches: each one takes a bottleneck away, and hands you back a new way to break and a new "this might be slightly out of date." The senior skill isn't the trick. It's saying what you bought and what you now owe.
When one fridge isn't enough you add more fridges (sharding), keep a spare in case one dies (replication), and put a snack drawer on the counter for stuff you grab all the time (caching). Each one helps. Each one is a new thing to keep tidy.
When the database creaks, don't jump to the scary fix. The cheapest fixes come first, and the answer depends on one question: are too many reads the problem, or too many writes?
flowchart TB
P(["one DB can't keep up"]) --> Q{"reads or writes?"}
Q -- "too many reads" --> R1["add read replicas"]
R1 --> R2["add a cache"]
Q -- "too many writes" --> W1["bigger box first"]
W1 --> W2["then shard
last resort · one-way door"]
classDef g fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef w fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
classDef a fill:#ffffff,stroke:#1F2937,color:#0E1726;
class R1,R2,W1 g;
class W2 w;
class P,Q a;
Replicas and caches help when lots of people are reading. Only sharding helps when too many writes hit at once. People often shard when they really had a reading problem, and a cache would have fixed it in an afternoon.
When a table is too big for one machine, you split the rows across several. Each machine holds a slice. A simple rule, the shard key, decides which row goes where.
flowchart TB R(["users table
too big for one machine"]) R --> S{"split by user_id"} S -- "id ends 0–3" --> A["Machine A"] S -- "id ends 4–6" --> B["Machine B"] S -- "id ends 7–9" --> C["Machine C"] classDef big fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef ask fill:#ffffff,stroke:#1F2937,color:#0E1726; classDef sh fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class R big; class S ask; class A,B,C sh;
One library got too big for one building. So you split the books across three branches, with a sign on the door: "A–H here, I–P there, Q–Z over there." Sharding is that sign, for rows.
The shard key is the column you split on. It decides two things, more or less forever: does the load spread evenly, and can a normal query be answered by one machine instead of all of them?
Good keys: user_id, tenant_id. Bad keys: anything with few values (status, country) or that piles up in one place (a date).
flowchart TB K(["key: user_id = 42"]) K --> R["RANGE
ids 0–100 live together → Shard A"] K --> H["HASH
scramble 42 → Shard C"] K --> D["DIRECTORY
look it up in a table → Shard B"] classDef a fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef b fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef c fill:#E6E8FF,stroke:#1F2937,color:#0E1726; class R a; class H b; class D c;
Range = shelved A–Z. Easy to grab "all of March", but the new-stuff shelf is always mobbed. Hash = a coat-check number. Perfectly even, but you can't browse "everything from March". Directory = a front desk that knows where each thing is. Flexible, but you ask the desk first.
Sharding promises even load. Real data is lumpy. If one value is far busier than the rest, its machine gets crushed while its neighbours sit idle. You added machines and fixed nothing.
flowchart LR T(["traffic"]) --> A["Shard A
1 celebrity user"] T --> B["Shard B"] T --> C["Shard C"] A --> X(["overloaded"]) classDef hot fill:#FEE4E2,stroke:#1F2937,color:#0E1726; classDef cool fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class A,X hot; class B,C cool;
The naive way (id % number-of-machines) has a nasty surprise: add one machine and almost every row has to move. The fix is a "ring" (consistent hashing): add a machine and only a small slice moves.
flowchart LR
subgraph NAIVE["id mod N · add a machine"]
direction TB
K1(["most keys move"]) --> BAD["huge, risky migration"]
end
subgraph RING["ring · add a machine"]
direction TB
K2(["one small slice moves"]) --> OK["quick and safe"]
end
classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
class BAD bad;
class OK good;
Picture a clock face with keys dotted around it. Each machine owns the keys in its arc. Add a machine and it takes over just one arc; everyone else keeps theirs. That's why Cassandra and DynamoDB use a ring instead of plain % N.
Sharding puts different rows on different machines. Replication puts the same rows on several machines. You do it for two reasons, and you should know which one you're buying.
flowchart LR W(["all writes"]) --> L["Leader
the real one"] L -- "copies changes" --> F1["Follower 1"] L -- "copies changes" --> F2["Follower 2"] F1 --> RD(["reads spread out"]) F2 --> RD classDef lead fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef fol fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef io fill:#ffffff,stroke:#1F2937,color:#0E1726; class L lead; class F1,F2 fol; class W,RD io;
Two reasons to photocopy your data. Stay up: if one machine dies, a copy takes over. Read more: spread millions of reads across the copies instead of hammering one. Same trick, two goals.
A write lands on the leader. Does it wait for a copy to confirm before telling the user "done"? That one yes/no is the whole trade-off.
flowchart LR
subgraph SYNC["Synchronous: safe, slower"]
direction LR
A1(["write"]) --> L1["leader"] --> Wt["wait for copy"] --> D1(["reply done"])
end
subgraph ASYNC["Asynchronous: fast, can lose a little"]
direction LR
A2(["write"]) --> L2["leader"] --> D2(["reply now"])
L2 -. "copy later" .-> F2b["follower"]
end
classDef wait fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
class Wt wait;
class D1,D2 ok;
Sync = wait until a copy has it, so nothing is lost if the leader dies a second later, but every write is slower. Async = reply instantly and copy after, fast, but a crash can lose the last few seconds. Most teams keep one sync copy and the rest async.
flowchart LR
subgraph LED["Leader-based: one writer"]
direction TB
CW1(["writes"]) --> LE["leader"]
LE --> Fa["copy"]
LE --> Fb["copy"]
end
subgraph LESS["Leaderless: any machine writes"]
direction TB
CW2(["writes"]) --> Na["node"]
CW2 --> Nb["node"]
CW2 --> Nc["node"]
end
classDef lead fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
classDef node fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
class LE lead;
class Na,Nb,Nc node;
Leader = one cashier takes all the money. Tidy, but if the cashier leaves, the queue stops until someone steps up. Leaderless (Dynamo, Cassandra) = many tills. You're never blocked, but two tills might sell the last item and you sort it out after.
A copy is always a touch behind the leader. Usually fine. Sometimes it causes a confusing bug: a user does a thing, then reads from a behind copy and can't see the thing they just did.
flowchart LR U(["user posts a comment"]) --> L["leader
has it"] L -. "lag: not copied yet" .-> F["follower
missing it"] U2(["same user reloads"]) --> F F --> Q(["my comment
is missing"]) classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef warn fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class L ok; class F,Q warn;
The fix is small: for a user's own recent actions, read from the leader, not a copy. Say it out loud: "I read from copies, except your own recent writes, those come from the leader." That sentence sounds like someone who has shipped this.
A cache is a small, fast copy of an answer you already worked out, kept close to whoever needs it. A request tries each layer in turn, and the earlier it gets an answer, the cheaper it is.
flowchart LR U(["request"]) --> B["browser
on the device"] B -- "miss" --> CDN["CDN / edge
near the user"] CDN -- "miss" --> APP["app cache
Redis"] APP -- "miss" --> DB["database
slow · the truth"] classDef fast fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef slow fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class B,CDN,APP fast; class DB slow;
It's the snack drawer on your counter. You don't walk to the shop (the database) every time you want a biscuit. You keep a few close by. The whole game: how many trips can I skip, and how do I stop the biscuits going stale?
This is the one you'll use almost always. On a read, look in the cache. Hit? Return it fast. Miss? Go to the database, then put the answer in the cache so next time is fast.
flowchart LR
R(["read"]) --> C{"in cache?"}
C -- "hit" --> F1(["return fast"])
C -- "miss" --> DB["read database"]
DB --> S["store in cache"]
S --> F2(["return"])
classDef g fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef w fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
classDef a fill:#ffffff,stroke:#1F2937,color:#0E1726;
class F1,F2 g;
class DB,S w;
class C a;
A cache is a copy, and the real data changes underneath it. Now the copy is wrong. You only have two ways to deal with it, and most cache bugs come from getting them tangled.
"There are only two hard things in computing: cache invalidation and naming things." It's a joke because it's true.
Each thing tonight has one signature way it fails. Being able to list them on the spot is exactly what makes you sound like you've run this for real.
One key answers a million requests a second from cache. It expires. Now a million requests miss in the same instant and all hit the database together, the exact flood the cache was hiding. The database falls over, things slow down, and it spirals.
flowchart LR E(["hot key expires"]) --> M["1M requests miss
at the same instant"] M --> DB["all hit the database
together"] DB --> X(["database overloads"]) classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726; classDef io fill:#ffffff,stroke:#1F2937,color:#0E1726; class M,DB,X bad; class E io;
Four short sentences. Run them on any scaling choice and you sound like you decided it, because you did. The cost sentence is the one juniors skip.
The fix for all four is the same: name the bottleneck first, then name the cost. Don't add a trick you can't bill.
Take one big table from your own app. Don't reach for a tool. Answer these three, and the design defends itself.
Drop your shard key + its worst hot spot in the chat. We'll run a few live.
Easy fixes first. The shard key is near-permanent, so choose it for even load. Copies are for more reads or staying up, pick which, and own the lag. Caches are a "might be a bit stale" promise, not free speed.