The prompt, scoped
"Design YouTube" is "ingest video, process it, and stream it to the world." We keep the spine: upload a video, transcode it into multiple renditions, store it, and serve playback over a CDN with adaptive bitrate. Out of scope, said up front: recommendations, comments, search ranking, monetization. The hard parts are the async pipeline and serving enormous read traffic cheaply.
Step 1
Requirements: functional and non-functional
The functional core:
- Upload a video (large file, resumable).
- Transcode it into a ladder of resolutions and bitrates, segmented for streaming.
- Store the original and the renditions durably.
- Stream playback with adaptive bitrate, globally, low latency.
Out of scope (say it): recommendations, comments, search, ads, live streaming.
The non-functional shape:
Two facts drive the design: writes are rare but heavy and slow (transcoding minutes of video), and reads are enormous and bandwidth-dominated. A video being "processing" for a minute after upload is fine; a buffering wheel during playback is not.
Maps to S2 (requirements) and S1 (read vs write optimisation, decisively read).
Step 2
Estimation: the numbers that bound the design
| Quantity | Assumption | Result |
|---|---|---|
| Uploads | ~500 hours of video per minute | heavy but rare; the transcode load |
| Views | billions per day | the bandwidth firehose, mostly served by CDN |
| Renditions | ~6 to 8 per video (144p to 4k) | storage multiplies several-fold over the original |
| Read : write | views vastly exceed uploads | optimise everything for read and for the CDN |
| The bottleneck | not QPS | egress bandwidth, and CPU for transcoding |
The number that decides the architecture
Egress bandwidth. Serving billions of video views from your own origin would be ruinous and slow. So the architecture is built around a CDN that absorbs almost all read traffic at the edge, and an async transcode pipeline that pre-builds every rendition so the edge has small, cacheable segments to serve. Bandwidth and CPU, not requests per second, are the budget.
Maps to S2 (capacity estimation, and knowing the real bottleneck).
Step 3
API: upload then manifest
| Endpoint | Does | Notes |
|---|---|---|
POST /v1/videos | start an upload | returns a presigned, resumable upload URL straight to object storage |
| upload complete | triggers processing | enqueues a transcode job; video status = processing |
GET /v1/videos/:id/manifest | get the playback manifest | HLS/DASH manifest listing renditions + segment URLs (on the CDN) |
| segment fetch | client pulls segments | directly from the CDN, not the API; the heavy path never hits your origin |
The client uploads straight to object storage (not through your API), and fetches segments straight from the CDN. Your services touch metadata and orchestration, never the bytes on the hot path.
Maps to S5 (API design): presigned uploads, a manifest, offloading bytes to storage and CDN. See the API cheat sheet.
Step 4
Data model and store choices
Video(id PK, uploader_id, title, status=processing|ready|failed, duration) Rendition(id PK, video_id, resolution, bitrate, codec, segment_path) ViewCount[video_id] -> counter approximate, high-write blobs (not rows) original upload ....... one large file -> object storage (S3-like) segments .............. many small files per rendition -> object storage = CDN origin stores Video / Rendition metadata .. ordinary lookups, joins -> relational / NoSQL Segments + originals ........ huge, immutable, served -> object storage behind a CDN View counts ................. high-write, approximate -> sharded counters / stream aggregation
Why these stores
- The bytes (original + segments) are immutable blobs, not database rows: object storage, which doubles as the CDN origin. Never put video in a relational store.
- Metadata (which renditions exist, where the segments are) is small and relational; it backs the manifest.
- View counts are high-write and tolerate approximation, so sharded counters or stream aggregation, never a row you increment under contention.
Maps to S3 (store from access patterns: blobs vs rows) and S4 (CDN as the read-scaling layer). See the data-stores cheat sheet.
Step 5
High-level architecture
Two lanes. The upload + transcode lane (top) takes an upload to object storage, enqueues a job, and a worker pool transcodes the original into a ladder of segmented renditions written back to storage. The playback lane (bottom) serves the manifest from the API and the segments from the CDN, which pulls from the object-storage origin only on a cache miss. The bytes never travel through your application servers.
Maps to S7 (architectural styles) and S6 (async pipeline, queues).
Step 6
Deep dive A: the upload and transcode pipeline
One upload becomes many files. The pipeline is asynchronous because transcoding minutes of 4k video takes real CPU time, and the user should not wait on it.
1. resumable upload -> original lands in object storage (chunked, retryable)
2. on complete -> enqueue a transcode job {video_id}
3. worker pool pulls a job:
a. read the original
b. transcode into a LADDER: 144p, 240p, 360p, 480p, 720p, 1080p, 4k
c. SEGMENT each rendition into ~2 to 10s chunks (HLS/DASH)
d. write segments + a manifest to object storage
e. mark video status = ready, emit "processed"
4. failure? retry the job (idempotent by video_id); poison -> DLQ
Why async, and why a ladder
Async because the work is slow and bursty: a queue plus an elastic worker pool absorbs upload spikes without making anyone wait. A ladder of pre-built renditions, rather than transcoding on demand, is what makes playback instant and adaptive: the segments already exist at every bitrate, so the player just picks one. You spend storage and upfront CPU to buy fast, cheap reads.
Maps to S6 (queues, idempotent jobs, DLQ) and S1 (latency vs cost: pre-build to read fast).
Step 6 · continued
Deep dive B: CDN and adaptive bitrate
Playback is the read firehose, and the CDN is what makes it affordable. Video is delivered as many small segments at multiple bitrates; the player adapts per segment to the bandwidth it is seeing.
adaptive bitrate (ABR)
- each rendition is split into short segments (say 4s)
- the manifest lists, per rendition, the segment URLs
- the player measures throughput and picks the bitrate PER SEGMENT
good network -> 1080p segment; network drops -> 480p segment
- result: it degrades resolution instead of buffering
CDN
- segments are immutable + cacheable -> perfect for edge caching
- popular video: served entirely from the nearest edge (origin untouched)
- long-tail video: first request misses, pulls from origin, then caches
- origin (object storage) sees a tiny fraction of total read traffic
The senior point
Immutable segments are why the CDN works so well: they can be cached forever at the edge with no invalidation problem. ABR is the other half: the player trades resolution for continuity, so a weak network gets a watchable lower-quality stream instead of a spinner. Together they turn an impossible bandwidth bill into a tractable one.
Maps to S4 (caching, the CDN as a read-scaling tier) and S1 (latency vs cost).
Step 6 · continued
Deep dive C: read scale and the long tail
Views are heavily skewed: a small set of videos is enormously popular, and a vast long tail is watched rarely. The design handles both ends differently, the same hot-key idea as the celebrity timeline.
the head (viral videos)
- hot at every edge; pre-warm popular content to edges proactively
- origin never sees this traffic; the edge absorbs it all
the long tail (rarely watched)
- first view in a region misses -> pulls from origin, caches at the edge
- tiered caching (regional cache in front of origin) shields the origin
from a thundering herd when something suddenly goes viral
view counting
- do NOT increment a row per view (contention + write load)
- aggregate a stream of view events, update counts approximately
Why this matters
If you only optimise for the head you melt the origin on the tail; if you only optimise for the tail you overpay for the head. Tiered caching plus proactive pre-warming covers both, and treating view counts as an approximate stream aggregation, not a hot row, keeps a viral video from hammering the database.
Maps to S4 (hot keys, tiered caching) and S9 (a viral video is a self-inflicted load spike).
Step 7
Reliability & failure design
| If this fails… | Mode | What the user sees |
|---|---|---|
| A CDN edge | degrade | routed to another edge; brief rebuffer, then continues |
| Transcode worker / job | retry | job is idempotent by video id; retried, or parked in a DLQ and alerted |
| Bandwidth drop (client) | fail open | ABR steps down to a lower bitrate instead of buffering |
| Upload interrupted | resume | resumable upload continues from the last chunk; nothing re-sent |
| Origin under herd | degrade | tiered cache shields it; worst case, throttle the cold tail |
Playback is built to degrade, not to fail: ABR drops resolution, edges fail over, and the origin is shielded by tiers of cache. The write path is durable-first: the original is safely in object storage before processing, and transcode jobs are idempotent and retried, with a DLQ for the ones that never succeed. Bounded everywhere with timeouts and breakers.
Maps to S9 (designing for failure): degrade playback, retry the pipeline. See the resilience cheat sheet.
Step 7 · continued
Observability & operability
- The SLI that defines video: rebuffer ratio (time spent buffering over watch time) and playback start latency. These are what users actually feel; a naive design watches CPU instead.
- Pipeline health: transcode queue depth and job success rate; a backlog means uploads sit in "processing".
- CDN economics: edge cache-hit ratio and origin egress. A falling hit ratio is both a cost and a reliability signal.
- The three pillars: metrics for rebuffer and hit ratio, traces across upload and transcode, structured logs keyed by video id.
- Alert on experience and cost: page on rebuffer-ratio burn, transcode backlog, and origin-egress spikes (a cache-miss storm), not raw CPU.
Maps to S10 (observability). See the observability cheat sheet.
Step 8
Trade-offs along the six core axes
| Axis (S1) | The call here | Why it is least-wrong |
|---|---|---|
| Read vs write optimisation | optimise hard for read | views dwarf uploads; pre-build everything reads need |
| Latency vs cost | CDN + pre-transcoded ladder | spend storage and a CDN bill to make playback instant and cheap to serve |
| Space vs time | store many renditions | keep 6 to 8 copies so the player never transcodes on demand |
| Consistency vs availability | availability + eventual | a video "processing" briefly is fine; playback must always work |
| Sync vs async | async transcode | decouple the slow, bursty work from the upload so no one waits |
| Simplicity vs capability | standard HLS/DASH + object store + CDN | use the boring, proven streaming stack; the novelty is all in scale |
The whole design is one shape: pre-build everything reads need (the rendition ladder), serve it from the edge (the CDN), and keep the slow, heavy work off the user's path (the async pipeline). Say that and the boxes follow.
Maps to S1 (the six core trade-offs) and S12 (present and defend).
Use this in the cohort
- Slot: Week 6 worked design. Contrast with Twitter: both are read-heavy, but here the read is bandwidth, solved by a CDN, not a precomputed timeline.
- Run it live: walk steps 1 to 5, then ask "how do you serve billions of views without melting your servers?" and let the room arrive at the CDN and pre-transcoding.
- The one sentence: "pre-transcode a ladder of immutable segments, serve them from a CDN, and keep the slow transcode async, so reads are cheap and instant and nobody waits on a write."