Design YouTube
Design YouTube at 2B users, 500 hours uploaded/minute. Covers resumable multipart upload to GCS, a chunked parallel transcoding pipeline producing 10 encodings per video (5 resolutions x 2 codecs), HLS/DASH adaptive bitrate playback via multi-tier CDN, search via Elasticsearch fed async by Kafka, and recommendation ranking via a candidate-then-rank two-stage model with pre-computed top-N cached in Redis. The deep dive walks through how transcoding keeps up with 500 hours/minute.
How it works
What are we designing? YouTube — a user-generated video sharing platform at massive scale. Anyone can upload a video; the system transcodes it into multiple resolutions/codecs, serves it to viewers via adaptive bitrate streaming, indexes it for search, and recommends related videos. Unlike Netflix (curated, ~100K titles), YouTube is UGC: 500 hours of video uploaded per minute, with wildly varying quality and popularity.
The defining challenges: massive ingestion volume (transcoding 500 hours/minute of uploads in real time), long-tail access patterns (most videos get <1000 views; a few go viral), and search + recommendation over billions of videos.
Functional requirements.
- Upload a video (any size, any format).
- Watch a video on demand; supports quality selection, scrub, captions.
- Like, comment, subscribe.
- Search by title / description / tags.
- 'Up next' recommendations.
- Live streaming (a stretch goal).
Non-functional requirements.
- Upload-to-available: < 5 min for a 10-min video (transcode pipeline).
- Time-to-first-frame: < 2 s for playback.
- Availability: 99.99%.
- Scale: 2B+ users, 1B hours watched/day, 500 hours uploaded/minute.
Non-goals. No YouTube TV, no Shorts (separate vertical), no community posts.
Capacity estimation.
Uploads. 500 hours/minute = 30K hours/hour = 720K hours/day. At ~1 GB/hour raw (compressed HD), 720K GB/day = 720 TB/day of raw uploads = ~260 PB/year.
Transcoding. Each video is transcoded into ~5 resolutions x 2 codecs = 10 encodings. So 7.2 PB/day of transcoded output. With 10x for transcode compute (CPU-hours per video hour), this is ~7M CPU-hours/day — distributed across thousands of worker instances.
Watch volume. 1B hours/day watched. At ~3 Mbps average stream = ~1.4 EB/day of egress. YouTube is ~10% of global internet traffic.
Storage. 720K hours/day raw + 7.2 PB transcoded = ~8 PB/day new content. After 3 years: ~9 EB. Stored on cold object storage (GCS) for originals; transcoded encodings on hot object storage and CDN edges.
Metadata. Each video ~10KB (title, description, tags, uploader, ts). 720K uploads/day = 7.2 GB/day metadata = ~2.6 TB/year. Search index ~5x that = 13 TB/year.
Search. 3B+ queries/day globally.
APIs.
POST /v1/upload/start -> { upload_id, presigned_url_parts[] } (multipart)
POST /v1/upload/complete (upload_id) -> kicks off transcoding
GET /v1/videos/:id -> metadata, available resolutions, captions
GET /v1/playback/:id -> manifest_url (HLS/DASH) + ad markers
POST /v1/videos/:id/like
POST /v1/videos/:id/comments
POST /v1/users/:id/subscribe
GET /v1/search?q=...&type=video
GET /v1/recommendations -> 'up next' list for the current user+videoUpload uses resumable multipart upload to GCS / S3 — each part can retry independently, and the upload can pause/resume across network blips.
Data model.
Videos (sharded SQL):
videos (id BIGINT PK, uploader_id, title, description, tags JSON,
duration_sec, upload_status ENUM('uploading','transcoding','ready','failed'),
view_count, like_count, created_at)
INDEX (uploader_id, created_at DESC)
FULLTEXT(title, description, tags)Encodings (object storage):
encodings (id, video_id, resolution, codec, bitrate_kbps, manifest_url)Comments (Cassandra, partitioned by video_id):
comments (video_id, comment_id TIMEUUID, author_id, text, like_count, ts,
PRIMARY KEY ((video_id), comment_id))Subscriptions (sharded SQL):
subscriptions (subscriber_id, channel_id, created_at,
PRIMARY KEY (subscriber_id, channel_id))
INDEX (channel_id) -- for 'who subscribes to me'Search index (Elasticsearch, fed by Kafka from uploads):
video_search: { video_id, title, description, tags, view_count, ts }Recommendations (pre-computed per user in Redis):
key: recs:{user_id}:{seed_video_id}
value: LIST of video_ids -- context-aware recs
TTL: 1hDeep dive: the transcoding pipeline and adaptive streaming.
Why transcoding is hard at YouTube's scale. 500 hours uploaded per minute means 500 hours of CPU work per minute (transcoding is roughly 1:1 real-time on a single core). To keep up, YouTube runs thousands of transcode workers (likely tens of thousands of CPU cores). Each video is split into N chunks (e.g. 10s each); each chunk is transcoded independently and in parallel. A 10-min video = 60 chunks, each transcoded in parallel — total wall-clock time ~10s per encoding.
The 10-encodings-per-video problem. Each video is encoded in:
- 5 resolutions: 144p, 360p, 720p, 1080p, 4K
- 2 codecs: H.264 (universal), VP9/AV1 (saves 30-50% bandwidth on capable devices) = 10 encodings per video. With 720K hours uploaded/day, that's 7.2M hours of transcoded output per day. YouTube does this with a huge fleet of spot/preemptible instances.
Chunked upload + resumable. A 1-hour 4K video is ~7 GB. If the upload fails at 99%, restarting from zero is unacceptable. YouTube uses GCS resumable upload (multipart): the client uploads the file in parts; each part can retry independently; the upload can pause and resume across hours.
Adaptive bitrate playback. Same as Netflix: HLS/DASH with 10s segments; player switches bitrate based on throughput. YouTube adds VP9/AV1 segments for capable devices, saving bandwidth (and YouTube's bandwidth bill).
Search indexing. Every upload publishes a Kafka event; an indexer writes the video's title, description, tags, and channel to Elasticsearch. Search returns video_ids; the player fetches metadata separately. View count is a ranking signal (boosted in the index periodically).
Recommendations. YouTube's recommendation model is famously a deep neural net over user history and video features. For our design: a candidate generator produces 500 candidate videos per (user, seed_video) pair from collaborative filtering; a ranker scores them with a DNN; top 20 are cached in Redis with a 1-hour TTL. The ranker weighs signals like watch time (not just clicks), recency, and channel affinity.
Captions. Auto-generated via speech-to-text (Whisper-style models) on the transcode worker. Captions are stored as WebVTT files served alongside segments.
Bottlenecks and failure modes.
-
Transcode backlog. A viral spike in uploads saturates the worker pool; new videos take 30+ minutes to become watchable. Mitigation: autoscale on queue depth; pre-emptible instances for capacity headroom; serve a low-res preview from the raw upload while transcoding.
-
Viral video hot edge. A new viral video gets 10M views in an hour; the CDN edge that first serves it saturates. Mitigation: CDN multi-tier caching; replicate to multiple edges; serve from origin on edge miss.
-
Comment hotspot. A viral video's comments table gets 100K writes/sec to one Cassandra partition. Mitigation: shard comments by sub-partition (video_id, comment_id_range).
-
Elasticsearch index lag. Recent uploads aren't searchable for 5-30s. Mitigation: accept it; show 'this video is being indexed' for the uploader.
-
Search index size. Billions of videos = petabyte-scale ES cluster. Mitigation: shard by year (recent videos get more replicas); archive old videos to a cold index.
-
Storage cost. 8 PB/day of new content is expensive. Mitigation: tiered storage (hot for popular, cold for long-tail); deduplicate identical encodings across formats; delete failed/copyright-taken-down uploads after grace period.
-
Live streaming. Live is a fundamentally different workload (no transcoding ahead of time; transcode on the fly). Mitigation: separate live pipeline with lower latency targets; HLS-LL or WebRTC for sub-second latency.
-
Copyright / content moderation. Every upload must be checked against a fingerprint database (Content ID). Mitigation: fingerprint on transcode; compare against Content ID DB; flag matches for review.
Scaling strategy and trade-offs.
Transcoding scaling. Spot/preemptible instances autoscaled on queue depth. Each worker pulls jobs from Kafka, transcodes a chunk, uploads to GCS.
CDN scaling. Multi-tier CDN: edge caches (closest to users) -> regional caches -> origin (GCS). Popular videos propagate up the tiers; long-tail content stays near origin.
Search scaling. Elasticsearch sharded by video_id hash, with time-based rollover indices. Old indices (videos >1 year) get fewer replicas (less queried).
Storage scaling. GCS / S3 with lifecycle policies: hot for first 30 days, nearline for 30-365 days, coldline after 1 year. Re-hydrate on view.
Multi-region. Uploads go to the nearest region; transcoded encodings are replicated globally. Search and recs run per-region.
Trade-offs made explicit.
- We chose async transcoding — gained fast upload responses, lost instant playback (5-min delay for new videos).
- We chose multi-codec (H.264 + VP9/AV1) — gained bandwidth savings, lost 2x transcoding cost.
- We chose ES for search — gained flexible ranking, lost a finicky operational system at petabyte scale.
- We chose pre-computed recs with 1h TTL — gained fast reads, lost freshness (a brand-new viral video won't appear in recs for up to 1h).
- We chose chunk-based transcoding — gained parallelism, lost the ability to seek within a not-yet-complete video.
YouTube receives 500 hours of uploads per minute. How can transcoding keep up?
Pick one answer.
A user uploads a 1-hour 4K video (~7 GB) and the upload fails at 99%. What upload strategy prevents restarting from zero?
Pick one answer.
Engineering mental model
Mental model. Think of Design YouTube as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Design YouTube mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Design YouTube, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = design_youtube(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Design YouTube
Change the variables below and predict what breaks first in Design YouTube. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Design YouTube, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Design YouTube. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Design YouTube?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Design YouTube, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Design YouTube, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Design YouTube: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Design YouTube. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Chunked parallel transcoding keeps up with 500 hours/min of uploads.
- +Resumable multipart upload handles multi-GB videos on flaky networks.
- +Multi-codec (H.264 + VP9/AV1) saves 30-50% bandwidth on capable devices.
- +Two-stage recommendation (candidate generation then ranking) is the standard pattern.
- −Transcode backlog on upload spikes — needs autoscaling + spot capacity.
- −Long-tail videos stay near origin (CDN miss) — slower first-byte for rare videos.
- −Petabyte-scale Elasticsearch is operationally hard.
- −Recommendations have a freshness lag (1h TTL on cached recs).
How this breaks in production
- Viral video saturating one CDN edge — needs multi-tier CDN with replication.
- Comment hotspot on viral videos — needs Cassandra sub-partitioning.
- Transcode backlog — needs autoscaling on queue depth.
- Search index lag (5-30s) for brand-new uploads.
- Live streaming fundamentally different — needs separate low-latency pipeline.
Don't fall into these traps
- •Single PUT upload for large videos — fails catastrophically on network blip.
- •Sequential per-video transcoding — can't keep up with 500 hours/min.
- •Single-codec encoding — wastes bandwidth on capable devices.
- •On-read recommendations — blows latency budget.
- •Single Elasticsearch cluster for billions of videos — no sharding strategy.
Real systems using this
How real systems implement this
- YouTube — GCS resumable upload, chunked parallel transcoding on Borg/GKE, multi-tier CDN, VP9/AV1 codecs, deep-learning recommendations. Documented in Google research papers.
- TikTok — Similar ingestion pipeline; shorter videos mean transcoding is faster per video but the upload rate is 10x higher. Recommendations heavily weighted toward watch completion rate.
- Vimeo — Smaller scale but higher per-video quality focus. Uses AWS MediaConvert for transcoding, CloudFront CDN.
Practice saying it out loud
- Q1Design YouTube. How do you transcode 500 hours of uploads per minute?
- Q2A user uploads a 7 GB 4K video and the upload fails at 99%. How do you prevent restarting from zero?
- Q3How do you make search not block the upload pipeline?
- Q4A video goes viral and saturates one CDN edge. What do you do?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Design Netflix