Design Instagram

What are we designing? An Instagram-style photo-sharing social network. Users post photos (with captions), follow other users, and browse a personalized feed of the most recent photos from people they follow. The system must handle massive read traffic (feed browsing dominates), large media uploads (photos and short videos), and real-time interactions (likes, comments).

The defining design challenge is the feed: when user A follows 500 people and opens the app, the system must assemble a personalized timeline of the latest ~1000 posts from those 500 people, sorted by recency, in under 200 ms — and it must do this for 500M daily users simultaneously. Feed generation strategy is the heart of this case study.

Functional requirements.

• Users can post a photo with a caption and optional location.
• Users can follow / unfollow other users.
• Users can browse their feed (recent posts from followed users, paginated).
• Users can like / comment on a post.
• Users can view a profile with their own posts.

Non-functional requirements.

• Feed latency: p99 < 200 ms (users expect instant app open).
• Photo upload: < 5 s end-to-end including processing.
• Availability: 99.99% (Instagram is daily-use; outages are news).
• Storage: photos are forever; we never delete user content.
• Scale: 500M DAU, 100M new posts/day, 10B feed reads/day.

Non-goals. No stories (24-hour content), no DM, no algorithmic ranking (v1 is chronological; ranking is layered on in design-news-feed).

Capacity estimation.

DAU & QPS. 500M DAU, average 20 feed refreshes/day = 10B feed reads/day = ~115K feed reads/sec average, peak ~500K/sec.

Posts. 100M posts/day, each ~500KB after compression (photo + thumbnail). 100M x 500KB = 50 TB/day of new media = ~18 PB/year. Stored on object storage (S3).

Feed metadata. Each post is ~1KB metadata (post_id, user_id, timestamp, caption hash, media_url). 100M x 1KB = 100GB/day of metadata = ~36 TB/year. Stored in sharded SQL.

Bandwidth. Photo upload: 100M x 500KB / 86400 = ~580 MB/s = 4.6 Gbps ingress. Feed read: each feed refresh returns ~20 posts x 100KB (compressed thumbnails) = 2MB; 500K/sec x 2MB = 1 GB/s = 8 Gbps egress from origin. With CDN serving photos, origin egress drops to ~10% (0.8 Gbps).

Feed cache. 500M users x their pre-computed feed (last 1000 post_ids) x 8 bytes = 4 GB. Trivially fits in Redis. But if we fan out writes, each new post by a user with 100 followers pushes the post_id into 100 feeds; with 100M posts/day x avg 200 followers = 20B fan-out writes/day = 230K writes/sec. Manageable.

APIs.

POST /v1/posts              (multipart: photo + caption + location)
GET  /v1/feed?cursor=...    -> { posts: [...], next_cursor }
GET  /v1/users/:id          -> profile + recent posts
POST /v1/users/:id/follow
DELETE /v1/users/:id/follow
POST /v1/posts/:id/like
POST /v1/posts/:id/comments

Photo upload uses presigned S3 URLs (valet keys) so the client uploads directly to object storage — the API server never proxies the photo bytes. After upload, the client calls POST /v1/posts with the S3 object key; the server kicks off an async transcoding pipeline and inserts the post metadata. The post appears in feeds only after transcoding completes (a few seconds).

Data model.

Users (sharded SQL):

users (id BIGINT PK, username, bio, avatar_url, created_at)

Posts (sharded SQL, sharded by user_id):

posts (id BIGINT PK, user_id, caption, media_url, location GEO,
       created_at, like_count, comment_count)
INDEX (user_id, created_at DESC)        -- for profile pages

Follows (sharded SQL or graph DB):

follows (follower_id, followee_id, created_at, PRIMARY KEY (follower_id, followee_id))
INDEX (followee_id)                     -- for 'who follows me'
INDEX (follower_id, created_at)         -- for 'who do I follow'

Feed cache (Redis sorted set per user):

key: feed:{user_id}
value: ZSET member=post_id, score=created_at   -- capped at top 1000

Photo storage (S3):

bucket: ig-media-prod
key:    {user_id}/{post_id}/{variant}    -- variant = orig|1080|720|480|thumb

Multiple resolutions are generated by the transcoding pipeline for adaptive delivery.

Deep dive: feed generation — fan-out on write vs read.

Fan-out on write (push model). When user P posts, the system looks up P's followers and pushes post_id into each follower's Redis feed (ZADD with score = post timestamp). Reads are O(1): just ZRANGE the user's feed.

Pros: feed reads are instant (single Redis call). Cons: write amplification. A user with 1M followers triggers 1M Redis writes per post. For Justin Bieber (100M followers), a single post = 100M Redis writes — that crushes the cluster.

Fan-out on read (pull model). When user U opens their feed, the system looks up the users U follows (say 500), fetches their recent posts from each user's post_id list (already indexed by (user_id, created_at)), merges by timestamp, returns the top 20.

Pros: no write amplification — posting is O(1). Cons: feed reads are expensive — 500 DB queries per refresh, even after caching.

Hybrid (our choice). Fan-out on write for normal users (most users have <500 followers); fan-out on read for celebrities (>10K followers). This caps the worst-case fan-out at 10K writes per post while keeping feed reads fast for 99% of users.

Concrete numbers. Average user has ~200 followers. 100M posts/day x 200 = 20B fan-out writes/day = 230K writes/sec. Top 0.1% of users (celebrities) average 10M followers — if we fanned them out, 1 post = 10M writes. By pulling them on read, we save ~10M writes per celebrity post and pay only ~500 extra DB queries per feed read for users who follow a celebrity (most users follow 1-5 celebrities).

Capping the feed. Each user's feed cache holds the last 1000 post_ids (ZSET, score = timestamp, ZREMRANGEBYRANK to evict beyond 1000). If the user scrolls past 1000, we fall through to the pull model for older posts.

Stale feed problem. With fan-out on write, if a user is unfollowed AFTER their post was pushed, the post lingers in the unfollower's feed. Mitigation: on unfollow, ZREM all of unfollowed-user's recent post_ids from the unfollower's feed. Cheap because the unfollower typically only has a few hundred of the unfollowed user's posts in their feed.

Deletion propagation. When a user deletes a post, we must remove it from every follower's feed. Mitigation: lazy deletion — mark the post deleted; feed reads filter deleted post_ids at hydration time. Periodic background job ZREMs stale deleted post_ids.

Bottlenecks and failure modes.

• Celebrity post spike. A celebrity with 100M followers posts; even at pull-on-read, the next minute's feed reads all hit the celebrity's post shard. Mitigation: cache the celebrity's recent posts in a separate hot-key cache; replicate to multiple shards.

• Redis feed cache failure. If Redis dies, every feed read falls through to the pull model — 500 DB queries per read at 500K reads/sec = 250M queries/sec. The DB dies. Mitigation: Redis cluster with replicas + circuit breaker that returns stale feeds with a 503-and-retry after 1s, rather than crushing the DB.

• Transcoding pipeline backlog. If transcode workers fall behind (viral video, instance failure), posts take minutes to appear in feeds. Mitigation: autoscale workers based on queue depth; show a placeholder thumbnail immediately so the post appears in feeds, then swap to full-res when transcode completes.

• S3 upload failures. If a presigned URL expires before the client finishes uploading a large video, the upload fails. Mitigation: use multipart upload with S3 (each part gets its own presigned URL; parts can retry independently).

• Hot post. A post goes viral — 1M likes in 5 minutes. The like counter row becomes a write hotspot. Mitigation: shard the counter across N rows (e.g. likes_0, likes_1, ... likes_9) and sum on read; or use a CRDT counter (Redis INCR is single-threaded — needs sharding for >10K writes/sec).

• Fan-out write storm. When a celebrity crosses 10K followers, our policy flips from push to pull. The transition must not retroactively remove their pushed posts from existing feeds. Mitigation: just stop pushing NEW posts; old pushed posts age out naturally.

Scaling strategy and trade-offs.

Database sharding. Shard posts and follows by user_id (so a user's posts and follow graph live on one shard). Cross-shard feed reads are reduced by the fan-out-on-write cache.

CDN for media. Photos are served from a CDN (Cloudflare, CloudFront). Cache hit rate >95% for popular posts; origin egress drops 10x. Cache key includes the variant resolution so the CDN serves the right size.

Multi-region. Deploy feed-service + Redis feed cache in 5+ regions. Posts DB is globally replicated (eventually consistent); writes go to a single primary region. Feed reads hit the local Redis cache.

Transcoding autoscaling. Scale transcode workers based on Kafka queue depth. Each worker pulls a job, transcodes (FFmpeg / libvips), uploads variants to S3, marks the post ready.

Trade-offs made explicit.

• We chose fan-out on write for normal users — gained instant feed reads, lost cheap celebrity posting (mitigated by hybrid).
• We chose chronological feed — gained predictability and simplicity, lost engagement optimization (algorithmic ranking adds 5-20% time-on-site but is a whole ML system of its own).
• We chose presigned S3 uploads — gained no photo bytes through API servers, lost the ability to do server-side validation before storage (mitigated by post-upload content moderation).
• We chose lazy deletion of feed entries — gained cheap delete, lost immediate consistency (a deleted post may appear briefly in feeds before hydration filters it).

Mental model. Think of Design Instagram as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Design Instagram mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.

Answer this without notes: When would you choose Design Instagram, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.