Design WhatsApp

What are we designing? WhatsApp — a mobile-first real-time chat application. Users send text/voice messages and media to individuals or groups. Messages must be delivered instantly when the recipient is online, queued and pushed when offline. The system must show delivery status (single tick = sent, double tick = delivered, blue tick = read), support group chats of up to 1024 members, and do all this at WhatsApp's legendary efficiency (a small engineering team serving 2B+ users).

The defining challenges: always-on connections (each user has a long-lived WebSocket to a chat server), message ordering (messages in a conversation must appear in the order they were sent, even across failures), and group fan-out (one message to a 1024-person group = 1024 deliveries).

Functional requirements.

• Send a text/voice/media message to a 1:1 or group chat.
• Receive messages in real-time when online.
• Receive push notifications when offline.
• Delivery and read receipts (single / double / blue ticks).
• Last-seen / online presence.
• Message history (last 30 days on server, full history on device).
• Groups up to 1024 members.

Non-functional requirements.

• Send-to-deliver latency: p99 < 1 s for online recipients.
• Connection count: 50M+ concurrent WebSocket connections per region.
• Availability: 99.9% (chat is high-stakes; outages are noticed).
• Durability: no message loss, even if the recipient's chat server crashes mid-delivery.
• End-to-end encryption: messages are encrypted on-device; the server never sees plaintext.
• Scale: 2B users, 100B messages/day.

Non-goals. No voice/video calls (separate media-relay system), no channels/broadcasts.

Capacity estimation.

Messages. 100B messages/day. Each ~200 bytes encrypted. 100B x 200 B = 20 GB/day of message text — tiny. The real cost is connections and metadata, not storage.

Concurrent connections. 2B users, ~30% online at peak = 600M concurrent WebSockets. A single chat-server host handles ~1M connections (Go + epoll), so we need ~600 chat servers globally. Realistic WhatsApp numbers are ~600 servers.

Connection bandwidth. Each WebSocket idles at ~50 bytes/sec of keepalive traffic. 600M x 50 B/s = 30 GB/s = 240 Mbps of pure keepalive. Message traffic adds ~5x peak.

Group fan-out. Average group has 10 members. 10% of messages go to groups of 100+; 1% go to groups of 1000+. The long tail of large groups dominates fan-out load. A single message to a 1024-person group = 1024 fan-out writes.

Push notifications. ~70% of messages go to offline recipients and trigger push via APNs/FCM. That's 70B push notifications/day = 800K push/sec peak. APNs/FCM rate limit is ~10K/sec per app, so we batch and shard across multiple FCM sender IDs.

Storage. Each message row is ~500 bytes (encrypted payload + metadata). 100B messages x 30 days retention = 3T messages x 500 B = 1.5 PB hot. Sharded Cassandra.

APIs. WhatsApp uses a custom binary protocol over WebSocket (NOT REST) to minimize byte overhead on mobile networks. Conceptually:

WS /v1/connect    (long-lived; auth token in handshake)
  -> frame: SEND  { conversation_id, encrypted_payload, client_msg_id }
  <- frame: ACK   { server_msg_id, ts }
  <- frame: RECV  { conversation_id, sender_id, encrypted_payload, server_msg_id }
  -> frame: DELIVERED  { server_msg_id }
  -> frame: READ       { server_msg_id }

REST /v1/media/upload_url      (presigned URL for media)
REST /v1/media/{media_id}      (fetch media)

Every frame is binary, length-prefixed, with a small header (sender_id, conversation_id, msg_id, ts). The client_msg_id is the client's UUID for idempotency — if the client retries after a network blip, the server deduplicates.

Data model.

Messages (Cassandra, partitioned by conversation_id, sorted by server_msg_id):

messages (
  conversation_id  TIMEUUID,
  server_msg_id    TIMEUUID,     -- Snowflake, sortable
  sender_id        BIGINT,
  encrypted_payload BLOB,        -- E2E encrypted; server never decrypts
  delivered_to     SET<BIGINT>,  -- which recipients have it
  read_by          SET<BIGINT>,
  ts               TIMESTAMP,
  PRIMARY KEY ((conversation_id), server_msg_id)
)

Conversations / groups (sharded SQL):

conversations (id BIGINT PK, type ENUM('1to1','group'), created_at)
group_members (conversation_id, user_id, role, joined_at,
               PRIMARY KEY (conversation_id, user_id))

Connection registry (Redis, mapping user_id -> chat-server-id):

key: presence:{user_id}
value: { chat_server_id, last_seen_ts }
TTL:   60s    -- refreshed by WebSocket keepalive

Offline message queue (per-user Kafka or Redis LIST):

key: pending:{user_id}
value: LIST of server_msg_ids awaiting delivery

On reconnect, the chat server drains this queue and pushes all messages.

Deep dive: message delivery, ordering, and the connection fabric.

Connection fabric. Each chat server runs an event loop (Go or Erlang/Elixir — WhatsApp famously uses Erlang) holding 1M WebSocket connections. Each connection is a lightweight process/goroutine. When a frame arrives, the server parses it and either: (a) ACKs to sender, (b) looks up recipient in Presence Redis, (c) forwards to the recipient's chat server via an internal RPC.

Message ordering. Messages in a conversation are ordered by server_msg_id, which is a Snowflake TIMEUUID. Cassandra stores them in this order, so a recipient fetching recent messages gets them in send order. For groups, all members see the same order because they all read from the same partition.

Idempotency. Each send includes a client_msg_id (UUID generated on the device). If the client retries after a network blip (didn't receive the server ACK), the chat server checks a short-TTL cache of recent client_msg_ids; if seen, it returns the same server_msg_id. This prevents duplicate messages on retries.

Delivery guarantees. The server's contract: once the server ACKs (single tick), the message is durable in the message store and will be delivered to the recipient eventually. If the recipient is online, immediately. If offline, on next reconnect (plus a push notification in the meantime).

Read receipts. When B opens the message, B's client sends a READ frame with the server_msg_id. B's chat server updates read_by in the messages table and forwards the read receipt to A's chat server, which pushes to A (turning the double tick blue). Read receipts can be disabled per-user privacy settings.

Group fan-out optimization. A naive group send to 1024 members looks up presence for each (1024 Redis lookups) and forwards to each chat server. We optimize: (1) batch the presence lookup (one Redis MGET for all 1024 user_ids), (2) group members by their chat_server_id, (3) send one batched RPC per distinct chat server containing all messages for its local members. This collapses 1024 forwards into ~10 (number of distinct chat servers).

End-to-end encryption. Each 1:1 conversation has a shared key derived via X3DH (Extended Triple Diffie-Hellman) on first contact. The server NEVER sees plaintext; it forwards opaque encrypted blobs. Group chats use a Sender Key framework so each sender has one key per group, not pairwise keys. Receipts and presence are NOT encrypted (the server needs them to route).

Bottlenecks and failure modes.

• Chat server failure mid-delivery. If the chat server hosting A's connection crashes after writing the message but before forwarding to B, B never receives it. Mitigation: a background 'sweeper' job compares the message store against delivered_to and re-delivers anything missing.

• Connection rebalancing. When a chat server dies, its 1M connections all reconnect to other servers within seconds. Mitigation: clients use exponential backoff with jitter to avoid a thundering herd; DNS / load balancer redistributes.

• Hot conversation. A 1024-person group with active chatters fans out constantly. Mitigation: rate-limit group sends (e.g. 1 msg/sec per sender); batch fan-out per chat server.

• Push notification rate limits. APNs and FCM rate-limit per-app. Mitigation: batch notifications per user (one push per 5 seconds of unread messages); use multiple FCM sender IDs for sharding.

• Presence Redis failure. Without presence, sends can't route to online users. Mitigation: Redis cluster with replicas; fail-open by treating the user as offline (queue + push).

• Cassandra hot partition. A 1024-person group's conversation_id partition receives all writes for that group; an active group can saturate one Cassandra node. Mitigation: split group messages across sub-partitions (e.g. conversation_id + day).

• Backpressure on offline users. A user who has been offline for a month has thousands of pending messages; on reconnect, draining them can saturate their device. Mitigation: cap the pending queue to last 1000 messages; pull older ones via pagination.

• Read-receipt storms. A user opens a 1024-person group chat after a week offline, generating 1024 read receipts. Mitigation: batch read receipts (one READ frame with the highest server_msg_id read).

Scaling strategy and trade-offs.

Connection scaling. Add chat servers horizontally. Each holds 1M WebSockets. 600M concurrent = 600 chat servers. Stateful per-connection; rebalancing handled by clients reconnecting with backoff.

Multi-region. Deploy chat servers in 10+ regions. Each user connects to the nearest region. Cross-region routing for messages between users in different regions goes through a backbone RPC.

Group scaling. For very large groups (1000+), use a tree-based fan-out: the sender's chat server forwards to N 'hub' chat servers, each of which fans out to their local members. This keeps fan-out latency logarithmic.

Message store scaling. Cassandra sharded by conversation_id. Each conversation's messages live on one partition (so reads are cheap). For very active groups, sub-partition by day.

Push notification scaling. Maintain multiple FCM sender IDs (each rate-limited separately); hash user_id -> sender_id to distribute load.

Trade-offs made explicit.

• We chose per-connection stateful chat servers — gained low-latency delivery (no Redis lookup on the hot path), lost easy rebalancing (mitigated by client backoff).
• We chose 30-day server retention — gained smaller storage, lost long-term server-side history (device is the source of truth).
• We chose E2E encryption — gained user privacy, lost server-side search and moderation (must be done on-device).
• We chose Snowflake TIMEUUID for ordering — gained no-coordinator uniqueness, lost total ordering across nodes in the same millisecond (rarely matters).
• We chose push notifications for offline delivery — gained reach users anywhere, lost dependency on APNs/FCM availability (out of our control).

Mental model. Think of Design WhatsApp as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Design WhatsApp mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.

Answer this without notes: When would you choose Design WhatsApp, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.