Design File Storage System

What are we designing? A Dropbox-style file storage and sync system. A user has files on their laptop; they want the same files on their phone, their desktop, and shared with collaborators. When a file changes on one device, the others should converge — usually within seconds, even on flaky networks, even when the same file was edited offline on two devices.

The defining challenges are (1) efficient transfer — you do not want to re-upload a 10 GB file when 1 KB changes, (2) efficient storage — if 10M users have the same 'install.pdf', you want one copy on disk, not 10M, and (3) conflict resolution — when two devices edit the same file offline, what does the merged result look like?

Functional requirements.

• A client (desktop / mobile / web) can upload, download, list, and delete files.
• Files sync across all of a user's devices within seconds of an edit.
• Files can be shared with other users (read or read-write).
• Large file edits transfer only the changed bytes (delta sync).
• File history is preserved (a user can restore a deleted file or an old version).
• Offline edits are supported; conflicts are surfaced to the user.

Non-functional requirements.

• Upload/download latency: small file < 1 s end-to-end; 1 GB file with 10% changed transfers < 30 s on residential broadband.
• Storage efficiency: dedup ratio > 30% across user base (realistic for typical corpora).
• Durability: 11 nines (objects replicated 3x across AZs, bit-rot detected via checksums).
• Availability: 99.9% (sync is not life-critical; brief outages are tolerable).
• Scale: 500M users, 100B files, exabyte-scale storage.

Non-goals. No real-time collaborative editing (that's Google Docs / CRDT territory), no end-to-end encryption by default (v1 is server-visible; E2EE is a separate, much harder problem because dedup across users breaks).

Capacity estimation.

Storage. 500M users x avg 20 GB stored = 10 EB raw. With 50% dedup (cross-user + intra-user + version-similarity), 5 EB actual. Replicated 3x = 15 EB on disk. Object storage on S3-class infrastructure.

Throughput. Avg user generates 1 MB/day of new chunks = 500M users x 1 MB = 500 GB/day ingress = ~6 MB/s — trivial. Peak: mornings 9-10 AM local time across each timezone = ~50x = 300 MB/s aggregate.

Metadata. 100B files x 1KB metadata = 100 TB. Chunk index: ~1T chunks (avg file = 10 chunks of 1 MB) x 64 bytes (hash + size + refcount) = 64 GB — fits in a sharded key-value store (Redis cluster + persistent backend).

Sync notifications. Every file change triggers a notification to all devices of all users who have access. Avg 2 devices per user + avg 1 share = 3 notifications per change. 100B files x 1 change/yr = 300B notifications/yr = ~10K/sec. Pushed via WebSocket / long-poll.

APIs.

# Client opens a file for upload (resumable, chunked)
POST /v1/files/upload_session
  -> { session_id, chunk_size: 4MB }

# Append chunks (each chunk has its own presigned URL)
PUT  <presigned S3 URL>   chunk_bytes
POST /v1/files/upload_session/:id/commit
  { chunks: [{hash, size, sequence}], path, parent_rev }
  -> { file_id, rev }

# Pull changes
GET  /v1/files/sync?cursor=...
  -> { changes: [...], next_cursor }

# Download a file (returns chunk URLs)
GET  /v1/files/:id
  -> { chunks: [{hash, url, size, sequence}] }

# Subscribe to push notifications
WS   /v1/notifications   -> server-pushed change events

The chunked upload with presigned S3 URLs is the same valet-key pattern as Instagram. The parent_rev in the commit call is what enables conflict detection: if the server's current rev != parent_rev, the server knows the client was operating on a stale version.

Data model.

Files (sharded SQL, sharded by owner_id):

files (
  id PK, owner_id, parent_id, name, type,
  rev, parent_rev, deleted, mtime, size
  INDEX (parent_id)                       -- list dir
  INDEX (owner_id, mtime)                 -- sync cursor
)

File versions (history):

file_versions (
  file_id, rev, created_at, chunks_blob, deleted, PRIMARY KEY (file_id, rev)
)

Chunks (content-addressed, global):

chunks (
  hash PK, size, refcount, storage_url, created_at
  INDEX (refcount)                        -- for GC sweep
)

Sync cursors per device (so a client can resume after offline):

sync_cursors (user_id, device_id, cursor PK, last_seen_at)

Block storage (S3):

bucket: nocap-blocks
key:    {first_2_hash_chars}/{hash}      -- sharded prefix for S3 partition distribution

Critical invariant: the chunks table uses content-addressed hashes. The same chunk uploaded by two different users lands on the same hash key, the refcount is incremented, and only one copy is stored in S3. This is the dedup win.

Deep dive: chunking and deduplication.

Fixed-size chunking. Split a file into fixed 4MB blocks. Simple. Problem: if a user inserts one byte at the start of a 1GB file, every subsequent chunk shifts and is 'new' — 250 chunks re-uploaded.

Content-defined chunking (CDC). Use a rolling hash (Rabin-Karp or Buzhash) to find chunk boundaries based on content, not byte offset. The boundary is wherever the rolling hash hits a specific value (e.g. hash & 0xFFFF == 0 — average 64KB chunk). Insertions only shift the chunk containing the insertion; all subsequent chunks realign to the same content-defined boundaries. This is what restic, borg, and ZFS dedup use.

Why dedup is a big deal. Real Dropbox measurements show 30-50% storage savings from (a) cross-user dedup (everyone has the same installers, memes, school handouts), (b) intra-user dedup (a user copies a 1GB folder — second copy is free), and (c) version dedup (saving a 10MB PDF after editing one line = 99% of chunks unchanged).

Reference counting and GC. Each chunk has a refcount = number of file_versions referencing it. When a file is deleted, refcounts on its chunks are decremented; a chunk is eligible for S3 deletion when refcount hits 0. The GC sweep runs nightly, deleting unreferenced chunks (with a grace period of 7 days for safety against in-flight commits).

Encryption trade-off. Dedup across users requires the server to see chunk hashes. If we client-side encrypt with a per-user key, two users with the same file produce different ciphertext and different hashes — dedup breaks. Dropbox chooses server-side dedup; Spider Oak and Tarsnap choose client-side encryption and accept worse dedup. This is a fundamental tension: dedup-vs-privacy.

Deep dive: sync protocol and conflict resolution.

Push protocol. Each file has a monotonically increasing revision number rev. When a client commits a change, it includes parent_rev (the rev it had when it started editing). The server: if parent_rev == server.rev, fast-forward and return new rev. If parent_rev != server.rev, the file changed on another device while we were editing — conflict.

Last-writer-wins + version history (Dropbox's actual approach). On conflict, the server commits the new version as the new head and renames the previously-current version to notes (conflicted copy, laptop's edit).md. Both versions are visible to the user, who manually merges. This is simple, predictable, and never loses data — but it pushes merge burden onto the user.

Operational Transform / CRDT (Google Docs approach). For rich-text collaborative editing, OT or CRDTs merge concurrent edits automatically. This is much harder to implement correctly (OT is famously tricky — Google spent years getting it right) and is overkill for files. Dropbox's deliberate choice: keep the model simple, surface conflicts, let humans merge.

Delta sync. On pull, the client sends its current chunk list for the file; the server responds with the set of chunks it doesn't have. The client downloads only new chunks via presigned S3 URLs and assembles the file locally. A 1KB edit to a 10GB file transfers 1 chunk (~4MB) — not 10GB.

Selective sync. Mobile clients don't want every file. Selective sync lets the user mark folders as 'online-only' — the metadata is synced, but chunks are not downloaded until the file is opened. This is implemented as a per-device flag on the file metadata; the client lazily fetches chunks on access.

Bottlenecks and failure modes.

• Hash collision. Two different chunks hash to the same SHA-256. Probability ~10^-60 — negligible, but if it happens, the second user silently corrupts the first. Mitigation: also store chunk size; on hash collision, treat as distinct (store under hash+size). For SHA-256 this is essentially never triggered.

• Metadata DB hotspot. A user with 1M files in one folder blows up directory listing. Mitigation: cap files per folder (Dropbox: 1M); shard by owner_id; cache listings.

• Notification storm. A shared folder with 1000 users gets 100 changes/min = 1M notifications/min. Mitigation: coalesce notifications within a 5s window; per-device rate limit.

• Upload of a chunk fails midway. Mitigation: presigned S3 multipart upload — each part retries independently; the commit only succeeds when all parts are uploaded.

• Client offline during commit. The commit is queued locally and retried when online. Mitigation: client stores a local pending-commits queue; idempotent commit (server dedups by client-generated commit_id).

• Garbage collection race. A chunk's refcount hits 0 and GC deletes it while a client is mid-upload-commit referencing it. Mitigation: 7-day grace period; refcount increments happen BEFORE upload completes (pre-reservation).

• Conflict floods. Two users editing a shared file every second. Mitigation: rate-limit commits per file; surface a 'too many edits, switch to live collaboration' message.

Scaling strategy and trade-offs.

Metadata sharding. Shard files by owner_id (a user's files all live on one shard — enables fast directory listing and sync cursor advances). 100B files / 1M users avg = 100K files per user; manageable per-shard.

Block storage. S3 with hash-prefix sharding ({hash[0:2]}/{hash}) to spread hot keys across S3 partition servers. S3 already does internal replication; we don't need a second replication layer.

Multi-region. Metadata DB is multi-region replicated (eventually consistent; conflicts are rare because a user typically edits from one region at a time). Block storage is global (S3 cross-region replication).

Client-side caching. The client keeps a local LMDB cache of chunk hashes it already has, so the server-side 'what chunks do you need' diff is computed client-side, not server-side. Saves server CPU and bandwidth.

Trade-offs made explicit.

• We chose chunk-level content-addressed dedup — gained 30-50% storage savings, lost the ability to do client-side per-user encryption (the dedup-vs-privacy tension).
• We chose last-writer-wins + conflicted copies — gained simplicity and predictability, lost automatic concurrent-edit merging (would require OT/CRDT, a separate engineering effort).
• We chose CDC over fixed-size chunking — gained better delta sync on insertions, lost some implementation complexity (rolling hash, edge cases).
• We chose 4MB chunk size — gained balance between per-chunk overhead and dedup granularity, lost some dedup efficiency vs 64KB chunks (but 64KB means 16x more chunk rows in the DB).
• We chose server-side rendering of metadata only — gained small payloads, lost the ability to do server-side search of file contents (would require a separate search index — see design-search-system).

Mental model. Think of Design File Storage System as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Design File Storage System mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Example: 5M writes/day × 1 KB/row ≈ 5 GB/day of logical data. Add indexes, replication, backups and growth headroom before sizing a real store.

Answer this without notes: When would you choose Design File Storage System, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.