Storage Estimation
Storage estimation computes how much disk the system will consume over a retention period. The base formula: writes/day × payload size × retention × replication factor = total storage. Text and structured data are small (KB-MB per record); media dominates (MB-GB per object). The estimate determines whether the data fits in a database, requires object storage, or requires sharding — and how much it will cost. Storage estimation is the second of the four capacity-estimation skills.
Foundational.
How it works
Storage estimation computes the total disk the system will consume, derived from the write rate, the payload size, the retention period, and the replication factor. The base formula:
Storage = writes/day × payload_size × retention_days × replication_factorThe replication factor (typically 3 for durable storage) accounts for replicas, indexes, and backups. The retention period determines whether you store data for 30 days, 1 year, 5 years, or forever — this dominates the total.
The single most important refinement: separate text and media. Text records are small (a tweet is ~500 B, a WhatsApp message is ~200 B, a database row is ~1 KB). Media objects are enormous (a photo is ~1-5 MB, a video is ~50 MB-1 GB). At the same DAU and write rate, a media-heavy system uses 1000-10000x more storage than a text-heavy system. Estimating them together produces wildly wrong numbers.
Example (Twitter):
- Text: 1B tweets/day × 500 B = 500 GB/day → 1.8 PB over 5 years (× 3 replicas = 5.5 PB)
- Media: 1B tweets × 20% have media × 1 MB avg = 200 TB/day → 730 PB over 5 years (× 3 replicas = 2.2 EB)
- Text fits in a database; media requires object storage. Media dominates by 400x.
This separation is also the architecture decision: text → relational DB or KV store; media → object storage (S3) with CDN delivery. The estimate tells you which components you need.
Where data lives — the architecture decision.
Storage estimation tells you which storage layer each piece of data requires:
-
In-memory (Redis/Memcached): the hot working set. Bounded by RAM (GB-TB). Sub-ms latency. Expensive (~$5/GB-month). Use for: hot cache, sessions, rate-limit counters.
-
NVMe SSD on a DB instance: structured data, indexes, transactions. Bounded by instance size (GB-TB). 1-5 ms latency. ~$0.10-0.50/GB-month. Use for: relational data, indexes, anything requiring queries.
-
Block storage (EBS, Persistent Disk): attached to a VM. GB-TB scale. ~$0.10/GB-month. Use for: database data files, application state.
-
Object storage (S3, GCS, Azure Blob): unstructured blobs, virtually unlimited scale. ~$0.023/GB-month (standard). 10-100 ms latency. Use for: media, backups, logs, large blobs.
-
Archive storage (S3 Glacier, Coldline): long-term retention, rare access. ~$0.004/GB-month. Hours to retrieve. Use for: compliance archives, old backups.
The decision rule: hot structured data → DB on SSD; hot unstructured data → object storage; cold data → archive. The estimate drives this directly:
- Total storage < 1 TB → fits on one DB instance, no sharding.
- Total storage 1-10 TB → DB with read replicas, or sharded.
- Total storage > 10 TB → object storage for unstructured; sharded DB for structured.
- Total storage > 1 PB → definitely object storage + CDN; probably multi-region.
Cost is the killer constraint. At 1 PB, S3 standard is ~$23k/month; at 100 PB, $2.3M/month. Storage tiers (Infrequent Access, Glacier) reduce this by 5-10x for cold data — that is why every media-heavy system tiers aggressively.
Production storage is rarely stored once. Durability requires replication: typically 3x for object storage (S3 stores 3 copies across facilities), 2-3x for database replicas (primary + 1-2 replicas for HA), plus backups (often another 1-2x). Indexes add ~30% overhead on top of raw data. A naive estimate of "1 PB of user data" becomes 3-5 PB once you account for replication, indexes, and backups. Always state your replication factor explicitly and multiply by it. The standard assumption for estimation: × 3 for replicas + × 1.3 for indexes/backups = ~4x the raw data.
Retention period — the storage amplifier.
The retention period has the largest single effect on total storage. The same per-day storage, retained for different periods, gives vastly different totals:
- 30 days: × 30
- 1 year: × 365
- 5 years: × 1825
- Forever: unbounded (grows linearly with time)
This is why retention policies matter operationally. Storing every tweet forever is 100x more storage than storing them for 2 weeks. Most real systems tier aggressively: hot data in SSD/object storage, warm data in infrequent-access tiers, cold data in archive tiers.
Example lifecycle (typical for a media system):
- 0-30 days: hot tier (S3 Standard) — fast access, higher cost.
- 30-90 days: warm tier (S3 IA) — 50% cheaper, slightly slower.
- 90 days+: cold tier (Glacier) — 80% cheaper, hours to retrieve.
A 5-year retention policy that tiers this way might cost 1/5th of putting everything in standard storage. The estimate tells you the upper bound; tiering tells you the achievable cost.
The interview version: state your retention period explicitly ("assume 5 years"), compute the upper bound, then mention tiering as a cost optimization. Interviewers love seeing the tiered lifecycle awareness — it shows operational maturity.
The storage estimation interview step-by-step.
- State your inputs. DAU, writes/user/day, average payload size (split text vs media), retention period, replication factor.
- Compute per-day storage. Writes/day × payload size, separately for text and media.
- Compute total over retention. Per-day × retention days × replication factor.
- Identify the dominant component. Usually media; sometimes text (rare).
- Translate to storage architecture. Text → DB; media → object storage; cold → archive.
- Estimate cost. Total GB × $/GB-month, possibly tiered.
For Twitter:
- Text: 1B tweets/day × 500 B × 1825 × 3 ≈ 2.7 PB
- Media: 200 TB/day × 1825 × 3 ≈ 1.1 EB
- Text → relational DB with replicas; media → S3 with CDN.
- Cost: ~$25k/month for text (DB storage), ~$2.5M/month for media (S3 standard). Tiering reduces media cost 5x.
The whole exercise takes 1-2 minutes. The numbers do not need to be exact; they need to drive the architecture decisions: "media requires object storage + CDN; text fits in a DB." The interviewer is testing whether you recognize that media and text have very different storage requirements and that media dominates.
You estimate a Twitter-like system: 1B tweets/day, 500 B per tweet, 20% with 1 MB media, 5-year retention, 3x replication. What is the total storage, and what is the architecture implication?
Pick one answer.
Why does almost every media-heavy production system use storage tiering (S3 Standard → S3 IA → Glacier)?
Pick one answer.
You estimate 100 GB of total storage for an internal HR application with 10k users. What architecture do you propose?
Pick one answer.
Engineering mental model
Mental model. Think of Storage Estimation as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Storage Estimation mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Storage Estimation, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = storage_estimation(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Example: 5M writes/day × 1 KB/row ≈ 5 GB/day of logical data. Add indexes, replication, backups and growth headroom before sizing a real store.
Interactive thought experiment: Storage Estimation
Change the variables below and predict what breaks first in Storage Estimation. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Storage Estimation, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Storage Estimation. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Storage Estimation?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Storage Estimation, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Storage Estimation, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Storage Estimation: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Storage Estimation. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Tells you whether data fits in a database, requires object storage, or requires sharding.
- +Drives the storage architecture decision (DB vs object storage vs archive).
- +Enables cost estimation — storage is the largest cloud cost for media-heavy systems.
- +Separates text (small) from media (huge), making the architecture clean.
- +Forces explicit retention policy — what gets stored for how long.
- −Requires estimating payload sizes, which vary by 10-1000x depending on assumptions.
- −Forgetting the replication factor underestimates by 3-5x.
- −Mixing text and media produces wildly wrong numbers — they differ by 1000x.
- −Cost estimates are sensitive to storage tier choices — must model tiering.
- −Captures steady-state, not growth — must project over time.
How this breaks in production
- Mixing text and media — off by 1000x.
- Forgetting replication factor — off by 3-5x.
- Forgetting indexes and backups — off by 30-100%.
- Sizing storage without considering cost — design is unaffordable.
- Not modeling storage tiers — overpays 5-10x for cold data.
- Assuming infinite retention — total storage grows unboundedly.
Don't fall into these traps
- •Treating all data as the same size — media dominates by 1000x.
- •Forgetting to multiply by replication factor (×3) and indexes (×1.3).
- •Sizing a relational database to hold media — they are object-storage workloads.
- •Not modeling storage tiering for cold data.
- •Estimating storage without retention period — "how much for 1 day" is useless.
- •Forgetting cost entirely — at 1 PB+, storage is the dominant cloud bill.
Real systems using this
How real systems implement this
- Netflix Open Connect content library — Netflix stores ~2 PB of video content in object storage, distributed to ISP-embedded Open Connect Appliances (OCAs). Storage tiering is essential: recent popular content on OCAs at ISPs, older content in regional S3, very old content in Glacier.
- AWS S3 storage classes — S3 offers Standard (~$0.023/GB-mo), Infrequent Access (~$0.0125), Glacier (~$0.004), Glacier Deep Archive (~$0.00099). Tiering across these is the canonical cost-optimization pattern for storage-heavy systems.
- Twitter / X storage architecture — Twitter stores tweet text in Manhattan (their internal KV store, sharded). Media (photos, videos) is in S3 with a CDN. The separation is exactly what the storage-estimation exercise prescribes.
- Dropbox storage architecture — Dropbox stores user files (largely media) in S3 and their own Magic Pocket infrastructure. Files are deduplicated and chunked — only changed chunks are stored. The estimate tells you the upper bound; dedup reduces the actual storage by ~30%.
Practice saying it out loud
- Q1Estimate the storage required for 5 years of YouTube uploads.
- Q2Estimate the storage for WhatsApp messages (text-only) over 1 year. What architecture?
- Q3Twitter with 200M DAU — what is the total storage over 5 years, split text and media?
- Q4A log retention system stores 1 TB/day of logs. How much storage for 7-year compliance? What tiers?
- Q5Your storage estimate gives 100 PB. What architecture and what approximate monthly cost?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Bandwidth Estimation