Cache Sizing
Cache sizing determines how much RAM to dedicate to the cache layer. The key insight is the Pareto distribution (80/20 rule): a small fraction of keys generates the majority of traffic, so caching just the hot set gives a high hit rate with modest memory. The standard methodology: estimate the working set (hot 1-5% of total data), size the cache to fit it (plus headroom), and verify the hit rate empirically. Over-sizing wastes RAM; under-sizing causes cache thrashing. Cache sizing is the fourth of the capacity-estimation skills and the trickiest because it depends on access distribution, not just totals.
Foundational.
How it works
Cache sizing answers: how much RAM should the cache layer have? The answer depends on a subtle property: access distribution. If every key is accessed equally, you must cache everything to get a high hit rate. If a small fraction of keys dominates traffic (the realistic case), you can cache just the hot set and achieve 90%+ hit rate with a fraction of the total data size.
This is the Pareto distribution (80/20 rule), generalized as the Zipfian distribution in caching theory. Real workloads almost always follow it:
- Twitter: 1% of users generate 50% of timeline reads.
- YouTube: 1% of videos generate 90% of views.
- E-commerce: 1% of products generate 80% of page views.
- News sites: a handful of headlines dominate all reads.
Because of this, the cache only needs to hold the hot set, not the entire dataset. The question becomes: what is the hot set, and how big is it?
The methodology:
- Estimate the total data size (from storage estimation).
- Estimate the hot set fraction (1-5% is a common assumption; 20% if very flat distribution).
- Compute the working set size = total × hot fraction × average entry size.
- Add headroom (1.5-2x) for fluctuations and growth.
- Verify empirically — deploy, measure hit rate, resize as needed.
The goal is not to cache everything; it is to cache enough to achieve the target hit rate (typically 90%+ for application caches, 99%+ for database buffer pools).
Working set — the size that actually matters.
The working set is the data being actively accessed — not the entire dataset, but the slice that is hot right now. This is what the cache must hold; everything else can stay on disk.
For a database, the working set is typically the hot indexes plus recently-written rows. For an application cache, it is the hot entities (active users, popular products, recent posts). For a CDN, it is the hot URLs.
The key insight: the working set is much smaller than the total data. A 1 TB database might have a 50 GB working set (5%). A 100 PB video catalog might have a 1 PB hot set (1%) — the trending and recently-uploaded content. The cache should be sized for the working set, not the total.
The cliff: if the cache size is below the working set, hit rate collapses. Below the cliff, the cache evicts hot entries to make room for other hot entries — thrashing. Every read becomes a miss, latency degrades to disk speed (10-100x slower). Above the cliff, hit rate is high and latency is uniform.
This is the most important cache-sizing principle: the cliff is sharp. A cache at 90% of the working set performs dramatically worse than a cache at 110% of the working set. There is no "partial credit" — you either fit the working set or you thrash. This is why cache sizing is an interview question and why under-provisioned caches are catastrophic in production.
Operationally: monitor the cache eviction rate. If evictions are high and hit rate is dropping, the cache is below the working set cliff — add RAM. If evictions are zero and hit rate is high, the cache is well-sized (or oversized). If the cache is using less than 50% of its allocated memory, it is over-provisioned — reclaim RAM.
The working set is not static. It grows during traffic peaks (more active users = more hot keys), shifts as content goes viral (a previously-cold key becomes hot), and grows over time as the system accumulates more active data. A cache sized exactly to the working set will thrash during peaks. The standard practice is to size the cache at 1.5-2x the measured working set, giving headroom for fluctuations and growth. Without headroom, a viral key or a peak traffic event pushes the cache below the cliff and latency spikes. With headroom, the system absorbs the fluctuation gracefully.
Estimating the hot set in interviews.
When you cannot measure the actual access distribution (as in an interview), assume the Pareto/Zipfian pattern and estimate from there:
- Estimate the total data size. (From storage estimation: total entries × average entry size.)
- Assume a hot fraction. For most workloads, 1-5% of keys generate 50-80% of traffic. For very skewed (celebrity accounts, viral content), use 1%. For flatter (B2B SaaS, internal tools), use 10-20%.
- Compute the working set. Total entries × hot fraction × average entry size.
- Add 1.5-2x headroom for fluctuations and growth.
- Translate to cache infrastructure. A 50 GB cache fits in one Redis instance; 200 GB needs a Redis cluster; 1 TB needs a sharded cluster.
Example (Twitter timeline cache):
- Total users: 200M, average timeline = 100 tweets × 500 B = 50 KB per timeline.
- Total if cached for everyone: 200M × 50 KB = 10 TB (way too much for RAM).
- Hot 1% of users (Pareto): 2M users × 50 KB = 100 GB working set.
- With 2x headroom: 200 GB cache.
- Infrastructure: a 3-node Redis cluster with ~70 GB each, or a 5-node cluster with 40 GB each for redundancy.
- Expected hit rate: 50-80% (the hot 1% generates 50% of traffic; cache absorbs that, misses fall through to DB).
This is the interview version: state your hot-fraction assumption, compute the working set, add headroom, and translate to infrastructure. The interviewer is testing whether you recognize that you do not cache everything — you cache the hot set.
Database buffer pool sizing — a special case.
The cache-sizing principles apply directly to database buffer pools:
- PostgreSQL
shared_buffers: the engine-managed cache. Rule of thumb: 25% of system RAM. The OS page cache handles the rest (cooperatively). - MySQL/InnoDB
innodb_buffer_pool_size: 50-75% of system RAM. InnoDB manages its own cache aggressively. - MS SQL Server
max server memory: ~75% of system RAM.
The target: fit the working set in the buffer pool. If the working set is 50 GB and the buffer pool is 64 GB, hit rate is 99%+ and the database feels instant. If the working set is 100 GB and the buffer pool is 64 GB, hit rate drops to ~70% and latency degrades 10-100x as reads hit disk.
This is why vertical scaling (more RAM) is so effective for databases up to a point: doubling the RAM can move the system from below the cliff to above it, with a 10-100x latency improvement. Beyond the working set, more RAM gives diminishing returns.
The diagnostic: check the buffer pool hit rate. PostgreSQL exposes this via pg_stat_database (blks_hit vs blks_read). MySQL via SHOW STATUS LIKE 'Innodb_buffer_pool%'. If hit rate is below 99%, the buffer pool is below the working set cliff — add RAM or shard. If hit rate is 99.99%+, the buffer pool is well-sized (or oversized).
The deeper lesson: cache sizing and vertical scaling are the same problem viewed from different angles. Sizing the cache is deciding how much RAM to give the cache layer; vertical scaling is deciding how much RAM to give the machine. The same cliff applies to both.
Your application has 100 GB of total data, with access following an 80/20 Pareto distribution (top 20% of keys generate 80% of traffic). What cache size gives approximately 80% hit rate, and what happens if you halve it?
Pick one answer.
Your Redis cache hit rate is 99.5%, eviction rate is zero, and the cache is using 30% of its allocated 100 GB. What does this tell you, and what should you do?
Pick one answer.
You have 200M users with timelines of 100 tweets each (50 KB per timeline). Total cached timelines = 10 TB. You assume the hot 1% of users generate 50% of traffic. What cache size do you propose, and what infrastructure?
Pick one answer.
Engineering mental model
Mental model. Think of Cache Sizing as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Cache Sizing mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Cache Sizing, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
value = cache.get(key)
if value is None:
value = load_from_origin(key)
cache.set(key, value, ttl=300)
return valueBack-of-the-envelope reasoning
Example: 100,000 requests/s at 90% cache hit rate means roughly 10,000 requests/s reach the origin. Raising the hit rate from 90% to 95% cuts origin traffic in half again.
Interactive thought experiment: Cache Sizing
Change the variables below and predict what breaks first in Cache Sizing. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Cache Sizing, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Cache Sizing. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Cache Sizing?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Cache Sizing, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Cache Sizing, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Cache Sizing, reason in this order: what data is hottest, where can it live closer to the caller, what makes it stale, and what happens on a miss or cache failure. A cache is an optimization boundary, not the source of truth.
Numerical sanity check
A useful first-order model is cache_load = request_rate × (1 - hit_rate). If traffic is 20,000 req/s and the hit rate is 90%, roughly 2,000 req/s still reaches the origin before considering misses caused by expiration or eviction.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
You have a read-heavy Cache Sizing path. Traffic doubles overnight. What metric would you inspect first, and what would convince you to add another cache layer?
Pick one answer.
What you gain, what you pay
- +Right-sized cache achieves 90%+ hit rate with modest RAM, by Pareto/Zipfian distribution.
- +Forces explicit identification of the hot set rather than caching blindly.
- +Avoids wasting RAM on cold data that is never read again.
- +Provides a measurable target (hit rate) to verify sizing empirically.
- +Works for both application caches and database buffer pools — same principle.
- −Requires estimating the access distribution, which varies by 10-100x depending on assumptions.
- −The cliff is sharp — under-sizing causes disproportionate thrashing.
- −Working set fluctuates — requires headroom and ongoing monitoring.
- −Real access patterns are not perfectly Pareto; some have multiple hot clusters.
- −Empirical verification requires production traffic; pre-production estimates are approximate.
How this breaks in production
- Cache below the working set cliff — hit rate collapses, latency degrades 10-100x.
- Cache over-provisioned — wasted RAM that could be used elsewhere.
- Working set grows beyond the cache during peak — sudden thrashing.
- Assuming uniform access distribution when reality is Pareto — oversizing 10-100x.
- Assuming Pareto when reality is flat (B2B SaaS with uniform users) — undersizing.
- Not monitoring eviction rate and hit rate — sizing decisions drift over time.
Don't fall into these traps
- •Caching the entire dataset — wastes RAM; the hot set is what matters.
- •Assuming uniform access distribution — reality is almost always Pareto/Zipfian.
- •Sizing without headroom — works in steady state, thrashes at peak.
- •Not monitoring hit rate — flying blind on whether the cache is correctly sized.
- •Treating cache sizing as a one-time decision — the working set grows and shifts.
- •Forgetting that database buffer pools are caches too — same sizing principle applies.
Real systems using this
How real systems implement this
- Netflix EVCache — Netflix's EVCache (a Memcached-based cluster) is sized to hold the hot set of user-state and content-metadata keys. They have publicly described how Pareto/Zipfian distribution governs their cache sizing — a small fraction of titles generates the majority of views, so the cache fits the hot set, not the full catalog.
- PostgreSQL `shared_buffers` default (25% of RAM) — PostgreSQL's documented recommendation of 25% of system RAM for `shared_buffers` is itself a Pareto-aware cache-sizing rule — it assumes the working set fits in ~25% of RAM, with the OS page cache handling the rest cooperatively. The 25% rule exists because most databases' working set is much smaller than the total data.
- Redis cluster sizing at Twitter / Stripe / GitHub — These companies size their Redis clusters based on measured hit rates and eviction rates, not on total data size. They start with an estimated hot set, deploy, monitor hit rate, and resize. This empirical refinement is the standard production cache-sizing methodology.
- CDN edge cache sizing — CDN POPs are sized to hold the hot set of URLs for their region — typically a small fraction of the global catalog. A 1 PB video catalog might have a 10 GB hot set at each POP (the locally-trending content), achieving 90%+ hit rate with modest per-POP storage.
Practice saying it out loud
- Q1Your system has 200M users with 50 KB timelines. What cache size do you propose, and what infrastructure?
- Q2Your Redis cache hit rate is 60%. What does this tell you, and what do you do?
- Q3Explain the Pareto distribution's effect on cache sizing. Why don't we cache everything?
- Q4Your cache hit rate is 99% with zero evictions but only 30% memory utilization. What does this mean, and what do you do?
- Q5How do you size a database buffer pool? What is the cliff?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Cache Aside (Lazy Loading)