Disaster Recovery
Disaster recovery (DR) is the plan for when an entire site, region, or service is lost — not a single failed instance. It's defined by two metrics: RTO (recovery time objective — how long until we're back up) and RPO (recovery point objective — how much data we can lose). DR strategies range from cold standby (cheap, slow) to hot multi-region (expensive, fast). The right choice depends on the cost of downtime vs the cost of preparedness.
How it works
Disaster recovery is the discipline of preparing for the worst: an entire data center or region going offline. It's distinct from high availability (HA), which handles individual component failures within a region. DR handles the scenario where the whole region is gone.
Two metrics define every DR plan:
- RTO (Recovery Time Objective): the maximum time between the disaster and the system being back up. RTO=1 hour means you tolerate at most 1 hour of downtime.
- RPO (Recovery Point Objective): the maximum amount of data you can lose. RPO=5 minutes means you tolerate losing at most 5 minutes of writes.
These metrics drive every decision. RTO=0 means hot multi-region active-active (very expensive). RPO=0 means synchronous cross-region replication (slow writes). Most systems pick tolerable values and design to those — not to zero, because zero is too expensive.
RTO and RPO are about expectations, not technology. You decide them based on business impact: how much does an hour of downtime cost? How much does an hour of lost data cost? Then you pick a strategy that meets them.
For an internal HR tool used during business hours, RTO=8 hours, RPO=24 hours might be fine — daily backups to S3, manual restore.
For a SaaS product with a 99.9% SLO, RTO=1 hour, RPO=5 minutes — warm standby with continuous replication.
For a payments system, RTO=5 minutes, RPO=0 — hot standby with synchronous cross-region replication.
The trap is over-engineering. A startup spending 50% of its cloud bill on multi-region active-active to chase 99.999% availability is wasting money — its customers would tolerate 99.9% fine. Match the DR strategy to the business need, not to a vanity SLO.
Backups are the foundation of every DR plan, even the most sophisticated. They're the last line of defense when replication fails (and it does). Key principles:
- 3-2-1 rule: 3 copies of data, on 2 different media, 1 off-site. In cloud terms: primary, replica, and a backup in a different region (or different cloud).
- Test restores. An untested backup is a hope, not a backup. Most DR plans fail because the restore step has never been rehearsed.
- Immutable backups. Ransomware and malicious insiders delete backups too. Use object lock / WORM storage so backups can't be deleted during the retention window.
- Versioned. Keep multiple points in time, not just the latest. Sometimes the latest backup is corrupted.
For databases, PITR (point-in-time recovery) lets you restore to any second within the retention window — useful for undoing a bad migration.
The dirty secret of disaster recovery is that most plans are theoretical. Runbooks are out of date, DNS changes haven't been tried, the DR database hasn't been promoted in months, and the credentials are stored in the region that's now down. Netflix's Chaos Monkey and the broader practice of Game Days exist precisely because untested DR plans fail when needed. Schedule a DR drill every quarter: simulate a region failure, run the failover, measure RTO and RPO against the targets, and fix what broke. If you can't run a drill in production, at least run one in staging.
What disasters should you plan for? Real-world examples:
- Region outages: AWS us-east-1 has had multi-hour outages in 2017, 2021, and 2023 affecting thousands of services. Multi-region DR is the only mitigation.
- Cloud provider outages: rare but possible. Multi-cloud is the answer, though most teams tolerate single-cloud risk.
- Data center loss: power, cooling, fire. Physical disasters.
- Ransomware: encrypts production data and backups. Immutable backups mitigate.
- Bad deployments: a migration that corrupts data. PITR lets you roll back.
- Misconfigured IAM: an admin accidentally revokes their own access. Break-glass accounts in a separate account mitigate.
Each disaster needs a different recovery strategy. A multi-region plan doesn't protect against ransomware unless the backups are immutable; immutable backups don't help if the deployment is bad.
Your SaaS product has an SLO of 99.9% (~8.7 hours downtime per year). You currently use a single region with daily backups. What DR strategy best matches your SLO?
Pick one answer.
Why is an untested backup not really a backup?
Pick one answer.
What's the difference between RTO and RPO?
Pick one answer.
Engineering mental model
Mental model. Think of Disaster Recovery as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Disaster Recovery mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Disaster Recovery, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = disaster_recovery(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Disaster Recovery
Change the variables below and predict what breaks first in Disaster Recovery. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Disaster Recovery, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Disaster Recovery. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Disaster Recovery?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Disaster Recovery, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Disaster Recovery, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Disaster Recovery: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Disaster Recovery. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Survives region-level and provider-level outages.
- +Bounds downtime (RTO) and data loss (RPO) to acceptable levels.
- +Forces explicit decisions about business continuity.
- +Backups protect against operational errors (bad migrations, deletions) that replication doesn't.
- −Cost scales steeply with lower RTO/RPO — hot multi-region is expensive.
- −Operational complexity: more regions, more replication, more things to break.
- −DR plans that aren't tested are worse than useless — they give false confidence.
- −Multi-region replication adds write latency (especially synchronous).
How this breaks in production
- Untested DR plans — restore fails when needed because runbooks are stale.
- Credentials stored in the failed region — can't access backups or DR infrastructure.
- Replication lag exceeds RPO — actual data loss is larger than the target.
- DNS caching extends effective RTO beyond what the failover time suggests.
Don't fall into these traps
- •Confusing backups with replication — both are needed; neither alone is enough.
- •Storing backups in the same region as production — a region outage takes them out too.
- •Choosing RPO=0 / RTO=0 for everything — usually overkill, always expensive.
- •Running DR drills in staging only — production has different data, traffic, and bugs.
Real systems using this
How real systems implement this
- Netflix — Active-active across multiple AWS regions. Each region is independent and serves live traffic. Regular Game Days simulate region failures to keep the DR plan tested.
- AWS Backup — Centralized backup service with cross-region and cross-account copies, point-in-time recovery, and immutable backups via object lock. Designed for 3-2-1 compliance.
Practice saying it out loud
- Q1Define RTO and RPO. How do they drive DR architecture?
- Q2Compare cold, warm, hot, and active-active DR strategies. When would you choose each?
- Q3Why is testing DR plans so important, and how would you do it?
- Q4How do you protect against ransomware in a DR plan?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Multi-Region Architecture