Distributed Systems Fundamentals
A distributed system is one in which multiple independent computers cooperate over a network to appear as a single coherent service to the outside world. The defining property is not geographical spread but partial failure: components you depend on can fail independently and unpredictably, and the network that connects them can drop, delay, or reorder messages at any time. Mastering distributed systems is largely about learning to design as if every assumption you would make on a single machine — shared memory, synchronous calls, reliable clocks — does not hold.
How it works
A distributed system has three properties that single-machine systems do not:
- No shared memory. Nodes communicate by passing messages over a network. They cannot read each other's memory directly; they can only exchange messages that may or may not arrive in order, or at all.
- Partial failure. A single node can crash while others keep running. The network can partition, leaving groups of nodes unable to communicate. The system as a whole is only 'up' if the survivors can keep serving, which is the entire point of distribution — and the entire source of its difficulty.
- No global clock. Even with NTP synchronization, clocks on different machines drift by milliseconds to seconds. You cannot use wall-clock time to determine which of two events happened first if they occurred on different nodes — you need logical clocks (Lamport timestamps, vector clocks) for that.
A multi-core CPU is not a distributed system in this sense — it shares memory and a single clock. A single Postgres instance behind a load balancer with one read replica is a distributed system the moment you write to the primary and read from the replica, because now you have network, partial failure, and clock skew in play.
Peter Deutsch and James Gosling cataloged the Eight Fallacies of Distributed Computing — assumptions that are always wrong and always bite:
- The network is reliable. (Cables get cut, switches fail, packets get dropped under load.)
- Latency is zero. (Even a local network round trip is sub-millisecond; a cross-continent call is 50-150ms.)
- Bandwidth is infinite. (Video and ML payloads saturate links.)
- The network is secure. (DDoS, MITM, hijacked BGP routes.)
- Topology doesn't change. (Auto-scaling, deploys, and DNS rebinding change it constantly.)
- There is one administrator. (Multi-cloud, multi-region, third-party SaaS.)
- Transport cost is zero. (Marshalling, TLS handshakes, proxy hops add CPU and latency.)
- The network is homogeneous. (Mixing gRPC, HTTP, custom binary protocols is the norm.)
Every distributed-systems bug is, at root, a violation of one of these. The defensive response is timeouts on every call, retries with exponential backoff and jitter, idempotency on every mutating operation, and circuit breakers to fail fast when a downstream is broken.
Two armies on opposite hills must coordinate an attack. They can only communicate via messengers through enemy territory — messengers may be captured. There is no protocol that lets both generals know with certainty that they have agreed to attack at the same time. The last messenger might be captured, leaving one general unsure whether the other received the confirmation. This is the foundational impossibility of distributed systems: you cannot achieve certain knowledge of agreement over an unreliable channel. Every consensus protocol (Paxos, Raft, 2PC) is a workaround — they reduce the probability of disagreement to arbitrarily small levels, never zero.
Distributed-systems engineers work in three layers of abstraction, each with stronger (and more expensive) guarantees:
- Best-effort delivery: messages may be lost, duplicated, or reordered. Used for telemetry and metrics — losing a data point is fine.
- At-least-once delivery: messages are not lost but may be duplicated. Used for queues, event streams, and almost every practical system. Requires idempotent consumers.
- Exactly-once semantics (EOS): messages are delivered exactly once, in order. Achieved only via consensus on every delivery — very expensive. Kafka transactions approximate EOS by combining at-least-once with transactional dedup on the consumer.
The stronger the guarantee, the higher the latency and the lower the throughput. Choosing the right level for each workload is the core design decision. A payment needs at-least-once plus idempotency, not exactly-once consensus on every byte. A sensor stream needs best-effort, not at-least-once.
Related abstractions: synchronous vs. asynchronous replication (do you wait for replicas before acknowledging the write?), strong vs. eventual consistency (do reads see the latest write?), and linearizability (does the system behave as if there were a single copy of the data?). The CAP theorem says you cannot have all of {linearizability, availability, partition tolerance} during a partition — partition tolerance is not optional on a real network, so the real choice is C vs A during partitions.
Distributed systems are harder to debug because there is no single call stack. A single user request might fan out across five services, hit three databases, and produce ten log lines on each — and the only way to assemble them is a shared correlation ID propagated through every hop. The triad of metrics, logs, and distributed traces (the 'three pillars of observability') exists precisely because local debugging is insufficient. Without distributed tracing, the classic 'the site is slow' bug becomes a multi-day hunt through log files. With it, the trace shows the slowest hop in seconds.
This is also why every distributed system design starts with operations in mind: how will I know if it is broken, how will I know if it is slow, and how will I trace a single user's request across the topology? These questions shape everything from request ID propagation to metric labeling to alert thresholds.
Which of the following is the defining property of a distributed system?
Pick one answer.
Why can't you use wall-clock timestamps to determine which of two events happened first if they occurred on different nodes?
Pick one answer.
What is the 'two generals problem,' and what is its practical implication?
Pick one answer.
Engineering mental model
Mental model. Think of Distributed Systems Fundamentals as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Distributed Systems Fundamentals mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Distributed Systems Fundamentals, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = distributed_systems_intro(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Distributed Systems Fundamentals
Change the variables below and predict what breaks first in Distributed Systems Fundamentals. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Distributed Systems Fundamentals, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Distributed Systems Fundamentals. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Distributed Systems Fundamentals?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Distributed Systems Fundamentals, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Distributed Systems Fundamentals, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Distributed Systems Fundamentals: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Distributed Systems Fundamentals. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Horizontal scalability — add more nodes to handle more load.
- +Fault tolerance — the system survives single-node failures.
- +Geographic distribution — serve users from the nearest region.
- +Independent deployment and scaling of components.
- +No single machine limits (CPU, memory, disk) on total system capacity.
- −Partial failure is the default — every design must handle it.
- −No global clock — ordering and consistency are hard problems.
- −Network is a new failure surface — partitions, latency, packet loss.
- −Operational complexity — observability, deployment, and debugging are harder.
- −Strong consistency is expensive or impossible (CAP theorem) — usually settle for eventual consistency.
- −No distributed transactions by default — 2PC is blocking; sagas are eventual.
How this breaks in production
- Network partition — split-brain if both halves accept writes; mitigated by quorum requirements (only the majority half can accept writes).
- Clock skew — events ordered incorrectly; mitigated by logical clocks or TrueTime.
- Cascade failure — one slow downstream causes thread pool exhaustion upstream; mitigated by circuit breakers and bulkheads.
- Gray failures — a node is partially alive (serving but slow); hard to detect, causes cascading timeouts; mitigated by adaptive health checks.
- Byzantine faults — nodes behave maliciously or arbitrarily; mitigated by Byzantine fault tolerance (only relevant in adversarial settings like blockchains).
Don't fall into these traps
- •Treating a distributed system like a single machine — assuming shared memory, synchronous calls, reliable clocks.
- •Ignoring the eight fallacies — especially 'the network is reliable' and 'latency is zero.'
- •Using wall-clock time for ordering across nodes — use logical clocks or version vectors.
- •Not propagating correlation IDs — debugging becomes a multi-day log-hunt.
- •Designing for the happy path only — every cross-node call must have a timeout, retry, and failure mode.
- •Forgetting that partial failure means 'maybe' — design every operation to be safe under ambiguous outcome (idempotency).
Real systems using this
How real systems implement this
- Google Spanner — Globally distributed SQL database that achieves external consistency via TrueTime (atomic clocks + GPS in every datacenter), Paxos-based replication, and 2PC across participant groups. The rare 'strongly consistent at global scale' system — and the engineering effort required to build it shows why most systems settle for eventual consistency.
- Amazon DynamoDB — AP-style distributed key-value store: writes accepted on any node, replicated via gossip, eventual consistency by default with optional strong reads from the primary. Illustrates the trade-off: high availability and write throughput, but reads may be stale.
- Apache Cassandra — Decentralized masterless AP database. Any node can accept any write; tunable consistency per query (ONE, QUORUM, ALL). No single leader — every node is equal, which removes the leader-election failure mode at the cost of stronger consistency being opt-in.
- Kubernetes — A distributed system for running distributed systems. The control plane (etcd for consensus via Raft, API server, scheduler, controllers) is itself distributed and partial-failure-aware. Node failures are expected; pods are rescheduled automatically.
Practice saying it out loud
- Q1What makes a system 'distributed'? What are the defining properties?
- Q2List the eight fallacies of distributed computing. Which one bites engineers most often?
- Q3What is the two generals problem? What is its practical implication for consensus?
- Q4Why is wall-clock time unreliable for ordering events across nodes? What do you use instead?
- Q5Your service calls a downstream that is sometimes slow. How do you prevent this from cascading into a full outage?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Replication