Distributed Systems Concepts Every Software Engineer Should Know

Every microservice call, every database write, every cache lookup in a modern system depends on trade-offs that were formalized decades ago in distributed systems theory. When an interviewer asks 'how would you handle a network partition' or 'why does this database favor availability over consistency,' they are testing whether you understand the reasons behind design decisions, not just the names of tools.

This guide covers the theory that shows up again and again in system design interviews: the CAP theorem, consistency models, consensus algorithms like Raft and Paxos, distributed transaction patterns, and the fault-tolerance techniques that keep large systems running when parts of them inevitably fail.

01CAP Theorem

The CAP theorem states that a distributed data store can only guarantee two of the following three properties at once: Consistency (every read receives the most recent write or an error), Availability (every request receives a non-error response, without guaranteeing it's the latest data), and Partition Tolerance (the system keeps working despite dropped or delayed messages between nodes).

The insight that matters in an interview is not the three letters themselves - it's that network partitions are a fact of life in any distributed system. Since partition tolerance isn't really optional, the actual decision you're making as a system designer is between Consistency and Availability during the partition. This is why CAP is often described more usefully as a CP-vs-AP choice.

Want to see these trade-offs applied to a real system? Walk through designing a rate limiter or a distributed cache, where CP vs AP choices directly affect the design.

02Consistency Models

"Consistency" is not one thing - it's a spectrum of guarantees about what a reader is allowed to see relative to writes happening elsewhere in the system. Picking the right model is a matter of matching guarantees to what your application actually needs, since stronger consistency generally costs more latency and availability.

In an interview, naming the model isn't enough - explain the cost. Strong consistency typically requires coordinating with a quorum or leader on every operation, which adds latency and can reduce availability during failures. Eventual consistency avoids that coordination, trading it for the possibility that two clients briefly see different answers.

03Consensus Algorithms

Consensus algorithms let a cluster of nodes agree on a single value or a single ordered sequence of operations, even when some nodes crash or messages are delayed. This is the machinery underneath every CP system: someone has to decide who the leader is and which writes actually 'happened.' Paxos was the original formalization of this problem, but it's notoriously difficult to reason about and implement correctly. Raft was designed later specifically to be understandable while providing the same guarantees, which is why most modern infrastructure builds on it.

Raft decomposes consensus into three separate, easier-to-reason-about sub-problems:

• Leader election - Time is divided into numbered terms. Nodes start as followers; if a follower doesn't hear from a leader within a timeout, it becomes a candidate and requests votes. Whichever candidate gets votes from a majority of nodes becomes leader for that term. Randomized timeouts prevent repeated split votes.
• Log replication - The leader is the only node that accepts new writes. It appends each write to its local log and replicates that log entry to followers. Once a majority of nodes have stored the entry, the leader considers it committed and applies it to the state machine.
• Safety - Raft guarantees only one leader can exist per term, and once an entry is committed by a majority, it can never be lost or overwritten - even if the leader crashes immediately after. A node can only become leader if its log is at least as up to date as a majority of the cluster's.

This leader-election-plus-replicated-log pattern is what powers etcd (and therefore Kubernetes' cluster state), CockroachDB, TiKV, and Consul. If you can walk through why a five-node Raft cluster can survive two node failures but not three (majority = 3 out of 5), you've demonstrated real understanding, not memorization.

04Distributed Transactions

A single business operation often has to touch multiple independently-owned services. Consider placing an order:

PlaceOrder() ->
  1. Deduct stock in Inventory Service
  2. Charge the customer in Payment Service
  3. Create a shipment in Fulfillment Service

If step 2 succeeds but step 3 fails, the system is left in an inconsistent state - money was charged but nothing will ship. A classic ACID transaction can't span these services because each owns its own database and shouldn't share locks with the others.

Two patterns solve this in practice:

• Two-Phase Commit (2PC) - A coordinator asks every participant to prepare (lock resources, confirm it can commit) and only commits once all participants vote yes; if any votes no, everyone rolls back. This gives strong atomicity but is blocking: if the coordinator crashes mid-protocol, participants can be stuck holding locks indefinitely. Rarely used across service boundaries in modern architectures because of this fragility.
• Saga pattern - Break the operation into a sequence of local transactions, each with a compensating transaction that undoes it if a later step fails. Deduct inventory (compensate: restore inventory), charge payment (compensate: refund), create shipment (compensate: cancel shipment). Sagas can be coordinated with a central orchestrator or run in a choreographed, event-driven style where each service reacts to the previous one's events.

Sagas trade strict atomicity for availability and service autonomy - the system is briefly inconsistent between steps, but it never blocks waiting on a coordinator, and every intermediate state has a well-defined recovery path. This is the pattern behind order processing at Uber, Amazon, and most microservice-based e-commerce systems.

05Fault Tolerance

Distributed systems must assume that machines, disks, and networks will fail - the goal is designing so that individual failures don't become outages. Three techniques come up constantly in interviews: replication, circuit breakers, and retries with backoff.

Replication

• Single-leader replication - One primary node accepts all writes and asynchronously (or semi-synchronously) streams them to replicas; reads can be spread across replicas to scale read throughput. Used by PostgreSQL streaming replication and MySQL. Simple to reason about, but failover requires promoting a new leader.
• Leaderless (quorum-based) replication - Any node can accept a write. Consistency is tuned with the formula W + R > N, where N is the number of replicas, W is how many must acknowledge a write, and R is how many are read from. Used by Cassandra, DynamoDB, and Riak - it trades a single point of coordination for tunable consistency per operation.

Health Checks and Circuit Breakers

Health checks let load balancers and orchestrators stop routing traffic to a node that's failing, before users notice. Circuit breakers apply the same idea at the client level, protecting a caller from a struggling downstream dependency:

• Closed - Requests flow normally, and failures are counted. If the error rate crosses a threshold, the breaker trips open.
• Open - Requests fail immediately without even attempting the call, protecting the failing service from added load and giving it time to recover. After a cooldown timeout, the breaker moves to half-open.
• Half-open - A small number of trial requests are allowed through. If they succeed, the breaker closes again; if they fail, it reopens and the timeout restarts.

This pattern is implemented in libraries like Resilience4j and built into service meshes such as Istio, and it's a common building block in circuit breaker design questions.

Retries with Exponential Backoff and Jitter

When a call fails transiently, retrying immediately just adds more load to an already-struggling service. Exponential backoff spaces out retries geometrically, and jitter randomizes the exact delay so that many clients retrying after the same failure don't all hammer the service in the same instant:

retry_delay = min(base_delay * 2^attempt + random_jitter, max_delay)

Without jitter, synchronized clients create a thundering herd - a wave of simultaneous retries that can re-trigger the very overload the retries were trying to recover from.

See these fault-tolerance patterns combined in a real design walkthrough: Crack the System Design Interview at FAANG covers how they show up in interview answers end to end.

06Conclusion

Distributed systems can feel overwhelming because of how many tools and acronyms surround them, but the underlying ideas are a small, learnable set: the CAP theorem forces a consistency-versus-availability choice whenever a partition happens; consistency models let you dial in exactly how strict that choice needs to be; consensus algorithms like Raft give a cluster a reliable way to agree on truth; sagas let multi-service operations stay resilient without global locks; and replication, circuit breakers, and backoff keep the whole thing standing when individual pieces fail.

The fastest way to make this knowledge stick is to apply it. Next time you sketch a system design, explicitly state which side of CAP you're choosing and why - that single habit is often what separates a strong interview answer from an average one.