Raft Consensus Algorithm Explained: Distributed Systems Tutorial

The CAP theorem states that a distributed data store can only guarantee two of the three properties: Consistency (all nodes see the same data at the same time), Availability (every request receives a response), and Partition-tolerance (the system continues functioning despite network partitions). In the PACELC interpretation, when no network partition exists, there is a trade-off between latency and consistency; however, when partitions do occur, the system must choose between availability and consistency. This leads to different database categories: SQL/ACID databases prioritize consistency and availability (CA), while NoSQL databases like Cassandra and Couchbase prioritize availability and partition-tolerance with eventual consistency (AP), and some systems like Memcached prioritize availability and partition-tolerance with relaxed consistency (BASE).

PACELC extends CAP by addressing trade-offs during both normal operation and partitions, creating four quadrants: always-consistent, latency-consistent, availability-consistent, and always-available. Unlike CAP's binary choice, PACELC acknowledges that most systems operate normally most of the time. Applying this framework: MySQL with async replication sacrifices latency for consistency normally and availability during partitions; with semi-sync, it sacrifices latency for consistency normally but availability during partitions. Zookeeper defaults to CP behavior (sacrificing availability for consistency during partitions) but offers linearizable consistency via sync calls. Cassandra provides tunable consistency levels allowing developers to configure different trade-offs between availability and consistency based on application needs.

The CAP Theorem states that a distributed system can only guarantee two of three properties simultaneously: (1) Consistency - all replicas show the same data at any time, (2) Availability - every request receives a response, (3) Partition Tolerance - the system continues operating despite network failures between nodes. In practice, most systems choose between Consistency and Availability when partitions occur, sacrificing one for the other. The PACELC Theorem extends CAP by recognizing that network partitions are rare events. It states that when partitions occur (P), systems must choose between Consistency and Availability. When no partitions occur (E), systems must choose between Latency and Consistency. For total order broadcast, all replicas must apply operations in the same order to prevent inconsistencies from concurrent writes. This is achieved through a leader that sequences operations, with all replicas applying them in the same order. Without this, different replicas might apply operations in different orders, leading to different final states.

The CAP Theorem, formulated by Eric Brewer, states that in distributed systems, a system can only achieve two of the three properties—Consistency (all nodes see the same data), Availability (requests receive responses), and Partition Tolerance (system continues functioning despite network failures)—but not all three simultaneously. The PACELC Theorem extends this by adding that even without network partitions, systems must choose between latency and consistency. Modern databases like Cassandra (AP), MySQL (CP), and DynamoDB (user-adjustable) exemplify these trade-offs, requiring designers to strategically select which properties best meet their system's requirements.

The CAP theorem, formulated by Eric Brewer, establishes that distributed data stores can provide only two of three guarantees: Consistency (every read receives the most recent write or error), Availability (every request receives a non-error response without guaranteeing freshness), and Partition Tolerance (continuing operation despite arbitrary message drops). The key distinction is that CAP consistency differs from ACID consistency—it focuses on receiving the latest value rather than maintaining database integrity through constraints. During network partitions, systems must choose between consistency and availability: canceling operations ensures consistency but decreases availability, while proceeding with operations maintains availability but risks inconsistency. This trade-off is unavoidable in wide-area networks where partitions are inevitable, though local networks with reliable hardware can achieve both consistency and availability simultaneously.

Raft handles configuration changes (adding or removing servers) without downtime through joint consensus. When a configuration change is needed, the old leader creates a joint configuration (c_old_new) including both old and new servers. This is broadcast to all servers. Once committed (requiring majority in both configurations), the system transitions to the new configuration. During the transition, any server from either configuration can be elected as leader, and separate majorities are needed in both configurations. This ensures new servers get all logs and can eventually vote. Raft also uses log compaction and snapshots to handle growing logs efficiently, storing snapshots in stable storage and using them to catch up followers that are behind the leader. Clients interact with Raft by first contacting a randomly chosen server, which returns the leader ID. Raft provides linearizability, where operations appear to execute instantaneously at some point between request and response. The serial number mechanism ensures that if a client retries a command after a leader crash, the server recognizes the serial number and responds without re-executing.

This section covers cluster configuration changes and joint consensus in Raft. Raft supports changing cluster configuration (adding/removing nodes) through the same consensus mechanism. Any configuration change requires a quorum vote, ensuring the change is committed before it takes effect. Raft supports Joint Consensus State, which allows adding and removing nodes simultaneously in a single configuration change. This requires collecting quorums from both the old and new configurations, ensuring the transition is atomic and no period exists where the cluster has no quorum. Raft is designed to be more understandable than Paxos, with simpler terminology and clearer separation of concerns (leader election, log replication, state machine application). For a cluster to function, it needs a quorum - more than half the nodes must be available. With 3 nodes, 2 must be available; with 8 nodes, 5 must be available. If a leader fails, followers can elect a new leader as long as a quorum remains available.

Log replication in RAFT: leader maintains nextIndex and matchIndex for each follower. Leader checks if follower's log matches at nextIndex; if mismatch, decreases nextIndex and resends entries; if match, sends entries from nextIndex. Follower applies entries and updates nextIndex. For log compaction, RAFT creates snapshots at specific indices capturing state machine values, then discards entries before the snapshot. This prevents log files from growing too large, reduces disk search time, and enables quick recovery by allowing servers to start from the latest state instead of replaying all logs.

Raft leader election involves: nodes have random timeouts to avoid simultaneous elections, the first node to receive votes becomes leader, leaders serve for a term and are re-elected if they become unresponsive. Log replication involves: leaders propose operations to followers, a quorum must acknowledge before the operation is committed, followers acknowledge received operations. Raft can optimize reads by allowing followers to serve read requests directly from their local state without going through the leader, as long as the data has been acknowledged by the leader. This optimization improves performance by reducing the number of round-trips needed for read operations. The key insight is that Raft requires permanent storage (disk) to ensure reliability, and the leader election process includes jitter to prevent thundering herd problems.

Long-running distributed systems face exponential log growth problems. Raft addresses this through log compaction via snapshots. Periodically, Raft asks the application to create snapshots of the current state at specific log indices. Once a snapshot is persisted, all log entries before that index can be safely discarded because the snapshot provides equivalent information about the application state. However, this introduces complexity: followers whose logs end before the leader's snapshot point cannot be caught up using normal append entries. The Install Snapshot RPC solves this by sending the leader's current snapshot directly to followers, allowing them to discard their entire log and start fresh from the snapshot. This enables efficient recovery while maintaining correctness.

The HashiCorp Raft library is a mature Go implementation used in production systems like Consul and Nomad. Key features include MIT licensing, good documentation, and high customizability. Integration requires implementing five functions: apply (to modify your state machine), snapshot (to encapsulate state), restore (to recover from snapshots), and release (optional). The library handles network communication between nodes, consensus protocol, leader elections, and snapshotting automatically once these hooks are implemented.

Distributed consensus is the fundamental challenge of ensuring multiple servers agree on a single state when processing concurrent client requests; Raft addresses this by having each server operate in one of three states (follower, candidate, or leader), where a leader is elected through majority voting and all state changes flow through the leader to ensure consistent ordering across the cluster, with the HashiCorp Raft library providing an accessible implementation that abstracts away the complexity of consensus algorithms.

Implementing Raft requires handling nodes with IDs and timeout parameters, receiving RPC messages with context objects representing state machine state, proposing changes through node objects, and processing ready() results including saving state to disk, committing entries, and flushing messages to other cluster members. The library allows in-memory operation of multi-node clusters. Unique ID generation proved problematic—users provided non-unique identifiers like localhost names, leading to internal confusion. etcd now generates UUIDs internally. Cloud storage sync() operations exhibit unpredictable latency due to network congestion, causing legitimate master elections. The reasonable solution is tuning timeout expectations for cloud environments. Future improvements include implementing MVCC for efficient non-blocking snapshots, improving read/write scalability through observer mode proxies that receive read-only log copies without participating in consensus, and protecting against abusive clients. Observer mode would allow fast promotion of new nodes by providing them with existing log state rather than requiring full replication.

Raft is a consensus algorithm that enables distributed systems to achieve strong consistency by ensuring all nodes have identical log entries through leader election and log replication, where a quorum (majority) must agree for any change to be committed; etcd implements Raft as a state machine that manages replicated logs, with the package designed to be minimalistic by not implementing network transport or persistent storage, requiring users to provide these components while ensuring deterministic behavior for easier testing and correctness verification.

CockroachDB is a distributed SQL database designed as a single binary that runs on any operating system. It separates storage from compute, using key-value storage (like etcd) and Raft consensus protocol. This architecture enables seamless node addition/removal without downtime. The system automatically creates a web interface for monitoring, shows SQL queries and network latency between nodes, and provides automatic recovery when nodes fail. Data locality allows administrators to tag tables for specific geographic regions to comply with regulations like GDPR.

Crash-tolerant protocols assume nodes fail silently but follow protocols correctly, suitable for systems like NASA space shuttles. Byzantine Fault Tolerant (BFT) protocols handle nodes that actively deviate, including double-signing or double-voting. BFT requires assuming at least two-thirds of voting power is honest. Crash-tolerant protocols like Raft/Paxos use simpler models, while BFT protocols like PBFT provide stronger security guarantees against malicious actors in permissioned blockchain environments.

A decentralized storage layer for blockchain networks can implement different levels of fault tolerance: Crash Fault Tolerance (CFT) handles honest nodes that may crash, while Byzantine Fault Tolerance (BFT) handles malicious nodes attempting to cheat. The Alpha release provides CFT with a centralized Atlas component (single point of failure), while the Beta release will implement BFT and remove the centralized component, providing Filecoin-like resilience where the network remains functional even if some nodes are malicious.

This section traces the development of Byzantine fault tolerance theory and its practical implementation. The Byzantine Generals Problem addresses fault tolerance when processes may fail arbitrarily or maliciously rather than simply crashing. Lamport assumed the worst-case scenario where failed processes could do absolutely anything, requiring algorithms that tolerate such behavior. Digital signatures proved crucial—they allow processes to verify message authenticity, preventing faulty processes from forging deceptive messages. Without digital signatures, four processes are needed to tolerate one Byzantine failure; with them, three suffice. The problem was motivated by real-world applications like aviation control systems, where computers must fly unstable aircraft that pilots cannot handle manually. The section then transitions to Paxos, an algorithm for building fault-tolerant distributed systems that handles crash failures rather than arbitrary Byzantine failures. The algorithm implements a state machine where replicas agree on the same sequence of commands, allowing systems to continue functioning when some computers fail. A key insight is distinguishing between algorithms (abstract specifications implementable in any language) and programs (concrete implementations)—working at higher abstraction levels separates concerns better than coding directly, as code conflates many issues irrelevant to concurrency.

Byzantine Fault Tolerance (BFT) is a critical property of distributed systems that enables networks to reach consensus and continue functioning correctly even when some nodes fail or act maliciously; this concept originated from the Byzantine Generals' Problem, a thought experiment where generals must coordinate attacks or retreats through unreliable communication channels, and is implemented in blockchain technology through consensus algorithms like Proof of Work to ensure trustless agreement among decentralized network participants.

Byzantine Fault Tolerant (BFT) consensus enables distributed systems to achieve agreement even when some participants behave maliciously or unpredictably. Unlike crash-fault tolerant protocols (Paxos, Raft) that assume nodes only fail by crashing, BFT protocols must handle nodes that can send conflicting messages, create network noise, or be controlled by attackers. The term 'Byzantine' originates from Leslie Lamport's work, drawing inspiration from the historical Byzantine Empire's internal conflicts. This capability is essential for blockchain technology, where trustless environments require robust consensus mechanisms to ensure system integrity despite potential adversaries.