What Is Split-Brain in Distributed Systems?

Split-brain — often called the split-brain problem — is a failure mode in which a network partition leaves two or more parts of a cluster unable to see each other, each concluding that the others have failed, and each continuing to act as the authoritative copy. The cluster has one name and one identity, but two brains — and both of them are making decisions.

(The term is borrowed from neurology, where it describes a brain whose hemispheres have been surgically separated. The distributed-systems meaning is unrelated beyond the metaphor.)

What makes split-brain dangerous is that nothing is broken. Every node is healthy, every process is running the code it was written to run, and every failure detector is behaving exactly as designed. The damage is emergent: two correct halves, each reasoning correctly from incomplete information, arrive at incompatible conclusions and both act on them.

How Split-Brain Happens

The sequence is always roughly the same, and it takes seconds.

1. The network partitions. A switch fails, a security group changes, an availability zone loses connectivity, or a GC pause stalls a node long enough to look dead. Nodes on each side can still talk to their neighbors; they just cannot reach the other side.

2. Failure detectors fire. Every node uses timeouts to decide whether its peers are alive. Neither side can distinguish "the other nodes have crashed" from "I cannot reach the other nodes" — this is the fundamental limitation, and no timeout value fixes it. Each side concludes the other is dead.

3. Both sides recover. Recovery logic is supposed to run when the leader dies. As far as the side without the leader is concerned, the leader has died, so it elects a new one. Meanwhile the original leader is still running, still healthy, and still accepting work.

4. Two authorities accept writes. Clients are partitioned too. Some reach the old leader, some reach the new one. Both sets of writes are accepted, acknowledged, and durably stored — in two different places.

What Goes Wrong: Divergent Writes and Lost Updates

While the partition lasts, both halves look fine from the inside. Clients get successful responses. Monitoring is green on both sides. The problem only becomes visible when the network heals — and by then the damage is already in the data.

• Divergent state. The same key has two different values, each of which was correctly derived from the writes that side saw. There is no version of the truth that includes both.
• Lost updates. Something has to reconcile the two histories, and the usual mechanism is that one side is declared the loser and resynchronizes from the winner. Every write the losing side acknowledged is silently discarded — including writes your application already told a user had succeeded.
• Duplicated side effects. If both halves ran the same scheduled job or held the same distributed lock, the work happened twice. Two shipments, two charges, two emails. Unlike divergent data, this cannot be repaired by picking a winner — the effect has already left the system.
• Corrupted aggregates. Counters, balances, and inventory levels that were incremented independently on both sides are not merely stale after the heal; they are wrong in a way that no single value can fix.

Reconciliation is harder than it sounds. Last-write-wins is easy to implement and throws data away. A semantic merge preserves more, but it requires application knowledge the datastore does not have — only your code knows whether two conflicting orders should be summed, deduplicated, or escalated to a human. This is why the industry answer is overwhelmingly prevention rather than repair.

Split-Brain vs. Network Partition vs. Failover

These three get used interchangeably in incident reviews, which makes post-mortems muddier than they need to be. They are three different things in a causal chain.

Framed against the CAP theorem, split-brain is what choosing availability over consistency looks like when it goes badly. A partitioned system must give something up. A CP system refuses writes on the minority side and accepts the downtime; an AP system keeps accepting writes everywhere and accepts the divergence. Split-brain is the AP choice made accidentally, by a system whose operators believed it was CP.

How Systems Prevent the Split-Brain Problem

There is no way to detect a partition from inside one — but there is a way to make sure only one side acts. Every mechanism below is a variation on the same idea: make authority something a node can lose without knowing why.

Quorum and majority. The dominant approach. A node may only act as leader if it can reach a strict majority of the cluster. Because two disjoint groups cannot both hold a majority of the same set, at most one side can ever be authoritative. The minority side must refuse to serve — which is exactly the availability sacrifice CAP predicts. This is why cluster sizes are odd: four nodes tolerate the same single failure as three, while adding a way to split 2–2 with no majority at all.

Fencing tokens. Quorum stops two leaders from being elected. It does not stop a leader that was legitimately elected, then paused for thirty seconds in garbage collection, from waking up and writing as though it were still in charge. A fencing token is a monotonically increasing number issued on each acquisition; the protected resource records the highest token it has seen and rejects anything lower. The stale leader's write is refused by the resource itself, without the resource needing to know anything about cluster membership. See Java FencedLock for the pattern in detail.

Witness and tiebreaker nodes. A two-datacenter deployment has no majority when the link between sites fails. A lightweight third participant in a third location — holding no data, existing only to vote — restores an odd count and breaks the tie.

STONITH. "Shoot The Other Node In The Head": before promoting, forcibly power off or isolate the old primary. Common in traditional HA clusters and storage, and available in the public clouds through fence agents that call the provider API to stop the instance — fence_aws, fence_gce, fence_azure_arm. It is much harder inside container orchestrators: Kubernetes has no built-in fencing agent, and its out-of-service taint requires an operator to confirm the node is already powered off before applying it.

Epoch numbers. Each successful election increments a cluster-wide epoch. Messages carry the epoch of their sender, and anything stamped with an old one is rejected on sight — the same principle as a fencing token, applied to cluster membership rather than to a single resource.

Split-Brain in Redis and Valkey

Redis and Valkey handle this differently depending on whether you run Sentinel or Cluster, and the difference matters more than most teams realize.

Sentinel. Two separate thresholds are at work, and conflating them is the most common Sentinel misconfiguration. The quorum value in sentinel monitor is only how many Sentinels must agree the master is unreachable to flag it objectively down. Actually running the failover requires authorization from at least a majority of all Sentinels — more, if you set the quorum higher than that majority — no matter how low the quorum is. That majority requirement is the split-brain guard, and it is why three Sentinels in three failure domains is the practical floor.

But note what this does and does not buy you. It prevents two replicas from being promoted. It does not stop the original master, isolated in the minority partition, from continuing to accept writes from whichever clients are isolated alongside it. Those writes are real, acknowledged, and doomed: when the partition heals, the old master is demoted to a replica of the newly promoted one and resynchronizes, discarding everything it accepted while alone.

The fix is to make the master police itself, on the master's own configuration:

min-replicas-to-write 1
min-replicas-max-lag 10

A master that cannot see at least one replica whose lag is within ten seconds starts rejecting write commands. Be precise about what this buys, because Redis is: the configuration file states that the option "does not GUARANTEE that N replicas will accept the write, but will limit the window of exposure for lost writes ... to the specified number of seconds." Lag is measured from the last replica acknowledgement, so writes can still be accepted and lost for up to the configured interval after a partition begins. It converts an unbounded loss window into a bounded one, which is the most any system with asynchronous replication can offer.

Cluster. Redis Cluster builds this in. Promoting a replica requires agreement from a majority of masters, and a master that cannot reach a majority stops serving its slots once cluster-node-timeout elapses. The self-demotion that Sentinel needs min-replicas-to-write to approximate is part of the Cluster protocol. If you are weighing the two, our Sentinel vs. Cluster comparison covers the wider trade-offs.

The caveat that applies to both. Redis replication is asynchronous. A master acknowledges a write to the client before the replica has confirmed it. Quorum bounds how long split-brain can last and how many nodes can be wrong; it does not close the window in which an acknowledged write exists on exactly one node that is about to be demoted. Anyone telling you quorum alone makes Redis failover lossless is overselling it.

Split-Brain and Distributed Locks

The sharpest practical consequence of that asynchronous-replication window is locking, because a lock is a claim of exclusivity and split-brain is exactly the loss of exclusivity.

The failure is small and specific. A client acquires a lock on the master. The master acknowledges, then fails or is partitioned away before the write reaches its replica. The replica is promoted with no record of the lock. A second client asks for the same lock, and the new master — correctly, given what it knows — grants it. Two clients now hold what both believe is an exclusive lock, and neither has any way to detect the other.

The Redlock algorithm was proposed to remove that single point of failure by requiring a majority across several independent masters. It remains one of the more contested designs in distributed systems. Martin Kleppmann argued that a lock guarding correctness needs fencing tokens; Salvatore Sanfilippo disputed that fencing is strictly necessary. The documentation has since moved: the Redis project's current guidance on distributed locks states plainly that "you should implement fencing tokens ... [this] applies to any distributed locking system." The reasoning is that no lock service can stop a client from pausing past its own lease expiry, so the protected resource has to be able to reject the stale writer on its own.

Handling Split-Brain With Redisson

Redisson is a Java client for Redis and Valkey, and several of its defaults exist specifically because of the failure described above.

Replica-synchronization checking on locks. After acquiring an RLock, Redisson verifies the lock reached at least one connected replica. If none acknowledge within the timeout, the lock is released and the acquisition fails with NoSyncedSlavesException — so a client never keeps a lock that reached no replica at all. This narrows the failover hazard substantially without a five-node Redlock deployment, and it is on by default:

Config config = new Config();
config.setCheckLockSyncedSlaves(true)   // default
      .setSlavesSyncTimeout(1000);      // milliseconds, default

RFencedLock when the lock guards correctness. Each acquisition returns a monotonically increasing token. Pass it to the protected resource, which records the highest token it has accepted and rejects anything lower — fencing out a holder that resumes after a long pause:

RFencedLock lock = redisson.getFencedLock("myLock");

Long token = lock.lockAndGetToken();
try {
    // pass token to the protected resource, which must reject any write
    // whose token is lower than the highest it has already accepted
    storage.write(data, token);
} finally {
    lock.unlock();
}

RFencedLock is available from Redisson 3.19.0 onward, and the older RedLock/RedissonRedLock object is deprecated. Our guide to the Redlock algorithm covers the migration and the safety debate behind it in full.

Automatic failover handling. Redisson discovers topology through Sentinel or Cluster and repoints to the promoted master on its own, so application code contains no failover logic and no stale connection to a demoted node.

Synchronous replication. Redis and Valkey expose this natively through the WAIT and WAITAOF commands, and Redisson builds on them: BatchOptions.sync() and TransactionOptions.syncSlaves() hold a batch or transaction until replicas confirm it. Redisson PRO extends the same idea to messaging, where Reliable Queue and Reliable PubSub take per-destination syncMode, syncTimeout, and syncFailureMode settings so an individual message can be held until replicas — and optionally the append-only file — acknowledge it.

Be clear about the limit, because Redis is: WAIT "does not make Redis a strongly consistent store", and it is "possible to still lose a write synchronously replicated to multiple replicas." Synchronous replication narrows the window described above very substantially. Nothing available on this architecture closes it outright.

Multi-datacenter topologies. Redisson PRO adds Multi Sentinel and Multi Cluster modes, which connect to several independent Sentinel or Cluster deployments — the configuration a cross-region setup needs, and the one most exposed to partitions. See the walkthroughs on regional replicas with multiple Sentinels and cross-datacenter replication; PRO can be evaluated with a free trial.

Split-Brain: Frequently Asked Questions

What Causes Split-Brain?

A network partition combined with automatic failover. When nodes lose connectivity, each side's failure detector reports that the other side is down, because a node cannot distinguish a crashed peer from an unreachable one. Each side then runs its recovery logic and elects its own leader, leaving two nodes accepting writes for the same dataset.

What Is the Difference Between Split-Brain and a Network Partition?

A network partition is the loss of connectivity itself — the cause. Split-brain is a specific consequence, where both sides of the partition continue operating as the authoritative copy. A partition handled correctly does not produce split-brain: the minority side detects it lacks a quorum and stops serving writes.

How Does Quorum Prevent Split-Brain?

Quorum requires a node to reach a strict majority of the cluster before acting as leader. Two disjoint groups cannot both hold a majority of the same set, so at most one side can ever be authoritative. The minority side must refuse writes, trading availability for consistency. Odd cluster sizes matter because an even split leaves neither side with a majority.

Can Redis Have a Split-Brain?

Yes. Sentinel's majority requirement prevents two replicas from being promoted, but an isolated original master keeps accepting writes until the partition heals, at which point it is demoted and those writes are discarded. Setting min-replicas-to-write and min-replicas-max-lag on the master makes it reject writes when it cannot reach enough replicas — a best-effort measure that bounds the loss window rather than eliminating it.

Does Redis Cluster Prevent Split-Brain?

Redis Cluster handles it better than Sentinel does. Promotion requires agreement from a majority of masters, and a master that cannot reach a majority stops serving its slots once cluster-node-timeout elapses. Because replication is still asynchronous, writes acknowledged in the moments before that timeout can be lost.

What Is a Fencing Token?

A monotonically increasing number issued each time a lock is acquired. The client passes it to the protected resource, which records the highest token it has seen and rejects any operation carrying a lower one. This blocks a stale lock holder that resumes after a pause, without the resource needing to know anything about cluster membership.

What Is STONITH?

"Shoot The Other Node In The Head" — forcibly powering off or isolating the old primary before promoting a replacement. It is common in traditional HA and storage clusters, and the public clouds support it through fence agents such as fence_aws, fence_gce, and fence_azure_arm. It is harder in container orchestrators: Kubernetes has no built-in fencing agent.

Similar terms

• CAP Theorem
• Redis Sentinel
• Redis Cluster
• Redis Sentinel vs Redis Cluster
• Redis Master-Slave Replication
• Distributed Lock
• Java FencedLock
• Redlock Algorithm
• Redis Lock
• Distributed Database
• Race Condition