Cluster Bus v2 - Decentralized Failure Detector by sushilpaneru1 · Pull Request #4127 · valkey-io/valkey

sushilpaneru1 · 2026-07-09T07:22:18Z

Implements a decentralized failure detection mechanism for multi-node shard that uses the replication stream (data-path) rather than relying solely on the centralized AE_ACK-based detector on the raft leader. For multi-node shard, this provides reliable signal on the actual replication health.

The failure detector uses two complementary mechanisms (either of - Centralized or decentralized):

Shard-level (replica ↔ primary): Primary and replica detect each other via the replication stream. The primary monitors repl_ack_time for connected replicas and tracks repl_unconnected_since for disconnected ones. The replica monitors last_interaction or repl_down_since for its primary. Asymmetric timeouts (T for primary, 1.5T for replica) ensure the primary proposes NODE_FAIL first.
Cluster-level (leader → whole shard): the raft leader detects a shard via AE_ACK when no member of that shard is responsive.

In case where shard nodes can't reach each other but can be reached by raft leader, replica will get marked NODE_FAIL by the primary as this replica is of no use to primary.

Testing

./runtest --verbose --dump-logs --cluster-raft --single tests/unit/cluster/


Test Summary: 497 passed, 0 failed

\o/ All tests passed without errors!

Cleanup: may take some time... OK

coderabbitai · 2026-07-09T07:22:26Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ee4fc7f7-7f80-4c28-9226-506423a71293

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

valkey-review-bot

I found a few correctness issues in the new failure detector. The main ones are that the apply-time validator is still too permissive, and the new timestamp handling can mark healthy replicas failed during sync or after a transient reconnect.

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

murphyjacob4 · 2026-07-25T00:23:31Z

+2. **NODE_FAIL for single-node shards**: for shards with no replicas,
+   there is no replication stream to monitor. If a peer hasn't
+   responded within `cluster-node-timeout`, the leader proposes
+   NODE_FAIL. When the node comes back and sends AE_ACK, the leader
+   proposes NODE_RECOVER.
+
+**Whole-shard-down fallback:** When ALL members of a multi-node shard
+have timed out on the leader's AE_ACK tracking, no shard member is


nit: Aren't these actually the same thing?

Just when there are no replicas, whole shard = 1 node.

I guess conceptually aligning these two may be useful. So we only have two modes of failure:

Shard-level Replica <-> Primary detection

Cluster-level Leader <-> (whole shard) detection

Yes you are right, will simplify the doc.

murphyjacob4 · 2026-07-25T00:25:24Z

+/* Grace period (ms) before proposing NODE_FAIL for a replica that is not in
+ * server.replicas. After a topology change (FAILOVER/SET_REPLICA_OF), a
+ * healthy replica needs time to establish its replication connection. */
+#define REPL_CONNECT_GRACE_PERIOD_MS 5000


Maybe max(5000, server.cluster_node_timeout)?

murphyjacob4 · 2026-07-25T00:29:00Z

+    Mark a node as failed. Proposed by a shard member via the
+    replication-stream detector, or by the raft leader via AE_ACK
+    for single-node shards and whole-shard-down fallback. The
+    proposer field enables apply-time validation (see Failure


Did I miss where apply-time validation is in the doc? I see that we describe it in NODE_RECOVER but this points to Failure Detection

there was dangling reference, updated the doc.

murphyjacob4 · 2026-07-25T00:34:43Z

-        if (node && node != myself) {
+        clusterNode *node = NULL;
+        clusterNode *proposer = NULL;
+        int valid_proposer = raftValidateFailRecoverEntry(e->data, &node, &proposer);


Aren't we subject to this style of failure?

R proposes FAILOVER

P proposes NODE_FAIL R

R is promoted to primary, P is demoted to replica. Shard epoch bumps to E+1

The stale NODE_FAIL R (proposed by P) is applied next.

The proposer validator sees P is a replica, and R is a primary. Since replica_and_primary is valid (a replica can fail its primary), it applies the failure, marking the new primary R as failed.

I think it would be trivially provable if we just made NODE_FAIL check (but not bump) on the epoch. If we do that, I don't think we actually need apply-time proposer validation. We could do the proposer validation at proposal time, and let the epoch ensure there is no conflicting changes that would invalidate that proposal time check.

Good catch. I have added epoch validation in apply time and proposer validation in proposal time. But for NODE_RECOVER I don't think we should do epoch validation. It can cause legitimate recovery to be delayed, causing availability regression on replicas.

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

github-actions Bot assigned sushilpaneru1 Jul 9, 2026

Cluster Bus V2 - Introduced replication stream based failure detector

c9d6698

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

sushilpaneru1 force-pushed the cluster-v2-failure-detector branch from e02768f to c9d6698 Compare July 9, 2026 07:27

sushilpaneru1 marked this pull request as draft July 9, 2026 07:31

valkey-review-bot Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread src/cluster_raft.c Outdated

Comment thread src/cluster_raft.c Outdated

Comment thread src/cluster_raft.c Outdated

Comment thread src/cluster_raft.c

sushilpaneru1 mentioned this pull request Jul 9, 2026

Raft Cluster: Replication Stream-Based Fault Detection (Decentralized Lease) #3862

Open

Address coderabbit concerns

37c330e

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

sushilpaneru1 force-pushed the cluster-v2-failure-detector branch from 28bb1b2 to 37c330e Compare July 9, 2026 19:53

sushilpaneru1 added 2 commits July 11, 2026 18:07

Update cluster-raft.md and some refactoring

999467e

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

Add whole shard down test

6297998

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

sushilpaneru1 marked this pull request as ready for review July 13, 2026 00:34

murphyjacob4 self-requested a review July 25, 2026 00:16

murphyjacob4 reviewed Jul 25, 2026

View reviewed changes

Add shard epoch validation for NODE_FAIL

eede0d7

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

sushilpaneru1 force-pushed the cluster-v2-failure-detector branch from c0fd464 to eede0d7 Compare July 27, 2026 17:59

Update cluster-raft.md

e52e850

Signed-off-by: Sushil Paneru <sushil.paneru1@gmail.com>

sushilpaneru1 requested a review from murphyjacob4 July 27, 2026 18:20

Navigation Menu

Uh oh!

Cluster Bus v2 - Decentralized Failure Detector - #4127

Cluster Bus v2 - Decentralized Failure Detector#4127
sushilpaneru1 wants to merge 6 commits into
valkey-io:cluster-v2from
sushilpaneru1:cluster-v2-failure-detector

sushilpaneru1 commented Jul 9, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 9, 2026 •

edited

Loading

Review skipped

Uh oh!

valkey-review-bot Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

murphyjacob4 Jul 25, 2026

Uh oh!

sushilpaneru1 Jul 27, 2026

Uh oh!

murphyjacob4 Jul 25, 2026

Uh oh!

murphyjacob4 Jul 25, 2026

Uh oh!

sushilpaneru1 Jul 27, 2026

Uh oh!

murphyjacob4 Jul 25, 2026

Uh oh!

sushilpaneru1 Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sushilpaneru1 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

valkey-review-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

murphyjacob4 Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

sushilpaneru1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

murphyjacob4 Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

murphyjacob4 Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

sushilpaneru1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

murphyjacob4 Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

sushilpaneru1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sushilpaneru1 commented Jul 9, 2026 •

edited

Loading

coderabbitai Bot commented Jul 9, 2026 •

edited

Loading