diff --git a/ratis-docs/src/site/markdown/designs/listener-delegation.md b/ratis-docs/src/site/markdown/designs/listener-delegation.md new file mode 100644 index 0000000000..c23b08cdf6 --- /dev/null +++ b/ratis-docs/src/site/markdown/designs/listener-delegation.md @@ -0,0 +1,183 @@ +--- +title: Listener Replication Delegation +summary: Delegate log replication to listeners from the leader to sufficiently caught-up followers +date: 2026-07-10 +jira: RATIS-2599 +status: proposed +author: Abhishek Pal +--- + + + +# Listener Replication Delegation (AppendEntries from Followers) + +## Summary + +This document describes the design for delegating log replication to listeners from the leader to sufficiently caught-up followers. +Instead of the leader sending AppendEntries RPCs directly to every listener, a caught-up follower can serve as the replication source for one or more listeners. This reduces leader load in clusters with many listeners and can reduce network hops for co-located listener-follower pairs. + +## Motivation + +In the current architecture, the leader maintains a `LogAppender` for every peer in the cluster — both followers (voting members) and listeners (non-voting members). +This means: + +1. **Leader bandwidth bottleneck:** In clusters with many listeners, the leader must send every log entry N times — once per listener. This concentrates network I/O on a single node. + +2. **Unnecessary network hops:** Listeners co-located with a follower (same rack/region) still receive data from a potentially remote leader. + +3. **Leader CPU overhead:** Serializing and managing per-listener LogAppender state consumes leader resources that could be used for consensus-critical work. + +The proposed solution offloads listener replication to followers that already have the data, freeing the leader to focus on consensus with voting members. + +## Design Overview + +``` +BEFORE (current): AFTER (proposed): +┌────────┐ ┌────────┐ +│ Leader │──appendEntries──► Follower1 │ Leader │──appendEntries──► Follower1 +│ │──appendEntries──► Follower2 │ │──appendEntries──► Follower2 +│ │──appendEntries──► Listener1 │ │──heartbeat-only──► Listener1 +│ │──appendEntries──► Listener2 │ │──heartbeat-only──► Listener2 +└────────┘ └────────┘ + │ (delegation notification) + ▼ + Follower1──appendEntries──► Listener1 + Follower2──appendEntries──► Listener2 +``` + +### Key Invariants + +1. **Leader authority:** The leader makes all delegation decisions. Followers do not self-elect to serve listeners. + +2. **Leader heartbeat continuity:** The leader continues sending full heartbeats (empty entries with `leaderCommit` and `commitInfos`) directly to delegated listeners. This maintains liveness detection and ensures listeners get accurate commit progress from the authoritative source. + +3. **Term safety:** Listeners validate that `leaderTerm >= currentTerm` in delegated entries. A deposed leader's follower will have a stale term, so listeners reject stale entries automatically. + +4. **Log consistency:** The standard Raft log consistency check (`previousLog` matching) still runs on the listener side preventing any divergence regardless of the source. + +5. **Full catch-up support:** The follower replicator can serve listeners from any `nextIndex` using its local log — not limited to near-real-time forwarding. + +6. **Automatic fallback:** If the delegate follower becomes unhealthy, the leader revokes delegation and resumes direct replication. The listener is unaware of the switch. + +## Detailed Design + +### 1. Protocol Changes + +Extend the existing `AppendEntriesRequest/Reply` messages with new fields. + +```protobuf +// Leader → Follower: "here are the listeners you should serve" +message ListenerAssignmentProto { + repeated RaftPeerProto assignedListeners = 1; + uint64 leaderTerm = 2; +} + +// Follower → Leader: "here's how each listener is progressing" +message ListenerProgressProto { + RaftPeerIdProto listenerId = 1; + uint64 matchIndex = 2; + uint64 nextIndex = 3; + uint64 commitIndex = 4; +} + +// Extended AppendEntriesRequestProto: +// field 16: ListenerAssignmentProto listenerAssignment +// field 17: bool fromFollower + +// Extended AppendEntriesReplyProto: +// field 8: repeated ListenerProgressProto listenerProgress +``` + +### 2. Configuration + +All configuration under `raft.server.listener.delegation.*`: + +| Key | Default | Description | +|-----|---------|-------------| +| `enabled` | `false` | Master switch for the feature | +| `max-listeners-per-follower` | `5` | Maximum listeners a single follower can serve | +| `follower-lag-threshold` | `100` | Max entries a follower can lag behind leader and still be eligible | +| `reassignment-interval` | `30s` | How often the leader re-evaluates assignments | + +### 3. Leader-side: Selection and Assignment + +**ListenerDelegationSelector** (stateless utility, mirrors `SnapshotSourceSelector`): +- Filters followers: `leaderLastIndex - follower.matchIndex <= lagThreshold` +- Ranks by: highest matchIndex, then freshest RPC response +- Load-balances: round-robin assignment respecting `maxListenersPerFollower` +- Returns `Map>` — empty map triggers leader-direct fallback + +**ListenerDelegationState** (mutable, held by `LeaderStateImpl`): +- Tracks `listener → delegateFollower` mapping +- Tracks `follower → [assignedListeners]` mapping +- Marks followers with pending notification changes +- Provides `revokeDelegation(listenerId)` and `clear()` (term change) + +**Leader LogAppender behavior for delegated listeners:** +- Switches to **heartbeat-only mode**: sends periodic heartbeats with accurate + `leaderCommit` and `commitInfos`, but no log entries +- Immediately resumes full mode on revocation + +### 4. Follower-side: Replication to Listeners + +**FollowerListenerReplicator** (daemon thread per assigned listener): +- Reads entries from the follower's local `RaftLog` +- Builds `AppendEntriesRequestProto` with `fromFollower=true` and the leader's term +- Tracks `nextIndex` / `matchIndex` per listener +- Handles `INCONSISTENCY` replies by adjusting `nextIndex` (full catch-up support) +- Uses the follower's `commitIndex` as `leaderCommit` (safe lower bound) +- Stops immediately on: term change, assignment revocation, `NOT_LEADER` reply +- If listener needs entries the follower no longer has (purged by snapshot): + reports failure to leader for snapshot-based recovery via RATIS-2428 + +**FollowerListenerReplicatorManager** (lifecycle manager): +- Processes `listenerAssignment` from leader's AppendEntries +- Starts/stops/updates replicators based on assignment changes +- Collects `ListenerProgressProto` reports for piggybacking on follower's reply + +### 5. Listener-side: Accepting Delegated Entries + +Modified `RaftServerImpl.appendEntriesAsync()`: + +- **When `fromFollower=true` AND node is LISTENER:** + - Validate `leaderTerm >= currentTerm` (reject stale terms) + - Do NOT call `state.setLeader(senderId)` — listener's known leader stays unchanged + - Do NOT call `changeToFollowerAndPersistMetadata` — listener stays a listener + - DO update `lastRpcTime` (prevents timeout alerts) + - DO proceed with standard log append and consistency checks +- **When `fromFollower=true` AND node is NOT LISTENER:** reject with `NOT_LEADER` +- **When `fromFollower=false`:** existing logic unchanged + +### 6. Leader-side: Progress Monitoring and Fallback + +**Progress processing:** +- When handling `AppendEntriesReply` from a delegate follower, extract + `listenerProgress` reports +- Update leader's own `FollowerInfo` for each reported listener (matchIndex, + commitIndex) + +**Health checks (every `reassignment-interval`):** +- Delegate follower's matchIndex fell behind `lagThreshold` → revoke +- Listener's matchIndex not advancing → revoke +- Delegate follower unresponsive → revoke +- Cooldown period prevents rapid reassignment oscillation + +**Revocation:** +- Resume leader's LogAppender for that listener (heartbeat-only → full) +- Notify follower with empty `assignedListeners` on next heartbeat +- Follower's `FollowerListenerReplicatorManager` stops the replicator