fix(cluster): PCM-X convert-queue closure — dirty-N flush, churn recovery, self-ACK, rejoin contract#62
Merged
Conversation
added 9 commits
July 18, 2026 18:37
…losed contract The L5 online-rejoin leg ran without the striping quartet and without a shared per-thread WAL root: a recycled ITL ref then has no derivable origin, and the L3 dead-member episode leaves HW remaster structurally blocked, wedging the rejoin episode in WAIT_CLUSTER. Arm the t/346 striping quartet (online_join / quorum_poll / join_convergence / xid_striping) plus wal_threads_root so the admission window and the verdict chain are actually exercised. Post-admission, the surviving origin still refuses to serve: its durable write-fence authority stays bound to the pre-rejoin epoch (survivor-side republish gap). Pin the interim contract instead of the historical 8-row read-back: the read must fail bounded with the exact TT-recycled error, verdict wire and origin fence refusals must grow, and the below-horizon / served counters must stay flat. The 8-row read-back becomes the future acceptance once the survivor republish lands. Spec: spec-5.22-online-join-cold-formation.md (Amend v1.1)
…rness CONNECTED proves the DATA channel, not application-protocol readiness: PCM-X formation follows HELLO asynchronously, so a test that writes through a live pair immediately after connect races the runtime gate. Add wait_for_pcm_x_active to ClusterPair (both runtimes must publish ACTIVE) and arm it in t/243 and t/247 before the first cross-node DML. Quad shared-catalog rigs additionally imply online_join + xid_striping, the substrate their formation now requires.
… shard The generic IC path treats dest=self as a successful no-op, but this application ACK is the drop proof a resource master that is also an S holder must consume before the PCM-X transfer can advance. Stage the local arm through the tag-sharded DATA ring so its LMS worker performs a real loopback dispatch with same-tag ordering; remote ACKs keep their direct DATA connection. The shard router accepts the ACK opcode for the local node accordingly.
… classification set_generation belongs to the holder-local tag, so two exact holders can legitimately publish the same numeric generation: complete the namespace with the authenticated source tuple before treating a stage begin as a duplicate, and require the previous holder set to have reached the graph-commit boundary before it may be replaced. A non-current source can only be replaying its last published set (STALE). The terminal retry driver scans responders from node zero each pass; once an earlier responder ACKed, a later responder may own the one reliable leg. Classify a completed responder before the generic cross-responder BUSY check so that leg cannot hide behind the first completed node, and treat a same-phase leg still armed after its responder bit committed as the structural split it is.
A dirty N page at the PCM-X N-source image prep is legitimate, not corruption evidence: relation extension (PageInit + MarkBufferDirty) and recovery redo both dirty a page before any PCM grant exists, so the first cluster-aware writer meets its own pre-grant dirt. The N-source contract serves STORAGE bytes and overwrites the resident copy, so consuming the page as-is would discard the newer local bytes (a lost write) — and judging it CORRUPT closed the whole runtime over a normal extension. Push the local bytes out first (WAL-first FlushBuffer under a shared content lock, reserve+pin taken under the header lock) and report BUSY: one flush converges the state and the image pump retries against a clean page. BM_IO_ERROR is judged before the dirty branch on both passes; the second-pass check stays BUSY since REVOKING already blocks data writes and residual dirt is idempotent hint traffic. The pcm_own source contract pins the new ordering.
…he runtime Four sites treated benign concurrent progress as corruption and fused the whole PCM-X runtime into RECOVERY_BLOCKED: * A follower WFG snapshot returning STALE is the same slot-churn signal (promotion / round advance) the claim site already recovers from with a refresh lookup. Route it to the exact re-dispatch via the new REFRESH_ROLE retry action instead of the kill path. * The cleanup CANCEL_LOCAL arm cancelled with the same stale handle and fused on STALE. Cancelling releases rather than confers authority, so rebuild the handle from the live slot (identity-exact lookup) and retry, bounded; a vanished membership is already terminal. * An ACTIVE_PROBE ticket observed CLAIMED with its GRD pending-X cookie absent can be another actor mid two-phase (a claimed cancel clears the cookie before it finalizes; the serve path clears by identity). The already-claimed ensure arm now mirrors the reserve-path recheck under the ticket domain lock before calling it BAD_STATE. * One dispatch BAD_STATE observation is damped per ticket in process- local memory; only a streak surviving consecutive periodic drive ticks reaches the fail-closed verdict, with lowest-streak recycling under table pressure so a real wedge keeps its count. CORRUPT / INVALID / NO_CAPACITY / COUNTER_EXHAUSTED still fuse immediately. Unit tests pin the retry-table row, the damping streak semantics, and the four recovery arms in source.
…queue lane t/248 verifies the falsified pg_xact fixture by reading the two on-disk status bits directly instead of txid_status(), whose local nextXid horizon check does not apply to remote-origin xids. t/365 upgrades the 2PC leg to the convert-queue contract: after churn forces terminal verdict resolution the queue transfers X to the peer, so the leg now asserts both cross-node increments chain onto the decided 2PC value instead of pinning the former deferred-image availability boundary, and accepts the queue reservation-conflict message as an explicit fail-closed judge. t/018, t/215/t/216 and t/401 refresh dump-key and emit-row counts for the lane observability additions.
t/400 (PCM-X convert-queue 4-node liveness) and t/404 (crash-rejoin stale-read/write fence) were in no nightly shard: the matrix covered 392-397 and 401-403, so neither ran in any CI tier. Give them their own shard — the quad formation and the full crash/restart cycle dominate the wall clock (L342: every new t/ file lands in a shard the same commit).
…n poll
A do{}while body is not a loop block in Perl, so the "last" in the
self-handoff drain poll was a runtime error — and it fired exactly when
the polled counters advanced (the healthy path), killing the test after
assert 57 with exit 29. Convert the poll to while(1) with an explicit
deadline exit. Never caught because t/400 was in no CI shard until the
stage7-queue-liveness-rejoin shard landed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes out the convert-queue lane for the S3 acceptance queue.
What
Test plan