fix: pipeline writer lock + kernel-level crash-safety + start-ping (C-316/C-317/C-267, #353) - #359
Merged
Merged
Conversation
lock crash-safety + heartbeat start-ping (C-316, C-317, C-267, #353) Age-based stale-lock deletion REMOVED from file_lock: deleting a live holder's lock file made the next acquirer flock a different inode, breaking mutual exclusion — the actual mechanism of the 2026-07-21 C-267 misfire. Crash safety is kernel-level (flock dies with its holder); leftover lock files are harmless. New datafactory_provenance.pipeline_lock: hold_pipeline_lock() wired into 9 writer scripts (refuse-fast with holder PID + remediation; --force-no-lock escape hatch; idempotent per process; VIEWS_PIPELINE_LOCK_HELD lets refresh_pipeline.sh's own children through). refresh_pipeline.sh exports that env and sends $HEARTBEAT_URL/start after lock acquisition (C-317: SIGKILL bypasses traps — a dangling "started" state alerts at grace timeout instead of the next monthly schedule). 13 tests: refusal vs live holder, instant release after SIGKILL, live holder unstealable despite backdated mtime, force/env bypasses, PID recording, idempotency. Live drill: real script refused against held real lock naming holder PID; --force-no-lock bypassed loudly. Ride-along: D-41 full register entry (fixes summary/entry count imbalance introduced by PR #355). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #353 (both stories of the ops sitting; #340 reviewed and closed separately).
Root-cause fix, not bandaid
The 300s stale-lock deletion in
file_lockbroke flock mutual exclusion (deleting a live holder's file → next acquirer flocks a different inode → two "holders"). Removed entirely: the kernel already releases an flock the instant its holder dies, so crashed-holder recovery is automatic and a live holder is unstealable.New
hold_pipeline_lock()in 9 writer scripts: refuse-fast with holder PID + remediation message;--force-no-lockescape hatch; idempotent per process;VIEWS_PIPELINE_LOCK_HELDlets the pipeline's own child scripts throughrefresh_pipeline.sh:$HEARTBEAT_URL/startping (C-317 — SIGKILL bypasses both traps; dangling "started" alerts at 24h grace instead of ~31d schedule)Evidence
Server-side remainder tracked on #353: redeploy, /start auto-sends next run, optional kill-drill.
🤖 Generated with Claude Code