From f6df334691a904aa569a9bf07481dd0ad73e71c2 Mon Sep 17 00:00:00 2001 From: Invoker Date: Mon, 17 Aug 2026 03:26:24 +0000 Subject: [PATCH] reflect: sibling-transcript check, template-identical caution, commit-as-you-go Three findings from a reflect pass on Invoker's fleet/a71f331 CI-repair incident (2026-08-17), where at least 4 uncoordinated workflows worked the same failing job over ~7 hours and two prior reflect passes on the same incident crashed non-zero before committing their own findings: - Step 3: check for a sibling reflect's surviving transcript before fanning out lenses, so a crashed sibling's real synthesis work (only visible in its raw JSONL, never committed) is read as prior art instead of re-derived from zero. - Step 3 (lens table): warn History (and any multi-attempt comparison) that commit agreement is not corroboration when both attempts were rendered from the same static template against the same anchor -- directly observed as two commits diffing to nothing but SHA/ timestamp/ID, converging on the same wrong answer. - Step 6: commit each applied skill edit immediately, not batched to the end -- directly observed as the cause of losing a sibling reflect's two drafted CLAUDE.md edits when its closing-summary turn was blocked by an unrelated Stop hook with no further turn. The step 6 addition also folds in the still-unlanded fix from branch `reflect-check-for-unlanded-prior-drafts` (checking git history for an unlanded prior reflect commit before drafting a new one) rather than leaving that finding to bit-rot on its own separate branch, and notes the residual gap it doesn't cover -- an uncommitted sibling draft -- which the new step 3 transcript check closes. Co-Authored-By: Claude Sonnet 5 --- skills/reflect/SKILL.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/skills/reflect/SKILL.md b/skills/reflect/SKILL.md index 4ef2117..1032a3e 100644 --- a/skills/reflect/SKILL.md +++ b/skills/reflect/SKILL.md @@ -80,6 +80,8 @@ A corpus-wide `top_sessions.py` pass is what actually surfaced a real, multi-wee ### 3. Spawn parallel reviewers +Before fanning out, check whether a sibling `reflect` already ran on the same incident and might still hold usable work: `ls ~/.claude/projects/ | grep -F `. This catches a case the step-6 unlanded-draft check (below) structurally can't: a sibling reflect that crashed *before* committing anything (e.g. denied by an unrelated Stop hook, killed mid-turn) leaves no git-log trace at all, but its transcript can still survive worktree teardown if the transcript directory itself wasn't torn down (see the degraded-mode note in step 1). If a sibling transcript exists, read its tail first — a crashed sibling that got as far as drafting synthesis findings (visible in its last assistant turn even with no commit to show for it) is prior art worth extending, not a reason to re-run all 5 lenses from zero. This was directly observed: two independent reflect passes on the same CI-repair incident (Invoker, 2026-08-17) crashed non-zero after completing real synthesis, and a third, unaware of either, re-derived the same facts from scratch before one of the crashed transcripts was found and read. + One message, parallel `Agent` calls (`subagent_type: general-purpose`), each given the transcript path (plus the cost-audit output for the Cost lens, and the git log for the History lens) and a distinct lens: | Lens | Looks for | @@ -90,6 +92,8 @@ One message, parallel `Agent` calls (`subagent_type: general-purpose`), each giv | History | For the files this session touched (especially ones it debugged, reworked, or was corrected on), run `git log --follow -p -- ` and `git blame` on the changed lines *before* judging the session in isolation. Check whether an earlier commit — particularly one tagged `Co-Authored-By: Claude` — already introduced this exact bug, papered over the same symptom, or reworked this same area before. A session that "fixed" something we broke ourselves, or that re-solved a problem a past session already solved, is a stronger and more accountable finding than anything visible in the transcript alone — it means a skill or a fix didn't actually stick. | | Divergent | Whatever the other lenses would miss — an unconventional angle, a blind spot, a pattern that only shows up zoomed out. | +**Caution for History (and any lens comparing multiple independent fix attempts):** two commits agreeing on the same approach is not corroborating evidence if both were generated from the same static prompt template/formula for the same anchor (same failing commit, same job, same rendered brief). Diff the commit message bodies (ignoring SHA/timestamp/task-ID fields) before treating agreement between attempts as a signal the approach is more likely correct — if they're template-identical, the agreement reflects a shared starting brief, not independent reasoning, and two wrong attempts converging on the same wrong answer looks identical to two correct ones agreeing. Directly observed: two Invoker CI-repair attempts from different, uncoordinated workflows landed the identical fix — which History separately confirmed was actually a regression, reversing an already-merged prior fix neither commit cited — with commit bodies diffing to nothing but SHA/timestamp/ID, because both were rendered from one formula against one anchor commit. + Each reviewer returns candidate learnings as: what happened (with a quote/reference), why it matters, and a suggested routing. For any finding that traces back to a point where the user (or a past self) had to intervene and correct the agent mid-task, the reviewer must first ask *how would this stop needing a correction at all* — not just "what skill line would have prevented it." Route toward the highest-value fix that actually applies, in this order: 1. **Categorical elimination** — a different architecture or data structure makes the mistake structurally impossible (e.g. a type that can't represent the invalid state, a function that can't be called in the wrong order, deleting the footgun API entirely). Always check this first; it's the only option that removes the problem rather than catching it. @@ -118,8 +122,10 @@ Present the full Accepted / Backlog / Route-to-automate-me / Rejected list to th ### 6. Apply the approved subset +- Before drafting a new edit, check whether the same lesson was already captured and drafted by an earlier `reflect` pass but never landed — a skill-edit commit is subject to the same disposable-branch/unmerged-PR risk as any other fix, and a lesson that only exists on an unmerged branch is not actually in effect. Search the target repo's history for a prior reflect commit touching the same file/section (e.g. `git log --all --grep="reflect" -i -- `) and check whether it's an ancestor of the branch you're landing on (e.g. `git merge-base --is-ancestor HEAD`). If a matching unlanded draft exists, prefer adapting/extending it — and note its commit/branch in the summary so a reviewer can consider closing the duplicate — over writing a third divergent copy of the same lesson. This only catches drafts that got as far as a commit; it will not find a sibling reflect's edit that was still sitting uncommitted when its session crashed — that case is what the transcript check at the start of step 3 is for. - Trivial edit (a corrected fact, a tightened sentence, a stale example): edit directly. - Substantive edit (a new section, a new principle, more than ~10 lines): write it out in full, matching the target skill's existing structure and tone, and show the diff before it's considered done. +- Commit each edit immediately after writing it, rather than batching every edit until the very end of this step. A later crash, an unrelated Stop-hook block on this task's own closing summary, or any other late-session interruption then loses nothing already applied — only whatever hadn't been written yet. Directly observed: a reflect pass drafted two substantive skill-edit findings, then had its closing summary turn blocked by an unrelated word-count hook with no further turn following; the drafted edits were never committed and were recoverable only because the session transcript happened to survive. - Backlog item: describe the concrete script/check/test to write, but don't write it as part of `reflect` itself — that's separate implementation work once the user confirms it's wanted. - Route-to-`automate-me` item: don't draft it here. Either invoke `automate-me` directly if the user wants it done now, or leave it as a named follow-up in the summary below.