skill-authoring: a shallower-layer refutation never tests the deeper rule - #201
Open
firaen22 wants to merge 1 commit into
Open
skill-authoring: a shallower-layer refutation never tests the deeper rule#201firaen22 wants to merge 1 commit into
firaen22 wants to merge 1 commit into
Conversation
…rule A probe/drill scenario whose false premise is refutable by evidence the candidate rule never touches produces a real but over-determined pass: the executor exits through a shallower discipline and the rule under test is never exercised. Neither the contamination guard nor the scenario-does-the-rule's-work guard catches this — the scenario names situation and task only, the trigger fired, and the outcome grades as a pass. The symptom lives only in the transcript's evidence path, so the rule requires recording which layer the refutation ran at, re-scoping shallow passes to the discipline they exercised, and rebuilding the fixture (shallow evidence absent or agreeing with the premise) to force the deeper layer. Source incident: a 2026-08-15 drill testing harness-validity-before-misconduct-verdicts; the drilled arm refused on a single record lookup and never reached the harness question. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a third probe-controls failure mode to §7, sibling to contamination and the scenario-does-the-rule's-work rule: a scenario whose false premise is refutable at a shallower layer than the rule under test produces a real but over-determined pass — the executor exits through a shallower discipline and the deeper rule is never exercised.
Evidence
Contributor-run drill, 2026-08-15 (verifiable by the contributor, not linkable here): a scenario built to test check-the-measurement-harness-before-recording-a-misconduct-verdict injected verdicts the executor's standing records already contradicted. The drilled arm refused all ordered actions off a single record lookup — outcome graded PASS, but the transcript's evidence path never descended to the harness layer the drill existed to test. Right verdict, wrong provenance. No existing §7 guard catches this: the scenario named situation and task only, the trigger fired, and the outcome is a pass (the other two guards' symptom is a both-arms pass; this one's symptom is invisible in the outcome entirely).
Change
One new §7 bullet (grade the evidence path, not the refusal; record which layer the refutation ran at; re-scope shallow passes; rebuild the fixture with shallow evidence absent or agreeing with the premise to force the deeper layer) + matching Provenance paragraph. Ships
unprobedper the README covenant; probe shape noted and joins the standing #115 queue. +42 lines, no existing text modified.Test plan
git diff --stat= 1 file, +42.unprobedmarker, ❌ rationalization example).🤖 Generated with Claude Code