From 9884e962d304f39571aa331ca9625b4ec2952075 Mon Sep 17 00:00:00 2001 From: Cleber Rangel Date: Tue, 11 Aug 2026 18:38:48 -0300 Subject: [PATCH] =?UTF-8?q?feat(eval):=20a=20muta=C3=A7=C3=A3o=20cara=20do?= =?UTF-8?q?=20eval=20passa=20pela=20guarda,=20e=20o=20STALE=20para=20a=20s?= =?UTF-8?q?u=C3=ADte=20antes=20do=20primeiro=20minuto=20de=20modelo?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Último item da #89. Provar que um caso de eval morde é mutar a cópia instalada da skill e rodar o modelo contra ela — e até aqui isso era feito à mão: copiar o repositório inteiro, editar com `perl -0pi`, conferir por `grep`, rodar a cópia. Dois modos de falha, os dois já observados neste repositório. A expressão que não casa faz a run acontecer contra o texto ORIGINAL e volta verde, que se lê como "o caso não morde" ao preço de uma run paga. E rodar uma cópia do `eval.sh` a partir de /tmp resolve `SKILL_ROOT` para `/`, que é o incidente pelo qual a guarda dentro do `fixture()` existe. `EVAL_MUTATE=''` tira a cópia e o hábito do caminho. A expressão é aplicada à cópia INSTALADA dentro de cada fixture, nunca a este repositório, e passa pelo `scripts/mutate.sh` — a única coisa aqui que compara checksum e aborta. Três decisões que o desenho força, e cada uma nasceu de um defeito: O aborto mora no PREFLIGHT, não no `fixture()`. O `fixture` roda dentro de uma substituição de comando, então um `exit` lá mata o subshell e a suíte segue com um caminho vazio. O preflight roda no processo da suíte, contra uma cópia descartável do arquivo, e sai 2 antes de qualquer fixture existir — 2 e não 1, porque isto é a suíte se recusando a medir, que é resposta diferente de um grader vermelho. A mutação sai em stderr. O `fixture` imprime no stdout o diretório que construiu e os chamadores capturam isso: uma linha de conversa no stdout vira parte do caminho. E o `run_arm` passou a recusar braço sem fixture. Medido, não temido: `( cd "" && pwd )` imprime o diretório atual e sai 0 — ou seja, a falha de qualquer fixture apontaria o modelo para o diretório de onde a suíte foi chamada, que é este repositório. Sete pisos novos (104 no total), e eles cobrem as duas direções: expressão que não casa PARA a suíte; expressão que casa passa (preflight que sempre aborta não é preflight); a edição chega à cópia que o braço lê; o braço mutado ainda começa com árvore limpa; e o `SKILL.md` deste repositório fica byte a byte igual depois de tudo. Quatro invariantes (504 no total) prendem a rota: o `eval.sh` tem de chamar o `mutate.sh`, não pode ter um caminho `perl -0pi` próprio ao lado, e os dois READMEs têm de nomear o interruptor — mecanismo que nenhum README nomeia é mecanismo que a próxima pessoa substitui por uma cópia editada à mão. Os três primeiros foram provados com mutação no próprio `mutate.sh`, num tar descartável: cada um reprova 503/504 quando a rota some. --- README.en.md | 13 +++ README.md | 13 +++ scripts/coherence_test.sh | 37 +++++++++ scripts/eval.sh | 169 +++++++++++++++++++++++++++++++++++++- 4 files changed, 231 insertions(+), 1 deletion(-) diff --git a/README.en.md b/README.en.md index 7669995..e85dbdc 100644 --- a/README.en.md +++ b/README.en.md @@ -203,6 +203,19 @@ against the **original** text, the green comes back, and it reads as "the case does not bite" when nothing was ever mutated. `mutation_test.sh` runs that guard's floors first and stops if they fail. +On the eval side the mutation is no longer made by hand: + +```bash +EVAL_MUTATE='s/\Q\E//' bash scripts/eval.sh yellow-run +``` + +The expression is applied to the copy **installed inside the fixture**, never to +this repository, and through the same `mutate.sh`. An expression that matches +nothing stops the suite **before** the first minute of model time, instead of +returning a green about a mutation that never happened. The old procedure — copy +the whole repository, edit, check by `grep`, run the copy — carried both failure +modes: the edit that does not apply, and the copy's `SKILL_ROOT` resolving to `/`. + `14/14 mutations caught` claims the **suite** notices the edit — not that the model's **behavior** would change. Those are different questions, and when both were finally measured on the same contract they disagreed: rewriting the RED diff --git a/README.md b/README.md index be7bf0a..db69e7e 100644 --- a/README.md +++ b/README.md @@ -200,6 +200,19 @@ direção — uma expressão que não casa faz a run acontecer contra o texto mutado. O `mutation_test.sh` roda os pisos dessa guarda antes de tudo e para se eles falharem. +Do lado do eval a mutação deixou de ser à mão: + +```bash +EVAL_MUTATE='s/\Q\E//' bash scripts/eval.sh yellow-run +``` + +A expressão é aplicada à cópia **instalada dentro do fixture**, nunca a este +repositório, e passa pelo mesmo `mutate.sh`. Uma expressão que não casa para a +suíte **antes** do primeiro minuto de modelo, em vez de devolver um verde sobre +uma mutação que nunca existiu. O procedimento antigo — copiar o repositório +inteiro, editar, conferir por `grep`, rodar a cópia — tinha os dois modos de +falha: a edição que não aplica e o `SKILL_ROOT` da cópia resolvendo para `/`. + `14/14 mutações pegas` afirma que a **suíte** percebe a edição — não que o **comportamento** do modelo mudaria. São perguntas diferentes, e quando as duas foram medidas no mesmo contrato deram respostas opostas: reescrever sozinha a diff --git a/scripts/coherence_test.sh b/scripts/coherence_test.sh index 4692567..414661d 100755 --- a/scripts/coherence_test.sh +++ b/scripts/coherence_test.sh @@ -1693,6 +1693,43 @@ check "phase 4 keeps the catalogued-operation form that keeps the two apart" \ "phase 4's subject changed shape; if it can now be written as refactor(consolidate) the phase 2 mark stops being a mark" +# 16.11 The expensive mutation goes through the guard (#89). Proving that an eval +# case bites means running the model against a mutated copy of the skill, and the +# way that fails is silent and expensive: an expression that matches nothing +# leaves the copy identical, the arm runs against the ORIGINAL text, and the +# green comes back reading as "the case does not bite". It happened once with a +# cheap text mutation and was caught only because a checksum was compared by +# hand; the same accident on an eval case costs a paid run and inverts a +# conclusion. +# +# So the route is asserted, not trusted. `eval.sh` has to hand the edit to +# `mutate.sh` — the one place that compares checksums and aborts — and it must +# not keep a `perl -0pi` path of its own beside it, because a second route is a +# route with no guard. +# +# The call is read with the comment LINES dropped and not with `bash_body`, +# which collapses quoted text to a single space: the path lives inside a quoted +# string, so the body scanner cannot see it by construction. Dropping the +# comments is what this check needs — the prose above must not be able to +# satisfy it. The negative check is the opposite case and uses `bash_body`, +# because `perl -0pi` appears in the comments of this very file. +check "eval.sh routes its mutation through mutate.sh" \ + "$(sed 's/^[[:space:]]*#.*$//' scripts/eval.sh | grep -q -F -- 'scripts/mutate.sh' && echo 0 || echo 1)" \ + "the mutation of the installed copy stopped going through the guard; a stale +expression then produces a green about a mutation that never happened, which is +what #89 was opened about" +check "eval.sh keeps no unguarded mutation path of its own" \ + "$(bash_body < scripts/eval.sh | grep -q -E 'perl +-0[a-z]*i' && echo 1 || echo 0)" \ + "eval.sh edits a file in place with perl again — a second route to the same +edit is a route with no checksum behind it" +for f in README.md README.en.md; do + check "$f documents the mutated-arm switch" \ + "$(LC_ALL=C grep -q -F -- 'EVAL_MUTATE' "$f" && echo 0 || echo 1)" \ + "the mechanism exists and no README names it, so the next person proving a +case bites copies the repository and edits it by hand — which is the procedure +whose two failure modes #89 measured" +done + # 16.10 The attribution ledger stays in step with the instrument (#82). The # document in docs/ records which rules are measured attributable, which are # measured non-attributable TODAY, and which were never measured — and the third diff --git a/scripts/eval.sh b/scripts/eval.sh index 92e5b75..9340d14 100755 --- a/scripts/eval.sh +++ b/scripts/eval.sh @@ -61,18 +61,48 @@ # refuses to mix with the deterministic suites. The vendored tree is built once, # outside the timed run, and copied per fixture. # +# Proving a case BITES is a mutation, and until #89 it was done by hand: copy the +# whole repository somewhere, edit SKILL.md with `perl -0pi`, grep the copy to +# check the edit landed, run the copy's own eval.sh. Two failure modes, both +# observed here. An expression that matches nothing runs the model against the +# ORIGINAL text and comes back green — read as "the case does not bite" when +# nothing was ever mutated, at the price of a paid run. And running a copy of +# this file from /tmp resolves SKILL_ROOT to `/`, which is the incident the +# guard inside fixture() exists for. +# +# EVAL_MUTATE removes the copy and the habit. The expression is applied to the +# INSTALLED copy inside each fixture — never to this repository — through +# scripts/mutate.sh, which aborts when the edit changes nothing. So the suite +# refuses to spend model time on an unmutated arm instead of reporting a green +# about a mutation that never happened. +# +# EVAL_MUTATE='s/\Q\E//' bash scripts/eval.sh yellow-run +# +# It only touches arms that carry the skill: the control arm has no copy to +# mutate, and that asymmetry is the point — the difference between the arms is +# what a mutated run measures. Every application is echoed with the case and the +# arm it landed on, and the diff with it, because a mutated run whose output +# reads like a clean one is evidence nobody can date afterwards — and because a +# greedy expression that swallows a section also moves the checksum, so "it +# changed" and "it changed what I meant" have to be two different readings. +# # Usage: bash scripts/eval.sh [case-name] # Env: EVAL_TURNS (default 20), EVAL_KEEP=1 to keep fixtures, # EVAL_FIXTURE_ROOT to move the fixture tree (the vendored knip lives # under it, in .vendor/, and survives between runs). The result envelope # of each arm is left in -.json there, and it outlives # EVAL_KEEP=0 — evidence that vanishes with the fixture is the defect -# #74 fixed. +# #74 fixed. EVAL_MUTATE='' mutates the installed +# copy of the skill in every arm that has one, through scripts/mutate.sh, +# and EVAL_MUTATE_FILE (default SKILL.md, relative to the installed copy) +# moves that edit to a reference instead. set -uo pipefail SKILL_ROOT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) TURNS=${EVAL_TURNS:-20} ONLY=${1:-} +MUTATE=${EVAL_MUTATE:-} +MUTATE_FILE=${EVAL_MUTATE_FILE:-SKILL.md} pass=0; fail=0; skipped=0 # Fixtures live under $HOME and not in $TMPDIR: a project-scoped skill in @@ -113,6 +143,56 @@ json_field() { run_completed() { [[ $1 == completed ]]; } +# mutate_installed — EVAL_MUTATE against the copy of the +# skill this arm will read, and nothing else. Three properties, each one a +# defect first. +# +# It edits the COPY. The mutation this suite needs is "what would the model do +# if the rule were not there", and the old way of asking it — mutate the working +# tree, or a whole clone of it, and remember to put it back — puts the answer +# and the repository on the same file. A mutation that outlives the run is a +# repository nobody can trust afterwards. +# +# It goes through mutate.sh. The guard there is the reason: an expression that +# matches nothing leaves the copy byte-identical, the arm runs against the +# ORIGINAL text, and the green that comes back reads as "the case does not +# bite". That is a false conclusion bought with a paid run, and it is the exact +# accident #89 was opened about. +# +# It writes to stderr. `fixture` prints the directory it built on stdout and the +# callers capture it, so a single line of chatter on stdout would become part of +# the path and every case would run in a directory that does not exist. +mutate_installed() { # + local dir=$1 arm=$2 target out rc + target="$dir/.claude/skills/codebase-cleanup/$MUTATE_FILE" + out=$("$SKILL_ROOT/scripts/mutate.sh" "$target" "$MUTATE" 2>&1); rc=$? + printf '%s\n' "$out" | sed "s|^| [$arm] |" >&2 + return $rc +} + +# And the same edit, tried once against a throwaway copy of the file before any +# fixture exists. This is where the abort can still be an abort: `fixture` runs +# inside a command substitution, so an `exit` there kills the subshell and the +# suite carries on with an empty path. Here the process is the suite, and a +# mutation that matches nothing stops it before the first minute of model time +# is spent. +mutation_preflight() { + [[ -n $MUTATE ]] || return 0 + local t rc + t=$(mktemp -d) || return 1 + mkdir -p "$t/.claude/skills/codebase-cleanup/$(dirname "$MUTATE_FILE")" + if ! cp "$SKILL_ROOT/$MUTATE_FILE" "$t/.claude/skills/codebase-cleanup/$MUTATE_FILE" 2>/dev/null; then + say "eval: EVAL_MUTATE_FILE=$MUTATE_FILE does not exist in $SKILL_ROOT" + rm -rf "$t"; return 1 + fi + say "== EVAL_MUTATE is set: every arm carrying the skill reads a MUTATED $MUTATE_FILE" + say " expr: $MUTATE" + mutate_installed "$t" preflight; rc=$? + rm -rf "$t" + [[ $rc -eq 0 ]] || say "eval: the expression changed nothing, so every arm would run against the ORIGINAL text — refusing to spend model time on it" + return $rc +} + # --------------------------------------------------------------------------- # vendor_knip — a node_modules tree carrying knip, built once and copied per # fixture. Three measurements decided this shape, and each one shows up in the @@ -345,6 +425,20 @@ EOF # reference being load-bearing. Measured, not imagined — that run is what # sent this comment here. [[ $2 == with-noref ]] && rm -f "$dir/.claude/skills/codebase-cleanup/${NOREF:-references/knip-config.md}" + # The mutation, if the run asked for one, and BEFORE the baseline commit so + # the arm starts on a clean tree — a copy edited afterwards is a tracked + # file modified and uncommitted, Step 0 refuses to start on a dirty tree, + # and the arm would then measure that refusal instead of the rule. + # + # `mutation_preflight` already proved this expression applies, so a failure + # here is a surprise and not the ordinary case: the fixture is destroyed + # rather than handed over, because a fixture whose mutation did not land is + # exactly the arm whose green means nothing. + if [[ -n $MUTATE ]] && ! mutate_installed "$dir" "$1-$2"; then + echo "eval: the mutation did not apply to $dir — refusing to hand over an arm that would run against the original text" >&2 + rm -rf "$dir" + return 1 + fi fi # A real project with node_modules has it ignored, and the protocol reads that # state: "confirm the repo ignores node_modules before the deps category — a @@ -399,6 +493,17 @@ EOF # turn-by-turn log, that is the flag, and this is the number. run_arm() { local dir=$1 name=$2 arm=$3 prompt=$4 + # A fixture that was never built must not become a run. `fixture` returns its + # path on stdout, so any failure inside it hands the caller an EMPTY string — + # and `cd ""` succeeds and stays put, which would point the model at whatever + # directory the suite was started from. That is this repository. Measured, not + # feared: `( cd "" && pwd )` prints the current directory and exits 0. + if [[ -z $dir || ! -f $dir/package.json ]]; then + LAST_JSON=; LAST_ERR=; LAST_RC=2; LAST_TURNS=; LAST_OUT= + LAST_OUTCOME=no_fixture + say " [$name/$arm] no fixture to run in — the arm is skipped, not run in the current directory" + return 1 + fi LAST_JSON="$FIXROOT/$name-$arm.json" LAST_ERR="$FIXROOT/$name-$arm.err" ( cd "$dir" && claude -p "$prompt" --max-turns "$TURNS" --output-format json ) >"$LAST_JSON" 2>"$LAST_ERR" @@ -998,6 +1103,63 @@ self_check() { [[ -e $nrskill/references/knip-config.md ]] && ok "floor: the other references survive the strip" || bad "floor: the other references survive the strip" "a second reference went missing, so the arm is not the controlled manipulation it claims to be" [[ -z $(git -C "$nrdir" status --porcelain 2>/dev/null) ]] && ok "floor: the with-noref arm starts on a clean tree" || bad "floor: the with-noref arm starts on a clean tree" "the strip happened after the baseline commit, so the run sees a tracked file deleted and uncommitted and Step 0 aborts before the protocol starts" + # EVAL_MUTATE, on the four properties that decide whether it is a mechanism or + # a decoration (#89). The whole point of routing the expensive mutation through + # mutate.sh is that a stale edit must STOP the suite instead of buying a green + # about a mutation that never happened — so a floor that only checked the happy + # path would be the same vacuous coverage this file keeps finding in itself. + local mroot="$t/mutroot"; mkdir -p "$mroot" + local saved_mut=$MUTATE saved_mfile=$MUTATE_FILE saved_mfixroot=$FIXROOT + local mdir mskill repo_before repo_after + repo_before=$(cksum < "$SKILL_ROOT/SKILL.md") + + # 1. An expression that matches nothing is refused BEFORE any fixture exists. + MUTATE='s/\Qthis sentence is not in SKILL.md\E/x/' + mutation_preflight >/dev/null 2>&1 \ + && bad "floor: a stale EVAL_MUTATE stops the suite before it spends model time" "the preflight accepted an expression that changes nothing, and every arm would then run against the original text" \ + || ok "floor: a stale EVAL_MUTATE stops the suite before it spends model time" + + # 2. And an expression that applies is accepted, so the guard is not simply + # refusing everything — a preflight that always aborts is no preflight. + MUTATE='s/\QGoal:\E/Objective:/' + mutation_preflight >/dev/null 2>&1 \ + && ok "floor: an EVAL_MUTATE that applies passes the preflight" \ + || bad "floor: an EVAL_MUTATE that applies passes the preflight" "the preflight refused an expression whose target is in SKILL.md, so no mutated run can ever start" + + # 3. The edit reaches the copy the model reads. + FIXROOT=$mroot + mdir=$(fixture floor-mutate with) + FIXROOT=$saved_mfixroot + mskill="$mdir/.claude/skills/codebase-cleanup/SKILL.md" + [[ -n $mdir ]] && LC_ALL=C grep -q 'Objective:' "$mskill" 2>/dev/null \ + && ok "floor: EVAL_MUTATE reaches the installed copy the arm reads" \ + || bad "floor: EVAL_MUTATE reaches the installed copy the arm reads" "the fixture was built from the unmutated text, which is the arm whose green means nothing" + [[ -n $mdir && -z $(git -C "$mdir" status --porcelain 2>/dev/null) ]] \ + && ok "floor: a mutated arm still starts on a clean tree" \ + || bad "floor: a mutated arm still starts on a clean tree" "the mutation landed after the baseline commit, so Step 0 aborts on a dirty tree and the arm measures that instead of the rule" + + # 4. And it never touches this repository. The old procedure edited a copy of + # the whole repo and put it back by hand; a mutation that outlives the run + # is the defect that procedure kept one mistake away. + repo_after=$(cksum < "$SKILL_ROOT/SKILL.md") + [[ $repo_before == "$repo_after" ]] \ + && ok "floor: a mutated run leaves this repository's SKILL.md untouched" \ + || bad "floor: a mutated run leaves this repository's SKILL.md untouched" "the mutation was applied to the source and not to the copy — the working tree now carries an edit nobody asked to keep" + + MUTATE=$saved_mut; MUTATE_FILE=$saved_mfile + rm -rf "$mroot" + + # The empty-fixture guard in run_arm, which is what keeps any of the failures + # above from becoming a model run in the wrong directory: `fixture` reports a + # failure by returning nothing, `cd ""` succeeds, and the arm would then run + # against the repository the suite was started from. + run_arm "" floor-empty with "dá uma faxina nesse projeto" >/dev/null 2>&1 \ + && bad "floor: an arm with no fixture is refused, not run in the current directory" "run_arm accepted an empty path — the next thing it does is start the model" \ + || ok "floor: an arm with no fixture is refused, not run in the current directory" + [[ $LAST_OUTCOME == no_fixture ]] \ + && ok "floor: the refused arm reports its own outcome instead of a grader's red" \ + || bad "floor: the refused arm reports its own outcome instead of a grader's red" "LAST_OUTCOME=$LAST_OUTCOME, so a missing fixture would read as a run that behaved badly" + # The subject search over a history long enough to make the old pipeline lose. # `grep -q` quits at the first match, git takes SIGPIPE, and `set -o pipefail` # turns that into a non-zero pipeline — so the commit that IS there reads as @@ -2031,6 +2193,11 @@ case_report_survives_disclosure() { # case that goes red at minute three because a download did not finish. vendor_knip || exit 1 +# And, when the run asks for a mutation, the proof that the edit applies — also +# before anything paid. Exit 2 and not 1: this is the suite refusing to measure, +# which is a different answer from a grader reporting red. +mutation_preflight || exit 2 + self_check case_yellow_stops_short case_red_does_not_act