🐛 fix(release): grade every cosign verify-blob invocation, not just one (D-160) - #90
Merged
Conversation
…SIGN, D-160) hack/release/install_cosign_pin_test.sh asserted an EXISTENTIAL property where the guarantee it publishes is UNIVERSAL. Every individual assertion was sound and the conjunction still certified nothing: a second, UNPINNED `cosign verify-blob` appended to any of the three graded files ran the gate to exit 0 — measured before the fix, hack/install.sh rc=0, SECURITY.md rc=0, hack/release/verify-artifacts.sh rc=0. Mechanism: `has_flag` grepped the whole folded extraction so a pinned sibling satisfied it; `one_value`'s `sort -u` collapsed the agreeing values so the D-153 drift comparison saw a consistent file; section 0 PRINTED the invocation count and asserted nothing about it; and 4b/5d's stub-log checks were positive-only greps, the same shape at runtime. SECURITY.md was worse than the other two — it fed only the drift comparison and had no invocation-level grading at all. This is the AUD2-S05 quorum defect one layer out: not a wrong assertion, a wrong quantifier. The property is now universal: every `cosign verify-blob` invocation in every graded file carries --certificate-oidc-issuer, --certificate-identity-regexp and --bundle, each with SECURITY.md's published value. Presence alone is not a pin — `--certificate-identity-regexp ''` carries the flag, matches every Fulcio identity ever issued, and slips past `sort -u` because an empty capture is not a line — so each invocation's own value is compared (WRONG-VALUE, distinct from UNPINNED). Classification fails CLOSED: an occurrence that is neither at command position nor a backticked prose mention is reported UNCLASSIFIABLE, never skipped. "Exactly one invocation per file", the remedy recorded when the finding was filed, was rejected as FALSE on this tree: SECURITY.md legitimately publishes two, one over the archive and one over checksums.txt. Anti-vacuity: sections 2b/2c run eight mutants against each of the three files plus a prose-exemption control, chosen for BRANCH coverage of the new predicate rather than variety of shape — second-unpinned (must red, name the file, quote the call, and produce exactly the two UNPINNED lines so pinned siblings are provably unflagged), second-pinned (must stay green AND raise the extractor's invocation count), off-command-position appended fully pinned so only the classifier can redden it, no-candidates, fold-broken (asserted NOT to also report UNPINNED, since conflating the two sends the maintainer to the wrong file), no-invocations, empty identity regexp, foreign issuer. The runtime twin has its own vacuity control. hack/install.sh, SECURITY.md and hack/release/verify-artifacts.sh are byte-unchanged: all four of their invocations were already pinned. This closes a hole in the checker, not in the checked. The stage was already wired into `task check` and pinned in CHECK_STAGES, so no wiring changed.
Independent review caught that the first D-160 fix REPRODUCED the very defect it was written to close, one scope down. `has_flag` grepped the whole FILE; the replacement grepped the whole LINE. Both are existential — only the scope shrank. UC-01 (P1). One folded line was graded as at most ONE invocation, so cosign verify-blob <both pins> --bundle a.json a && cosign verify-blob --bundle EVIL.json evil began with `cosign verify-blob` (not UNCLASSIFIABLE), carried --bundle (not FOLD-BROKEN), carried both flag strings (not UNPINNED) and yielded the published values (not WRONG-VALUE): GREEN, in same-line, `;`-separated and backslash-folded forms, on all three graded files. The two shell scripts were incidentally caught by the runtime twin at §5d; SECURITY.md is a document and has no runtime twin, so the FULL gate exited 0 there while printing "EVERY cosign verify-blob invocation ... is pinned" — false as printed, on the file the row itself calls "the copy-paste recipe adopters run by hand". D-153's exact incident class. Closed by grading per OCCURRENCE: occurrence_count + a MULTI-OCCURRENCE violation that fails closed on any line carrying two calls. The file-level trigger stays the literal `cosign verify-blob` (widening it would red on prose), but counting INSIDE an already-triggered line matches whole-word `verify-blob`, so a second call spelled `cosign verify-blob` or `"$COSIGN" verify-blob` and chained onto a pinned one is caught too. Whole words, not substrings: a substring count reds on a correctly pinned call whose bundle is named `verify-blob-test.sigstore.json` — a false positive on a plausible filename, sitting on the branch the whole fix rests on. That control is now in the gate. UC-02 (P1). extract_issuer/extract_identity anchor on a greedy `.*`, so the LAST value on a line wins: a hostile `--certificate-identity-regexp ''` placed FIRST was masked by a correct value placed second. The extraction direction favoured the attacker. Closed by flag_value_tokens, which returns EVERY value of every flag on the occurrence; each must equal the published one. UC-03 (P2, the erosion path). A CORRECTLY pinned call written with double quotes was refused as `WRONG-VALUE ... identity=<empty/unparsable> ... (an empty regexp matches every Fulcio identity)` — a true refusal with a false reason, and the repair that message invites is widening the extractor. Both quote styles are now accepted; a BARE value is diagnosed as UNQUOTED-VALUE, a quoting defect, never as a wrong or empty one. A gate that misdiagnoses erodes faster than one that refuses. UC-04 (P3). D-160's "rc=0, rc=0, rc=0" figure is splice-shape-sensitive; the qualifier now sits in the same sentence as the number. Also corrects the recorded remedy in the backlog: "assert the extracted invocation count is exactly 1 per graded file" is factually wrong and would red main on landing, because SECURITY.md legitimately publishes two invocations (`:86` over the archive, `:93` over checksums.txt). Marked do-not-implement so a later lane does not pick it up. The three graded files remain byte-unchanged. Verification: 17 probes against pin_violations sourced verbatim (the harness now self-tests that the function exists, after an earlier version reported GREEN because it did not), 100 in-gate assertions, and 62 external whole-gate mutant rows including all three chaining operators on all three files.
… hole (UC-05..UC-09) UC-05 (P1). occurrence_count compared whitespace-delimited fields to `verify-blob`, found one, and concluded SINGULARITY — when all it had established is that it did not find a second. Absence of evidence read as evidence of absence. `cosign "verify-blob"`, `cosign 'verify-blob'` and `cosign verify-blob""` are the same command to the shell, and each hid a chained unpinned call from the count. Calibrated against the round-2 gate (9040c7a) on all three files at whole-gate level: rc=0, rc=0, rc=0 — worse than first reported. `A && B` short-circuits and `set -e` does not fire on a non-final member of an AND-list, so even the two shell scripts' runtime twin missed it; SECURITY.md has no runtime twin at all. Closed by stripping shell quotes before the field compare. This is A SPELLING PATCH, NOT A TERMINATOR, and it is labelled as one in the code. A structural terminator was sought and ruled out with evidence: refusing any graded line that carries a command separator would red the REAL files, because hack/install.sh and hack/release/verify-artifacts.sh both legitimately end their genuine invocation with an or-else `die` tail. UC-07 (the durable fix, a change of posture rather than of logic). Since no structural terminator exists, completeness is not available and must not be claimed. The PASS banner was printed unconditionally and asserted "EVERY cosign verify-blob OCCURRENCE ... is pinned" plus "a second unpinned call reddens whether it sits on its own line or is chained onto a pinned one" — a universal claim bound to no assertion. Under UC-05 it did not merely overstate: on a green run it printed something FALSE, which is strictly worse than printing nothing, and it is this repo's #1 defect class wearing a different hat. The banner now reports OBSERVED counts per file (occurrences found, classified and graded, prose-exempt, refused), lists only ASSERTED properties each of which is shown to fail on a mutant, and states the NOT-ASSERTED residual in the same breath. When a fifth spelling turns up, the gate is then INCOMPLETE rather than WRONG, and incomplete is recoverable. UC-06 (P3). A trailing comment mentioning verify-blob fired "has 2 verify-blob calls chained on ONE line" — fail-closed but false as printed, since cosign_candidates only drops lines whose FIRST non-space char is `#`. The UC-03 misdiagnosis shape recurring inside the UC-01 fix. The message now says TOKENS, names both causes, and names the correct repair. Excluding trailing comments was rejected: it needs quote-aware parsing, the machinery that produced UC-01/02/03. UC-08 (P3). shellcheck SC1087 x2 silenced with ${flag}. UC-09. Two stage-count figures reported during this lane ("25", then "28") were both wrong and neither came from a committed file — they counted `task: [` lines in a run log, which includes nested task invocations. Measured: Taskfile check: has 21 `- task:` entries, CHECK_STAGES has 21 stage names, they agree, and both files are byte-identical to origin/main. The three graded files remain byte-unchanged. Verification: 109 in-gate assertions; the quoted-spelling shapes added to §2b's spelling loop for all three files; 12 full-gate rows for the quoted spellings plus the filename-collision false-positive control; and a calibration run proving each of those 9 mutants was rc=0 against the round-2 gate, so they are not merely always-red.
TEXT ONLY. Zero non-comment, non-echo lines changed: no predicate, no occurrence_count, no exemption logic, no parsing added. R3-01 (P1). The paragraph 68f7b9d added to make the gate honest contained a sentence that is false: "Any line carrying more than one verify-blob token is REFUSED (MULTI-OCCURRENCE), never graded on its first call." A candidate line that is NOT at command position but contains the backtick-delimited `cosign verify-blob` span is exempted BEFORE occurrence_count runs, so the refusal never reaches it. Measured, spliced into SECURITY.md:57: See `cosign verify-blob`: cosign verify-blob --bundle A.json a && cosign verify-blob --bundle EVIL.json evil occurrence_count = 2 by the definition the banner states two sentences earlier; pin_violations EMPTY; whole gate rc=0 — while printing that sentence, and while reporting the line among those "exempted as backticked prose" when one of the two is a live command. That last part is the UC-06 "false as printed" species this lane itself grades as a defect. Four corrections, at every site that carried the claim: 1. The refusal sentence is scoped to lines AT COMMAND POSITION, and says explicitly that it does not reach a line exempted under residual (1). 2. The exemption is disclosed as residual (1), ahead of the variable/eval/ fragment spellings, because it is the residual that is actually REACHABLE: it keys on the backtick-delimited substring occurring ANYWHERE on the line, not on the line being prose. The unreachable spellings are demoted to (2), with the "not present in these files today" claim verified by grep. 3. OBSERVED no longer calls an exempted line "prose". The branch a line took is a fact; "prose" was an inference, and R3-01 is where that inference is wrong. It now reads "NOT GRADED (exempted because the line contains a backticked mention; such a line may still carry live calls)". 4. The same falsified claim is corrected in the D-160 row, in pin_violations' own header comment, and in the 2c framing comment — 2c grades the backticks-vs-words axis only, and now says so instead of claiming the hatch is narrow outright. Narrowing the exemption is NOT attempted: it needs quote- and markdown-aware parsing, the machinery that produced UC-01/02/03. One claim in my own new text was caught and corrected before commit: "reproduced in a shell script too" was ambiguous enough to read as whole-gate rc=0. Measured both ways — the STATIC defect does reproduce there (candidates=2, invocations=1, pin_violations empty), but the whole gate reds at 4b for an unrelated runtime reason, so the end-to-end green is specific to the document, which has no runtime twin. D-160 now states both halves.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
hack/release/install_cosign_pin_test.shgraded the D-153 cosign signer pinexistentially — some
cosign verify-blobinvocation in each graded file carriesthe pin — where the property the project publishes is universal. A second,
unpinned
cosign verify-blobleft the gate at exit 0 on all three graded files:hack/install.sh,hack/release/verify-artifacts.sh, andSECURITY.md— the last ofwhich is the recipe adopters copy-paste by hand, and which had no invocation-level
grading at all, only the cross-file drift comparison.
That matters here specifically: D-153 records an incident where this project's
published cosign instructions rejected its own releases for every version since
v0.2.0.
How
pinned one (
&&,;,|, backslash-folded) can no longer hide behind its sibling.SECURITY.md's publishedissuer/identity pair, not merely flag presence —
--certificate-identity-regexp ''carries the flag and matches every Fulcio identity.
UNCLASSIFIABLE, never silently skipped.PASS:banner is replaced by a three-part report —OBSERVED (counts, explicitly "not proofs of absence"), ASSERTED (only
mutation-proven properties), NOT ASSERTED (the residuals). The denominator
provably cannot be made complete, so the gate now reports what it classified rather
than asserting what exists.
Scope
The three graded files are byte-unchanged. This closes a hole in the checker,
not in the checked — all four of their invocations were already correctly pinned. No
task wiring changed; the stage was already in
task checkand pinned inCHECK_STAGES.Evidence
OK:assertions.task checkgreen;task changelog-verifygreen.returned APPROVE with no P0/P1.
Records D-160.