build: make the gate reproducible, pinned and complete - #344
Merged
Conversation
The gate was written down in three places and executed in one, and the one execution left several of the repo's own tools unreached. A Makefile now holds the definition and .github/workflows/gate.yml calls its targets, so `make gate` and the CI job are the same commands rather than two lists that agree until somebody edits one. check-coverage.sh mints its coverage profile with mktemp instead of a fixed relative cover.out. Two runs sharing that path interleaved into a profile that was neither run's: three concurrent runs on one tree reported 15220, 15219 and 15215 statements for a tree whose real total is 4942. It also cds to the repo root, so running it from a subdirectory can no longer measure a subtree and report a pass. The suite now runs under -race, in the same execution the coverage profile comes from rather than a second one. scripts/fuzz.sh gives every fuzz target a bounded search, and scripts/bench.sh runs the benchmarks and refuses a run that measured nothing. golangci-lint is pinned to the release CI resolves today, with the version defined once in the Makefile and read back by the workflow. CheckPath's read-error branch is covered by a unix socket rather than a chmod 0o000 file, so it no longer depends on not being root; a pristine checkout previously failed the 100% gate under euid 0 by exactly one statement. BenchmarkAnchorWalk asked to be compared against a whole compile, which did not exist; BenchmarkCompile_Petstore is that number, alongside marshal, unmarshal and validate benchmarks. BenchmarkAnchorWalk itself had never executed — its fixture path was two levels short and it skipped, silently and with exit 0.
…d-complete-gate # Conflicts: # .github/workflows/gate.yml # CLAUDE.md # scripts/check-coverage.sh
The bounded search finds a real compiler defect in about a second from a cleared
corpus: an anyOf whose only branch is {"type":"null"} lowers to a union that
declares no variants, which irverify rejects and `morphic compile` reports
nothing about. It reproduces on main through the CLI and the harness, so it is
not this change's to fix, but a gate step that reddens on every run until it is
would keep every unrelated PR red. Filed as #416 and quarantined the same way
the search's other known finding was.
FuzzCanonicalWords_Properties comes out of the quarantine. It was held back for
#336, which is closed: CanonicalWords("ℤℤA") is idempotent now, and a 30s search
of that target from a cleared corpus runs 2.5M executions and reports nothing.
A quarantine that outlives its bug reads as though it were still protecting
something while it quietly stops a target from ever running, so the comment now
says plainly that closing the issue is what retires the entry — nothing in the
script can check that the reason still holds.
Prerequisites of a single target are eligible to run concurrently under -j, so `make -j gate` both lost the CI order this file exists to mirror and let `fuzz` write a reproducer into testdata/ while `coverage` was running `go test ./...` over the same tree.
The block introduced itself as the checks CI runs and listed nine commands, none of them the make targets the workflow now calls, plus a `go test ./...` step the gate does not have and a per-package coverage claim the script does not make. It becomes the same one-command pointer CLAUDE.md and micro-compiler-plan.md carry. Departs from the note in this branch that left README to #64/#65: those cover the wider drift, but this change is what makes this particular block false, so it does not get to leave it that way. Closes #415.
The header claimed the same commands and stopped there, which was the weaker half: .NOTPARALLEL is what makes the sequence match too. It also called lint the one step not running a Makefile target, while the step that reads the pin back runs one — it is the one *check* that does not.
The comment said two concurrent runs inflate the total several times over. That was true when it was written and is not now: the block merge landed in #381 counts a block once however many times it appears, and five staggered runs against a shared cover.out all report the same 6132 on this tree. What the unique path still buys is that a run judges the blocks it produced. Two runs sharing one path truncate each other mid-write, and a profile missing the blocks another run had already written reads as a pass when those were the uncovered ones.
Both files said every check in the workflow runs a Makefile target. `lint` does not — the golangci-lint action runs it, at the version the Makefile pins, which is the arrangement gate.yml already spells out and these two flattened into a universal. They also disagreed about whether the order matched; it does, and both now say so.
OmarAlJarrah
added a commit
that referenced
this pull request
Aug 13, 2026
A worker that stopped partway through left zero values in its results, and the comparison reported those as documents compiled differently — which sends a reader looking for a corrupted lowering when nothing lowered at all. Planting an early break made it print that for 74 of 77 specs. The loop now names the case and says so. The length check above it stays: it cannot fail while runCorpus returns either an error or a full-length slice, and it is what keeps the indexing below from panicking should that ever change. Its message no longer claims to be about finishing the corpus, which is the check that was just added. The Engine doc said the race property is pinned "only under -race" and left the reader to wonder whether anything runs it. #344 landed -race in the coverage step, so it does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The gate was written down in three places, executed in one, and that one execution never reached
several of the repo's own tools.
One definition (#90). A
Makefileholds the gate; every check in.github/workflows/gate.ymlruns one of its targets.
make gateand the CI job are the same commands in the same order byconstruction.
.NOTPARALLEL:keeps that true undermake -j, where prerequisites of one target areotherwise eligible to run at once — which would both lose the order and let
fuzzwrite areproducer into
testdata/whilecoveragewas runninggo test ./...over the same tree.A profile that is this run's (#59).
check-coverage.shmints its profile withmktempinsteadof a fixed relative
cover.out.go test -coverprofiletruncates that file when it starts andappends each package's blocks as the package finishes — measured, the file grows from 368 KB two
seconds in to 1.07 MB at the end — so two runs sharing the path truncate each other's output
mid-write. The script also
cds to the repo root, so running it from a subdirectory can no longermeasure a subtree and still report a pass.
What changed since this branch was opened: the inflated totals this section used to quote no
longer occur. #381 landed a merge that counts each block once, so duplicated blocks are absorbed
before they reach the total, and five staggered runs sharing one
cover.outnow all report 6132 onthis tree. What is left is the truncation itself, and that still costs a verdict: a profile missing
blocks another run had already written reads as a pass, because the blocks it lost cannot fail.
A checkout that passes as root (#59).
TestCheckPath_UnreadableFileIsErrorused achmod 0o000file and skipped under euid 0, which left
internal/harness/path.go:34-35uncovered there — apristine checkout failed the 100% gate by exactly one statement for anyone building as root, as a
container commonly does. It now uses a unix socket: refusing to open one for reading is not a
permission check, so no euid bypasses it and the skip is gone.
A pinned linter (#61).
golangci-lint-actionhad noversion:, so it installed whatever itresolved as latest. The pin (
v2.12.2) is defined once in the Makefile and read back by theworkflow, which is also what the two nolint steps need: they ask golangci-lint which linters it is
running, and that only describes the lint step if it is the same build.
make lintwarns — it doesnot fail — when the local binary differs.
Race and fuzz (#77).
-racerides along with the coverage run rather than getting a second fullsuite execution; the profile is identical either way.
scripts/fuzz.shgives each fuzz target abounded
-fuzztimesearch, deriving the target list from the source so a new target is fuzzed themoment it lands, and a finding is uploaded as an artifact so the reproducer survives a red run.
Benchmarks that run (#78).
BenchmarkCompile_Petstoreis the whole-compile numberanchorindex_bench_test.gohas been asking readers to compare against, alongside marshal, unmarshaland validate benchmarks.
scripts/bench.shruns them all once in the gate and refuses a run thatmeasured nothing.
What was already fixed, and is not claimed here
check-coverage.shhas usedset -euo pipefailwith a barego testsincea0ca360, so a failing test cannot be swallowed; and there is no per-package nameparsing left to misname anything — the gate prints profile coordinates.
golangci-lint runexits 0 againstgo 1.26.3withv2.12.2. Only the pin was missing.
FuzzCycleDetectoralready has 2committed reproducers.
narrower; there are 6 benchmark functions now, the newest arriving with perf(compilers/openapi): index the mappings a pointer descends #379.
Deliberately out of scope
FuzzCompileand
FuzzLowerSchemaseed from the whole committed spec corpus viaf.Add, and the twoir/targets seed from adversarial rune and numeric-literal tables. Files under
testdata/fuzz/wouldduplicate those seeds rather than add coverage; that directory is where a found reproducer gets
pinned, which is what
FuzzCycleDetector's two entries are.FuzzLowerSchemais held out of the gate's search, named inscripts/fuzz.shwith the issuethat must close first. A bounded run finds a real compiler defect in about a second from a cleared
corpus: an
anyOfwhose only branch is{"type":"null"}lowers to a union that declares novariants, which
irverifyrejects andmorphic compilereports nothing about. It reproduces onmainthrough both the CLI and the harness, so it is not a build change's to fix — filed as openapi: a single null branch lowers to a union with no variants #416.Its seeds still run on every
go test.criteria.
Test plan
Every claim below was run on this branch with
mainmerged in.#90, the gate.
make gateexits 0 in about 90 s and leaves the worktree clean, running fmt,vet, lint, both nolint checks, build, the coverage counter's own verifier, coverage, fuzz and
benchmarks. The CI step order and
make gate's prerequisite order were compared field by fieldrather than by eye, and match.
#59, the profile. Watching the file
go test -coverprofilewrites, with a cleared test cache:so the append-as-you-go mechanism is real. Three staggered
./scripts/check-coverage.shruns reportall 6132 statements covered.each, as does a serial baseline. Five staggered runs against a copypatched back to a shared
cover.outalso report 6132 — the miscount that motivated this is nowabsorbed by #381's block merge, which is why the summary states the truncation instead.
That a lost block costs a verdict was checked rather than reasoned, by dropping one from a profile
the gate rejects:
#59, root. The socket test is what covers the branch, checked by block rather than by suite:
#77, race. A planted package-level counter written from two goroutines makes
make coveragefail with one
WARNING: DATA RACE; removing it restores green. The race and non-race profiles areidentical — 4322 blocks, 6132 statements, 6132 hit, both ways — which is why one run collects both.
The 300 s timeout is about 30× the slowest package under
-race(internal/harness, 9.96 s).#77, fuzz.
make fuzzreportsfuzzed 4 of 5 target(s), 10s each. Renaming everyfunc Fuzzmakes it exit 1 with
no fuzz target was searched; adding a name to the quarantine that is not atarget makes it exit 1 with
quarantine names FuzzNotAThing, which is not a fuzz target; remove it.#78, benchmarks.
make bench-smokeruns all 6 and each reportsns/op. Plantingb.Skipinone
fails with
these benchmarks skipped, so they measured nothing; renaming everyfunc Benchmarkfails with
no benchmark reported a result.BenchmarkAnchorWalk's fixture path was two directorylevels short, so it skipped — silently, and with exit 0 — and had never executed: on
mainit stillprints
PASSwith nons/op, and the path it reads does not exist. Fixed here rather than filed,because the benchmark step this change adds is what surfaced it; its
b.Skipfcalls are nowb.Fatalffor the same reason.make benchexits 0 in about 35 s and gives the ratioanchorindex_bench_test.goasks a reader tocheck, which is the whole reason
BenchmarkCompile_Petstoreis here:so the
$dynamicAnchorwalk costs about 1/540th of a whole compile — the design's claim that theindex stays a memo rather than being derived at entry, with both halves measured the same way.
#61, pin.
make -s print-lint-versionprintsv2.12.2on one line, which is what the workflowputs in the action's
version:input.make lint GOLANGCI_LINT_VERSION=v9.9.9prints the mismatchwarning and still runs; at the pin it is silent.
The quarantine that was already stale.
FuzzCanonicalWords_Propertieswas held back for #336,which is closed.
CanonicalWords("ℤℤA")is idempotent now, and a 30 s search of that target from acleared corpus runs 2.5M executions and reports nothing, so it comes out of the quarantine. The
script cannot check that a held-back target's reason still holds, so the comment now says plainly
that closing the issue is what retires the entry.
The one step no run has exercised.
upload fuzz findingsonly fires on a red fuzz step, so agreen gate skips it and no run here has executed it. Its condition and glob were checked by reading
rather than by running:
if: failure() && steps.fuzz.outcome == 'failure'reaches it only whenfuzzitself is what failed, and**/testdata/fuzz/**covers the paths Go actually wrote duringthis work.
if-no-files-found: ignorekeeps it from adding a second failure to an already red job.Merging
mainin.check-coverage.shgained an argument form in #381 and amktempcleanuptrap here. The trap expands
$cover_fileat exit, and the argument form reassigns it, so the scriptdeleted the profile its caller passed in — which broke
verify-coverage-count.shoutright, 15checks and every mutation. The profile's source is now settled in one place and the trap is
installed only on the path that mints the file. Checked both ways: the caller's profile survives,
COVER_FILEstill keeps its own, and the verifier reportsall checks passed.Scope
CLAUDE.md,README.mdanddocs/micro-compiler-plan.mdeach carried a copy of the gate's steplist under a promise that it was what CI runs. All three now point at
make gateand theMakefileinstead. README's block also carried a
go test ./...step the gate does not have and a per-packagecoverage claim the script does not make, which is #415.
Closes #59
Closes #61
Closes #77
Closes #78
Closes #90
Closes #415