Skip to content

build: make the gate reproducible, pinned and complete - #344

Merged
OmarAlJarrah merged 8 commits into
mainfrom
build/reproducible-and-complete-gate
Aug 13, 2026
Merged

build: make the gate reproducible, pinned and complete#344
OmarAlJarrah merged 8 commits into
mainfrom
build/reproducible-and-complete-gate

Conversation

@OmarAlJarrah

@OmarAlJarrah OmarAlJarrah commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

The gate was written down in three places, executed in one, and that one execution never reached
several of the repo's own tools.

One definition (#90). A Makefile holds the gate; every check in .github/workflows/gate.yml
runs one of its targets. make gate and the CI job are the same commands in the same order by
construction. .NOTPARALLEL: keeps that true under make -j, where prerequisites of one target are
otherwise eligible to run at once — which would both lose the order and let fuzz write a
reproducer into testdata/ while coverage was running go test ./... over the same tree.

A profile that is this run's (#59). check-coverage.sh mints its profile with mktemp instead
of a fixed relative cover.out. go test -coverprofile truncates that file when it starts and
appends each package's blocks as the package finishes — measured, the file grows from 368 KB two
seconds in to 1.07 MB at the end — so two runs sharing the path truncate each other's output
mid-write. The script also cds to the repo root, so running it from a subdirectory can no longer
measure a subtree and still report a pass.

What changed since this branch was opened: the inflated totals this section used to quote no
longer occur. #381 landed a merge that counts each block once, so duplicated blocks are absorbed
before they reach the total, and five staggered runs sharing one cover.out now all report 6132 on
this tree. What is left is the truncation itself, and that still costs a verdict: a profile missing
blocks another run had already written reads as a pass, because the blocks it lost cannot fail.

A checkout that passes as root (#59). TestCheckPath_UnreadableFileIsError used a chmod 0o000
file and skipped under euid 0, which left internal/harness/path.go:34-35 uncovered there — a
pristine checkout failed the 100% gate by exactly one statement for anyone building as root, as a
container commonly does. It now uses a unix socket: refusing to open one for reading is not a
permission check, so no euid bypasses it and the skip is gone.

A pinned linter (#61). golangci-lint-action had no version:, so it installed whatever it
resolved as latest. The pin (v2.12.2) is defined once in the Makefile and read back by the
workflow, which is also what the two nolint steps need: they ask golangci-lint which linters it is
running, and that only describes the lint step if it is the same build. make lint warns — it does
not fail — when the local binary differs.

Race and fuzz (#77). -race rides along with the coverage run rather than getting a second full
suite execution; the profile is identical either way. scripts/fuzz.sh gives each fuzz target a
bounded -fuzztime search, deriving the target list from the source so a new target is fuzzed the
moment it lands, and a finding is uploaded as an artifact so the reproducer survives a red run.

Benchmarks that run (#78). BenchmarkCompile_Petstore is the whole-compile number
anchorindex_bench_test.go has been asking readers to compare against, alongside marshal, unmarshal
and validate benchmarks. scripts/bench.sh runs them all once in the gate and refuses a run that
measured nothing.

What was already fixed, and is not claimed here

Deliberately out of scope

  • No committed seed corpora for the four bare fuzz targets. They are not seedless: FuzzCompile
    and FuzzLowerSchema seed from the whole committed spec corpus via f.Add, and the two ir/
    targets seed from adversarial rune and numeric-literal tables. Files under testdata/fuzz/ would
    duplicate those seeds rather than add coverage; that directory is where a found reproducer gets
    pinned, which is what FuzzCycleDetector's two entries are.
  • FuzzLowerSchema is held out of the gate's search, named in scripts/fuzz.sh with the issue
    that must close first. A bounded run finds a real compiler defect in about a second from a cleared
    corpus: an anyOf whose only branch is {"type":"null"} lowers to a union that declares no
    variants, which irverify rejects and morphic compile reports nothing about. It reproduces on
    main through both the CLI and the harness, so it is not a build change's to fix — filed as openapi: a single null branch lowers to a union with no variants #416.
    Its seeds still run on every go test.
  • The compiler bug above is not fixed here. openapi: a single null branch lowers to a union with no variants #416 carries the reproducer and the two acceptance
    criteria.

Test plan

Every claim below was run on this branch with main merged in.

#90, the gate. make gate exits 0 in about 90 s and leaves the worktree clean, running fmt,
vet, lint, both nolint checks, build, the coverage counter's own verifier, coverage, fuzz and
benchmarks. The CI step order and make gate's prerequisite order were compared field by field
rather than by eye, and match.

#59, the profile. Watching the file go test -coverprofile writes, with a cleared test cache:

t= 2s size=368457
t= 9s size=757569
t=18s size=1068748   (final)

so the append-as-you-go mechanism is real. Three staggered ./scripts/check-coverage.sh runs report
all 6132 statements covered. each, as does a serial baseline. Five staggered runs against a copy
patched back to a shared cover.out also report 6132 — the miscount that motivated this is now
absorbed by #381's block merge, which is why the summary states the truncation instead.

That a lost block costs a verdict was checked rather than reasoned, by dropping one from a profile
the gate rejects:

with the uncovered block   Coverage gate failed: 1 of 6132 statements uncovered   exit 1
with it truncated away     Coverage gate passed: all 6131 statements covered      exit 0

#59, root. The socket test is what covers the branch, checked by block rather than by suite:

                                              stmts hits
path.go:34.17,36.4   with the test               1    1
path.go:34.17,36.4   with the test skipped       1    0

#77, race. A planted package-level counter written from two goroutines makes make coverage
fail with one WARNING: DATA RACE; removing it restores green. The race and non-race profiles are
identical — 4322 blocks, 6132 statements, 6132 hit, both ways — which is why one run collects both.
The 300 s timeout is about 30× the slowest package under -race (internal/harness, 9.96 s).

#77, fuzz. make fuzz reports fuzzed 4 of 5 target(s), 10s each. Renaming every func Fuzz
makes it exit 1 with no fuzz target was searched; adding a name to the quarantine that is not a
target makes it exit 1 with quarantine names FuzzNotAThing, which is not a fuzz target; remove it.

#78, benchmarks. make bench-smoke runs all 6 and each reports ns/op. Planting b.Skip in
one
fails with these benchmarks skipped, so they measured nothing; renaming every func Benchmark
fails with no benchmark reported a result. BenchmarkAnchorWalk's fixture path was two directory
levels short, so it skipped — silently, and with exit 0 — and had never executed: on main it still
prints PASS with no ns/op, and the path it reads does not exist. Fixed here rather than filed,
because the benchmark step this change adds is what surfaced it; its b.Skipf calls are now
b.Fatalf for the same reason.

make bench exits 0 in about 35 s and gives the ratio anchorindex_bench_test.go asks a reader to
check, which is the whole reason BenchmarkCompile_Petstore is here:

BenchmarkAnchorWalk-12          432920      2756 ns/op      3104 B/op      75 allocs/op
BenchmarkCompile_Petstore-12       805   1490671 ns/op   1348592 B/op   14363 allocs/op

so the $dynamicAnchor walk costs about 1/540th of a whole compile — the design's claim that the
index stays a memo rather than being derived at entry, with both halves measured the same way.

#61, pin. make -s print-lint-version prints v2.12.2 on one line, which is what the workflow
puts in the action's version: input. make lint GOLANGCI_LINT_VERSION=v9.9.9 prints the mismatch
warning and still runs; at the pin it is silent.

The quarantine that was already stale. FuzzCanonicalWords_Properties was held back for #336,
which is closed. CanonicalWords("ℤℤA") is idempotent now, and a 30 s search of that target from a
cleared corpus runs 2.5M executions and reports nothing, so it comes out of the quarantine. The
script cannot check that a held-back target's reason still holds, so the comment now says plainly
that closing the issue is what retires the entry.

The one step no run has exercised. upload fuzz findings only fires on a red fuzz step, so a
green gate skips it and no run here has executed it. Its condition and glob were checked by reading
rather than by running: if: failure() && steps.fuzz.outcome == 'failure' reaches it only when
fuzz itself is what failed, and **/testdata/fuzz/** covers the paths Go actually wrote during
this work. if-no-files-found: ignore keeps it from adding a second failure to an already red job.

Merging main in. check-coverage.sh gained an argument form in #381 and a mktemp cleanup
trap here. The trap expands $cover_file at exit, and the argument form reassigns it, so the script
deleted the profile its caller passed in — which broke verify-coverage-count.sh outright, 15
checks and every mutation. The profile's source is now settled in one place and the trap is
installed only on the path that mints the file. Checked both ways: the caller's profile survives,
COVER_FILE still keeps its own, and the verifier reports all checks passed.

Scope

CLAUDE.md, README.md and docs/micro-compiler-plan.md each carried a copy of the gate's step
list under a promise that it was what CI runs. All three now point at make gate and the Makefile
instead. README's block also carried a go test ./... step the gate does not have and a per-package
coverage claim the script does not make, which is #415.

Closes #59
Closes #61
Closes #77
Closes #78
Closes #90
Closes #415

The gate was written down in three places and executed in one, and the one
execution left several of the repo's own tools unreached.

A Makefile now holds the definition and .github/workflows/gate.yml calls its
targets, so `make gate` and the CI job are the same commands rather than two
lists that agree until somebody edits one.

check-coverage.sh mints its coverage profile with mktemp instead of a fixed
relative cover.out. Two runs sharing that path interleaved into a profile that
was neither run's: three concurrent runs on one tree reported 15220, 15219 and
15215 statements for a tree whose real total is 4942. It also cds to the repo
root, so running it from a subdirectory can no longer measure a subtree and
report a pass.

The suite now runs under -race, in the same execution the coverage profile comes
from rather than a second one. scripts/fuzz.sh gives every fuzz target a bounded
search, and scripts/bench.sh runs the benchmarks and refuses a run that measured
nothing. golangci-lint is pinned to the release CI resolves today, with the
version defined once in the Makefile and read back by the workflow.

CheckPath's read-error branch is covered by a unix socket rather than a chmod
0o000 file, so it no longer depends on not being root; a pristine checkout
previously failed the 100% gate under euid 0 by exactly one statement.

BenchmarkAnchorWalk asked to be compared against a whole compile, which did not
exist; BenchmarkCompile_Petstore is that number, alongside marshal, unmarshal
and validate benchmarks. BenchmarkAnchorWalk itself had never executed — its
fixture path was two levels short and it skipped, silently and with exit 0.
…d-complete-gate

# Conflicts:
#	.github/workflows/gate.yml
#	CLAUDE.md
#	scripts/check-coverage.sh
The bounded search finds a real compiler defect in about a second from a cleared
corpus: an anyOf whose only branch is {"type":"null"} lowers to a union that
declares no variants, which irverify rejects and `morphic compile` reports
nothing about. It reproduces on main through the CLI and the harness, so it is
not this change's to fix, but a gate step that reddens on every run until it is
would keep every unrelated PR red. Filed as #416 and quarantined the same way
the search's other known finding was.

FuzzCanonicalWords_Properties comes out of the quarantine. It was held back for
#336, which is closed: CanonicalWords("ℤℤA") is idempotent now, and a 30s search
of that target from a cleared corpus runs 2.5M executions and reports nothing.
A quarantine that outlives its bug reads as though it were still protecting
something while it quietly stops a target from ever running, so the comment now
says plainly that closing the issue is what retires the entry — nothing in the
script can check that the reason still holds.
Prerequisites of a single target are eligible to run concurrently under -j, so
`make -j gate` both lost the CI order this file exists to mirror and let `fuzz`
write a reproducer into testdata/ while `coverage` was running `go test ./...`
over the same tree.
The block introduced itself as the checks CI runs and listed nine commands, none
of them the make targets the workflow now calls, plus a `go test ./...` step the
gate does not have and a per-package coverage claim the script does not make. It
becomes the same one-command pointer CLAUDE.md and micro-compiler-plan.md carry.

Departs from the note in this branch that left README to #64/#65: those cover the
wider drift, but this change is what makes this particular block false, so it
does not get to leave it that way. Closes #415.
The header claimed the same commands and stopped there, which was the weaker
half: .NOTPARALLEL is what makes the sequence match too. It also called lint the
one step not running a Makefile target, while the step that reads the pin back
runs one — it is the one *check* that does not.
The comment said two concurrent runs inflate the total several times over. That
was true when it was written and is not now: the block merge landed in #381
counts a block once however many times it appears, and five staggered runs
against a shared cover.out all report the same 6132 on this tree.

What the unique path still buys is that a run judges the blocks it produced. Two
runs sharing one path truncate each other mid-write, and a profile missing the
blocks another run had already written reads as a pass when those were the
uncovered ones.
Both files said every check in the workflow runs a Makefile target. `lint` does
not — the golangci-lint action runs it, at the version the Makefile pins, which
is the arrangement gate.yml already spells out and these two flattened into a
universal. They also disagreed about whether the order matched; it does, and
both now say so.
@OmarAlJarrah
OmarAlJarrah merged commit 3d40d98 into main Aug 13, 2026
1 check passed
@OmarAlJarrah
OmarAlJarrah deleted the build/reproducible-and-complete-gate branch August 13, 2026 19:29
OmarAlJarrah added a commit that referenced this pull request Aug 13, 2026
A worker that stopped partway through left zero values in its results, and the
comparison reported those as documents compiled differently — which sends a
reader looking for a corrupted lowering when nothing lowered at all. Planting an
early break made it print that for 74 of 77 specs. The loop now names the case
and says so.

The length check above it stays: it cannot fail while runCorpus returns either an
error or a full-length slice, and it is what keeps the indexing below from
panicking should that ever change. Its message no longer claims to be about
finishing the corpus, which is the check that was just added.

The Engine doc said the race property is pinned "only under -race" and left the
reader to wonder whether anything runs it. #344 landed -race in the coverage
step, so it does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment