Make the correctness evidence reproducible in CI, and pin the shareability guarantee

[jonyoder]

Test and CI only. No behaviour change, no tag.

The correctness evidence behind the last few performance changes — a 4,007-resolution equivalence transcript, production differentials, race repros — was run ad hoc on a laptop. None of it ran in CI. There is already a concrete instance of that failure mode: go-python-packaging shipped a race-safety fix whose regression test was inert in its own CI, because CI ran plain go test; a revert would have passed green.

1. go test -race ./... in CI

2. The shareability guarantee is pinned

PackageMetadata.SupportsPython's doc was relaxed in 0.6.0 to tell callers a parsed version.Version is safe to share. That was true and it was backed by prose. index/shared_version_test.go now shares one parsed value across eight goroutines and covers both of the paths 0.6.0 fixes separately:

• packable versions, which never touch part.Parts (the packed integer key);
• the padParts fallback, which about a quarter of distinct real versions take under an entirely separate safety argument.

Fixture strings are chosen for the shape the hazard needs — trailing zero on the shared operand, shared operand strictly shorter — and TestSharedVersionFixturesReachBothPaths asserts that shape rather than trusting the comment, including an allocation probe that proves which of the two paths each entry actually took (packed allocates nothing; padParts copies).

Discrimination, verified

Each subtest was run against a scratch copy with go-python-packaging pinned back to v0.5.0. All seven report WARNING: DATA RACE there and are clean at v0.6.0.

test	v0.5.0	v0.6.0
TestSharedParsedVersionIsRaceFree/packable	DATA RACE	clean
.../unpackable-local	DATA RACE	clean
.../unpackable-epoch	DATA RACE	clean
.../unpackable-long-release	DATA RACE	clean
TestSupportsPythonSharedTargetIsRaceFree/packable-less-than	DATA RACE	clean
.../packable-greater-than	DATA RACE	clean
.../unpackable-epoch	DATA RACE	clean

⚠️ Three corrections that fell out of that

Pinned to v0.5.0 with the padding race live, the entire existing suite passed under -race. Nothing in it shared a parsed version.Version between goroutines:

• resolver/concurrency_test.go was named in pep440set/verpos_race_test.go as "the test that reaches it, through cross-group Compare in ranking". It does not, and it says so now. It deliberately shares nothing parsed, and MockIndex.Versions re-parses from stored string keys on every call, so no fixture edit could have changed that.
• RSFIndex.Versions does the same. No index in this module can hand two goroutines an aliased value.
• concurrentIndex's doc said its Requires-Python was there "so Specifiers.Check runs". It is not: the provider converts Requires-Python to a version set in interpreterDependency and never calls Check, and Specifiers.Check has no non-test caller outside SupportsPython — which is an API for external callers, not a step of a resolve.

TestConcurrentResolutionsShareOneParsedInterpreter is added and labelled as non-discriminating: it shares the one thing a real caller shares, and measured against v0.5.0 it reports zero races, for the three reasons above. It is kept for the caller-visible shape and as a tripwire if any of them stops holding — not counted as coverage.

3. The equivalence transcript runs in CI

TestResolutionTranscriptMatchesGolden converts the two-build diff into a build-against-golden diff on the committed excerpt: 125 cases per PR, compared byte for byte against resolver/testdata/excerpt-transcript.txt. It fails rather than skips when the fixture is missing, and CI fails if it skips anyway — the same pattern as the producer-output step, added for the same reason. The CI guard matches the run's mode, not just the summary's shape, so a nightly that lost GPR_TRANSCRIPT_FULL fails instead of silently re-running the PR subset.

writeTranscript is shared with TestDumpResolutions, so the CI check and the env-gated full-snapshot tool cannot drift into rendering the same resolution differently.

⚠️ The full-snapshot differential genuinely cannot run in CI — it needs the 981 MB snapshot and two builds, and took hours. That is stated in the file rather than papered over. What is reproducible is the same property at excerpt scale: weaker in coverage, identical in sensitivity.

4. Anti-vacuity

The minimums are on discriminating cases, not cases compared: resolutions that pinned a dependency (Deep), that activated an extra, and failures that derived a conflict. Failures whose whole report is "this package is absent" are counted as Vacuous and credited to nothing — that class is exactly what an earlier differential counted as coverage.

Current excerpt, per-PR set: transcript[fast]: 125 cases, 45 discriminating (31 deep, 14 derivations); 72 shallow, 2 extras, 8 vacuous, 0 unparseable, 0 timeouts.

Verified by mutation, not by argument:

mutation	caught
Newest.Less inverted (oldest-first ranking)	yes — flask 3.1.3 → flask 0.11, and extras fell to 1, tripping the minimum
markerWithPre rendering changed (report text only)	yes — sha256 mismatch on pandas numpy<1.26

5. CI runtime, measured on the runner

step	before	after
Test (go test ./...)	10 s	22 s
Race (go test -race ./...)	—	120 s
Equivalence transcript	—	14 s
build job total	28 s	178 s

The -race step is 80% of the increase. To hold the equivalence half to 14 s, 20 excerpt packages that account for nearly all of the sweep's cost are held out of the per-PR run (95 s full vs 6 s subset, locally) and covered by a new nightly job against a second golden file.

Second commit: fixes from review

⚠️ The first commit's full golden file contained a TIMEOUT line for hypothesis — a committed assertion about wall clock, i.e. exactly the fragility the by-name exclusion exists to avoid, reintroduced through the nightly, and it would have made that job red on a runner. hypothesis turns out not to finish at all on this excerpt: it exceeds a ten-minute per-case bound. It moves to a new unboundedExcerptPackages list excluded from both modes, because no deadline makes its entry a fact about the resolver rather than about the machine. transcriptStats now counts timeouts and asserts there are none — the check that would have caught this.

⚠️ The first commit's cost figures were wrong: "88 s for all 139 packages" had measured 137, with the two worst already excluded. Corrected throughout, and the CHANGELOG now quotes runner numbers rather than laptop ones.

Also corrected: the format header was written only by the golden test, leaving TestDumpResolutions — the tool that produced the v1 transcripts the header exists to distinguish — the one unmarked artifact; minCases was described as "roughly half" when it is 72%; firstDifference reported a differing case header as its own context.

Notes

• Failure reports are stored as lines=N bytes=N sha256=… plus the conclusion line. Full text would make the golden file 1.15 MB — larger than the RSF fixture it derives from — of which 1.0 MB is nine reports enumerating whole version sets. The hash keeps every byte of sensitivity; GPR_TRANSCRIPT_REPORTS=full is there for investigating a mismatch.
• A # gpr-transcript v2 header prevents a new transcript being silently diffed against the v1 files behind the published 0.6.0 figures.

---

Third commit: second round of review findings

Two were defects in the checks themselves:

• ⚠️ caseTimeout was 10 minutes — exactly go test's default -timeout — and none of the three ci.yml steps that run this test passes -timeout. A genuinely hung case would trip the package timeout and panic the binary before writeTranscript could record a TIMEOUT and before stats.Timeouts was evaluated. The assertion added in the second commit, credited there as "the check that would have caught hypothesis", could not fire in the PR job at all; only the nightly (-timeout 60m) was unaffected. Now 2 minutes — ~6× above the worst measured case, ~5× below the package default.
• ⚠️ The ERROR class was counted nowhere and asserted nowhere — the exact defect stats.Timeouts had just been added to fix. It is reachable (resolver.go:127, :205, :212) and incremented only Cases. With minDeepResolutions at 15 against an actual 31, half the deep cases could have degraded into ERROR and every minimum would still have passed. Now counted, asserted zero, and the classes are asserted to sum to Cases so this cannot recur a third time.

One overclaim, measured false: sharedInterpreter / sharedInterpreterConstraint claimed their spellings were "the entire test" and that >=3.8 would have made it green at v0.5.0. It is green either way — Specifiers.Check is never reached from a resolve. Same overclaim this branch corrected in concurrentIndex one commit earlier, two files over.

Figures corrected against fresh measurement: the full-report golden is 1,465,675 bytes, not 1.15 MB, and its largest report is ipykernel at 293 KB, not prompt-toolkit at 227 KB; ipykernel/ipython are 19.9 s and 18.7 s ("38 s each" was the pair); two stale 21s; "roughly fifteen" used the transcript's ratio, not the repository's; the CHANGELOG quoted the first commit's build job (178 s) rather than the head's (168 s).

Three rationales were wrong despite correct conclusions: the ci.yml summary grep catches a run that never executed, not one that compared nothing; the nightly's concurrency block serialises rather than deduplicates; and fixtureDepth = 4 is a dependency-depth criterion, so "none of that criteria mentions dependency depth" was false.

Also documented: the race detector collapses repeated stacks, so running the whole index package against a broken pin shows one failing subtest, not four. That is deduplication, not three uncovered paths — each was confirmed individually.

---

Fourth commit: third round of review findings

⚠️ The caseTimeout margin was a laptop margin. Two minutes was justified as "about six times above the worst measured case" — 120 s over a 21.5 s figure measured on an M4 Max, weighed against a bound that has to hold on a runner. Scaled by the observed ratio (go test ./... is 7.1 s locally, 23 s on the runner), the worst case is nearer 70 s and the real margin was 1.7×, on a check whose failure mode is exactly the machine-dependent TIMEOUT the previous commit removed.

Fixed from both ends rather than by trading one margin against the other:

• caseTimeout is 5 minutes, ~4.3× the runner-adjusted worst case;
• ci.yml passes an explicit -timeout 20m on all three invocations that reach this test, so the package budget no longer depends on go test's default and can no longer collide with caseTimeout by coincidence. Five minutes is still under that default, so a bare local go test ./... is armed too.

Newly documented because it is load-bearing and was not: the PR job survives this bound only because slowExcerptPackages holds the expensive cases out of the fast set — it runs under -race, which costs a further ~4×.

The cost table is re-measured whole, under one methodology (cold single-package resolve, fresh index each — an upper bound on marginal sweep cost), rather than patched two entries at a time. The previous note called the figures "lower bounds" and said they "do not sum to the 95 s total"; both were backwards. Cold consistently exceeds in-sweep, and the twenty sum to 106 s against a 95–100 s sweep. All twenty figures moved. hypothesis has now been measured against 20 s, 60 s, 3 min and 10 min bounds and exceeded all four, against 21.5 s for the slowest package that finishes.

Three claims corrected for overreach rather than error: the Errors/Timeouts assertions were justified as catching cases that would "vanish", but the golden comparison catches them too — what the counters add is a message naming the class, plus cover for a golden regenerated while cases were timing out (which is how one got committed). And the sum assertion was credited with catching the uncounted ERROR class; it is a runtime check that would have sat green through both commits where ERROR was uncounted and empty. It catches the co-occurrence, not the omission.

Smaller: Errors cannot be a cancelled context (caseCtx.Err() is tested first, routing those to Timeouts) — the reachable case is the solver giving up; the sum failure message now explains a negative count rather than printing one; String() documents that extras overlaps deep by design; three stale "order of magnitude" phrases; and the CHANGELOG no longer implies its per-step figures reconcile to the second.

---

Fifth commit: fourth round of review findings

Two behavioural defects, both verified fixed by running them:

• ⚠️ writeTranscript continued past a timeout, so N hung cases cost N × caseTimeout and the guard's reach depended on how many things hung. At 5 minutes exactly one hang fits inside a bare go test's 10-minute package budget; at the 2 minutes it replaced, four did — so raising the constant made the local path strictly worse, and the compensating -timeout had been added only to ci.yml. Hangs also arrive in classes, not singly (14 of the 125 fast cases already sit above a 200 ms bound), so a regime shift would hang many at once and panic the binary rather than fail with a message. The sweep now halts on the first timeout. Verified with a 50 ms bound: stops at case 6 of 125, sum still balances.
• ⚠️ A failing regeneration still wrote its golden. assertTranscriptIsDiscriminating uses t.Errorf, and the GPR_UPDATE_TRANSCRIPT branch ran unconditionally afterwards — so a regeneration whose sweep timed out wrote the TIMEOUT line straight into the golden, signalled only by exit status. That is how one got committed two commits ago. It now refuses to write when the run has already failed. Verified: golden checksum unchanged after a failing regeneration attempt.

Figures corrected, all of them mine:

claim	was	is
-race cost	"~4×"	4.5× idle, 7.7× loaded
ipykernel if moved to fast set	275 s (guard would not fire)	310–530 s vs a 300 s guard — straddling it
adding slow set to every PR	"four or five times"	~13× (the old figure added a laptop delta to a runner baseline)
cold table vs sweep	106 s vs 95–100 s total	106 s vs 89–104 s marginal, over-count 2–19%
CI runtime	single samples	ranges over 6 baseline + 4 branch runs

The -race understatement inverted its own paragraph's conclusion, and the "four or five times" error is the exact laptop-vs-runner mistake the caseTimeout note calls out by name, made a hundred lines above it.

String() now renders the partition as an explicit sum with the overlay outside it — its previous doc claimed extras sat "after the semicolon with the other non-class counts", of which there were none. CI greps updated to match.

A note on the evidence itself

While this PR was being written, concurrent work on the same machine destroyed three of five baseline rounds belonging to a different measurement running alongside it. The surviving files still existed, still parsed, and would still have produced a plausible-looking median. Nothing in the output would have said it was wrong.

That is the sharper version of this PR's thesis: an ad hoc measurement is not merely unrepeatable, it is undetectably corruptible. Every figure quoted here was re-derived afterwards from freshly created, uniquely-prefixed directories, and the CI timings come from the GitHub API rather than from any local file. A CI check has neither problem — it reruns on demand and derives its inputs from the committed tree. That reasoning is now recorded at the top of equivalence_test.go.

---

Sixth commit: fifth round of review findings

⚠️ A real bug: the t.Failed() write guard only caught poisonings a counter notices. GPR_TRANSCRIPT_REPORTS=full inlines every failure report and moves no counter at all — so the run passes and writes a golden 20× larger (240 KB fast; by renderFailure's own figure 1.47 MB in full mode, larger than the RSF fixture, in a public repo).

It is reachable by following the instructions, not by misusing them: renderFailure's doc and the golden-mismatch message both tell the reader to re-run with GPR_TRANSCRIPT_REPORTS=full to investigate, and doing so with GPR_UPDATE_TRANSCRIPT still exported from the previous command silently rewrites the baseline from a green run. Now refused explicitly — verified blocked, golden checksum unchanged. The previous commit's claim that the counters "PREVENT a poisoned golden" is corrected to what they actually do: they prevent the ones a counter can see.

⚠️ The "~13×" and "310–530 s straddles the guard" figures scaled a compile-free workload by a compile-inclusive ratio. The 2.4–3.2× came from go test ./..., which on the runner compiles every package; the marginal cost of twenty more resolutions contains no compile at all. That is the same species of error as the draft it replaced.

The like-for-like number was available all along — the equivalence step runs with -v, so CI logs the test body:

	local	runner	ratio
equivalence test body	5.6–6.0 s	11.2–14.1 s (4 runs)	1.9–2.5×

Recomputed: the extra sweep is 169–260 s and the repository's test time goes to roughly 190–280 s — an order of magnitude, stated as such rather than to two figures. The straddle band becomes 184–414 s against the 300 s guard, so the conclusion survives — and that is the point. It is now the conclusion that doesn't depend on where in the band you land, rather than one produced by picking a convenient multiplier, which both previous drafts did, in opposite directions.

Halting turned one hang into five errors, four naming the wrong cause. A hang at case 6 of 125 reported "the excerpt or the corpus has shrunk" plus three collapsed minimums — none of which happened. Coverage minimums are now skipped when the sweep halted, and the timeout error says so. Verified: one diagnosis, not five.

Also: writeTranscript's doc comment had been left attached to the transcriptRun struct inserted inside it, so godoc read the function's documentation as the type's — a comment describing something other than what it's attached to, in a commit series about exactly that. And the halt rationale claimed it "costs nothing real" while the paragraph above it argued the class size is the diagnostic signal; halting does lose that, and now says so.

CI range widened to 143–186 s over five runs; per-step ranges re-checked across all five and unchanged.

---

Rebased onto 11da678 (gpp v0.7.0) — and the discrimination re-verified

main moved under this PR: #46 rewrote pep440set's comparison ladder and bumped go-python-packaging to v0.7.0, and #47 dated the release. Branch collapsed to a single commit and rebased; conflicts in CHANGELOG.md and pep440set/verpos_race_test.go resolved by keeping both sides rather than picking one — main's updated mechanism note plus this branch's correction, and my entries under [Unreleased] with main's dated [0.7.0] section preserved below them.

⚠️ The fail-without-fix evidence was gathered against a different implementation than the one this now merges onto, so it was re-run rather than assumed. On the rebased branch, pinned back to gpp v0.5.0:

test	subtests	result
TestSharedParsedVersionIsRaceFree	4	7 races each — all still discriminate
TestSupportsPythonSharedTargetIsRaceFree	3	2 races each — all still discriminate

Also re-checked at v0.7.0: the packable/unpackable allocation guards still hold, so both the packed-key path and the padParts fallback are still genuinely covered; and the goldens still match, which independently says the pep440set rewrite changed no rendered version-set text.

The index package does not import pep440set, so the rewritten verPos ladder is not on the path these tests reach — now checked rather than asserted.

⚠️ A self-inconsistency of mine that survived five review rounds: concurrency_test.go listed TestConcurrentResolutionsShareOneParsedInterpreter among "the tests that DO cover it", while that test's own doc says at length that it measures zero races at v0.5.0 and does not cover the hazard. Corrected.

Figures now carry their load

The full sweep measured 231 s during this rebase against the 95–110 s in the comment — same machine, load average 63.

	idle M4 Max	same machine, load avg 63
fast (125 cases)	5.6–6.0 s	21.6 s
full (145 cases)	95–110 s	231 s
ratio	16–19×	10.7×

Both the absolute times and the ratio move with load, so the split decision now rests on the order of magnitude — stable across both columns — rather than on a multiplier that isn't. The nightly job is noted as the only direct measurement of that sweep on a runner; everything in the test file is local or extrapolated.

index/types.go's "this module requires v0.6.0" is updated, and that paragraph now says why it is no longer only a paragraph.