fix(e2e): build wt from the working tree instead of guessing the binary - #142
Merged
Conversation
The runner resolved the binary under test by scanning for a pre-existing ./bin/wt and then falling back to whatever wt was on PATH, and never built anything. Both let a run silently report on a binary unrelated to the source being tested: bin/ is gitignored so a stale build survives a rebase, and the PATH fallback picks up a system-wide install. That produces failures which are not real, and - more dangerous - passes that hide a regression, because a binary predating the regression never exercises it. Without -wt, build from the module root into a temp directory and test that. Keep -wt as the explicit escape hatch; CI already passes it, so only the local default changes. Reject a -wt path that is missing or a directory up front rather than failing opaquely inside every scenario. Adds unit tests for the orchestrator, which had none, pinning both fallbacks as removed. Closes #141
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #142 +/- ##
==========================================
+ Coverage 42.48% 43.14% +0.66%
==========================================
Files 34 34
Lines 3580 3597 +17
==========================================
+ Hits 1521 1552 +31
+ Misses 2059 2045 -14
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wtfrom the working tree when--wtis not given, instead of scanning for a pre-existingbin/wtand then falling back to whateverwtis onPATH--wtis unchanged and remains the explicit escape hatch, which is what CI usesCloses #141
Why
findWtBinarynever built anything. It took the first of./bin/wt,./wt,bin/wt, thenexec.LookPath("wt"). Both fallbacks misfired on a developer machine while verifying #139:bin/wt.bin/is gitignored, so a build from earlier in the same worktree survives a rebase. After rebasing ontomain(which had picked up fix(hooks): require approval before running hooks from a committed .wt.toml #134), the leftover binary had notrustcommand, and the two trust scenarios failed withunknown command "trust" for "wt".PATHfallback. In a fresh worktree with nobin/, the runner silently selected/opt/homebrew/bin/wt— v0.1.33, predating both feat(clone): addwt clonewith repo_root + repo_pattern placement #127 and fix(hooks): require approval before running hooks from a committed .wt.toml #134. Four scenarios failed there, since that build has nowt cloneeither.In both cases the scenarios and the source were correct; the failures were an artefact of testing the wrong binary. The dangerous case is the inverse: a stale binary that predates a regression passes scenarios that should fail.
CI never hit this because the workflow always passes
--wt=bin/wt, so the auto-detection path was exercised only by developers running the suite locally — exactly the people with a stalebin/wtor a Homebrewwtinstalled.What changed
findWtBinary→resolveWtBinary, returning(path, cleanup, error). Without-wtit builds intoos.MkdirTempand removes it afterwards; with-wtit uses the given path verbatim.cmd.Dirset to the module root (go list -m -f {{.Dir}}), so invoking the runner from a subdirectory still builds the right package.mainsplit intorun() intsodefer cleanup()runs on every exit path, including the failure exit.-wtpath that is missing or a directory is now a clear error up front, rather than an opaque exec failure repeated inside every scenario.Test coverage
New
e2e/run_test.go— the orchestrator had no unit tests:wtonPATH(decoy script; skipped on Windows)bin/wt— the other half of e2e: runner silently tests a stale or unrelated binary when --wt is not given #141-wtpath rather than falling back-wt-wtpath verbatimRed-green verified: restored the
PATHfallback, confirmedTestResolveWtBinaryIgnoresWtOnPathfailed; restored the./bin/wtcandidate scan, confirmedTestResolveWtBinaryIgnoresStaleBinArtifactfailed withresolveWtBinary selected the stale bin/ artifact instead of building. Both restored.Test plan
go test ./...— all passgo vet ./...— cleangofmt -l .— cleango run e2e/run.gofrom a worktree with nobin/— 251 passed, 0 failed, 13 skipped (previously 4 phantom failures in this exact setup)CI impact
None.
.github/workflows/ci.ymlalready passes--wt=bin/wt/--wt=bin/wt.exeon every platform, so only the local default path changes behaviour.