Skip to content

fix(e2e): build wt from the working tree instead of guessing the binary - #142

Merged
timvw merged 1 commit into
mainfrom
fix/e2e-build-binary
Aug 20, 2026
Merged

fix(e2e): build wt from the working tree instead of guessing the binary#142
timvw merged 1 commit into
mainfrom
fix/e2e-build-binary

Conversation

@timvw

@timvw timvw commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • The e2e runner now builds wt from the working tree when --wt is not given, instead of scanning for a pre-existing bin/wt and then falling back to whatever wt is on PATH
  • Both silent fallbacks are removed: they let a run report on a binary unrelated to the source being tested
  • --wt is unchanged and remains the explicit escape hatch, which is what CI uses

Closes #141

Why

findWtBinary never built anything. It took the first of ./bin/wt, ./wt, bin/wt, then exec.LookPath("wt"). Both fallbacks misfired on a developer machine while verifying #139:

  1. Stale bin/wt. bin/ is gitignored, so a build from earlier in the same worktree survives a rebase. After rebasing onto main (which had picked up fix(hooks): require approval before running hooks from a committed .wt.toml #134), the leftover binary had no trust command, and the two trust scenarios failed with unknown command "trust" for "wt".
  2. PATH fallback. In a fresh worktree with no bin/, the runner silently selected /opt/homebrew/bin/wt — v0.1.33, predating both feat(clone): add wt clone with repo_root + repo_pattern placement #127 and fix(hooks): require approval before running hooks from a committed .wt.toml #134. Four scenarios failed there, since that build has no wt clone either.

In both cases the scenarios and the source were correct; the failures were an artefact of testing the wrong binary. The dangerous case is the inverse: a stale binary that predates a regression passes scenarios that should fail.

CI never hit this because the workflow always passes --wt=bin/wt, so the auto-detection path was exercised only by developers running the suite locally — exactly the people with a stale bin/wt or a Homebrew wt installed.

What changed

  • findWtBinaryresolveWtBinary, returning (path, cleanup, error). Without -wt it builds into os.MkdirTemp and removes it afterwards; with -wt it uses the given path verbatim.
  • The build runs with cmd.Dir set to the module root (go list -m -f {{.Dir}}), so invoking the runner from a subdirectory still builds the right package.
  • main split into run() int so defer cleanup() runs on every exit path, including the failure exit.
  • A -wt path that is missing or a directory is now a clear error up front, rather than an opaque exec failure repeated inside every scenario.
  • The resolved binary is printed along with how it was obtained, so a surprising result is traceable to the artefact that produced it.

Test coverage

New e2e/run_test.go — the orchestrator had no unit tests:

Red-green verified: restored the PATH fallback, confirmed TestResolveWtBinaryIgnoresWtOnPath failed; restored the ./bin/wt candidate scan, confirmed TestResolveWtBinaryIgnoresStaleBinArtifact failed with resolveWtBinary selected the stale bin/ artifact instead of building. Both restored.

Test plan

  • go test ./... — all pass
  • go vet ./... — clean
  • gofmt -l . — clean
  • go run e2e/run.go from a worktree with no bin/251 passed, 0 failed, 13 skipped (previously 4 phantom failures in this exact setup)
  • Codex review loop, 3 rounds → no actionable findings remaining

CI impact

None. .github/workflows/ci.yml already passes --wt=bin/wt / --wt=bin/wt.exe on every platform, so only the local default path changes behaviour.

The runner resolved the binary under test by scanning for a pre-existing
./bin/wt and then falling back to whatever wt was on PATH, and never
built anything. Both let a run silently report on a binary unrelated to
the source being tested: bin/ is gitignored so a stale build survives a
rebase, and the PATH fallback picks up a system-wide install.

That produces failures which are not real, and - more dangerous - passes
that hide a regression, because a binary predating the regression never
exercises it.

Without -wt, build from the module root into a temp directory and test
that. Keep -wt as the explicit escape hatch; CI already passes it, so
only the local default changes. Reject a -wt path that is missing or a
directory up front rather than failing opaquely inside every scenario.

Adds unit tests for the orchestrator, which had none, pinning both
fallbacks as removed.

Closes #141
@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 52.94118% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.14%. Comparing base (e241cf1) to head (8401242).

Files with missing lines Patch % Lines
e2e/run.go 52.94% 24 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #142      +/-   ##
==========================================
+ Coverage   42.48%   43.14%   +0.66%     
==========================================
  Files          34       34              
  Lines        3580     3597      +17     
==========================================
+ Hits         1521     1552      +31     
+ Misses       2059     2045      -14     
Files with missing lines Coverage Δ
e2e/run.go 5.62% <52.94%> (+5.62%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timvw
timvw merged commit af50e8c into main Aug 20, 2026
16 checks passed
@timvw
timvw deleted the fix/e2e-build-binary branch August 20, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

e2e: runner silently tests a stale or unrelated binary when --wt is not given

1 participant