test: add a hosted-runner e2e path alongside the self-hosted one - #689
test: add a hosted-runner e2e path alongside the self-hosted one#689mdozhdev wants to merge 4 commits into
Conversation
e2e-tests-local needs Docker and the iOS Simulator on one machine, which GitHub-hosted macOS runners cannot provide, so it runs on a self-hosted Mac as a single shard at ~2h per run. Add the same suite on macos-latest with the regtest stack on an ubuntu runner, reached over Tailscale. Both jobs reuse build-local, so the app is built once. e2e-tests-local is untouched and e2e-status still gates only on it, so the new job reports without blocking until it has earned replacing it. regtest-stack and e2e-tests-remote must not depend on each other: the stack job only finishes once the tests are done, so a dependency either way deadlocks. Requires secrets.TS_AUTHKEY and the suite fixes in synonymdev/bitkit-e2e-tests#207. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lets the new jobs be exercised on this draft without marking it ready and without occupying the self-hosted Mac, which e2e-tests-local would. Revert before merging. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ated Reverts the draft-guard change, which did not work: detect-changes, build-local and e2e-branch are all draft-guarded too, so the new jobs skipped along with their dependencies. Instead park e2e-tests-local, so this PR can be marked ready and exercise e2e-tests-remote without tying up the shared self-hosted Mac for hours. e2e-status treats a skipped shard as a failure, so it will report red until this is restored. Revert before merging. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Greptile SummaryThe PR adds a hosted macOS E2E path that connects over Tailscale to a regtest stack on an Ubuntu runner, while temporarily parking the self-hosted path. The stack lifetime can end before the remote test job’s configured timeout.
Confidence Score: 4/5The workflow should not merge until the regtest stack is guaranteed to remain available for the full remote-test lifetime; the mutable Tailscale action reference is also worth hardening. The stack job can exit at 150 minutes while the remote job remains active for up to 180 minutes, destroying the backend runner and breaking remaining tests. Files Needing Attention: .github/workflows/e2e-tests.yml
|
| Filename | Overview |
|---|---|
| .github/workflows/e2e-tests.yml | Adds the cross-runner E2E workflow, but its 150-minute stack hold can terminate before the 180-minute remote test job and its Tailscale action is not immutably pinned. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Build[build-local] --> Stack[regtest-stack on Ubuntu]
Build --> Tests[e2e-tests-remote on macOS]
Branch[e2e-branch] --> Stack
Branch --> Tests
Stack -->|Tailscale services and LND credentials| Tests
Hold[150-minute hold loop] --> Stack
Tests -->|up to 180 minutes| Result[Remote E2E result]
Reviews (1): Last reviewed commit: "test: TEMPORARY - park e2e-tests-local w..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| deadline=$(( SECONDS + 9000 )) | ||
| while (( SECONDS < deadline )); do |
There was a problem hiding this comment.
When the remote E2E job runs longer than 150 minutes, this deadline ends the hold loop while the remote job can continue for up to 180 minutes. The Ubuntu runner is then destroyed along with the regtest services and Tailscale connection, causing the remaining tests to lose access to LND, Electrum, and bitcoind.
| - uses: tailscale/github-action@v3 | ||
| with: | ||
| authkey: ${{ secrets.TS_AUTHKEY }} | ||
| hostname: regtest-${{ github.run_id }} |
There was a problem hiding this comment.
The new Tailscale step passes TS_AUTHKEY to an action referenced by the mutable v3 tag, so repointing or compromising that tag would expose the tailnet credential to unreviewed action code. Pinning the action to a reviewed commit would make the executed code immutable. How this was verified: Both new jobs provide secrets.TS_AUTHKEY directly to tailscale/github-action@v3.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
The hold loop was sized for a single-spec run and gave up after 150 minutes. The full shard with three attempts takes about three hours, so the stack was torn down 32 seconds before the first of seven "connect ETIMEDOUT ...:43782" errors in attempt 3, and the job still reported success. Raise the deadline and both job timeouts past the worst realistic case, and fail loudly if the stack is ever torn down while tests are running. Also drop @hardware_wallet from the remote shard: ensureTrezorEmulator() shells out to docker compose on the machine running the tests, which has no Docker here. Tracked with the trezor-emu job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 participant
Description
e2e-tests-localneeds Docker and the iOS Simulator on one machine, which GitHub-hosted macOS runners cannot provide. It therefore runs on the self-hosted Mac as a single shard, ~2h per run — against ~20m for the same suite on Android's hosted Linux runners.Adds the same suite on
macos-latest, with the regtest stack on anubuntu-latestrunner reached over Tailscale:regtest-stacke2e-tests-remoteBoth reuse
build-local, so the app is still built once.Nothing existing changes.
e2e-tests-localis untouched, ande2e-statusstill gates only on it — the new job reports but does not block, until it has earned replacing it.The two new jobs must not depend on each other:
regtest-stackonly finishes once the tests are done, so a dependency either way deadlocks.Pairs with
synonymdev/bitkit-e2e-tests#208 — same branch name, so
determine-e2e-branchselects it automatically.Verified
Full
@lightningspec green on this arrangement, 39 min end to end. This PR runs the full 18-tag shard against a remote stack for the first time — that result is the thing to watch.Linked Issues/Tasks