fix(ci): harden verify against flaky packages.microsoft.com apt source (#2975)#2986
Closed
rysweet wants to merge 1 commit into
Closed
Conversation
…update (#2975) The `verify` workflow intermittently reds the default branch when the GitHub-hosted runner's packages.microsoft.com apt repositories serve an invalid InRelease file: E: ... InRelease Clearsigned file isn't valid, got 'NOSPLIT' E: The repository '... noble InRelease' is no longer signed. Any `apt-get update` then exits 100 and fails the job. This hit two main runs at 20:43 UTC 2026-07-07 from a single cause: - run 28897370109 (pre-commit) — rust-runner-prep `apt-get update -qq` - run 28897367724 (e2e-dashboard) — `playwright install --with-deps` It "self-recovered" only because the Microsoft repo happened to be valid on the next run; it is a recurring, non-deterministic source of spurious red, not a code regression. None of the CI jobs need the Microsoft repos (they install mold + Playwright browser deps from the Ubuntu archive), so add scripts/ci-harden-apt.sh to strip any apt source referencing packages.microsoft.com before every `apt-get update`. Wired into: - .github/actions/rust-runner-prep (covers pre-commit + install-real) - .github/workflows/verify.yml e2e-dashboard (before playwright --with-deps) The script is idempotent and non-fatal (removing an unneeded source can only help). Validated with shellcheck, actionlint, and a sandboxed functional + idempotency test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Owner
Author
|
Superseded by #4115 (same CI-health steward goal, current instance). #4115 carries the same |
rysweet
added a commit
that referenced
this pull request
Jul 16, 2026
…om apt source (#2975) (#4115) GitHub-hosted runner images ship `packages.microsoft.com` apt sources that periodically serve an invalid `InRelease` file ("NOSPLIT" / "no longer signed"). When that happens every `apt-get update` exits 100 and turns the default-branch `verify` workflow red even though nothing in our code changed — a recurring spurious failure in the CI-health steward's scope (issue #2975). Fix: a small, idempotent, non-fatal `scripts/ci-harden-apt.sh` strips the Microsoft apt sources (both `sources.list.d/*.list|*.sources` and monolithic `sources.list`) BEFORE any `apt-get update`. Wired into every network-fetching apt consumer: - `.github/actions/rust-runner-prep/action.yml` — before the mold `apt-get update` - `.github/workflows/verify.yml` (e2e-dashboard) — before `playwright install --with-deps` (`apt-get clean` needs no hardening — it fetches no repos.) None of our jobs need the Microsoft repos; they install only Ubuntu-archive packages. Regression rigor (which the prior stale duplicate PR #2986 lacked): - tests/ci_harden_apt_wiring.rs — asserts the script exists + is executable and that the hardening step is wired BEFORE the apt consumer in both files. - tests/gadugi/ci-harden-apt.{sh,yaml} — outside-in qa-team scenario that exercises the script in an isolated fake apt tree (Microsoft-only removal, third-party/Ubuntu preservation, idempotency) and runs the Rust guard. The script takes APT_SOURCES_LIST_D / APT_SOURCES_LIST / CI_HARDEN_APT_SUDO env overrides so the behaviour is verifiable with no sudo and no network. Supersedes and dedupes the stale PR #2986 (same steward goal, older instance). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2975.
Problem (grounded in run data, not assumed)
verifyintermittently reds the default branch when the GitHub-hostedrunner's
packages.microsoft.comapt repositories serve an invalidInRelease:Any
apt-get updateon the runner then exits 100 and fails the job. Twomainverifyruns failed at 20:43 UTC 2026-07-07 from this single cause:pre-commitrust-runner-prep→apt-get update -qqe2e-dashboardnpx playwright install --with-deps(internalapt-get update)Failed to install browsers, exit 100The next
mainrun (21:00,28898321760) was green only because theMicrosoft repo happened to be valid again. This is a recurring, non-deterministic
source of spurious red — not a code regression that "self-recovered."
Fix
None of the CI jobs need the Microsoft repos; they install
moldand Playwrightbrowser deps from the Ubuntu archive. Add
scripts/ci-harden-apt.shto stripany apt source referencing
packages.microsoft.combefore everyapt-get update,and wire it into both entry points:
.github/actions/rust-runner-prep/action.yml— beforeapt-get update -qq(coverspre-commitandinstall-real).github/workflows/verify.ymle2e-dashboardjob — beforenpx playwright install --with-depsThe script is idempotent (safe to re-run; skips already-disabled lines) and
non-fatal (removing an unneeded source can only help). It handles both
sources.list.d/*.list/*.sources(deb822) files and the monolithicsources.list.Evidence for merge-ready criteria
1. Tests / behavioral validation. This is a CI-infrastructure change with no
product runtime surface, so gadugi/qa-team scenarios do not apply. The script was
validated with a sandboxed functional + idempotency test (never touching the
host
/etc/apt) asserting: MS.listand deb822.sourcesfiles removed;google-chrome.listand Ubuntu-archive lines preserved verbatim; active MS linein
sources.listdisabled exactly once; pre-commented MS lines untouched; RUN 2reports "nothing to do" (idempotent). All assertions PASS.
2. Docs / changed surfaces. Changed surfaces are internal CI only
(
.github/**+scripts/ci-harden-apt.sh). No user-facing runtime, CLI, or APIsurface changes — no end-user docs required. Rationale is documented inline in
each workflow step and the script header.
3. Quality audit (SEEK → VALIDATE → FIX).
bash -n,shellcheck(clean),actionlint(clean), YAML parse (both files). VALIDATE: pass.sources.listbranch re-commented already-disabled MS lines, stacking# disabled-by-ci:prefixes. FIX: restrict the condition +sedto active (non-comment) MS lines only.4. CI. Full
verifyruns on this branch/PR; will confirm 100% green beforemarking ready. (CI green here also proves the fix is safe:
moldand Playwrightdeps still install from the Ubuntu archive after the MS sources are removed.)
6. Focused diff. 3 files, +79/-0: one new script + two one-step wire-ins.
No unrelated edits.