Skip to content

fix(ci): harden verify against flaky packages.microsoft.com apt source (#2975)#2986

Closed
rysweet wants to merge 1 commit into
mainfrom
engineer/steward-ci-github-actions-health-across-all-gov-e06d9e64-1783461322-93de84
Closed

fix(ci): harden verify against flaky packages.microsoft.com apt source (#2975)#2986
rysweet wants to merge 1 commit into
mainfrom
engineer/steward-ci-github-actions-health-across-all-gov-e06d9e64-1783461322-93de84

Conversation

@rysweet

@rysweet rysweet commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Fixes #2975.

Problem (grounded in run data, not assumed)

verify intermittently reds the default branch when the GitHub-hosted
runner's packages.microsoft.com apt repositories serve an invalid InRelease:

E: Failed to fetch https://packages.microsoft.com/.../InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: The repository '... noble InRelease' is no longer signed.

Any apt-get update on the runner then exits 100 and fails the job. Two main
verify runs failed at 20:43 UTC 2026-07-07 from this single cause:

Run Job Failing step Failure
28897370109 pre-commit rust-runner-prepapt-get update -qq exit 100 (NOSPLIT)
28897367724 e2e-dashboard npx playwright install --with-deps (internal apt-get update) Failed to install browsers, exit 100

The next main run (21:00, 28898321760) was green only because the
Microsoft repo happened to be valid again. This is a recurring, non-deterministic
source of spurious red — not a code regression that "self-recovered."

Fix

None of the CI jobs need the Microsoft repos; they install mold and Playwright
browser deps from the Ubuntu archive. Add scripts/ci-harden-apt.sh to strip
any apt source referencing packages.microsoft.com before every apt-get update,
and wire it into both entry points:

  • .github/actions/rust-runner-prep/action.yml — before apt-get update -qq (covers pre-commit and install-real)
  • .github/workflows/verify.yml e2e-dashboard job — before npx playwright install --with-deps

The script is idempotent (safe to re-run; skips already-disabled lines) and
non-fatal (removing an unneeded source can only help). It handles both
sources.list.d/*.list/*.sources (deb822) files and the monolithic
sources.list.

Evidence for merge-ready criteria

1. Tests / behavioral validation. This is a CI-infrastructure change with no
product runtime surface, so gadugi/qa-team scenarios do not apply. The script was
validated with a sandboxed functional + idempotency test (never touching the
host /etc/apt) asserting: MS .list and deb822 .sources files removed;
google-chrome.list and Ubuntu-archive lines preserved verbatim; active MS line
in sources.list disabled exactly once; pre-commented MS lines untouched; RUN 2
reports "nothing to do" (idempotent). All assertions PASS.

2. Docs / changed surfaces. Changed surfaces are internal CI only
(.github/** + scripts/ci-harden-apt.sh). No user-facing runtime, CLI, or API
surface changes — no end-user docs required. Rationale is documented inline in
each workflow step and the script header.

3. Quality audit (SEEK → VALIDATE → FIX).

  • Cycle 1 — SEEK: static analysis. bash -n, shellcheck (clean), actionlint (clean), YAML parse (both files). VALIDATE: pass.
  • Cycle 2 — SEEK: sandboxed functional test. FOUND a real idempotency bug — the sources.list branch re-commented already-disabled MS lines, stacking # disabled-by-ci: prefixes. FIX: restrict the condition + sed to active (non-comment) MS lines only.
  • Cycle 3 — VALIDATE: re-ran static + sandboxed functional + idempotency (double-run) tests → clean; RUN 2 = "nothing to do". Zero remaining findings.

4. CI. Full verify runs on this branch/PR; will confirm 100% green before
marking ready. (CI green here also proves the fix is safe: mold and Playwright
deps still install from the Ubuntu archive after the MS sources are removed.)

6. Focused diff. 3 files, +79/-0: one new script + two one-step wire-ins.
No unrelated edits.

…update (#2975)

The `verify` workflow intermittently reds the default branch when the
GitHub-hosted runner's packages.microsoft.com apt repositories serve an
invalid InRelease file:

  E: ... InRelease  Clearsigned file isn't valid, got 'NOSPLIT'
  E: The repository '... noble InRelease' is no longer signed.

Any `apt-get update` then exits 100 and fails the job. This hit two main
runs at 20:43 UTC 2026-07-07 from a single cause:
  - run 28897370109 (pre-commit)   — rust-runner-prep `apt-get update -qq`
  - run 28897367724 (e2e-dashboard) — `playwright install --with-deps`

It "self-recovered" only because the Microsoft repo happened to be valid on
the next run; it is a recurring, non-deterministic source of spurious red,
not a code regression.

None of the CI jobs need the Microsoft repos (they install mold + Playwright
browser deps from the Ubuntu archive), so add scripts/ci-harden-apt.sh to
strip any apt source referencing packages.microsoft.com before every
`apt-get update`. Wired into:
  - .github/actions/rust-runner-prep (covers pre-commit + install-real)
  - .github/workflows/verify.yml e2e-dashboard (before playwright --with-deps)

The script is idempotent and non-fatal (removing an unneeded source can only
help). Validated with shellcheck, actionlint, and a sandboxed functional +
idempotency test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rysweet

rysweet commented Jul 15, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #4115 (same CI-health steward goal, current instance). #4115 carries the same packages.microsoft.com apt-hardening fix rebased on latest main, plus the regression rigor this PR lacked: a Rust wiring guard (tests/ci_harden_apt_wiring.rs) and a validated+run gadugi-test qa-team scenario (tests/gadugi/ci-harden-apt.{yaml,sh}). Closing to keep one PR per distinct failure (#2975).

@rysweet rysweet closed this Jul 15, 2026
rysweet added a commit that referenced this pull request Jul 16, 2026
…om apt source (#2975) (#4115)

GitHub-hosted runner images ship `packages.microsoft.com` apt sources that
periodically serve an invalid `InRelease` file ("NOSPLIT" / "no longer signed").
When that happens every `apt-get update` exits 100 and turns the default-branch
`verify` workflow red even though nothing in our code changed — a recurring
spurious failure in the CI-health steward's scope (issue #2975).

Fix: a small, idempotent, non-fatal `scripts/ci-harden-apt.sh` strips the
Microsoft apt sources (both `sources.list.d/*.list|*.sources` and monolithic
`sources.list`) BEFORE any `apt-get update`. Wired into every network-fetching
apt consumer:
  - `.github/actions/rust-runner-prep/action.yml` — before the mold `apt-get update`
  - `.github/workflows/verify.yml` (e2e-dashboard) — before `playwright install --with-deps`
(`apt-get clean` needs no hardening — it fetches no repos.) None of our jobs
need the Microsoft repos; they install only Ubuntu-archive packages.

Regression rigor (which the prior stale duplicate PR #2986 lacked):
  - tests/ci_harden_apt_wiring.rs — asserts the script exists + is executable and
    that the hardening step is wired BEFORE the apt consumer in both files.
  - tests/gadugi/ci-harden-apt.{sh,yaml} — outside-in qa-team scenario that
    exercises the script in an isolated fake apt tree (Microsoft-only removal,
    third-party/Ubuntu preservation, idempotency) and runs the Rust guard.

The script takes APT_SOURCES_LIST_D / APT_SOURCES_LIST / CI_HARDEN_APT_SUDO env
overrides so the behaviour is verifiable with no sudo and no network.

Supersedes and dedupes the stale PR #2986 (same steward goal, older instance).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: verify flakes when runner's packages.microsoft.com apt repo is unsigned (NOSPLIT)

1 participant