fix(evergreen-tracks): authenticate Hub tag reads — anonymous pagination now blocked past offset 1000

[sfreudenthaler]

What happened

The daily evergreen-tracks-promote cron failed on 2026-08-11 (run 31481784007) and would have failed every morning from here on. It died in the plan (dry-run) job, before gate/apply, so no tag was moved — the alert's "one track may have moved" caveat did not apply this time.

Root cause

Docker Hub now refuses anonymous pagination past offset 1000:

GET https://hub.docker.com/v2/namespaces/dotcms/repositories/dotcms/tags?page=11&page_size=100
403 {"message":"pagination offset too large for anonymous requests; sign in to page further"}

registry.list_tags() read the Hub API with no auth at all ("Public repos need no auth") and walked every page. dotcms/dotcms has 7,831 tags = 79 pages, so it 403s on page 11.

This is a Docker-side change: the 2026-08-10 run walked all 79 pages in 44s, the 2026-08-11 run died after 6s, and no evergreen-tracks commit landed in between.

Also broken, same read path — the latest promote in cicd_6-release.yml, which would have failed on the next GA cut, and the admin (hold/taint) workflow.

The fix

executor.hub_login() already minted a Hub JWT for deletes, and every calling workflow already had DOCKER_USERNAME/DOCKER_TOKEN for docker/login-action — the read path just never used them. Authenticated requests aren't subject to the offset cap.

• registry.py — one login per walk, JWT on every page request. Auth stays optional so tests and small repos need no creds; a creds-less 403 now raises a message naming the cause instead of a bare HTTPError.
• promote (plan + apply) and release latest-promote — export the two secrets. The admin workflow already did, so it self-heals.
• docker/login-action does not cover the Hub API, which is why being "logged in" wasn't enough.

Secondary fix: the failure was undiagnosable

Both promote steps captured the engine with 2>/dev/null ("drop uv's stderr chatter"), which discarded the whole traceback. CI showed a bare Process completed with exit code 1 and the Slack alert carried no cause — the root cause had to be re-derived by running the tool locally. stderr now goes to a file (it must stay off stdout, which is the plan text apply diffs) and is printed on failure.

Verification

• uv run pytest — 68 passed, including two new tests: every tag request carries the JWT when creds are set, and a creds-less 403 explains itself.
• Live CI run on this branch (31534540024): the plan job walked all 79 pages authenticated in ~29s and produced a real plan — standard -> 26.07.28-01. Cancelled at the approval gate so nothing applied.

Merging this makes tomorrow's 10:00 UTC cron pass and apply that pending standard move.

Closes: #37025

🤖 Generated with Claude Code

https://claude.ai/code/session_01PqQqV1iv96Db8AzBSWTXV4

[claude]

Claude finished @sfreudenthaler's task in 1m 56s —— View job

---

Code Review

I reviewed the diff against origin/main. The core fix is sound: list_tags now attaches a Hub JWT per walk (one login, reused across all pages), auth stays optional, and the anonymous-403 path raises a self-explaining error. Tests cover both the JWT-attached and creds-less-403 cases, and the workflow stderr capture is a real diagnosability win.

New Issues

• 🟡 Medium: .github/actions/core-cicd/evergreen-tracks/tests/test_registry.py:12 — the pre-existing test_list_tags_paginates_and_returns_name_digest no longer clears DOCKER_USERNAME / DOCKER_TOKEN. Before this PR list_tags never read the environment, so the test was env-independent. Now _auth_headers() reads both vars; if either is set in the runner/dev shell, the test triggers hub_login() → a POST https://hub.docker.com/v2/users/login that this test never registers with responses, so responses raises ConnectionError and the test fails. That makes a previously deterministic test flaky for anyone with Docker creds exported. Add monkeypatch.delenv("DOCKER_USERNAME", raising=False) / delenv("DOCKER_TOKEN", raising=False) (accept the monkeypatch fixture like the other two tests do), or use an autouse fixture that isolates these vars for the whole module. Fix this →

• 🟡 Medium: .github/actions/core-cicd/evergreen-tracks/src/evergreen_tracks/registry.py:55-60 — the anonymous-403 branch fires on any creds-less 403 and always attributes it to the offset>1000 cap ("this repo has more tags than that"). If Hub ever returns a 403 for a different reason on page 1 (e.g. rate limiting), the message points at the wrong cause. The suggested remediation (set creds) is still reasonable, so this is only about message accuracy.

  • Assumption: Docker Hub uses 401/404 (not 403) for private/nonexistent repos, so in practice the offset cap is the realistic 403 cause here.
  • What to verify: whether an anonymous read of a valid public repo can 403 for reasons other than the offset cap. If not, this is a non-issue.

Notes (non-blocking, no action needed)

• One-login-per-walk relies on the Hub JWT outliving a full 79-page listing (~29s observed). Fine at current scale; if tag counts grow enough that a walk approaches the JWT lifetime, a mid-walk 403/401 with headers set would fall through to a bare raise_for_status(). Not worth guarding now.
• No circular-import risk from registry importing executor (executor doesn't import registry). ✅

Nothing here blocks merge. The Medium test-isolation item is worth fixing so the suite stays deterministic across environments.
· branch issue-37025-evergreen-hub-auth