Skip to content

perf(evergreen-tracks): plan promotions from filtered reads instead of a 79-call full tag scan - #37030

Merged
sfreudenthaler merged 2 commits into
issue-37025-evergreen-hub-authfrom
issue-37028-narrow-reads
Aug 12, 2026
Merged

perf(evergreen-tracks): plan promotions from filtered reads instead of a 79-call full tag scan#37030
sfreudenthaler merged 2 commits into
issue-37025-evergreen-hub-authfrom
issue-37028-narrow-reads

Conversation

@sfreudenthaler

@sfreudenthaler sfreudenthaler commented Aug 11, 2026

Copy link
Copy Markdown
Member

Stacked on #37027 (the Hub auth outage fix). Review that one first — this branches off it, so the diff here is only the efficiency work.

Problem

Moving one floating tag paged the entire repo. dotcms/dotcms has 7,832 tags and Hub caps page_size at 100 (250 and 1000 are silently ignored), so every run was 79 calls / ~45s — against a Hub budget of 180 requests / 60s per IP. The promoter has no 429 handling, so a single rate-limited page is a hard failure and, since #37027, a Slack page.

Most of that listing is noise: 1,946 *SNAPSHOT branch builds, 677 *_lts. Promotion needs a few GA CalVer tags and three marker names.

Approach

Same technique the in-cluster reconciler already uses in dotCMS/infrastructure-as-code (reconcile.py::resolve_track) — Hub's name= substring filter:

  • one call per requested track → the <track> digest, and its <track>_hold marker for free (both match the track name as a substring)
  • one call per month of history → that month's GA releases and their <version>_tainted markers, which share the version's month prefix

A month is one page (6–28 tags).

--tracks is now resolved before any registry read, so the release pipeline's latest-only promote no longer pays to resolve standard and trailing.

Measured against the live registry

path before after
daily standard,trailing 79 calls 4
--tracks latest (release pipeline) 79 calls 2
all three tracks 79 calls 5
CI plan job wall-clock 29s 2.9s

Plans are byte-identical to the full scan's — standard -> 26.07.28-01, digest c2526c56…, matching the authenticated full-scan run 31534540024. Verified in CI on this branch: 31544087963 (read registry state in 4 filtered call(s)), cancelled at the approval gate so nothing applied.

The correctness argument worth reviewing

Everything rests on fetching months newest-first and contiguously.

planner.plan's forward-only guard only bites when a track already sits on something newer than the newest eligible release. Any such version lives in a month at or above the one the walk stopped on — so it has necessarily already been read. A track on something older resolves to "move forward", which is the same decision the full listing produces. Nothing here can move a track backwards that the full scan would have held.

The tempting "optimization" that breaks this is starting the walk at the threshold cutoff month instead of today (why read months newer than the cutoff?). test_promote_narrow_reads_still_refuse_to_move_a_track_backwards fails on exactly that mutation.

I originally also ported the reconciler's push-month lookup to name each track's current version. Mutation testing showed removing it changed no promotion outcome — the argument above is why it was dead weight — so it's gone, along with the Tag.month plumbing it needed.

Fallback

Returns to the full listing when the walk window holds nothing promotable, so a broken read path surfaces as the expensive right answer rather than a quiet "no moves" stall — which is precisely how tracks would silently stop advancing. admin deliberately keeps the full scan: rare, human-triggered, and it addresses arbitrarily old versions.

Tests

73 pass (68 existing unchanged + 5 new). The new ones were mutation-tested — each of these breaks at least one:

mutation caught by
skip months newer than the cutoff backwards-move + walk-back tests
never walk back a month 3 tests
drop the empty-window fallback fallback test
stop month-filtering 3 tests
ignore --tracks when reading scoping test

ruff is not wired into CI for this package; I checked my changes are net-neutral against the baseline anyway (21 findings before and after, same rules, shifted line numbers).

Closes: #37028

🤖 Generated with Claude Code

https://claude.ai/code/session_01PqQqV1iv96Db8AzBSWTXV4

This PR fixes: #37028

@claude

claude Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Claude finished @sfreudenthaler's task in 33s —— View job


I'll analyze this and get back to you.

sfreudenthaler and others added 2 commits August 12, 2026 12:04
…ll tag scan

Moving one floating tag paged the entire repo: 7,832 tags, page_size capped at
100 by Hub, so 79 calls and ~45s against a budget of 180 requests/60s per IP.
The promoter has no 429 handling, so one rate-limited page is a hard failure
and a Slack page.

Hub's `name=` substring filter narrows it to what promotion actually needs:

  * one call per requested track -> the `<track>` digest, and its `<track>_hold`
    marker for free (both match the track name as a substring)
  * one call per month of history -> that month's GA releases AND their
    `<version>_tainted` markers, which share the version's month prefix

A month is one page (6-28 tags). Measured against dotcms/dotcms: the daily
standard+trailing promote is 4 calls, `--tracks latest` is 2, all three is 5 --
down from 79, and the plan is byte-identical to the full scan's
(`standard -> 26.07.28-01`, same digest).

`--tracks` is now resolved BEFORE any registry read, so the release pipeline's
latest-only promote no longer pays to resolve standard and trailing.

Correctness rests on fetching months newest-first and CONTIGUOUSLY. planner's
forward-only guard only bites when a track sits on something NEWER than the
newest eligible release, and any such version is in a month at or above the one
the walk stopped on -- already read. A track on something older resolves to
"move forward", the same decision the full listing makes. A mutation that
starts the walk at the threshold cutoff instead of today (skipping newer
months) is caught by test_promote_narrow_reads_still_refuse_to_move_a_track_backwards.

Falls back to the full listing when the walk window holds nothing promotable,
so a broken read path surfaces as the expensive right answer instead of a quiet
"no moves" stall. `admin` keeps the full scan: rare, human-triggered, and it
addresses arbitrarily old versions.

Closes: #37028

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqQqV1iv96Db8AzBSWTXV4
`_jwt`'s lru_cache is process-global, so it outlives a test. Proven with two
tests using identical credentials and different tokens: the second saw the
first's JWT. Harmless today (only one test authenticates) but a trap for the
next auth test, and the same class of non-determinism as the ambient-creds
issue already fixed in conftest — so it belongs in the same fixture.

The cache itself stays: in production the CLI is one-shot, so one login per
process is exactly right, and it is what keeps a full-listing fallback from
re-authenticating on all 79 pages.

Closes: #37028

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqQqV1iv96Db8AzBSWTXV4

@wezell wezell left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving this rather than #37027

@sfreudenthaler
sfreudenthaler merged commit cd0eb0e into issue-37025-evergreen-hub-auth Aug 12, 2026
31 checks passed
@sfreudenthaler
sfreudenthaler deleted the issue-37028-narrow-reads branch August 12, 2026 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

evergreen-tracks: promoter does a 79-call full tag scan to move one tag — port the reconciler's name= month filter

2 participants