perf(evergreen-tracks): plan promotions from filtered reads instead of a 79-call full tag scan - #37030
Merged
sfreudenthaler merged 2 commits intoAug 12, 2026
Conversation
Contributor
|
Claude finished @sfreudenthaler's task in 33s —— View job I'll analyze this and get back to you. |
…ll tag scan
Moving one floating tag paged the entire repo: 7,832 tags, page_size capped at
100 by Hub, so 79 calls and ~45s against a budget of 180 requests/60s per IP.
The promoter has no 429 handling, so one rate-limited page is a hard failure
and a Slack page.
Hub's `name=` substring filter narrows it to what promotion actually needs:
* one call per requested track -> the `<track>` digest, and its `<track>_hold`
marker for free (both match the track name as a substring)
* one call per month of history -> that month's GA releases AND their
`<version>_tainted` markers, which share the version's month prefix
A month is one page (6-28 tags). Measured against dotcms/dotcms: the daily
standard+trailing promote is 4 calls, `--tracks latest` is 2, all three is 5 --
down from 79, and the plan is byte-identical to the full scan's
(`standard -> 26.07.28-01`, same digest).
`--tracks` is now resolved BEFORE any registry read, so the release pipeline's
latest-only promote no longer pays to resolve standard and trailing.
Correctness rests on fetching months newest-first and CONTIGUOUSLY. planner's
forward-only guard only bites when a track sits on something NEWER than the
newest eligible release, and any such version is in a month at or above the one
the walk stopped on -- already read. A track on something older resolves to
"move forward", the same decision the full listing makes. A mutation that
starts the walk at the threshold cutoff instead of today (skipping newer
months) is caught by test_promote_narrow_reads_still_refuse_to_move_a_track_backwards.
Falls back to the full listing when the walk window holds nothing promotable,
so a broken read path surfaces as the expensive right answer instead of a quiet
"no moves" stall. `admin` keeps the full scan: rare, human-triggered, and it
addresses arbitrarily old versions.
Closes: #37028
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqQqV1iv96Db8AzBSWTXV4
`_jwt`'s lru_cache is process-global, so it outlives a test. Proven with two tests using identical credentials and different tokens: the second saw the first's JWT. Harmless today (only one test authenticates) but a trap for the next auth test, and the same class of non-determinism as the ambient-creds issue already fixed in conftest — so it belongs in the same fixture. The cache itself stays: in production the CLI is one-shot, so one login per process is exactly right, and it is what keeps a full-listing fallback from re-authenticating on all 79 pages. Closes: #37028 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqQqV1iv96Db8AzBSWTXV4
sfreudenthaler
force-pushed
the
issue-37028-narrow-reads
branch
from
August 12, 2026 16:05
59689a7 to
11ab662
Compare
sfreudenthaler
merged commit Aug 12, 2026
cd0eb0e
into
issue-37025-evergreen-hub-auth
31 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Moving one floating tag paged the entire repo.
dotcms/dotcmshas 7,832 tags and Hub capspage_sizeat 100 (250 and 1000 are silently ignored), so every run was 79 calls / ~45s — against a Hub budget of 180 requests / 60s per IP. The promoter has no 429 handling, so a single rate-limited page is a hard failure and, since #37027, a Slack page.Most of that listing is noise: 1,946
*SNAPSHOTbranch builds, 677*_lts. Promotion needs a few GA CalVer tags and three marker names.Approach
Same technique the in-cluster reconciler already uses in dotCMS/infrastructure-as-code (
reconcile.py::resolve_track) — Hub'sname=substring filter:<track>digest, and its<track>_holdmarker for free (both match the track name as a substring)<version>_taintedmarkers, which share the version's month prefixA month is one page (6–28 tags).
--tracksis now resolved before any registry read, so the release pipeline's latest-only promote no longer pays to resolvestandardandtrailing.Measured against the live registry
standard,trailing--tracks latest(release pipeline)Plans are byte-identical to the full scan's —
standard -> 26.07.28-01, digestc2526c56…, matching the authenticated full-scan run 31534540024. Verified in CI on this branch: 31544087963 (read registry state in 4 filtered call(s)), cancelled at the approval gate so nothing applied.The correctness argument worth reviewing
Everything rests on fetching months newest-first and contiguously.
planner.plan's forward-only guard only bites when a track already sits on something newer than the newest eligible release. Any such version lives in a month at or above the one the walk stopped on — so it has necessarily already been read. A track on something older resolves to "move forward", which is the same decision the full listing produces. Nothing here can move a track backwards that the full scan would have held.The tempting "optimization" that breaks this is starting the walk at the threshold cutoff month instead of today (why read months newer than the cutoff?).
test_promote_narrow_reads_still_refuse_to_move_a_track_backwardsfails on exactly that mutation.I originally also ported the reconciler's push-month lookup to name each track's current version. Mutation testing showed removing it changed no promotion outcome — the argument above is why it was dead weight — so it's gone, along with the
Tag.monthplumbing it needed.Fallback
Returns to the full listing when the walk window holds nothing promotable, so a broken read path surfaces as the expensive right answer rather than a quiet "no moves" stall — which is precisely how tracks would silently stop advancing.
admindeliberately keeps the full scan: rare, human-triggered, and it addresses arbitrarily old versions.Tests
73 pass (68 existing unchanged + 5 new). The new ones were mutation-tested — each of these breaks at least one:
--trackswhen readingruffis not wired into CI for this package; I checked my changes are net-neutral against the baseline anyway (21 findings before and after, same rules, shifted line numbers).Closes: #37028
🤖 Generated with Claude Code
https://claude.ai/code/session_01PqQqV1iv96Db8AzBSWTXV4
This PR fixes: #37028