docs(design): CI build performance roadmap + implementer notes (#363)

[scottschreckengaust]

Summary

Documents the CI build-performance roadmap in docs/design/CI_BUILD_PERFORMANCE.md (with its generated Starlight mirror). This satisfies the "keep CI_BUILD_PERFORMANCE.md current" DoD bullet of umbrella issue #363 — it is docs-only and implements no perf lever.

Relates to #363

#363 is an umbrella issue and stays OPEN after this docs PR merges. The remaining performance levers are tracked as separate ADR-003 child issues (see below), not implemented here.

What the doc covers

• How the build time is spent — per-task end offsets from a merge_group run on the 4-core runner; //cdk:test is the long pole, //cdk:synth:quiet the next binding constraint.
• Why //cdk:test was expensive — the transpile-only (isolatedModules) fix (item Installation: docker image inspect fails #1, Done) and the src/-vs-test/ type-check tradeoff it introduced.
• Recommendations table — items Installation: docker image inspect fails #1/#1b Done, open levers Docs: user guide hard-codes us-east-1 #3–feat: add Iteration 3e for memory security and integrity (OWASP ASI06) #5, chore(deps): bump defu from 6.1.4 to 6.1.6 in /docs #6 (cache investigation), chore(project): update structure #7 (Free Disk Space); with re-ranked sequencing.
• Implementer notes — required-check-on-merge_group constraints, the Free Disk Space dominant-overhead finding, the cache-independent-install finding, corrected fixed_overhead boundary + breakdown, coverage-threshold sync, runner sizing.
• Item Docs: specify that you can't use the agent with the canned repo #2 sharding design (deferred) — jest --shard mechanism, recomputed shard-vs-wall-time table, the fan-out + aggregate-gate pattern with the correct build (agentcore) required context and the always()-plus-step gate, cross-shard coverage merge, open design questions.
• Measurement protocol — every change must report a real 4-core CI before/after.

Review remediation (theagenticguy)

Reviewer supplied measured merge_group data; all figures below cite the runs they came from — 30412723686 (Build 171s, //cdk:test 125.08s / 136 suites), 30409171662 (186s), 30400147506 (148s), all 4-core, 2026-07-28/29. Seven inline comments addressed and replied in-thread:

1. Stale ledger — split so perf(test): skip redundant jest type-check via isolatedModules (#357) #359 (type-check removal, 2026-06-16) and perf(test): disable Lambda bundling in CDK unit-test synths (~15x per synth) (#366) #371 (Lambda-bundling disable, merged 2026-06-17) each get credit; added a "Current baseline" table (build ~148–186s, //cdk:test ~125s) with the run IDs, replacing the stale ~346s/~298s post-Installation: docker image inspect fails #1 column.
2. "Paid for twice" scope — tsc --build includes src/** only, so src/ went twice→once but test/ went once→zero; noted the eslint tsconfig.dev.json backstop and a deferred tsc --noEmit -p tsconfig.dev.json lever.
3. Idle-window claim — corrected to ~29s (synth ends +142s vs //cdk:test +171s); named //cdk:synth:quiet as the next binding constraint.
4. fixed_overhead — restated as 120–220s measured from job start (not ~95s from install).
5. Shard math — recomputed with measured inputs → 4-way lands at/above the whole build; demoted Docs: specify that you can't use the agent with the canned repo #2 to deferred (revisit trigger //cdk:test > ~250s).
6. Inverted if: — replaced with GitHub's documented pattern: job if: always() + gate moved into a step keying on failure || cancelled; explained why bare contains() never runs.
7. Check-run names — corrected to build (agentcore) (build.yml compute_type matrix + ruleset 14980587, confirmed PR chore(deps): actions: bump the all-actions group across 1 directory with 4 updates #672); flagged as a REQUIRED-CONTEXT change, not workflow-only.

Re-review remediation round 2 (theagenticguy)

Re-review confirmed the 7 findings above are fixed + resolved, then opened 3 new inline comments plus a dropped nav finding. All addressed and replied in-thread; corrections cross-checked against per-step job timings (gh api .../jobs/<id> .steps[].started_at/completed_at), not log arithmetic:

1. Cache-fix is net-neutral-to-slower — install cost is cache-independent (was rec chore(deps): bump defu from 6.1.4 to 6.1.6 in /docs #6's over-promoted premise). Comparing a miss merge_group run (30412723686) vs a hitting pull_request run (30419878840) with the identical yarn.lock hash: Cache * restore 1s→9s, yarn Done in 48.35s→55.09s, Install dependencies 56s→60s — the hit path is net slower. Cause: mise.toml [tasks.install] runs yarn install --check-files, which re-verifies every restored file; uv sync installs 127 pkgs in 121ms so a venv cache can't pay for itself. Merge-queue cache misses are also largely structural (PR-run caches are scoped to refs/pull/.../merge). Rec chore(deps): bump defu from 6.1.4 to 6.1.6 in /docs #6 reframed as a --check-files/earn-their-restore investigation and demoted in sequencing; every "cache saves time" / "shaves the job-setup tax" claim removed.
2. Free Disk Space is the dominant addressable overhead — added as new ledger item chore(project): update structure #7, ahead of chore(deps): bump defu from 6.1.4 to 6.1.6 in /docs #6 (and Installation: docker image inspect fails #1 in sequencing). Per-step share: run 30412723686 (job 90452344735) 148s/220s = 67%; run 30419878840 (job 90474296240) 97s/179s = 54%; run 30418865914 (job 90471207016) 75s/158s = 47%. Framed as the drop-or-narrow experiment the measurement protocol demands; the Node 22.23.1 re-download is a 4-5s smell, no-ENOSPC caveat preserved. Feeds the shard math (cut it → fixed_overhead 120-220s→60-100s).
3. fixed_overhead breakdown named inline — the Mechanism sentence now reads "120-220s, of which Free Disk Space is 75-148s and install is a cache-independent ~56-60s — so the overhead is mostly addressable." The ~250s shard trigger is made conditional on overhead staying 120-220s (move it down if chore(project): update structure #7 lands).
4. Dropped nav finding (no thread) — the Architecture sidebar in docs/astro.config.mjs is an explicit list, so the page would not appear in nav (Astro still builds the route, so //docs:build didn't catch it). Added slug entries architecture/ci-build-performance and architecture/bedrock-cost-attribution (the latter was also missing). Confirmed against the generated mirror filenames; astro check + //docs:build (70 pages) both pass.

Docs-only — perf levers are tracked as child issues

The remaining levers are not implemented in this PR. They are proposed as ADR-003 child issues of #363 (for the orchestrator to open), re-ranked per the round-2 findings:

1. Drop / narrow the Free Disk Space step — 47–67% of per-job overhead; the single biggest addressable pre-build cost. Run as a measured experiment (peak disk + 4-core before/after).
2. Bump the default runner (4-core → 8/16-core) — via vars.DEFAULT_RUNNER_LABEL; helps both //cdk:test and //cdk:synth:quiet.
3. Path-filtered builds — make //cdk:test conditional (e.g. dorny/paths-filter) so docs/CLI/agent-only PRs skip it, while build (agentcore) still reports.
4. Gate coverage to merge_group — skip collectCoverage on pull_request push; keep thresholds enforced on merge_group.
5. Investigate yarn install --check-files / CI caches — not a cache-hit-rate win: measured install cost is cache-independent (net-neutral-to-slower on a hit). A "delete or fix --check-files" investigation.
6. Shard the CDK suite — deferred on current numbers (overhead swamps a ~125s suite); re-cost if //cdk:test exceeds ~250s, and note the trigger moves down if the Free Disk Space step is cut.

Testing

• mise //docs:sync — no drift (committed Starlight mirror matches the regenerated output).
• pre-commit run --files ... — sync docs → Starlight mirrors Passed, astro check (docs) Passed, gitleaks Passed.
• mise //docs:build — full Astro/Starlight build Complete (70 pages); both new sidebar slugs resolve (architecture/ci-build-performance, architecture/bedrock-cost-attribution).
• Rebased onto latest origin/main; clean, no conflicts. Scoped gitleaks on origin/main..HEAD — no leaks (the 3 full-history findings are pre-existing aws-account-id false positives unrelated to this diff).

🤖 Generated with Claude Code

[scottschreckengaust]

✅ Acceptance summary (for the reviewer)

#363 DoD mapped to this change — this PR satisfies the "keep CI_BUILD_PERFORMANCE.md current" bullet of the #363 umbrella only; it is docs-only (source + regenerated Starlight mirror, no code/CI/workflow changes).

• Every factual CI-claim in the doc re-verified against current origin/main (runner resolution via vars.DEFAULT_RUNNER_LABEL, build on both pull_request + merge_group, isolatedModules, coverage thresholds in contracts/coverage-thresholds.json) — all accurate.
• /review_pr = approve-with-nits, zero blocking; both nits fixed (wall-time ~170s (4-way); suite count well over a hundred).
• Uses Relates to #363 (NOT Closes): perf(ci): remaining build-performance levers (shard, coverage-gate, runner, path-filter) — follow-ups to #357 #363 is an umbrella that stays open — the remaining perf levers are tracked as the child issues below.

🔀 Merge guidance (for the reviewer)

Independent — no ordering constraint. Cluster ci-perf-docs (disjoint from every other PR in this batch); touches only docs/design/CI_BUILD_PERFORMANCE.md + its generated mirror. Safe to merge in any order relative to the other batch PRs. Action: review + approve at will; native auto-merge is disabled repo-wide, so this awaits your manual squash-merge.

🤖 orchestrator note (agent) — promotion is orchestrator-driven; merge remains a human action.

[theagenticguy]

Review: docs-only, but the numbers it enshrines are stale enough to send the next implementer the wrong way

We verified every load-bearing claim against the repo at 6e7ae0f and against real merge_group run logs. The structure, the required-check reasoning, and the coverage-merge caveat are genuinely good, and the Starlight mirror is byte-identical to node docs/scripts/sync-starlight.mjs output, so the mutation gate will pass. Four things need fixing before merge.

1. The headline numbers describe a build that no longer exists. The doc's ~346s build / ~298s //cdk:test were accurate on 2026-06-16. PR #371 ("disable Lambda bundling in CDK unit-test synths, ~15x per synth", merged 2026-06-17, one day later) cut them again. Measured on 4-core merge_group runs from the last week:

	Doc claims	Measured 2026-07-28/29
build step	~346s	148s, 171s, 176s, 186s, 192s
//cdk:test	~298s	~125s (run 30412723686: Time: 125.08 s, 136 suites)
//cdk:test share	~86%	~74%

The doc credits the entire 710s->346s win to item #1 and never mentions #371. A reader concludes the long pole is 298s when it is 125s.

2. That staleness inverts the doc's central recommendation. Sharding is ranked first on the strength of a 298s long pole and ~95s per-job overhead. Both inputs are wrong in the direction that kills the case. See the inline comment on the shard table: at today's numbers, 4-way sharding lands each shard above the current whole-build wall time. The doc should either re-measure and re-rank, or state plainly that #2's case depends on numbers that need re-verification first.

3. Two concrete defects in the recommended workflow snippet would break the merge queue if implemented verbatim. Both are inline: the if: condition is inverted, and the aggregate job name does not match this repo's actual required context (build (agentcore), not build).

4. The page will not appear in the docs nav. docs/astro.config.mjs lists the Architecture sidebar entries explicitly rather than autogenerating them, so a new docs/design/*.md needs a matching { slug: 'architecture/ci-build-performance' } entry. Astro will still build the route, which is why mise //docs:build passed. Note BEDROCK_COST_ATTRIBUTION is missing from that list too, so this looks like a repeat oversight worth fixing in the same pass.

One process note: this PR is still a draft. Marking it ready for review would let the required checks report.

Commands used to verify: gh run view <id> --log on runs 30412723686, 30409171662, 30400147506; gh api repos/.../rulesets/14980587; node docs/scripts/sync-starlight.mjs diffed against the committed mirror.

[theagenticguy]

Re-review: all 7 inline findings genuinely fixed. The revision introduced one new wrong recommendation, and one original finding was dropped.

We re-verified every claim in the revised doc against the repo at a91b763 and against fresh run logs, including authoritative per-step timings from gh api .../actions/jobs/<id> rather than log-timestamp arithmetic.

The 7 inline threads are accurately resolved — we resolved them ourselves. Each was checked against the file, not just against the reply:

Thread	Claim	Verified
Stale ledger	#1/#1b split, baseline table with all 3 run IDs	✅ lines 38-60
Type-check scope	src/ twice→once, test/ once→zero, eslint backstop, deferred tsc --noEmit lever	✅ lines 27-32
Idle window	per-task end offsets, ~29s not ~200s, synth named next constraint	✅ lines 7-21
fixed_overhead boundary	job-start measurement, 120-220s, extra steps enumerated	✅ lines 80, 96
Shard math	recomputed table, demoted to Deferred, ~250s trigger	✅ lines 100-106
Inverted if:	always() on job + gate moved to step	✅ lines 136-149
Check-run names	build (agentcore), both corrections, required-context framing	✅ lines 110-123

The Starlight mirror is byte-identical to node docs/scripts/sync-starlight.mjs output, so the mutation gate passes.

What blocks merge now. The revision promoted our cache observation into new recommendation #6 and ranked it first. We only reported that caches miss on merge_group; we never established that fixing that would save time. It does not, and we should have said so the first time rather than leaving an unmeasured aside to be promoted. Measured, same yarn.lock hash on both runs:

	caches missing (30412723686)	caches hitting (30419878840)
Cache * restore steps	1s	9s
yarn install --check-files	Done in 48.35s	Done in 55.09s
Install dependencies step	56s	60s

The cache-hit path is slower. mise.toml runs yarn install --check-files, which re-verifies every file and defeats a restored node_modules. So #6 as written would send an implementer after a win that does not exist, which is the same failure mode as the original stale-numbers finding.

The actual dominant cost is the Free Disk Space step: 148s / 97s / 75s out of 220s / 179s / 158s total pre-build time, so 47-67% of the overhead the shard math depends on. It also deletes /opt/hostedtoolcache, after which Setup Node.js re-downloads Node 22.23.1. Details inline.

And one original finding got no reply and no fix: the doc page still will not appear in the site nav. docs/astro.config.mjs is unchanged on this branch and its Architecture sidebar is an explicit list, so docs/design/CI_BUILD_PERFORMANCE.md needs a matching { slug: 'architecture/ci-build-performance' } entry. Astro still builds the route, which is why mise //docs:build passes and nothing catches it. BEDROCK_COST_ATTRIBUTION is still missing from that list too.

These are all small edits. Nothing structural is wrong with the doc anymore, and the sharding design and required-check section are now genuinely good implementer references.

Verified with: gh api repos/.../actions/jobs/{90452344735,90474296240,90471207016}, runs 30412723686 30419878840 30418865914 30409171662 30400147506, and node docs/scripts/sync-starlight.mjs diffed against the committed mirror.

[theagenticguy]

Approve

All 10 findings across both prior rounds are fixed and verified against the file at 0152332, not against the replies. All 10 threads are now resolved.

This round's 3 threads:

Finding	Verified
Rec #6 premise contradicted by measurement	✅ reworded to the --check-files investigation; merge-ref scoping named as structural; miss-vs-hit table cited (restore 1s→9s, yarn 48.35s→55.09s, install 56s→60s); uv sync 121ms; "shaves the 120-220s tax" claim removed; demoted to position 5
Free Disk Space is the real overhead driver	✅ new item #7 ahead of #6 and #1 in sequencing; all three per-step figures with job IDs (67% / 54% / 47%); sourced to gh api .../actions/jobs/<id>; Node re-download framed as a ~4-5s smell; no-ENOSPC caveat preserved; drop-or-narrow experiment with peak-disk reporting
fixed_overhead breakdown nit	✅ Mechanism names 75-148s disk + ~56-60s cache-independent install; re-measure-before-costing instruction added; #2's trigger carries the matching conditional

The nav finding from round one is fixed too — docs/astro.config.mjs now carries architecture/ci-build-performance, and it picked up the bedrock-cost-attribution entry that was missing before. Verified end to end rather than by inspection: the Architecture sidebar is 20 slugs against 20 pages with no orphans in either direction, and dist/architecture/architecture/index.html renders href="/sample-autonomous-cloud-coding-agents/architecture/ci-build-performance/", so the page is genuinely reachable from the nav.

Gates run locally against the branch:

• node docs/scripts/sync-starlight.mjs → committed mirror byte-identical, so "Fail build on mutation" will pass
• astro build → 70 pages, Complete
• astro check → 0 errors, 0 warnings, 0 hints
• link-check.sh → exit 0
• Every ](#…) anchor in the doc resolves against the rendered HTML IDs

All 8 checks on the PR are green, including build (agentcore).

One note for whoever picks up #7, carried forward rather than blocking: the Free Disk Space timing varies 75-148s across the three sampled runs, which is wide enough that the 47-67% range is a sample rather than a stable estimate. Worth pulling a larger sample before sizing the experiment, and the doc's measurement protocol already asks for that.

Good revision. The doc now argues against its own original recommendation where the data says so, cites run IDs for every number, and the required-check section is a genuinely useful trap map for the next implementer.