perf(ci): shard the CDK jest suite (jest --shard) — DEFERRED, follow-up to #363

[scottschreckengaust]

Parent: #363 (build-performance umbrella). Spun from #363's DoD by the resolve-open-issues orchestrator; needs maintainer approved before implementation (ADR-003).

⚠️ Re-framed 2026-07-29 after measured review of PR #364 (@theagenticguy). This lever was originally filed as "the biggest remaining win." Measurement shows it is currently net-neutral-to-slower and should be DEFERRED behind cheaper levers. The design below is kept because it is the part people get wrong; the priority is what changed.

Status: DEFERRED (documented design, do not implement yet)

Why deferred — the numbers moved

The original ~298s //cdk:test figure was stale. After PR #371 (disable Lambda bundling in CDK unit-test synths) the suite is ~125s (136 suites) on the 4-core merge_group runner (runs 30412723686 / 30409171662 / 30400147506, 2026-07-28/29). Substituting today's inputs into the doc's own slice + fixed_overhead formula (per-job overhead ~120–220s, and caches currently miss on every run):

Shards	Slice	+ overhead	Shard wall
1 (today)	~125s	n/a	whole build ~148–192s
2	~63s	120–220s	~183–283s
4	~31s	120–220s	~151–251s

A 4-way shard lands at or above the current entire build — before counting the aggregate job's own overhead, extra runner minutes, and coverage-merge complexity. Two further reasons it cannot help yet:

• Synth is the next binding constraint (//cdk:synth:quiet finishes ~142s), so sharding //cdk:test below ~142s buys nothing until synth is also addressed.
• Caches miss on every merge_group run (node_modules=miss, venv=miss, jest=miss) — fixing cache hit-rate is cheaper and lowers this lever's own crossover point.

Do these FIRST (they dominate sharding on current numbers)

1. Fix cache hit-rate (new issue perf(ci): fix merge_group cache misses (node_modules/venv/jest miss on every run) — follow-up to #363 #684 — see below) — cheapest; also improves sharding's economics.
2. perf(ci): bump default runner (4→8/16-core) — follow-up to #363 #677 — bump default runner (vertical scaling: jest maxWorkers ≈ cores, no per-job overhead duplication). Strictly better than sharding until the largest single runner is saturated.
3. perf(ci): path-filtered builds — skip //cdk:test on non-CDK PRs — follow-up to #363 #678 — path-filtered builds (skip //cdk:test on non-CDK PRs entirely).

Revisit trigger

Reconsider sharding only when //cdk:test serial time exceeds ~250s on the then-current runner AND vertical scaling (#677) is exhausted AND caches are warm — i.e. one runner's cores can no longer hold the needed parallelism, or the suite exhausts a runner's memory/disk. At that point horizontal fan-out becomes the sole remaining lever.

Design (kept for when the trigger fires — these are the fragile parts)

• Matrix job runs N shards; the aggregate job must keep the compute_type: [agentcore] matrix so its check-run name stays build (agentcore) — the required context in ruleset 14980587. A job named plain build emits check build, which does NOT match the required build (agentcore) context and deadlocks the merge queue (the failure feat(ci): require secrets/deps/SAST on every PR — make the merge gate enforceable (incident #313 class) #327 + build.yml's header warn about). This is a REQUIRED-CONTEXT change, not workflow-only.
• Adding a shard dimension yields check names build (agentcore, 1), build (agentcore, 2), … — do NOT mark individual shards required (multiplies required contexts).
• Aggregate gate must use always() + a step-level check, NOT a job-level if: contains(...) — contains() is not a status function, so if: ${{ contains(needs.*.result,'failure') }} silently resolves to success() && contains(...) (a contradiction) and the job never runs, admitting failing shards. Correct form:

  build:
    needs: [build-shard]
    if: ${{ always() }}
    steps:
      - name: Gate on shard results
        if: ${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}
        run: exit 1

• Merge per-shard coverage before threshold enforcement so coverageThreshold still holds on the combined result.
• Run self-mutation / drift checks once in the aggregate job, not per shard.

Secondary (non-wall-clock) benefits, if near the crossover anyway

• Retry only the failing shard instead of the whole suite; isolate a flaky suite's blast radius.

Refs #363