Skip to content

chore(agents): one home per rule, a parallel gate, and a leaner browser suite - #705

Merged
JArmandoAnaya merged 6 commits into
mainfrom
docs/iteration-test-policy
Aug 19, 2026
Merged

chore(agents): one home per rule, a parallel gate, and a leaner browser suite#705
JArmandoAnaya merged 6 commits into
mainfrom
docs/iteration-test-policy

Conversation

@JArmandoAnaya

@JArmandoAnaya JArmandoAnaya commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Working with a coding agent here has meant loading a policy corpus that repeated itself and waiting on checks that re-proved each other. This pull request attacks both, in four commits on top of the policy change it started as.

The test-execution policy

The standing policy made a full local run the default and --fast the exception, which is the reason the loop was slow. This changes it. During iteration only the tests pertinent to the change are run: the file's own suite, the module's suite, or a named test. bash scripts/check.sh is the gate, CI runs the same checks on every pull request, and locally the gate runs once, immediately before a pull request is opened. The direction comes from the decision on the check-command subset issue (#674). Two obligations come with it: reading what CI answered is part of pushing, and two categories of change earn the full gate whatever their size — anything touching state, gating or progress, and any change to the shape of a published wire model — because every narrow signal is blind to both by construction.

The instruction corpus

The skill set goes from fourteen files to eleven and from 121 KB to 78 KB, with every invariant surviving as a rule stated once in the file that owns it. The check doctrine existed in five places and now lives in AGENTS.md alone, pointed to by the skills that used to restate it. public-communications and issue-pr-writing, which each instructed the reader to read the other before posting anything, are one skill, public-writing. The typescript and react-19 skills are gone: their repo-specific content — the generated client, ESM packaging, the compiler-enforced React rules — moved into nodejs-setup, and the rest was language tutorial the toolchain already enforces. refactor-protocol keeps every rule at roughly a quarter of the text, and kernel-architecture now names all four import contracts, including the wire, jobs and inference walls its list had fallen behind on.

CLAUDE.md had drifted from AGENTS.md — it was missing the merge ban, among other things — because it was a hand-maintained copy that nothing regenerated. scripts/setup_agents.sh now links CLAUDE.md → AGENTS.md and prunes symlinks whose skill no longer exists, so the drift class is closed rather than corrected.

CI

The python job ran pytest serially: 510 of its 560 seconds, which made it about ninety percent of every pull request's wall clock. It now passes -n auto, the same distribution the local gate has used since the suite-reduction program (#517) measured that parallelism, not pruning, is what the suite's 63 ms mean responds to. The duplicate kernel-only mypy invocation is gone — the kernel's strict flags are per-module configuration, so mypy src/visionset already checks it strictly — and the annotator e2e job runs four Playwright workers to match its runner's cores instead of the config's fallback of two. The first run on this branch is the measurement, and it is smaller than the local numbers promised: pytest went from 510 to 476 seconds and the annotator e2e job from 253 to 227, leaving the run's wall clock roughly where it was. The suite that parallelizes 6× on a developer machine gains little on a four-core runner whose disk every workspace-opening worker contends for — the same lesson the reduction program recorded from the other direction, that this suite's cost lives somewhere deletion and now worker count cannot reach. The changes stay: they cost nothing, and the mypy dedup and comment consolidation stand on their own. All fourteen check names are unchanged, because each is a required status context in the main ruleset.

Tests

The reduction is bounded by what #517 already proved: the corpus has no fat tail, and its ratified pruning catalog was worth under two tenths of a second. So this pass read only the two slices that program never saw. The ~780 Python tests added since its catalog turned out to be as lean as the original corpus — 615 added functions yielded one merge and no deletions. The browser suite was read as a pyramid question and loses 37 of 278 tests, almost all in gallery.spec and inference.spec, where the deleted test re-asserts — sometimes title for title — what the gallery and inference vitest suites already prove against the same stub shapes. Everything that needs a real browser stays: computed CSS, layout, focus, reload, pointer, touch and wheel delivery, router navigation, the per-endpoint wire exercisers, and the whole cycle walk, untouched. The two byte-identical tests #517 ratified for removal and had to restore are also gone: their blocker — one adapter statement covered only as a function of the suite's workspace count — no longer exists, and the whole-package missed-statement count was measured identical with and without them (219 of 10549, both ways).

Found, not fixed

The e2e job matrix, the dev-image job's re-runs, and the double build_dist.sh build (wheel job and 30-minute flow) each re-execute work another job performs; each carries a documented distinct guarantee or would need a needs: coupling to remove, and all are left as they are. publish-pypi.yml remains a hand-synchronized copy of the wheel job's setup, as its header says. Consolidating small jobs (the 13-second OpenAPI gate, the e2e matrix) is blocked by the ruleset's required-context list and would need that list changed in the same window. Of #517's three unshipped consolidation families, none should ship: the CLI not-found family remains mis-catalogued (its members assert exit 0, exit 2, a message, and two domain refusals), and the listing-contract family remains declined by the recorded decision.

Test plan

Targeted, per the policy this pull request establishes; CI is the comprehensive gate.

uv run pytest tests/architecture                                59 passed
uv run pytest tests/kernel/test_annotation_service.py           91 passed
uv run pytest tests/kernel/test_schema_service.py tests/inference/test_weights.py
                                                                116 passed, 4 skipped
uv run pytest --collect-only                                    3980 tests collected
uv run ruff check .                                             All checks passed!
uv run ruff format --check tests/ scripts/                      175 files already formatted
pnpm test:scripts                                               102 pass, 0 fail
pnpm --filter @visionset/app lint                               exit 0 (eslint + tsc e2e)
CI=1 pnpm exec playwright test --list                           Total: 241 tests in 20 files
CI=1 pnpm exec playwright test showcase.spec.ts                 5 passed
bash scripts/check.sh nope                                      exit 2, verdict INCOMPLETE
bash -n scripts/check.sh scripts/setup_agents.sh                clean

The coverage comparison for the two deleted duplicates ran the full suite under uv run --with pytest-cov in the primary checkout (which carries the local-inference extra), before and after: 219 missed of 10549 statements, identical.

The standing policy made a full local run the default and `--fast` the
exception, which is what made the loop slow. During iteration only the tests
pertinent to the change are run; `bash scripts/check.sh` is the gate, CI runs
it on every pull request, and locally it runs once before a pull request is
opened or when it is asked for.

Two obligations come with that. Reading what CI answered is part of pushing,
because a narrow local run plus an unread CI result is not a checked change.
And two categories earn the full gate whatever their size — anything touching
state, gating or progress, and any change to the shape of a published wire
model — because every narrow signal is blind to both by construction.

The check tables now say per row what they do not cover, so a reader learns
from the frontend row itself that it runs no browser, and from the mypy row
that it reads the kernel alone.
The skill corpus taught through incident narratives — the mutation-harness
story, the orphaned spin-loop story, the docker volume stories — and repeated
its policy: the checks doctrine existed in five places, the PR rules in three,
and two skills were generic language tutorials the toolchain already enforces.
Every invariant survives as a rule stated once, in the file that owns it.

AGENTS.md owns the check policy and the commit/PR invariants; skills point to
it. public-communications and issue-pr-writing, which demanded to be read
together, are one skill: public-writing. typescript and react-19 are gone —
their repo-specific lines live in nodejs-setup, the rest was tsc/eslint/React
Compiler restated. refactor-protocol keeps every rule at a quarter of the
text. kernel-architecture now names all four import contracts, including the
wire, jobs and inference walls its list had fallen behind on.

setup_agents.sh prunes symlinks whose skill is gone and links
CLAUDE.md -> AGENTS.md, so the hand-copied twin that had already drifted —
missing the merge ban among other things — cannot exist again.
The python job ran pytest serially — 510 of its 560 seconds, ~90% of every
pull request's wall clock — while the local gate has run the same suite under
`-n auto` since the suite-reduction program measured that parallelism, not
pruning, is the only thing that makes it faster. CI now passes `-n auto`
too, resolving to its runner's own core count.

The kernel-only mypy invocation is gone: the kernel's strict flags are
per-module configuration in pyproject.toml, so `mypy src/visionset` already
checks the kernel strictly and the second run added nothing. The annotator
e2e job runs four Playwright workers to match the runner's four cores; the
config's CI fallback of two was sized for a smaller runner generation.

The no-`--with-deps` rationale now lives once, in the annotator e2e job;
the cycle and bench jobs point to it. Job names and the check roster are
unchanged — all fourteen are required status contexts on main.
A semantic pass over the two slices the reduction program (#517) never
reviewed — the ~780 Python tests added since its catalog, and the browser
suite as a pyramid question — with every deletion justified against the named
test that keeps the behavior covered.

The browser suite loses 37 of 278 tests, almost all in gallery.spec and
inference.spec, where the deleted test re-asserts — sometimes title for
title — what the gallery and inference vitest suites already prove against
the same stub shapes. Everything that needs a real browser stays: computed
CSS, layout, focus, reload, pointer/touch/wheel delivery, router navigation,
the per-endpoint wire exercisers, and the whole cycle walk. The Python side
confirms the program's finding: 615 added test functions yielded one merge
(the pre-labeled landing and the human takeover were two walks of the same
fixture) and no deletions.

The two byte-identical tests the program ratified for removal and had to
restore are gone. Their blocker — one adapter statement covered only by the
suite's workspace count — no longer exists: the reopen-by-recency and draft
work added since covers that lifecycle deliberately, and the whole-package
missed-statement count is measured identical with and without them.
The header retold the stories behind the rules — the piped-tail incident, the
Node 26 localStorage failure, the port-collision history — at a length that
buried the rules themselves. Every invariant survives, shorter: the pipeline
trap, the docs group's opt-in trade, the two browser suites and why the cycle
run is not a luxury, CI=1, the git ls-files index caveat, the out-of-scope
groups, the verdict-line contract and the banner's quiet case.

One correction rather than a compression: the pytest comment claimed CI runs
the suite serially by choice, which stopped being true when the python job
took -n auto.
@JArmandoAnaya JArmandoAnaya changed the title docs(skills): iteration runs the pertinent tests, the gate runs once chore(agents): one home per rule, a parallel gate, and a leaner browser suite Aug 19, 2026
@JArmandoAnaya
JArmandoAnaya merged commit 7edf98b into main Aug 19, 2026
15 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the docs/iteration-test-policy branch August 19, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant