feat(test-placement): add an authoring step and close the L1/L2 gap - #73
Merged
schultzp2020 merged 1 commit intoJul 24, 2026
Conversation
The skill could route a developer to "L2, in rhdh" and then have nothing to say. Steps 2-3 conclude with a *layer*, but Step 4 was organised by *repo*, and the `rhdh` section covered only L3, L4a and L4b. L1 and L2 existed nowhere except as two cells in the Step 3 ladder table -- `mockServices`, `backend-test-utils` and `frontend-test-utils` appeared zero times in the whole skill. The decision output and the creation guidance did not line up. Two changes: **Step 4** -- the `rhdh` section is now ordered cheapest-first to match the ladder (L1, L2, L3, L4a, L4b) and gains the missing L1 and L2 entries, each naming a template to copy that exists on `main` today: - L1: plugins/scalprum-backend/src/service/router.test.ts - L2: plugins/scalprum-backend/src/service/router.integration.test.ts **Step 5 (new)** -- turns the skill from an advisor into an author. It is a procedure, not a knowledge dump: open the template named in Step 4, consult `rhdh: docs/testing.md`, mirror the imports and setup, run the layer's command, then apply a per-layer checklist. That split is deliberate. The Backstage test surface moves -- `@backstage/test-utils` was renamed to `@backstage/frontend-test-utils` -- so an API name pinned in a skill goes stale silently and nobody bumps a skill when a package is renamed. Pointing at a file on `main` cannot go stale the same way, and a glob fallback covers the case where the template itself is moved. Detailed API references stay in the repo; only the decision and the procedure live here. Step 5 also states that a test must be justified by the failure it catches rather than by a coverage delta, since Codecov is `informational: true` in both rhdh and rhdh-plugins and cannot block a PR. The frontmatter description gains "how do I write a test for this" style triggers, which the previous where-only trigger list did not cover. All eight rhdh paths cited were verified against `main`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The skill could route a developer correctly to "L2, in
rhdh" — and then have nothing to say.Steps 2–3 conclude with a layer, but Step 4 was organised by repo, and its
rhdhsection covered only L3, L4a and L4b. L1 and L2 existed nowhere except as two cells in the
Step 3 ladder table. Across the whole skill,
mockServices,backend-test-utilsandfrontend-test-utilsappeared zero times.So the decision output and the creation guidance did not line up. I suspect that mismatch
is why L1/L2 fell through — it follows from the organising axis, not from an oversight.
What changed
Step 4 — the
rhdhsection is reordered cheapest-first to match the Step 3 ladder(L1 → L2 → L3 → L4a → L4b) and gains the missing L1 and L2 entries. Each names a template
that exists on
maintoday:plugins/scalprum-backend/src/service/router.test.tsplugins/scalprum-backend/src/service/router.integration.test.tsStep 5 (new) — turns the skill from an advisor into an author. It is a procedure,
not a knowledge dump: open the template Step 4 named, consult
rhdh: docs/testing.md,mirror the imports and setup, run the layer's command, apply a per-layer checklist.
Why the API details are not in the skill
Deliberate. The Backstage test surface moves —
@backstage/test-utilswas renamed to@backstage/frontend-test-utils, and the package went 0.5.1 → 0.6.1 inside 123 commits ofmain. Nobody bumps a skill when a package is renamed, so an API name pinned here goesstale silently and starts telling people to import something that no longer exists.
A path to a file on
maincannot go stale the same way, and Step 5 carries a glob fallbackfor the case where the template itself is moved. Same rule the skill already follows for
layer-migration-matrix.md: the decision lives in the skill, the detailed reference livesin the repo.
This depends on rhdh#5140
(RHIDP-13234), merged, which added
docs/testing.md— the L1–L3 authoring reference Step 5points at. Added to References.
One opinionated line
Step 5 states that a test must be justified by the failure it catches, not by a coverage
delta — Codecov is
informational: trueon both the project and patch statuses inrhdhand
rhdh-plugins, so no coverage number can block a PR. Worth encoding, since a PRjustified that way has already been rejected in review.
Also
The frontmatter description gains "how do I write a test for this" / "write a test for X"
triggers. The previous list was all where-questions, which would not fire for someone
asking the skill to actually write the test.
Verification
rhdhpaths cited in the diff verified againstmainvia the GitHub API.namematches the directory and the description is 638 chars(
tests/unit/test_skill_structure.pyrequires20 < len <= 1024).Per
AGENTS.mdthis is a behaviour change, so it wants a patch tag after merge.