Skip to content

feat(test-placement): add an authoring step and close the L1/L2 gap - #73

Merged
schultzp2020 merged 1 commit into
redhat-developer:mainfrom
gustavolira:feat/test-placement-authoring
Jul 24, 2026
Merged

feat(test-placement): add an authoring step and close the L1/L2 gap#73
schultzp2020 merged 1 commit into
redhat-developer:mainfrom
gustavolira:feat/test-placement-authoring

Conversation

@gustavolira

Copy link
Copy Markdown
Member

The problem

The skill could route a developer correctly to "L2, in rhdh" — and then have nothing to say.

Steps 2–3 conclude with a layer, but Step 4 was organised by repo, and its rhdh
section covered only L3, L4a and L4b. L1 and L2 existed nowhere except as two cells in the
Step 3 ladder table. Across the whole skill, mockServices, backend-test-utils and
frontend-test-utils appeared zero times.

So the decision output and the creation guidance did not line up. I suspect that mismatch
is why L1/L2 fell through — it follows from the organising axis, not from an oversight.

What changed

Step 4 — the rhdh section is reordered cheapest-first to match the Step 3 ladder
(L1 → L2 → L3 → L4a → L4b) and gains the missing L1 and L2 entries. Each names a template
that exists on main today:

Layer Template
L1 plugins/scalprum-backend/src/service/router.test.ts
L2 plugins/scalprum-backend/src/service/router.integration.test.ts

Step 5 (new) — turns the skill from an advisor into an author. It is a procedure,
not a knowledge dump: open the template Step 4 named, consult rhdh: docs/testing.md,
mirror the imports and setup, run the layer's command, apply a per-layer checklist.

Why the API details are not in the skill

Deliberate. The Backstage test surface moves — @backstage/test-utils was renamed to
@backstage/frontend-test-utils, and the package went 0.5.1 → 0.6.1 inside 123 commits of
main. Nobody bumps a skill when a package is renamed, so an API name pinned here goes
stale silently and starts telling people to import something that no longer exists.

A path to a file on main cannot go stale the same way, and Step 5 carries a glob fallback
for the case where the template itself is moved. Same rule the skill already follows for
layer-migration-matrix.md: the decision lives in the skill, the detailed reference lives
in the repo.

This depends on rhdh#5140
(RHIDP-13234), merged, which added docs/testing.md — the L1–L3 authoring reference Step 5
points at. Added to References.

One opinionated line

Step 5 states that a test must be justified by the failure it catches, not by a coverage
delta — Codecov is informational: true on both the project and patch statuses in rhdh
and rhdh-plugins, so no coverage number can block a PR. Worth encoding, since a PR
justified that way has already been rejected in review.

Also

The frontmatter description gains "how do I write a test for this" / "write a test for X"
triggers. The previous list was all where-questions, which would not fire for someone
asking the skill to actually write the test.

Verification

  • All eight rhdh paths cited in the diff verified against main via the GitHub API.
  • Frontmatter is valid YAML; name matches the directory and the description is 638 chars
    (tests/unit/test_skill_structure.py requires 20 < len <= 1024).
  • Diff is +39 / −7 on one file; the skill is 152 lines.

Per AGENTS.md this is a behaviour change, so it wants a patch tag after merge.

The skill could route a developer to "L2, in rhdh" and then have nothing to
say. Steps 2-3 conclude with a *layer*, but Step 4 was organised by *repo*,
and the `rhdh` section covered only L3, L4a and L4b. L1 and L2 existed
nowhere except as two cells in the Step 3 ladder table -- `mockServices`,
`backend-test-utils` and `frontend-test-utils` appeared zero times in the
whole skill. The decision output and the creation guidance did not line up.

Two changes:

**Step 4** -- the `rhdh` section is now ordered cheapest-first to match the
ladder (L1, L2, L3, L4a, L4b) and gains the missing L1 and L2 entries, each
naming a template to copy that exists on `main` today:

- L1: plugins/scalprum-backend/src/service/router.test.ts
- L2: plugins/scalprum-backend/src/service/router.integration.test.ts

**Step 5 (new)** -- turns the skill from an advisor into an author. It is a
procedure, not a knowledge dump: open the template named in Step 4, consult
`rhdh: docs/testing.md`, mirror the imports and setup, run the layer's
command, then apply a per-layer checklist.

That split is deliberate. The Backstage test surface moves -- `@backstage/test-utils`
was renamed to `@backstage/frontend-test-utils` -- so an API name pinned in a
skill goes stale silently and nobody bumps a skill when a package is renamed.
Pointing at a file on `main` cannot go stale the same way, and a glob fallback
covers the case where the template itself is moved. Detailed API references
stay in the repo; only the decision and the procedure live here.

Step 5 also states that a test must be justified by the failure it catches
rather than by a coverage delta, since Codecov is `informational: true` in
both rhdh and rhdh-plugins and cannot block a PR.

The frontmatter description gains "how do I write a test for this" style
triggers, which the previous where-only trigger list did not cover.

All eight rhdh paths cited were verified against `main`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@schultzp2020 schultzp2020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Jul 24, 2026
@schultzp2020
schultzp2020 merged commit e841099 into redhat-developer:main Jul 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants