Skip to content

Add a REFERENCE.md freshness check to pr_tests.yml and roll it out fleet-wide - #58

Merged
silug merged 12 commits into
simp:mainfrom
hcaballero2:issue_46-reference.md-ci
Aug 12, 2026
Merged

Add a REFERENCE.md freshness check to pr_tests.yml and roll it out fleet-wide#58
silug merged 12 commits into
simp:mainfrom
hcaballero2:issue_46-reference.md-ci

Conversation

@hcaballero2

@hcaballero2 hcaballero2 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Closes #46. Also reconciles the pr_tests.yml template with the deployed fleet (#41).

This started as @hcaballero2's freshness check (the first four commits, rebased onto main) and grew into the full rollout package, following our convention of combining the template change and the workflow run in one PR so they're tested together:

  1. Reconcile the pr_tests.yml template with the deployed fleet — the template was behind the workflows actually deployed (checkout@v7, Ruby 3.2/3.4, the enabled Ruby Style job, the OpenVox spec matrix, parallel_spec, and the acceptance job). The template now matches the fleet, with one difference: the reference job's step is bundle exec rake validate:strings (this PR's freshness check) instead of the deployed regenerate-and-diff block.

  2. preserve_blocks for merge_gha_workflows — acceptance jobs are per-repo territory (simplib and x2go run custom docker matrices), so the merge task can now be told to keep whole blocks from the existing file (preserve_blocks: [jobs.acceptance]). Blocks are spliced by Psych node line spans before the scalar-grafting pass, so template comments/formatting still can't be disturbed. Repos without their own acceptance job get the template's vagrant-based one.

  3. Rewrite the generate_reference_md task — the 2023-era task self-committed, took a metadata.json path it only used to find the repo, and had a NameError on non-simp gems. It now takes repo_path, runs bundle install + bundle exec rake strings:generate:reference (gems shared across repos via .vendor/bundle), reports {changed}, and leaves committing to the git_commit_changes stage. The plan stage filters to project_type == pupmod.

  4. Session config 20260811-reference-md.yaml (now latest) — merges pr_tests.yml (scoped, with preserve_blocks: [jobs.acceptance]) and regenerates every module's REFERENCE.md so the new check starts green fleet-wide.

Verified:

  • 211 plain rspec examples + 18 BoltSpec plan examples, all green (new specs cover block preservation/insertion/idempotency and the rewritten task via a faked BUNDLER_EXE)
  • ci-managed-file-contract.sh passes; all plans parse and load
  • Local e2e against real clones of pupmod-simp-aide and pupmod-simp-simplib: aide's diff is exactly the reference-step swap, simplib gets the new reference job with its custom acceptance matrix untouched, and both repos' REFERENCE.md regenerated with no drift

🤖 Generated with Claude Code


Update after the first fleet run (72 PRs): two defects surfaced and are fixed here:

  • 9 repos gained the template's acceptance job despite having none — 5 of them (simp_banners, simp_options, site, tuned, vnc) have no spec/acceptance suites at all, so the job could only fail (example: [puppetsync] Add a REFERENCE.md freshness check to PR tests pupmod-simp-simp_options#134). preserve_blocks is now repo-authoritative in every direction: the template's block is removed when the repo has none, so a repo only gets an acceptance job by adding one itself.
  • rubygem-simp-compliance_engine was classified as a pupmod (it carries a pupmod-shaped metadata.json alongside its gemspec) and its fully custom pr_tests.yml was flattened into the module template. Gemspec detection now runs before pupmod detection, so gems stay out of module-template syncs. Its PR was closed.

Remediation: just re-run bolt plan run puppetsync from this branch. The 63 correct repos come out unchanged (their feature branches already match, so the GitHub stages skip them); the 8 affected repos rebuild from master and force-push corrected branches over their open PRs; compliance_engine is filtered out by the classifier fix. Verified locally against real clones of simp_options (reference job added, no acceptance job, REFERENCE.md regenerated) and compliance_engine (rejected as rubygem).

Hazel Caballero and others added 8 commits August 11, 2026 17:29
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The deployed fleet's pr_tests.yml evolved past the template (enabled
Ruby Style job, OpenVox 8.x/9.x-preview spec matrix, parallel_spec,
pupmod:build, modern runner/action/ruby values, dropped stale header
docs) — so a template-canonical workflow merge would have reverted
deployed improvements. Adopt the deployed content as the template,
with one deliberate difference: the REFERENCE.md job checks freshness
via `bundle exec rake validate:strings` (issue simp#46's intent) instead
of the deployed regenerate-and-diff block.

The acceptance job included here is the fleet-standard variant
(vagrant/libvirt, almalinux9+10); repos with custom acceptance
matrices (e.g. simplib) are preserved by the merge stage's new
preserve_blocks support (added separately).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The merge_gha_workflows task grows a preserve_blocks parameter: dot-paths
(e.g. jobs.acceptance) whose blocks in the EXISTING file replace the
template's wholesale — or are appended when the template lacks them.
Acceptance jobs are per-repo territory (simplib and x2go run custom
docker matrices), so a template-canonical pr_tests.yml refresh must not
flatten them.

Blocks are spliced by Psych node line spans (key.start_line to
value.end_line) before the existing scalar-grafting pass, so template
comments and formatting still can't be disturbed by construction.
Configured per session via puppetsync.plans.sync.merge_github_workflows.preserve_blocks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2023-era task took a metadata.json path it only used to find the
repo, self-committed its result (bypassing git_commit_changes), and hit
a NameError on non-simp-owned gems. It now takes repo_path, runs
bundle install + bundle exec rake strings:generate:reference under
Bundler.with_unbundled_env (gems shared across repos via
<repo>/../../.vendor/bundle), leaves the commit to git_commit_changes,
and reports {changed} from git status. The plan stage (rewired to
repo_path in the previous commit) now also filters to project_type
pupmod, since only Puppet modules have a REFERENCE.md.

Specs fake BUNDLER_EXE with a stub script; spec_helper's run_task takes
an optional env hash for that.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scoped to pr_tests.yml with preserve_blocks: [jobs.acceptance], plus a
generate_reference_md stage so every module's REFERENCE.md is fresh when
the new check starts gating PRs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@silug
silug force-pushed the issue_46-reference.md-ci branch from 762216b to 5874d77 Compare August 11, 2026 17:45
@silug silug changed the title Add REFERENCE.md freshness check to pr_tests.yml (validate:strings) Add a REFERENCE.md freshness check to pr_tests.yml and roll it out fleet-wide Aug 11, 2026
silug and others added 2 commits August 11, 2026 18:33
The first fleet run of the reference-md session added the template's
vagrant acceptance job to 9 repos that had none — 5 of them
(simp_banners, simp_options, site, tuned, vnc) have no spec/acceptance
suites at all, so the new job could only fail. preserve_blocks now means
the EXISTING file is authoritative for the listed dot-paths in every
direction: its block replaces the template's, is appended when the
template lacks it, and the template's block is REMOVED when the repo has
none. A repo only gets an acceptance job by adding one itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ta.json

rubygem-simp-compliance_engine carries a pupmod-shaped metadata.json
alongside its gemspec, and pupmod detection ran first — so the fleet run
classified it as a pupmod and flattened its fully custom pr_tests.yml
(version-consistency checks and all) into the module template. The
gemspec now wins: rubygem detection runs before pupmod, keeping gems out
of module-template syncs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
silug and others added 2 commits August 11, 2026 19:46
Its CI isn't green yet, and the GitHub PR stages can't use a
fork-network sibling like silug/puppet-selinux until simp#83 is fixed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repo-specific override (needed for its per-directory spec matrix and
extra acceptance suites) was still the pre-OpenVox 2023 template, so the
fleet run replaced the repo's modern workflow with it — reverting the
OpenVox spec matrix and dropping the reference job it was supposed to
add (caught validating simp/pupmod-simp-simp#385). The override is now
the repo's current master workflow plus the reference job: exactly the
change every other repo received.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@silug
silug marked this pull request as ready for review August 12, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a REFERENCE.md freshness check to pr_tests.yml (bundle exec rake validate:strings)

2 participants