Skip to content

Pull RPM build containers from ghcr.io/simp/simp-<os>-build (fixes fleet-wide RPM build failures) - #48

Closed
silug wants to merge 1 commit into
mainfrom
release-rpms-ghcr-build-containers
Closed

Pull RPM build containers from ghcr.io/simp/simp-<os>-build (fixes fleet-wide RPM build failures)#48
silug wants to merge 1 commit into
mainfrom
release-rpms-ghcr-build-containers

Conversation

@silug

@silug silug commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Problem

release_rpms.yml resolves its builder image as:

simp_builder_docker_image: 'docker.io/simpproject/simp_build_${{ github.event.inputs.build_container_os }}:latest'

Docker Hub's simpproject/ namespace has no EL8/9/10 build images. The only relevant ones are:

Image Last pushed
simpproject/simp_build_centos7 2024-08-09
simpproject/simp_build_centos8 2023-06-26

(plus older centos6 and *_ruby3_1 variants). So no value of build_container_os can select a current container — simpproject/simp_build_el8 does not exist.

Building against those years-old images fails with:

NameError: uninitialized constant JSON::Fragment

JSON::Fragment was added in the json gem 2.9; the stale images ship older. As of 2026-07-28 this is failing in 70 of 71 non-archived repos that have run the workflow, spanning 2026-06-09 → 2026-07-24 — i.e. through the entire 7.0.0/8.0.0 release wave.

Fix

Point at the current containers, which are public and rebuilt weekly:

Image Tags
ghcr.io/simp/simp-el8-build latest, 2026042820260720
ghcr.io/simp/simp-el9-build latest, 2026042820260720
ghcr.io/simp/simp-el10-build latest, 2026061820260727

Three changes:

  1. simp_builder_docker_imageghcr.io/simp/simp-${os}-build:${tag} — note both the registry host and the image-name shape differ from the old form.
  2. New build_container_tag input (default latest) so a dated tag can be pinned for reproducible rebuilds.
  3. build_container_os default centos8el8 (lowest currently supported EL, so the widest RPM compatibility).

This is already proven in production

pupmod-simp-mockup has carried exactly this configuration for a while and is the only repo with it. The single green release_rpms.yml run anywhere in the org came from it: simp/pupmod-simp-simplib 5.0.3 (build os: el10), dispatched through mockup's workflow via target_repo on 2026-06-22 — signing and release-asset attachment included. This PR ports that working configuration into the baseline so the other 78 repos get it.

Diffed against pupmod-simp-mockup's copy; this PR is the mockup delta minus the runner/action bumps, which #42 already owns (see below).

Relationship to #42

#42 also touches this file, but only bumps runs-on: ubuntu-20.0424.04, actions/github-script@v6@v9, and actions/checkout@v5@v7. It does not touch the image reference, the input defaults, or add build_container_tag. The two changes are complementary and deliberately kept separate.

⚠️ Merge-order note: my image-line hunk ends around template line 259 and one of #42's github-script@v9 hunks starts around line 259, so whichever merges second may need a trivial context resolution. There is no semantic conflict — keep both sides (the ghcr.io image line and the @v9 bump).

Deliberately not included

  • verbose input. mockup un-comments the verbose workflow input, but its build step still hardcodes verbose: 'no' #${{ github.event.inputs.verbose }}, so the input would be exposed while doing nothing. Left as-is here rather than shipping a no-op knob fleet-wide — worth either wiring up properly or leaving commented, as a separate decision.
  • The prerelease-tag defect. release_rpms.yml:105's tag regex captures the prerelease counter as a separate optional group from the prerelease word and has no release (-R) component, which is consistent with simp-core#864 / simp-doc#461 (simp-6.6.0-Alpha.el8.noarch.rpm instead of simp-6.6.0-Alpha10-1…). Same file, separate defect — kept out to keep this reviewable.

Verification suggested before syncing to all 78 repos

A release does not have to wait on the sync: because mockup's workflow accepts target_repo + target_repo_token, RPMs can be built and attached for any repo through mockup today. Recommended order:

  1. Dispatch with dry_run=yes against one module and confirm the container pulls and the build runs.
  2. One real simple module, then one with native/provider bits.
  3. Confirm SIMP_CORE_REF_FOR_BUILDING_RPMS (reportedly still 6.5.0-1) is still viable — a real build exercises it immediately.
  4. Then sync this template out.

@hcaballero2 hcaballero2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skeptical review — I tried to break this and mostly couldn't. What I verified before believing the PR body:

  • All three images exist and pull anonymously: docker manifest inspect ghcr.io/simp/simp-el{8,9,10}-build:latest all succeed (and a dated tag, simp-el8-build:20260720, too). No ghcr.io auth wrinkle for public runners.
  • The head-branch file is byte-identical to pupmod-simp-mockup's proven copy except the verbose input block and the runs-on/github-script/checkout versions — exactly the deltas the body claims to defer.
  • The workflow is workflow_dispatch-only, so github.event.inputs.build_container_tag is always populated; no empty-tag edge case from other triggers.
  • This is the only file in puppetsync referencing simpproject (org code search), so nothing else keeps pointing at the dead registry.
  • The failure diagnosis is real: simplib run 29344542051 (2026-07-14) dies with NameError: uninitialized constant JSON::Fragment under ruby-2.7.8/json_pure-2.6.3 inside the old container.

Two substantive concerns:

1. Syncing this template as-is would regress fleet repos to a retired runner. The baseline still carries runs-on: ubuntu-20.04 (deferred to #42), but the fleet's deployed copies have already moved on — e.g. simplib's .github/workflows/release_rpms.yml line 95 is runs-on: ubuntu-24.04, and mockup's likewise. GitHub retired ubuntu-20.04 hosted runners in April 2025, so a puppetsync run of this file before #42 lands would overwrite working 24.04 fleet copies with a workflow that fails at scheduling — trading a broken build step for a workflow that never starts. The body's "trivial context resolution" framing undersells this: #42 (or at least its runs-on hunk) is a hard prerequisite for syncing, not just a merge-order nicety. Suggest either folding the runs-on bump into this PR, or stating explicitly in the body/CHANGELOG that this file must not be synced until #42 is in.

2. Free-text build_container_os now composes nonexistent image names on stale muscle memory. See inline comment — the very run I pulled logs from was dispatched with build os: centos7 two weeks ago.

Minor: json mechanics aside (the constant is actually raised from json_pure 2.6.3's ext shim, not the 2.9 gem itself), the "container toolchain too old" root cause stands, so no change requested there.

description: "Build container OS"
required: true
default: 'centos8'
default: 'el8'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the valid values are now exactly el8/el9/el10, consider making this a type: choice input:

      build_container_os:
        description: "Build container OS"
        required: true
        type: choice
        options: [el8, el9, el10]
        default: 'el8'

With free text, anyone dispatching with old muscle memory (centos8, centos7) gets ghcr.io/simp/simp-centos7-build:latest — a nonexistent image whose pull failure only surfaces deep inside the build/signing job. This isn't hypothetical: simplib's most recent release_rpms run (29344542051, 2026-07-14) was dispatched with build os: centos7. choice also works fine for the cross-repo target_repo dispatch path (the API rejects values outside options, which is the point). If you'd rather keep it free-text for forward-compat with future ELs, at least name the valid values in the description.

The docker.io/simpproject/simp_build_<os> images this template referenced have
no EL8/9/10 variants -- the newest are simp_build_centos7 (pushed 2024-08-09)
and simp_build_centos8 (2023-06-26). Builds against those die with
'NameError: uninitialized constant JSON::Fragment' (json >= 2.9 required).

Point at ghcr.io/simp/simp-el{8,9,10}-build, which are public and rebuilt
weekly, and add a build_container_tag input so a dated tag can be pinned.
Matches the proven configuration already deployed in pupmod-simp-mockup.
@silug
silug force-pushed the release-rpms-ghcr-build-containers branch from ea957dc to a5f04ae Compare August 10, 2026 15:36
silug added a commit that referenced this pull request Aug 10, 2026
A session config can now limit the workflow merge to specific files
(puppetsync.plans.sync.merge_github_workflows.files), enabling targeted
template rollouts. Motivation: the first live smoke of the merge stage
showed the deployed fleet's pr_tests.yml is AHEAD of the template
(REFERENCE.md freshness job, Ruby Style job, acceptance matrix — see
#41/#46), so an unscoped template-canonical refresh would revert
deployed improvements until the template is reconciled. Scoping lets
single-file rollouts (like the release_rpms.yml ghcr change, PR #48)
proceed safely in the meantime.

Adds BoltSpec coverage for the stage itself (previously untested at
the plan level): default merges every templated workflow and skips
untemplated ones; the files list scopes it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@silug

silug commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #80, which carries this exact template change (rebased onto current main) together with the puppetsync session config that rolls it out — combined per our convention so the change and its delivery are reviewed and tested as one unit. All of this PR's context (problem analysis, mockup provenance, #42 merge-order note, pre-sync verification checklist) is preserved in #80's description.

@silug silug closed this Aug 10, 2026
@silug
silug deleted the release-rpms-ghcr-build-containers branch August 10, 2026 16:21
silug added a commit to silug/simp-puppetsync that referenced this pull request Aug 10, 2026
First dynamic-inventory session (repolist=github-org via the latest
symlink): rolls out PR simp#48's release_rpms.yml build-container change
fleet-wide, scoped to that single file via
merge_github_workflows.files — the pr_tests.yml template is currently
behind the deployed fleet (simp#41/simp#46), so unscoped workflow syncs stay
off until the template is reconciled.

Stage plan (verified via list_pipeline_stages): install_gems, checkout,
merge_github_workflows (scoped), git_commit_changes, then the GitHub
fork/remote/push/PR stages. Approve/merge plans configured; run them
with repolist=generated-20260731-release-rpms-ghcr.

Smoke-tested end-to-end against pupmod-simp-aide and pupmod-simp-acpid
(see simp#79): one-file commits matching PR simp#48's diff exactly, Renovate
values preserved, second run fully unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
silug added a commit that referenced this pull request Aug 11, 2026
…eet-wide RPM build failures) (#80)

* Add session config: fleet rollout of release_rpms.yml ghcr change

First dynamic-inventory session (repolist=github-org via the latest
symlink): rolls out PR #48's release_rpms.yml build-container change
fleet-wide, scoped to that single file via
merge_github_workflows.files — the pr_tests.yml template is currently
behind the deployed fleet (#41/#46), so unscoped workflow syncs stay
off until the template is reconciled.

Stage plan (verified via list_pipeline_stages): install_gems, checkout,
merge_github_workflows (scoped), git_commit_changes, then the GitHub
fork/remote/push/PR stages. Approve/merge plans configured; run them
with repolist=generated-20260731-release-rpms-ghcr.

Smoke-tested end-to-end against pupmod-simp-aide and pupmod-simp-acpid
(see #79): one-file commits matching PR #48's diff exactly, Renovate
values preserved, second run fully unchanged.

* Pull RPM build containers from ghcr.io/simp/simp-<os>-build

The docker.io/simpproject/simp_build_<os> images this template referenced have
no EL8/9/10 variants -- the newest are simp_build_centos7 (pushed 2024-08-09)
and simp_build_centos8 (2023-06-26). Builds against those die with
'NameError: uninitialized constant JSON::Fragment' (json >= 2.9 required).

Point at ghcr.io/simp/simp-el{8,9,10}-build, which are public and rebuilt
weekly, and add a build_container_tag input so a dated tag can be pinned.
Matches the proven configuration already deployed in pupmod-simp-mockup.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants