Do not let unattended-upgrades install kernels it cannot load - #292
Open
akuzminsky wants to merge 1 commit into
Open
Do not let unattended-upgrades install kernels it cannot load#292akuzminsky wants to merge 1 commit into
akuzminsky wants to merge 1 commit into
Conversation
Development environment only; sandbox and modules/ follow after testing. Automatic-Reboot is false everywhere (nothing in Hiera overrides the class default), so a kernel installed by unattended-upgrades can never be loaded. It sits installed-but-not-running until something reboots the host, and on warm-pool runners nothing ever does -- a hibernation resume is not a boot, which is the premise profile::github_runner's boot upgrade is already built on. Observed 2026-08-01 on sandbox runner ip-10-1-1-96 (ami-0ad90a31b549bce10): Exec[gha-boot-security-upgrade] pulled linux-aws, linux-headers-aws and linux-image-aws from 6.17.0-1019 to 7.0.0-1009, spent ~54s of the 1200s lifecycle-hook budget doing it, and the host then ran 6.17.0-1019 for the rest of its life. canonical-livepatch was covering the running kernel, which is the actual protection. It also left a "Pending kernel upgrade" debconf note that had no tty to render on, so cloud-init-output.log carried an alarming-looking whiptail failure and a pile of Debconf perl warnings -- all cosmetic, but indistinguishable at a glance from a real fault. Blacklisting the kernel removes that noise at the source: with running == newest-installed, needrestart has nothing to hint about. Patterns verified against the real package names from that host's dpkg.log: 8/8 kernel packages blocked, 0 collateral, identically under re.match and re.search -- so the behaviour does not depend on which unattended-upgrades uses. linux-libc-dev and util-linux stay upgradable, as they should. Gated on Automatic-Reboot rather than hardcoded: enable that for a role and kernels become upgradable again, because then something will actually load them. Kernels otherwise reach these hosts via AMI rebuilds plus ASG cycling. Refs #289 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stops
unattended-upgradesinstalling kernels on hosts that can never load them.environments/developmentonly; sandbox andmodules/follow after testing.Why
Automatic-Rebootisfalseeverywhere — nothing in Hiera overrides the class default — so a kernel installed by unattended-upgrades sits installed-but-not-running until something reboots the host. On warm-pool runners nothing ever does: a hibernation resume is not a boot, which is the same premiseprofile::github_runner's boot upgrade is already built on.Observed 2026-08-01 on sandbox runner
ip-10-1-1-96(ami-0ad90a31b549bce10):Exec[gha-boot-security-upgrade]pulled 6.17.0-1019 → 7.0.0-1009, spent ~54s of the 1200s lifecycle-hook budget, and the host then ran 6.17.0-1019 for the rest of its life.canonical-livepatchwas covering the running kernel, which is the actual protection:It also produced this in
cloud-init-output.log, which looks like a fault but isn't:That's needrestart's kernel hint hitting debconf with no tty during provisioning. Blacklisting the kernel removes it at the source — with running == newest-installed, needrestart has nothing to hint about, so no separate
kernelhintsoverride is needed.Verification
Patterns checked against the real package names from that host's
dpkg.log:Identical under both, so the behaviour does not depend on which one unattended-upgrades uses internally.
linux-libc-devandutil-linuxstay upgradable — they are userspace and do get real security fixes.Both template branches render correctly (
automatic_reboottrue and false).Design note
Gated on
Automatic-Rebootrather than hardcoded: turn that on for a role and kernels become upgradable again, because then something will actually load them. Kernels otherwise reach these hosts through AMI rebuilds plus ASG cycling, which is the documented strategy.Refs #289
🤖 Generated with Claude Code