Skip to content

Do not let unattended-upgrades install kernels it cannot load - #292

Open
akuzminsky wants to merge 1 commit into
mainfrom
no-kernel-in-unattended-upgrades
Open

Do not let unattended-upgrades install kernels it cannot load#292
akuzminsky wants to merge 1 commit into
mainfrom
no-kernel-in-unattended-upgrades

Conversation

@akuzminsky

Copy link
Copy Markdown
Member

Stops unattended-upgrades installing kernels on hosts that can never load them. environments/development only; sandbox and modules/ follow after testing.

Why

Automatic-Reboot is false everywhere — nothing in Hiera overrides the class default — so a kernel installed by unattended-upgrades sits installed-but-not-running until something reboots the host. On warm-pool runners nothing ever does: a hibernation resume is not a boot, which is the same premise profile::github_runner's boot upgrade is already built on.

Observed 2026-08-01 on sandbox runner ip-10-1-1-96 (ami-0ad90a31b549bce10):

/var/log/unattended-upgrades/unattended-upgrades.log
01:38:11  Starting unattended upgrades script
01:38:18  Packages that will be upgraded: linux-aws linux-headers-aws linux-image-aws
01:39:05  All upgrades installed

Exec[gha-boot-security-upgrade] pulled 6.17.0-1019 → 7.0.0-1009, spent ~54s of the 1200s lifecycle-hook budget, and the host then ran 6.17.0-1019 for the rest of its life. canonical-livepatch was covering the running kernel, which is the actual protection:

kernel: 6.17.0-1019.19~24.04.1-aws
running: true
livepatch: checkState: checked

It also produced this in cloud-init-output.log, which looks like a fault but isn't:

┌──── Pending kernel upgrade ────┐
│ The currently running kernel version is 6.17.0-1019-aws which is not the
│ expected kernel version 7.0.0-1009-aws.
Failed to open terminal.debconf: whiptail output the above errors, giving up!
Use of uninitialized value $ret in scalar chomp at .../ConfModule.pm line 133

That's needrestart's kernel hint hitting debconf with no tty during provisioning. Blacklisting the kernel removes it at the source — with running == newest-installed, needrestart has nothing to hint about, so no separate kernelhints override is needed.

Verification

Patterns checked against the real package names from that host's dpkg.log:

re.match:   kernel blocked 8/8   collateral 0/6
re.search:  kernel blocked 8/8   collateral 0/6

Identical under both, so the behaviour does not depend on which one unattended-upgrades uses internally. linux-libc-dev and util-linux stay upgradable — they are userspace and do get real security fixes.

Both template branches render correctly (automatic_reboot true and false).

Design note

Gated on Automatic-Reboot rather than hardcoded: turn that on for a role and kernels become upgradable again, because then something will actually load them. Kernels otherwise reach these hosts through AMI rebuilds plus ASG cycling, which is the documented strategy.

Refs #289

🤖 Generated with Claude Code

Development environment only; sandbox and modules/ follow after testing.

Automatic-Reboot is false everywhere (nothing in Hiera overrides the class
default), so a kernel installed by unattended-upgrades can never be loaded. It
sits installed-but-not-running until something reboots the host, and on
warm-pool runners nothing ever does -- a hibernation resume is not a boot, which
is the premise profile::github_runner's boot upgrade is already built on.

Observed 2026-08-01 on sandbox runner ip-10-1-1-96
(ami-0ad90a31b549bce10): Exec[gha-boot-security-upgrade] pulled linux-aws,
linux-headers-aws and linux-image-aws from 6.17.0-1019 to 7.0.0-1009, spent ~54s
of the 1200s lifecycle-hook budget doing it, and the host then ran 6.17.0-1019
for the rest of its life. canonical-livepatch was covering the running kernel,
which is the actual protection. It also left a "Pending kernel upgrade" debconf
note that had no tty to render on, so cloud-init-output.log carried an
alarming-looking whiptail failure and a pile of Debconf perl warnings -- all
cosmetic, but indistinguishable at a glance from a real fault.

Blacklisting the kernel removes that noise at the source: with running ==
newest-installed, needrestart has nothing to hint about.

Patterns verified against the real package names from that host's dpkg.log:
8/8 kernel packages blocked, 0 collateral, identically under re.match and
re.search -- so the behaviour does not depend on which unattended-upgrades uses.
linux-libc-dev and util-linux stay upgradable, as they should.

Gated on Automatic-Reboot rather than hardcoded: enable that for a role and
kernels become upgradable again, because then something will actually load them.
Kernels otherwise reach these hosts via AMI rebuilds plus ASG cycling.

Refs #289

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant