From 7412097d0b589280be0cf51b1c053bf8381b3731 Mon Sep 17 00:00:00 2001 From: Oleksandr Kuzminskyi Date: Fri, 31 Jul 2026 20:14:05 -0700 Subject: [PATCH] Do not let unattended-upgrades install kernels it cannot load Development environment only; sandbox and modules/ follow after testing. Automatic-Reboot is false everywhere (nothing in Hiera overrides the class default), so a kernel installed by unattended-upgrades can never be loaded. It sits installed-but-not-running until something reboots the host, and on warm-pool runners nothing ever does -- a hibernation resume is not a boot, which is the premise profile::github_runner's boot upgrade is already built on. Observed 2026-08-01 on sandbox runner ip-10-1-1-96 (ami-0ad90a31b549bce10): Exec[gha-boot-security-upgrade] pulled linux-aws, linux-headers-aws and linux-image-aws from 6.17.0-1019 to 7.0.0-1009, spent ~54s of the 1200s lifecycle-hook budget doing it, and the host then ran 6.17.0-1019 for the rest of its life. canonical-livepatch was covering the running kernel, which is the actual protection. It also left a "Pending kernel upgrade" debconf note that had no tty to render on, so cloud-init-output.log carried an alarming-looking whiptail failure and a pile of Debconf perl warnings -- all cosmetic, but indistinguishable at a glance from a real fault. Blacklisting the kernel removes that noise at the source: with running == newest-installed, needrestart has nothing to hint about. Patterns verified against the real package names from that host's dpkg.log: 8/8 kernel packages blocked, 0 collateral, identically under re.match and re.search -- so the behaviour does not depend on which unattended-upgrades uses. linux-libc-dev and util-linux stay upgradable, as they should. Gated on Automatic-Reboot rather than hardcoded: enable that for a role and kernels become upgradable again, because then something will actually load them. Kernels otherwise reach these hosts via AMI rebuilds plus ASG cycling. Refs #289 Co-Authored-By: Claude Opus 5 (1M context) --- debian/changelog | 6 +++ .../52unattended-upgrades-infrahouse.erb | 37 +++++++++++++++++-- 2 files changed, 39 insertions(+), 4 deletions(-) diff --git a/debian/changelog b/debian/changelog index 57ac518..8dabf91 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +puppet-code (0.1.0-1build323) noble; urgency=medium + + * commit event. see changes history in git log + + -- root Sat, 01 Aug 2026 03:14:07 +0000 + puppet-code (0.1.0-1build322) noble; urgency=medium * commit event. see changes history in git log diff --git a/environments/development/modules/profile/templates/unattended_upgrades/52unattended-upgrades-infrahouse.erb b/environments/development/modules/profile/templates/unattended_upgrades/52unattended-upgrades-infrahouse.erb index 2df1797..d2cdcdf 100644 --- a/environments/development/modules/profile/templates/unattended_upgrades/52unattended-upgrades-infrahouse.erb +++ b/environments/development/modules/profile/templates/unattended_upgrades/52unattended-upgrades-infrahouse.erb @@ -1,7 +1,36 @@ // Managed by Puppet (profile::unattended_upgrades). Do not edit. // Augments the distro default /etc/apt/apt.conf.d/50unattended-upgrades: the -// -security origin stays enabled there; here we only pin reboot behaviour. -// Per-host package blacklisting is done by the relevant profile via its own -// apt.conf.d drop-in (entries append to Unattended-Upgrade::Package-Blacklist). +// -security origin stays enabled there; here we pin reboot behaviour and skip +// kernels when they cannot be loaded. Further per-host blacklisting is done by +// the relevant profile via its own apt.conf.d drop-in (entries append to +// Unattended-Upgrade::Package-Blacklist). -Unattended-Upgrade::Automatic-Reboot "<%= @automatic_reboot %>"; \ No newline at end of file +Unattended-Upgrade::Automatic-Reboot "<%= @automatic_reboot %>"; +<% unless @automatic_reboot -%> + +// Automatic-Reboot is off, so a newly installed kernel can never be loaded: it +// sits installed-but-not-running until something reboots the host, and on +// warm-pool GitHub runners nothing ever does -- a hibernation resume is not a +// boot. Installing it anyway costs bootstrap time against the ASG lifecycle +// hook budget, leaves the host on the old kernel regardless (livepatch covers +// the running one), and makes needrestart raise a debconf "Pending kernel +// upgrade" note with no tty to render on during provisioning, which surfaces in +// cloud-init-output.log as an alarming-looking whiptail failure. +// +// Observed 2026-08-01 on a sandbox runner: Exec[gha-boot-security-upgrade] +// pulled linux-aws 6.17.0-1019 -> 7.0.0-1009, spent ~54s doing it, and the host +// then ran 6.17.0-1019 for the rest of its life anyway. +// +// Kernels reach these hosts through AMI rebuilds plus ASG cycling instead. +// +// Deliberately gated on Automatic-Reboot rather than hardcoded: turn that on +// for a role and kernels become upgradable again, because then something will +// actually load them. +Unattended-Upgrade::Package-Blacklist { + "linux-aws"; + "linux-image-"; + "linux-headers-"; + "linux-modules-"; + "linux-tools-"; +}; +<% end -%>