CI: bound apt calls with timeouts and install ccache from staged bundle - #11219
CI: bound apt calls with timeouts and install ccache from staged bundle#11219julek-wolfssl wants to merge 3 commits into
Conversation
ccache-setup ran a raw apt-get update, so jobs whose ghcr .deb bundle had already installed cleanly still reached the mirror. On 2026-08-19 that step stalled on archive.ubuntu.com for 9-43 min and took out 11 jobs. The .deb is already in /var/cache/apt/archives - install-apt-deps stages the whole bundle - so install it with --no-download first and only fall back to the mirror. ccache was missing from the 22.04-minimal and linuxkm bundles; add it. Neither the fallback in install-apt-deps nor any of the raw apt sites had a timeout, so a wedged mirror hung instead of failing and the retry loops never fired. ci-deps-image already solved this for itself (Acquire timeouts plus `timeout`); apply the same to the consumers, and to the linuxkm producer job that was still missing it. Also recover from a dpkg interrupted by a kill, and bound the bundle pull.
There was a problem hiding this comment.
Pull request overview
This PR hardens CI against stalled Ubuntu apt mirrors by adding explicit Acquire timeouts plus timeout wrappers and bounded retry loops, and reduces mirror dependence by installing ccache offline from staged .deb bundles when available.
Changes:
- Add bounded apt update/install retry logic with timeouts across multiple GitHub Actions workflows and scripts.
- Install
ccachepreferentially viaapt-get --no-downloadfrom staged bundles, falling back to mirrored installs only if needed. - Include
ccachein the Ubuntu 22.04 minimal package list and the linuxkm.debbundle closure.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/sbom.yml | Wrap apt operations with timeouts + retry helper to avoid mirror hangs in SBOM job. |
| .github/workflows/falcon-interop.yml | Add apt retry helper with timeouts for falcon interop/backends CI legs. |
| .github/workflows/cross-library.yml | Add bounded apt retry loops with timeouts for container-based cross-library builds. |
| .github/workflows/ci-deps-image.yml | Add ccache to linuxkm bundle package set and bound apt calls with timeouts. |
| .github/scripts/zephyr-4.x/zephyr-test.sh | Add apt timeouts to best-effort host package install to prevent silent stalls. |
| .github/ci-deps/packages-ubuntu-22.04-minimal.txt | Add ccache to the 22.04 minimal bundle package list. |
| .github/actions/install-apt-deps/action.yml | Bound docker pull and add apt timeouts + dpkg recovery in apt fallback path. |
| .github/actions/ccache-setup/action.yml | Prefer offline ccache install from staged .deb bundle; add timed retry fallback. |
Suppressed comments (2)
.github/workflows/falcon-interop.yml:188
- This job has
timeout-minutes: 20, but the apt retry function allows up to ~50 minutes (3 attempts × (120s update + 900s install) plus backoff). Reduce the retry budget so it can complete within the job timeout (e.g., 2 attempts with a smaller install timeout).
APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
-o Acquire::https::Timeout=30)
apt_retry() {
local i
for i in 1 2 3; do
.github/workflows/cross-library.yml:152
- Same retry-budget concern in this job: 3 attempts × a 900s install timeout can exceed
timeout-minutes: 25, so retries may never complete. Tighten the attempt count and/or per-attempt install timeout so the loop can finish within the job budget.
APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
-o Acquire::https::Timeout=30)
for i in 1 2 3; do
if timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -q && \
timeout -k 10 900 apt-get "${APT_OPTS[@]}" install -y \
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
cross-library's containers are bare images with no bash, so Actions runs the step under sh; the bash array APT_OPTS=(...) was a syntax error and took out all six cross-library builds. Use a plain word-splitting variable. The retry budget also has to fit the caller's timeout-minutes - install-apt-deps is used by jobs with 4 minutes - or the last attempt is cut off before it can report. Two attempts at 60s/300s instead of three at 120s/900s. The Acquire timeouts are what actually detect a wedge, so the outer bound only backstops apt wedging outside its own I/O loop, and apt resumes from archives/partial/. Also run dpkg --configure -a before the offline ccache probe, not only in the fallback loop: an interrupted dpkg would otherwise push it to the mirror.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.
Suppressed comments (6)
.github/actions/install-apt-deps/action.yml:109
- These per-attempt limits do not fit the action's existing four-minute consumers: one attempt may take 360 seconds, and the new default allows two attempts. On a stalled install, the workflow-level timeout cancels the job before
timeout, the retry, or the final error can occur, contradicting the stated fail-fast behavior. Shorten/configure these command deadlines or raise every affected caller's job timeout.
if sudo timeout -k 10 60 apt-get "${APT_OPTS[@]}" update -q && \
sudo timeout -k 10 300 apt-get "${APT_OPTS[@]}" install -y \
$NO_REC ${{ inputs.packages }}; then
.github/actions/ccache-setup/action.yml:68
- As above,
sudodoes not reliably preserve the exportedDEBIAN_FRONTEND. If recovery or installation needs debconf input, this retry path can block rather than reaching its timeout/error handling. Pass the variable explicitly to the privileged commands.
sudo dpkg --configure -a >/dev/null 2>&1 || true
if sudo timeout -k 10 60 apt-get "${APT_OPTS[@]}" update -q && \
sudo timeout -k 10 180 apt-get "${APT_OPTS[@]}" install -y \
--no-install-recommends ccache; then
.github/actions/install-apt-deps/action.yml:56
- The 300-second pull timeout is longer than several callers' entire four-minute job budget (for example,
.github/workflows/cyrus-sasl.yml:57andrng-tools.yml:56). If GHCR wedges—the path this timeout is intended to handle—the job is cancelled before this command times out, so the apt fallback and diagnostic never run. Use a pull deadline that fits the shortest consumer, or make it configurable and update those callers' budgets.
if ! timeout -k 10 300 docker pull -q "$IMG" >/dev/null 2>&1; then
.github/actions/ccache-setup/action.yml:55
- The exported
DEBIAN_FRONTENDis normally removed bysudo; the previous implementation passed it explicitly aftersudo. Consequently both recovery and the offline install can open a debconf prompt and hang CI. Pass the variable throughsudofor both commands.
This issue also appears on line 65 of the same file.
sudo dpkg --configure -a >/dev/null 2>&1 || true
if sudo apt-get install -y --no-install-recommends \
--no-download ccache; then
.github/actions/install-apt-deps/action.yml:109
- The shell export is filtered by
sudo, so the new dpkg recovery and timed install do not reliably run noninteractively. A package configuration prompt would bypass the intended retry behavior and can consume the job budget; passDEBIAN_FRONTENDexplicitly throughsudo.
This issue also appears on line 107 of the same file.
# A previous attempt killed mid-unpack leaves dpkg needing this.
sudo dpkg --configure -a >/dev/null 2>&1 || true
if sudo timeout -k 10 60 apt-get "${APT_OPTS[@]}" update -q && \
sudo timeout -k 10 300 apt-get "${APT_OPTS[@]}" install -y \
$NO_REC ${{ inputs.packages }}; then
.github/workflows/ci-deps-image.yml:198
- This helper actually invokes the command three times: after both loop attempts fail, the trailing
"$@"runs a third attempt. For the 300-second install this can consume another five minutes and violates both the two-attempt comment and the 20-minute budget calculation.
retry() { local i; for i in 1 2; do "$@" && return 0; sleep 5; done; "$@"; }
|
retest this please |
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #11219
No scan targets match the changed files in this PR. Review skipped.
The per-attempt deadlines did not fit callers with a four minute timeout-minutes, so a wedged mirror cancelled the job before the loop could report it. Spend a configurable budget across the attempts instead and set one that fits on the short jobs. sudo resets the environment, so an exported DEBIAN_FRONTEND never reached apt-get and a debconf prompt could block until the timeout.
ccache-setupran a rawapt-get updateeven when the ghcr.debbundlehad already installed cleanly, so jobs still hit the mirror. On 2026-08-19
this stalled on archive.ubuntu.com for 9-43 min and took out 11 jobs.
--no-downloadfrom the staged bundle in/var/cache/apt/archivesfirst (already staged byinstall-apt-deps), andonly fall back to the mirror if that fails.
timeoutto every raw apt call ininstall-apt-deps's fallback and the linuxkm producer job, matching whatci-deps-imagealready does, so a wedged mirror fails fast instead ofhanging with no retry.