Skip to content

CI: bound apt calls with timeouts and install ccache from staged bundle - #11219

Open
julek-wolfssl wants to merge 3 commits into
wolfSSL:masterfrom
julek-wolfssl:ci/offline-ccache-and-apt-timeouts
Open

CI: bound apt calls with timeouts and install ccache from staged bundle#11219
julek-wolfssl wants to merge 3 commits into
wolfSSL:masterfrom
julek-wolfssl:ci/offline-ccache-and-apt-timeouts

Conversation

@julek-wolfssl

Copy link
Copy Markdown
Member
  • ccache-setup ran a raw apt-get update even when the ghcr .deb bundle
    had already installed cleanly, so jobs still hit the mirror. On 2026-08-19
    this stalled on archive.ubuntu.com for 9-43 min and took out 11 jobs.
  • Install ccache with --no-download from the staged bundle in
    /var/cache/apt/archives first (already staged by install-apt-deps), and
    only fall back to the mirror if that fails.
  • Add ccache to the 22.04-minimal and linuxkm bundles, where it was missing.
  • Add Acquire timeouts plus timeout to every raw apt call in
    install-apt-deps's fallback and the linuxkm producer job, matching what
    ci-deps-image already does, so a wedged mirror fails fast instead of
    hanging with no retry.
  • Recover from a dpkg interrupted by a kill, and bound the bundle pull.

ccache-setup ran a raw apt-get update, so jobs whose ghcr .deb bundle had
already installed cleanly still reached the mirror. On 2026-08-19 that step
stalled on archive.ubuntu.com for 9-43 min and took out 11 jobs. The .deb is
already in /var/cache/apt/archives - install-apt-deps stages the whole bundle
- so install it with --no-download first and only fall back to the mirror.
ccache was missing from the 22.04-minimal and linuxkm bundles; add it.

Neither the fallback in install-apt-deps nor any of the raw apt sites had a
timeout, so a wedged mirror hung instead of failing and the retry loops never
fired. ci-deps-image already solved this for itself (Acquire timeouts plus
`timeout`); apply the same to the consumers, and to the linuxkm producer job
that was still missing it. Also recover from a dpkg interrupted by a kill,
and bound the bundle pull.
Copilot AI lite review requested due to automatic review settings August 20, 2026 08:55
@julek-wolfssl julek-wolfssl self-assigned this Aug 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens CI against stalled Ubuntu apt mirrors by adding explicit Acquire timeouts plus timeout wrappers and bounded retry loops, and reduces mirror dependence by installing ccache offline from staged .deb bundles when available.

Changes:

  • Add bounded apt update/install retry logic with timeouts across multiple GitHub Actions workflows and scripts.
  • Install ccache preferentially via apt-get --no-download from staged bundles, falling back to mirrored installs only if needed.
  • Include ccache in the Ubuntu 22.04 minimal package list and the linuxkm .deb bundle closure.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
.github/workflows/sbom.yml Wrap apt operations with timeouts + retry helper to avoid mirror hangs in SBOM job.
.github/workflows/falcon-interop.yml Add apt retry helper with timeouts for falcon interop/backends CI legs.
.github/workflows/cross-library.yml Add bounded apt retry loops with timeouts for container-based cross-library builds.
.github/workflows/ci-deps-image.yml Add ccache to linuxkm bundle package set and bound apt calls with timeouts.
.github/scripts/zephyr-4.x/zephyr-test.sh Add apt timeouts to best-effort host package install to prevent silent stalls.
.github/ci-deps/packages-ubuntu-22.04-minimal.txt Add ccache to the 22.04 minimal bundle package list.
.github/actions/install-apt-deps/action.yml Bound docker pull and add apt timeouts + dpkg recovery in apt fallback path.
.github/actions/ccache-setup/action.yml Prefer offline ccache install from staged .deb bundle; add timed retry fallback.
Suppressed comments (2)

.github/workflows/falcon-interop.yml:188

  • This job has timeout-minutes: 20, but the apt retry function allows up to ~50 minutes (3 attempts × (120s update + 900s install) plus backoff). Reduce the retry budget so it can complete within the job timeout (e.g., 2 attempts with a smaller install timeout).
          APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
                    -o Acquire::https::Timeout=30)
          apt_retry() {
            local i
            for i in 1 2 3; do

.github/workflows/cross-library.yml:152

  • Same retry-budget concern in this job: 3 attempts × a 900s install timeout can exceed timeout-minutes: 25, so retries may never complete. Tighten the attempt count and/or per-attempt install timeout so the loop can finish within the job budget.
          APT_OPTS=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30
                    -o Acquire::https::Timeout=30)
          for i in 1 2 3; do
            if timeout -k 10 120 apt-get "${APT_OPTS[@]}" update -q && \
               timeout -k 10 900 apt-get "${APT_OPTS[@]}" install -y \

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/actions/ccache-setup/action.yml Outdated
Comment thread .github/workflows/falcon-interop.yml
Comment thread .github/workflows/cross-library.yml Outdated
Comment thread .github/scripts/zephyr-4.x/zephyr-test.sh
Comment thread .github/workflows/sbom.yml
Comment thread .github/workflows/ci-deps-image.yml Outdated
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

MemBrowse Memory Report

gcc-arm-cortex-m0plus

  • FLASH: .text +1,864 B (+2.9%, 66,855 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m3

  • FLASH: .text +1,792 B (+1.4%, 125,651 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4

  • FLASH: .text +1,920 B (+0.9%, 204,271 B / 262,144 B, total: 78% used)

gcc-arm-cortex-m4-baremetal

  • FLASH: .text +1,792 B (+2.7%, 69,347 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-crypto-only

  • FLASH: .text +1,856 B (+1.1%, 177,962 B / 262,144 B, total: 68% used)

gcc-arm-cortex-m4-dtls13

  • FLASH: .text +1,920 B (+1.0%, 185,596 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-min-ecc

  • FLASH: .text +1,728 B (+2.8%, 64,261 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-openssl-compat

  • FLASH: .rodata +56 B, .text +2,560 B (+0.3%, 779,156 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pkcs7

  • FLASH: .text +2,112 B (+1.0%, 217,070 B / 262,144 B, total: 83% used)

gcc-arm-cortex-m4-pq

  • FLASH: .text +2,304 B (+0.8%, 300,252 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-rsa-only

  • FLASH: .text +2,112 B (+0.6%, 330,672 B / 1,048,576 B, total: 32% used)

gcc-arm-cortex-m4-sp-math

  • FLASH: .text +1,728 B (+2.8%, 64,261 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-tls12

  • FLASH: .text +1,792 B (+1.4%, 126,451 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4-tls13

  • FLASH: .text +2,240 B (+0.9%, 240,993 B / 262,144 B, total: 92% used)

gcc-arm-cortex-m7

  • FLASH: .text +1,920 B (+0.9%, 204,271 B / 262,144 B, total: 78% used)

gcc-arm-cortex-m7-pq

  • FLASH: .text +2,240 B (+0.7%, 301,148 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m7-tls13

  • FLASH: .text +2,240 B (+0.9%, 241,057 B / 262,144 B, total: 92% used)

linuxkm-pie

  • Data: __patchable_function_entries +104 B (+0.4%, 26,688 B)

linuxkm-standard

  • Data: __patchable_function_entries +32 B (+0.1%, 49,488 B)

stm32-sim-stm32h753

  • FLASH: .text +1,856 B (+1.0%, 188,208 B / 2,097,152 B, total: 9% used)

cross-library's containers are bare images with no bash, so Actions runs the
step under sh; the bash array APT_OPTS=(...) was a syntax error and took out
all six cross-library builds. Use a plain word-splitting variable.

The retry budget also has to fit the caller's timeout-minutes - install-apt-deps
is used by jobs with 4 minutes - or the last attempt is cut off before it can
report. Two attempts at 60s/300s instead of three at 120s/900s. The Acquire
timeouts are what actually detect a wedge, so the outer bound only backstops
apt wedging outside its own I/O loop, and apt resumes from archives/partial/.

Also run dpkg --configure -a before the offline ccache probe, not only in the
fallback loop: an interrupted dpkg would otherwise push it to the mirror.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (6)

.github/actions/install-apt-deps/action.yml:109

  • These per-attempt limits do not fit the action's existing four-minute consumers: one attempt may take 360 seconds, and the new default allows two attempts. On a stalled install, the workflow-level timeout cancels the job before timeout, the retry, or the final error can occur, contradicting the stated fail-fast behavior. Shorten/configure these command deadlines or raise every affected caller's job timeout.
          if sudo timeout -k 10 60 apt-get "${APT_OPTS[@]}" update -q && \
             sudo timeout -k 10 300 apt-get "${APT_OPTS[@]}" install -y \
               $NO_REC ${{ inputs.packages }}; then

.github/actions/ccache-setup/action.yml:68

  • As above, sudo does not reliably preserve the exported DEBIAN_FRONTEND. If recovery or installation needs debconf input, this retry path can block rather than reaching its timeout/error handling. Pass the variable explicitly to the privileged commands.
              sudo dpkg --configure -a >/dev/null 2>&1 || true
              if sudo timeout -k 10 60 apt-get "${APT_OPTS[@]}" update -q && \
                 sudo timeout -k 10 180 apt-get "${APT_OPTS[@]}" install -y \
                   --no-install-recommends ccache; then

.github/actions/install-apt-deps/action.yml:56

  • The 300-second pull timeout is longer than several callers' entire four-minute job budget (for example, .github/workflows/cyrus-sasl.yml:57 and rng-tools.yml:56). If GHCR wedges—the path this timeout is intended to handle—the job is cancelled before this command times out, so the apt fallback and diagnostic never run. Use a pull deadline that fits the shortest consumer, or make it configurable and update those callers' budgets.
        if ! timeout -k 10 300 docker pull -q "$IMG" >/dev/null 2>&1; then

.github/actions/ccache-setup/action.yml:55

  • The exported DEBIAN_FRONTEND is normally removed by sudo; the previous implementation passed it explicitly after sudo. Consequently both recovery and the offline install can open a debconf prompt and hang CI. Pass the variable through sudo for both commands.

This issue also appears on line 65 of the same file.

          sudo dpkg --configure -a >/dev/null 2>&1 || true
          if sudo apt-get install -y --no-install-recommends \
               --no-download ccache; then

.github/actions/install-apt-deps/action.yml:109

  • The shell export is filtered by sudo, so the new dpkg recovery and timed install do not reliably run noninteractively. A package configuration prompt would bypass the intended retry behavior and can consume the job budget; pass DEBIAN_FRONTEND explicitly through sudo.

This issue also appears on line 107 of the same file.

          # A previous attempt killed mid-unpack leaves dpkg needing this.
          sudo dpkg --configure -a >/dev/null 2>&1 || true
          if sudo timeout -k 10 60 apt-get "${APT_OPTS[@]}" update -q && \
             sudo timeout -k 10 300 apt-get "${APT_OPTS[@]}" install -y \
               $NO_REC ${{ inputs.packages }}; then

.github/workflows/ci-deps-image.yml:198

  • This helper actually invokes the command three times: after both loop attempts fail, the trailing "$@" runs a third attempt. For the 300-second install this can consume another five minutes and violates both the two-attempt comment and the 20-minute budget calculation.
          retry() { local i; for i in 1 2; do "$@" && return 0; sleep 5; done; "$@"; }

@julek-wolfssl

Copy link
Copy Markdown
Member Author

retest this please

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #11219

No scan targets match the changed files in this PR. Review skipped.

The per-attempt deadlines did not fit callers with a four minute
timeout-minutes, so a wedged mirror cancelled the job before the loop
could report it. Spend a configurable budget across the attempts instead
and set one that fits on the short jobs.

sudo resets the environment, so an exported DEBIAN_FRONTEND never
reached apt-get and a debconf prompt could block until the timeout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants