COMP: Reduce Linux CI disk pressure and harden ccache management

[hjmjohnson]

Reduce Linux CI disk pressure: free preinstalled software, compact ccache before
Cache@2 tar, harden cleanup steps. Fixes recurring near-ENOSPC in ITK.Linux.Python.

Commit summary

Commit 1 — COMP: Align Linux.Python CI disk management with Linux CI pattern

• Restructure AzurePipelinesLinuxPython.yml to match the pattern already in
  AzurePipelinesLinux.yml: conditional 1d/4d ccache eviction inside the build
  step, build-tree removal after diagnostic steps.
• Reduce CCACHE_MAXSIZE from 8G to 5G.

Commit 2 — COMP: Free unused preinstalled software in Linux Azure CI jobs

• Add "Free preinstalled software" step (first step, before checkout) to both
  Linux pipelines: remove Android SDK, GHC/GHCUP, .NET, Swift, CodeQL, Boost.
• Prune unused Docker images.

Commit 3 — COMP: Harden Linux CI ccache and disk-cleanup after code review

• Add ccache -c to the condition: always() cleanup step so a restored
  oversized cache is compacted before Cache@2 archives it (the missing step
  that would let ENOSPC recur on the first post-merge run).
• Extend success eviction window 1d → 3d: ccache does not refresh entry mtimes
  on cache hits, so the 1d window was evicting warm entries used this build.
• Delete false comment "ccache refreshes timestamps on hit" from Linux.yml.
• Move CCACHE_MAXSIZE to the pipeline variables block so ccache --show-config
  and ccache --show-stats report the operative limit.
• Add set -e to "Free preinstalled software" and cleanup steps.
• Mark docker image prune -af with || true (daemon absence is non-fatal).
• Fix Boost removal path to include headers (/usr/local/include/boost) and
  libraries (/usr/local/lib/libboost_*); the old path only removed CMake
  find-package configs (~1 MB, not ~200 MB).
• Add rm -rf $(Agent.BuildDirectory)/ITK-dashboard to cleanup step.
• Use df -h / consistently (was bare df -h in "Free preinstalled software").
• Mark ccache eviction calls || true to express that maintenance failure does
  not override the build exit code.

[greptile-apps]

Greptile Summary

This PR hardens disk management across all Linux, macOS, and Windows Azure CI pipelines to address recurring ENOSPC failures during Cache@2 post-job archives. The core changes are: promoting CCACHE_MAXSIZE from per-step env: to pipeline-level variables:; adding a "Free preinstalled software" step to reclaim ~5–15 GB before any build work; and replacing the old one-off eviction workaround in LinuxPython with a consistent pattern across all pipelines.

• CCACHE eviction windows extended 1d → 3d on success — ccache does not refresh entry mtimes on hits, so the 1d window was silently evicting warm entries used during the current build.
• ccache -c added to the always() cleanup step across all five pipelines to compact the cache before Cache@2 archives it; however, all three Linux/macOS cleanup steps use set -e without guarding ccache -c with || true, so a compaction failure will abort before rm -rf runs.
• Boost removal path corrected to include headers (/usr/local/include/boost) and libraries (/usr/local/lib/libboost_*) instead of only the ~1 MB CMake find-package configs."

Confidence Score: 3/5

The disk-pressure improvements are well-motivated, but the cleanup step across Linux, LinuxPython, and macOS pipelines has a logic gap that can reproduce the original ENOSPC failure on the first run after merge.

All five pipelines share a new condition:always() cleanup step that runs ccache -c under set -e without a || true guard. If ccache -c exits non-zero — most plausible on a full disk, the exact failure mode this PR addresses — the step aborts before rm -rf removes the build tree, leaving several gigabytes on disk and allowing Cache@2 to hit ENOSPC exactly as before. A one-character fix (ccache -c || true) on each of the three Linux/macOS cleanup steps removes the risk entirely.

AzurePipelinesLinux.yml, AzurePipelinesLinuxPython.yml, and AzurePipelinesMacOSPython.yml all need || true added to ccache -c in their cleanup steps. AzurePipelinesBatch.yml and AzurePipelinesWindowsPython.yml share the same ccache -c gap but are lower risk because their rm -rf calls are already protected with || true.

Important Files Changed

Filename	Overview
Testing/ContinuousIntegration/AzurePipelinesLinux.yml	Moves CCACHE_MAXSIZE to pipeline variables, adds Free preinstalled software step, hardens eviction logic with 3d/4d windows and
Testing/ContinuousIntegration/AzurePipelinesLinuxPython.yml	Replaces the old CCACHE_MAXSIZE=6.5G override workaround with a clean pipeline-variable declaration (5G), aligns eviction pattern with Linux.yml, adds Free preinstalled software step; same set -e + unguarded ccache -c issue in the new cleanup step.
Testing/ContinuousIntegration/AzurePipelinesMacOSPython.yml	Adds macOS-specific Free preinstalled software step (dotnet + CoreSimulator runtimes), moves CCACHE_MAXSIZE to pipeline variables, adds eviction logic and cleanup step; same set -e + unguarded ccache -c fragility as Linux files.
Testing/ContinuousIntegration/AzurePipelinesBatch.yml	Moves CCACHE_MAXSIZE to pipeline variables; adds a new cleanup step using $AGENT_JOBSTATUS for eviction, ccache -c, and rm -rf with
Testing/ContinuousIntegration/AzurePipelinesWindowsPython.yml	Moves CCACHE_MAXSIZE to pipeline variables and adds cleanup step with $AGENT_JOBSTATUS-based eviction, ccache -c, and

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Free preinstalled software\nrm android/ghc/dotnet/swift/boost\ndocker image prune -af"] --> B["checkout + install dependencies"]
    B --> C["Cache@2 restore ccache + ExternalData"]
    C --> D["ccache zero-stats, evict-older-than 7d, show-config"]
    D --> E["Build and test\nctest -S dashboard.cmake"]
    E --> F{ctest_rc == 0?}
    F -- success --> G["ccache evict-older-than 3d or true"]
    F -- failure --> H["ccache evict-older-than 4d or true"]
    G --> I["exit ctest_rc"]
    H --> I
    I --> J["ccache show-stats - condition always"]
    J --> K["Diagnostics + JUnit + Publish results"]
    K --> L["Free build tree - condition always\nccache -c\nrm -rf build tree + ITK-dashboard\ndf -h /"]
    L --> M["Cache@2 post-job save"]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["Free preinstalled software\nrm android/ghc/dotnet/swift/boost\ndocker image prune -af"] --> B["checkout + install dependencies"]
    B --> C["Cache@2 restore ccache + ExternalData"]
    C --> D["ccache zero-stats, evict-older-than 7d, show-config"]
    D --> E["Build and test\nctest -S dashboard.cmake"]
    E --> F{ctest_rc == 0?}
    F -- success --> G["ccache evict-older-than 3d or true"]
    F -- failure --> H["ccache evict-older-than 4d or true"]
    G --> I["exit ctest_rc"]
    H --> I
    I --> J["ccache show-stats - condition always"]
    J --> K["Diagnostics + JUnit + Publish results"]
    K --> L["Free build tree - condition always\nccache -c\nrm -rf build tree + ITK-dashboard\ndf -h /"]
    L --> M["Cache@2 post-job save"]

Loading

Comments Outside Diff (1)

1. Testing/ContinuousIntegration/AzurePipelinesLinux.yml, line 152-154 (link)

   ccache --show-stats runs before compaction, so reported size is pre-compact

   The ccache --show-stats step runs before the "Free build tree" step that runs ccache -c. Disk usage metrics visible in the logs will reflect the state after eviction but before compaction, which can look larger than the final cache that Cache@2 actually archives. The same ordering exists in AzurePipelinesLinuxPython.yml and AzurePipelinesMacOSPython.yml. A second ccache --show-stats at the end of the cleanup step would let operators confirm the final archived cache size.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "COMP: Apply ccache and disk-cleanup hard..." | Re-trigger Greptile}