Skip to content

ops(runners): periodic and low-disk-triggered cleaning for self-hosted runners - #6

Closed
goastler wants to merge 1 commit into
mainfrom
ops/runner-disk-maintenance
Closed

ops(runners): periodic and low-disk-triggered cleaning for self-hosted runners#6
goastler wants to merge 1 commit into
mainfrom
ops/runner-disk-maintenance

Conversation

@goastler

Copy link
Copy Markdown
Member

The self-hosted fleet is non-ephemeral and nothing ever reclaims its disk. Every job leaves build state behind — a single Protect checkout is ~17 GB of target/, over half of it incremental state that is worthless once the job ends — so a 60 GB runner fills after a handful of branches and then fails jobs late, with a linker error (ar: … No space left on device) that looks like anything but a disk problem. That persistence is also exactly why save_cargo_cache skips the GitHub cache backend on self-hosted: local disk is the cache there, and nobody prunes it.

Adds runner-maintenance/:

  • runner-disk-clean.sh — the sweep. Incremental dirs, cold target/ trees, cold workspace checkouts, ~/.cargo/registry/src (regenerated from the .crate archives, which are kept), cold node_modules, and dangling docker layers.
  • runner-disk-clean.{service,timer} — nightly at 03:30 with a 30m jitter, Persistent=true, Nice=19 + idle IO so it never competes with a running build. Runs as the runner's user, not root.
  • job-started-hook.sh — for ACTIONS_RUNNER_HOOK_JOB_STARTED. No-ops unless free space is under MIN_FREE_GB (default 15), then sweeps with a 1-day cut instead of 7. Checking before the job means the reclaim happens before the build burns its full time.

Everything is age-based, never a wipe. Jobs run back to back on these hosts and share ~/.cargo, so state a running job might hold has to survive; the age cut (7 days scheduled, 1 day under pressure) is far longer than any single job. The hook also never fails a job that is merely short on space — a build that would have squeaked through shouldn't be pre-emptively killed; the warning line is the signal that the fleet needs a bigger disk.

Deliberately not reusing the per-ecosystem actions from #4 here — those delete shared preinstalled toolchains, which on a persistent runner is permanent and breaks every later job.

Nothing has been installed on any runner; this is the artefact only. Deploying it needs a decision on where runner config should live (there is no ansible inventory covering the fleet today).

Related: #5, Protect#450.

@goastler

Copy link
Copy Markdown
Member Author

Superseded by https://github.com/prosopo/gh-runner/pull/17, which does this from the two job hooks that already exist in the gh_runner_native ansible role.

Three things changed rather than being ported:

  • No MIN_FREE_GB. The target is the largest disk label the host advertises — that label is a promise GitHub schedules against, so it is exactly the amount that has to be free.
  • No timer. Disk only fills because jobs run, so a job starting or ending is the whole set of moments worth checking, and a hook knows whether a build is in flight where a timer has to guess.
  • No mtime sweep. The runner executes one job at a time, so every workspace but the current one is idle by definition. mtime was wrong in the dangerous direction anyway: a long cargo build writes into target/debug/deps/ and never touches the workspace directory above it.

Also worth recording why the location moved: this repo holds composite actions, so the runner's user, work dir and cargo home all had to be restated here as defaults, and all three were wrong for the fleet — actions-runner vs ghrunner, $RUNNER_HOME/_work vs /var/lib/actions-runner/_work, $RUNNER_HOME/.cargo vs /opt/rust. df on a work dir that does not exist returns nothing, which read as 0 GB free, so the unit would have looked installed and healthy while sweeping paths that were not there. In the ansible role those values are defined once.

@goastler goastler closed this Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant