ops(runners): periodic and low-disk-triggered cleaning for self-hosted runners - #6
Closed
goastler wants to merge 1 commit into
Closed
ops(runners): periodic and low-disk-triggered cleaning for self-hosted runners#6goastler wants to merge 1 commit into
goastler wants to merge 1 commit into
Conversation
Member
Author
|
Superseded by https://github.com/prosopo/gh-runner/pull/17, which does this from the two job hooks that already exist in the Three things changed rather than being ported:
Also worth recording why the location moved: this repo holds composite actions, so the runner's user, work dir and cargo home all had to be restated here as defaults, and all three were wrong for the fleet — |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The self-hosted fleet is non-ephemeral and nothing ever reclaims its disk. Every job leaves build state behind — a single Protect checkout is ~17 GB of
target/, over half of it incremental state that is worthless once the job ends — so a 60 GB runner fills after a handful of branches and then fails jobs late, with a linker error (ar: … No space left on device) that looks like anything but a disk problem. That persistence is also exactly whysave_cargo_cacheskips the GitHub cache backend on self-hosted: local disk is the cache there, and nobody prunes it.Adds
runner-maintenance/:runner-disk-clean.sh— the sweep. Incremental dirs, coldtarget/trees, cold workspace checkouts,~/.cargo/registry/src(regenerated from the.cratearchives, which are kept), coldnode_modules, and dangling docker layers.runner-disk-clean.{service,timer}— nightly at 03:30 with a 30m jitter,Persistent=true,Nice=19+ idle IO so it never competes with a running build. Runs as the runner's user, not root.job-started-hook.sh— forACTIONS_RUNNER_HOOK_JOB_STARTED. No-ops unless free space is underMIN_FREE_GB(default 15), then sweeps with a 1-day cut instead of 7. Checking before the job means the reclaim happens before the build burns its full time.Everything is age-based, never a wipe. Jobs run back to back on these hosts and share
~/.cargo, so state a running job might hold has to survive; the age cut (7 days scheduled, 1 day under pressure) is far longer than any single job. The hook also never fails a job that is merely short on space — a build that would have squeaked through shouldn't be pre-emptively killed; the warning line is the signal that the fleet needs a bigger disk.Deliberately not reusing the per-ecosystem actions from #4 here — those delete shared preinstalled toolchains, which on a persistent runner is permanent and breaks every later job.
Nothing has been installed on any runner; this is the artefact only. Deploying it needs a decision on where runner config should live (there is no ansible inventory covering the fleet today).
Related: #5, Protect#450.