Announce a task that exhausts its restart cap - #66
Merged
Conversation
MAX_RESTARTS is the last thing between a pathological task and an unbounded requeue loop, and reaching it says something definite: the doctor has stopped intervening, and the task will hold its slot until walltime producing nothing. It said so with one lowercase `restarts exhausted; leaving it` in the middle of a verbose sweep. That is the same shape of silent ending as a TIME_LIMIT kill mailed to nobody, which is what this tooling exists because of. Seen again on a fifth campaign, running the pre-2.8.3 classifier. swfcalib task 41767827_48 wedged in PSOCK worker startup and was requeued three times. Segmenting its accumulated log by attempt, since SLURM appends across requeues, gives 3 of 8 workers through package loading on the first attempt, 3 of 8 on the second, 1 of 8 on the third, then all 8 on the fourth, which ran 1.42 hours and completed. About 4.5 hours to deliver one 95-minute batch, and nothing surfaced it except reading the sweep log by hand afterwards. The cap now emits a `!! EXHAUSTED` token with the task, its classification, the node and the restart count; the sweep summary gains an `exhausted=` counter; and MAIL_TO, when set, receives one message per newly exhausted task. Once per task, not once per sweep. The doctor re-probes every ten minutes and an exhausted task stays exhausted, so an undeduplicated alert would repeat until walltime and train the reader to skip it, which would leave the state as silent as it was. The campaign-scoped ledger already in STATE_FILE is the dedup key, and its `exhausted <taskid>` rows cannot collide with the node-offense rows, which are matched on a leading node name. Behaviour is unchanged for every task below the cap, and the escalation is report-only in the sense that matters: it still declines to requeue.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MAX_RESTARTSis the last thing between a pathological task and an unbounded requeue loop, and reaching it says something definite: the doctor has stopped intervening, and the task will hold its slot until walltime producing nothing. It said so with one lowercaserestarts exhausted; leaving itin the middle of a verbose sweep, which is the same shape of silent ending as a TIME_LIMIT kill mailed to nobody.The README already flagged this in the third-campaign write-up ("That cap was doing work it was never meant to do"), and a fifth campaign hit it again.
What happened
swfcalibtask41767827_48, running the pre-2.8.3 classifier, wedged in PSOCK worker startup and was requeued three times. SLURM appends across requeues, so one task log holds every attempt; segmenting it by the workflow's start markers and counting workers that finished loading statnet:About 4.5 hours to deliver one 95-minute batch. Nothing surfaced it except reading the sweep log by hand afterwards. (The node exclusions those events also wrote are the thing #62 fixed; this PR is only about the terminal state being audible.)
Change
!! EXHAUSTEDtoken carrying task, classification, node and restart count, replacing the lowercase line.exhausted=counter in the sweep summary alongsideconfirmed_starved_requeued=andcleared_as_transient=.MAIL_TO, when set andmailis available, gets one message per newly exhausted task.Emitted once per task, not once per sweep. The doctor re-probes every ten minutes and an exhausted task stays exhausted, so an undeduplicated alert would repeat until walltime and train the reader to skip it, leaving the state as silent as before. The campaign-scoped ledger already in
STATE_FILEis the dedup key.Notes
exhausted <taskid>rows cannot collide with the node-offense rows: those are matched with^$nodeand^$node cpustarv$, both anchored on a leading node name.MAIL_TOis unset by default, so this is opt-in and no existing deployment starts mailing.Testing
bash -nclean;devtools::load_all()clean. The exhausted branch was exercised standalone across repeated sweeps: it alerts once for a task at the cap, stays silent on the next two sweeps for the same task, alerts separately for a second task, leaves a task under the cap on the requeue path, and lands the counter at 2.