Stop reporting an idle task as CPU-starved, and stop blaming its node - #62
Merged
Conversation
`dstate == 0` defaulted to `cpustarv` at any CPU level. That is wrong on its own terms: losing a share of the cores to a competitor reads near 50%, and a task with no competitor and no I/O wait reads near zero. There is no reading at which "starved" means "idle". Measured on a 64-task swfcalib array over two days: 127 confirmed events, 100 of them at 0-3% against 23 at the genuine 51% signature, and every one of the 310 probe readings had dstate=0 so the filesystem branch never fired. 120 requeues across 79 of 192 tasks, which is one to two orders of magnitude above every prior campaign and is what prompted the look. Two independent checks confirmed the misdiagnosis. The nodes blamed most (node18, node19, node20, node21, node22) were IDLE with zero processes when checked afterwards, so the exclusions those events wrote rested on nothing. And the requeued tasks had never begun work: measuring output produced before each kill gave exactly four values across 56 tasks, 59, 126, 193 and 260 lines, the same startup banner repeating once per requeue cycle, with 260 the end of package loading. One inspected directly had sat at `Attaching package: 'dplyr'` for 93 minutes. So these were tasks wedged in R startup loading a library off shared storage, requeued into starting the same load again. Requeue stays the right action and did eventually clear them. Blaming the node was not, and it is actively harmful: excluding healthy nodes on false evidence shrinks the pool and concentrates the next attempt onto fewer machines, which makes real contention worse. `dstate == 0` with CPU at or below HUNG_CPU is now `hung`: requeued promptly, node not implicated, matching what the classifier already did for the same condition when one process happened to be in D-state. Above HUNG_CPU and below the floor stays `cpustarv` and still escalates to exclusion on repeats. The confirmation line now reports the classification rather than a blanket "STARVED", since the label is what sends an operator hunting a competitor that was never there. The CONFIRMED token is unchanged so existing greps still match. Verified in report-only mode against a live 23-task array on the cluster. At the default HUNG_CPU=5 all 23 idle tasks classify hung with no node implicated; at HUNG_CPU=0 the 1% tasks cross back into cpustarv and the 0% stay hung, so the boundary sits exactly where it is documented. Under the old code all 23 were cpustarv with node exclusion.
#61 landed on main first and took the 2.8.2 slot for the swfcalib `::` change, so the classifier fix moves to 2.8.3. NEWS keeps both entries in order; nothing else conflicted.
smjenness
added a commit
that referenced
this pull request
Jul 29, 2026
Both conflicts were version numbering only. #62 landed as 2.8.3 rather than the 2.8.2 it was written as, because #61 took that slot first, so NEWS here carried a stale 2.8.2 header for the same entry. DESCRIPTION keeps 2.9.0, which is the feature release above it. No code conflicted: #62's classifier changes were already the base of this branch. Smoke-tested the merged scripts in report-only mode against the live campaign. Both feature sets are present and working together: chain tasks report their real footprint and classify `threaded`, the EpiModel-shaped task still judges normally, and the confirmation line carries the classification from #62.
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dstate == 0defaulted tocpustarvat any CPU level. That is wrong on its own terms: losing a share of the cores to a competitor reads near 50%, and a task with no competitor and no I/O wait reads near zero. There is no reading at which "starved" means "idle".Measured on a 64-task swfcalib array over two days: 127 confirmed events, 100 of them at 0-3% against 23 at the genuine 51% signature. All 310 probe readings had
dstate=0, so the filesystem branch never fired. 120 requeues across 79 of 192 tasks, one to two orders of magnitude above every prior campaign, which is what prompted the look.Two independent checks confirmed the misdiagnosis:
Attaching package: 'dplyr'for 93 minutes.These were tasks wedged in R startup loading a library off shared storage, being requeued into starting the same load again. Requeue stays the right action and did eventually clear them, one completing 1.6 hours after its restart. Blaming the node was not, and it is actively harmful: excluding healthy nodes on false evidence shrinks the pool and concentrates the next attempt onto fewer machines, which makes real contention worse.
Change
dstate == 0with CPU at or belowHUNG_CPUis now classifiedhung: requeued promptly, node not implicated. That matches what the classifier already did for the identical condition when one process happened to be in D-state. AboveHUNG_CPUand below the floor stayscpustarvand still escalates to exclusion on repeats.The confirmation line reports the classification (
CONFIRMED HUNG,CONFIRMED CPU-STARVED,CONFIRMED IO-STALLED) rather than a blanketCONFIRMED STARVED, since the label is what sends an operator hunting a competitor that was never there. TheCONFIRMEDtoken is unchanged so existing log greps still match.Verification
Report-only against a live 23-task array. At the default
HUNG_CPU=5all 23 idle tasks classify hung with no node implicated. AtHUNG_CPU=0the 1% tasks cross back intocpustarvand the 0% stay hung, so the boundary sits exactly where it is documented. Under the old code all 23 werecpustarvwith node exclusion.Version 2.8.2, with a NEWS entry and a fourth field-evidence section in the doctor README.