Skip to content

Stop reporting an idle task as CPU-starved, and stop blaming its node - #62

Merged
smjenness merged 2 commits into
mainfrom
dev/doctor-idle-not-starved
Jul 29, 2026
Merged

Stop reporting an idle task as CPU-starved, and stop blaming its node#62
smjenness merged 2 commits into
mainfrom
dev/doctor-idle-not-starved

Conversation

@smjenness

Copy link
Copy Markdown
Collaborator

dstate == 0 defaulted to cpustarv at any CPU level. That is wrong on its own terms: losing a share of the cores to a competitor reads near 50%, and a task with no competitor and no I/O wait reads near zero. There is no reading at which "starved" means "idle".

Measured on a 64-task swfcalib array over two days: 127 confirmed events, 100 of them at 0-3% against 23 at the genuine 51% signature. All 310 probe readings had dstate=0, so the filesystem branch never fired. 120 requeues across 79 of 192 tasks, one to two orders of magnitude above every prior campaign, which is what prompted the look.

Two independent checks confirmed the misdiagnosis:

  • The nodes blamed most (node18 through node22) were IDLE with zero processes when checked afterwards, so the exclusions those events wrote rested on nothing.
  • The requeued tasks had never begun work. Measuring output produced before each kill gave exactly four values across 56 tasks, 59, 126, 193 and 260 lines, which is the same startup banner repeating once per requeue cycle, with 260 the end of package loading. One inspected directly had sat at Attaching package: 'dplyr' for 93 minutes.

These were tasks wedged in R startup loading a library off shared storage, being requeued into starting the same load again. Requeue stays the right action and did eventually clear them, one completing 1.6 hours after its restart. Blaming the node was not, and it is actively harmful: excluding healthy nodes on false evidence shrinks the pool and concentrates the next attempt onto fewer machines, which makes real contention worse.

Change

dstate == 0 with CPU at or below HUNG_CPU is now classified hung: requeued promptly, node not implicated. That matches what the classifier already did for the identical condition when one process happened to be in D-state. Above HUNG_CPU and below the floor stays cpustarv and still escalates to exclusion on repeats.

The confirmation line reports the classification (CONFIRMED HUNG, CONFIRMED CPU-STARVED, CONFIRMED IO-STALLED) rather than a blanket CONFIRMED STARVED, since the label is what sends an operator hunting a competitor that was never there. The CONFIRMED token is unchanged so existing log greps still match.

Verification

Report-only against a live 23-task array. At the default HUNG_CPU=5 all 23 idle tasks classify hung with no node implicated. At HUNG_CPU=0 the 1% tasks cross back into cpustarv and the 0% stay hung, so the boundary sits exactly where it is documented. Under the old code all 23 were cpustarv with node exclusion.

Version 2.8.2, with a NEWS entry and a fourth field-evidence section in the doctor README.

`dstate == 0` defaulted to `cpustarv` at any CPU level. That is wrong on its own
terms: losing a share of the cores to a competitor reads near 50%, and a task
with no competitor and no I/O wait reads near zero. There is no reading at which
"starved" means "idle".

Measured on a 64-task swfcalib array over two days: 127 confirmed events, 100 of
them at 0-3% against 23 at the genuine 51% signature, and every one of the 310
probe readings had dstate=0 so the filesystem branch never fired. 120 requeues
across 79 of 192 tasks, which is one to two orders of magnitude above every
prior campaign and is what prompted the look.

Two independent checks confirmed the misdiagnosis. The nodes blamed most
(node18, node19, node20, node21, node22) were IDLE with zero processes when
checked afterwards, so the exclusions those events wrote rested on nothing. And
the requeued tasks had never begun work: measuring output produced before each
kill gave exactly four values across 56 tasks, 59, 126, 193 and 260 lines, the
same startup banner repeating once per requeue cycle, with 260 the end of
package loading. One inspected directly had sat at `Attaching package: 'dplyr'`
for 93 minutes.

So these were tasks wedged in R startup loading a library off shared storage,
requeued into starting the same load again. Requeue stays the right action and
did eventually clear them. Blaming the node was not, and it is actively harmful:
excluding healthy nodes on false evidence shrinks the pool and concentrates the
next attempt onto fewer machines, which makes real contention worse.

`dstate == 0` with CPU at or below HUNG_CPU is now `hung`: requeued promptly,
node not implicated, matching what the classifier already did for the same
condition when one process happened to be in D-state. Above HUNG_CPU and below
the floor stays `cpustarv` and still escalates to exclusion on repeats.

The confirmation line now reports the classification rather than a blanket
"STARVED", since the label is what sends an operator hunting a competitor that
was never there. The CONFIRMED token is unchanged so existing greps still match.

Verified in report-only mode against a live 23-task array on the cluster. At the
default HUNG_CPU=5 all 23 idle tasks classify hung with no node implicated; at
HUNG_CPU=0 the 1% tasks cross back into cpustarv and the 0% stay hung, so the
boundary sits exactly where it is documented. Under the old code all 23 were
cpustarv with node exclusion.
#61 landed on main first and took the 2.8.2 slot for the swfcalib `::` change,
so the classifier fix moves to 2.8.3. NEWS keeps both entries in order; nothing
else conflicted.
@smjenness
smjenness merged commit 5d0dd36 into main Jul 29, 2026
2 checks passed
@smjenness
smjenness deleted the dev/doctor-idle-not-starved branch July 29, 2026 11:47
smjenness added a commit that referenced this pull request Jul 29, 2026
Both conflicts were version numbering only. #62 landed as 2.8.3 rather than the
2.8.2 it was written as, because #61 took that slot first, so NEWS here carried
a stale 2.8.2 header for the same entry. DESCRIPTION keeps 2.9.0, which is the
feature release above it. No code conflicted: #62's classifier changes were
already the base of this branch.

Smoke-tested the merged scripts in report-only mode against the live campaign.
Both feature sets are present and working together: chain tasks report their
real footprint and classify `threaded`, the EpiModel-shaped task still judges
normally, and the confirmation line carries the classification from #62.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant