On a 64-task swfcalib array the doctor requeued 41 times across two wave iterations, roughly 128 task-runs. That is about a 32% intervention rate against the usual sub-0.1%. Every one of them was a false positive, and the cause is that the hung/starved classifier reads med_cpu but never top_cpu, even though the probe collects both.
Ran on 2.9.2, project LA-PrEP-2026, array 41779950 / 41780428.
Evidence
Every SUSPECT line the doctor raised over the campaign:
| measure |
value |
top_cpu >= 90 |
136 of 136 (129 at 99%, 7 at 100%) |
med_cpu |
134 at 0%, 2 at 1% |
dstate |
0 throughout |
nproc |
9 throughout (master + 8 workers, the healthy footprint) |
A representative line:
SUSPECT 41779978 node121 jobid=41779978 taskid=41779950_27 nproc=9 med_cpu=0 top_cpu=99 dstate=0
So in every case a process was pinned at a full core, nothing was blocked on I/O, and no process had died. None of these tasks was hung.
Mechanism
In the judging loop, med and dst are extracted from the probe string but top is not:
med=$(sed -n 's/.*med_cpu=\([0-9]*\).*/\1/p' <<< "$rest") # line 228
dst=$(sed -n 's/.*dstate=\([0-9]*\).*/\1/p' <<< "$rest") # line 229
and the classifier then decides on those two alone:
exclude_node=1; kind=cpustarv # line 263
if [ "${dst:-0}" -eq 0 ] && [ "${med:-100}" -le "$HUNG_CPU" ]; then # line 264
kind=hungThe comment on that branch reads "Nothing blocked and nothing running", but only the first half is tested. top is parsed at line 181 for the MULTICORE_CPU threaded exemption and is not carried into the judging loop, so a task with eight idle workers and one at 99% takes the hung path.
The median is a sound health metric for a netsim_scenarios array, where every worker runs one simulation for the whole task and they all stay busy together. It is not sound for a task whose worker activity is uneven: the median over nine processes goes to 0 as soon as fewer than half are active, which for swfcalib step 4 happens routinely rather than exceptionally.
Why the task logs do not disambiguate this
Worth recording, because it sent me the wrong way first and it is quoted in the source. The comment on the hung branch cites 2026-07-29 evidence that "every requeued task's log stopped inside package loading, none had begun simulating".
That inference does not hold for swfcalib step 4. A healthy, completed task's entire 601-line log is package-loading boilerplate and nothing else:
32 statnet.org banner lines
16 "The following objects are masked from 'package:base'"
16 snctrl masking notices
9 renv messages (9 = master + 8 workers)
The model function emits no per-simulation output, so "the log stopped inside package loading" is the appearance of every task in this workflow whether it is working or wedged. It is a null signal here, not evidence.
Proposed change
Extract top in the judging loop and require it to be low before calling a task hung. A task with a process at or near a full core is doing work by definition, whatever the median says:
top=$(sed -n 's/.*top_cpu=\([0-9]*\).*/\1/p' <<< "$rest")
...
if [ "${dst:-0}" -eq 0 ] && [ "${med:-100}" -le "$HUNG_CPU" ] && [ "${top:-100}" -le "$ACTIVE_CPU" ]; thenwith something like ACTIVE_CPU=${ACTIVE_CPU:-50}, above the noise a genuinely idle task shows and well below one core. A task at med_cpu=0 top_cpu=99 then falls through to the existing cpustarv path, which is also wrong for it, so the cleaner form may be to spare it outright with a log line naming the uneven-worker case.
Fixing it here rather than dropping swfcalib from a project's PATTERN seems better: the pattern approach fixes one project and leaves the same trap for the next workflow whose workers drain unevenly.
Impact
The requeues discarded real work rather than rescuing stuck tasks. Live tasks reached Restarts=2 against MAX_RESTARTS=3, and the first wave iteration took about 2.5h against an expected 1.6h. No nodes were wrongly excluded, since the 2.8.3 split correctly sets exclude_node=0 on this path, so the damage was confined to lost compute.
The doctor is off for the rest of this campaign.
On a 64-task
swfcalibarray the doctor requeued 41 times across two wave iterations, roughly 128 task-runs. That is about a 32% intervention rate against the usual sub-0.1%. Every one of them was a false positive, and the cause is that the hung/starved classifier readsmed_cpubut nevertop_cpu, even though the probe collects both.Ran on 2.9.2, project LA-PrEP-2026, array
41779950/41780428.Evidence
Every SUSPECT line the doctor raised over the campaign:
top_cpu >= 90med_cpudstatenprocA representative line:
So in every case a process was pinned at a full core, nothing was blocked on I/O, and no process had died. None of these tasks was hung.
Mechanism
In the judging loop,
medanddstare extracted from the probe string buttopis not:and the classifier then decides on those two alone:
The comment on that branch reads "Nothing blocked and nothing running", but only the first half is tested.
topis parsed at line 181 for theMULTICORE_CPUthreaded exemption and is not carried into the judging loop, so a task with eight idle workers and one at 99% takes the hung path.The median is a sound health metric for a
netsim_scenariosarray, where every worker runs one simulation for the whole task and they all stay busy together. It is not sound for a task whose worker activity is uneven: the median over nine processes goes to 0 as soon as fewer than half are active, which forswfcalibstep 4 happens routinely rather than exceptionally.Why the task logs do not disambiguate this
Worth recording, because it sent me the wrong way first and it is quoted in the source. The comment on the hung branch cites 2026-07-29 evidence that "every requeued task's log stopped inside package loading, none had begun simulating".
That inference does not hold for
swfcalibstep 4. A healthy, completed task's entire 601-line log is package-loading boilerplate and nothing else:The model function emits no per-simulation output, so "the log stopped inside package loading" is the appearance of every task in this workflow whether it is working or wedged. It is a null signal here, not evidence.
Proposed change
Extract
topin the judging loop and require it to be low before calling a task hung. A task with a process at or near a full core is doing work by definition, whatever the median says:with something like
ACTIVE_CPU=${ACTIVE_CPU:-50}, above the noise a genuinely idle task shows and well below one core. A task atmed_cpu=0 top_cpu=99then falls through to the existingcpustarvpath, which is also wrong for it, so the cleaner form may be to spare it outright with a log line naming the uneven-worker case.Fixing it here rather than dropping
swfcalibfrom a project'sPATTERNseems better: the pattern approach fixes one project and leaves the same trap for the next workflow whose workers drain unevenly.Impact
The requeues discarded real work rather than rescuing stuck tasks. Live tasks reached
Restarts=2againstMAX_RESTARTS=3, and the first wave iteration took about 2.5h against an expected 1.6h. No nodes were wrongly excluded, since the 2.8.3 split correctly setsexclude_node=0on this path, so the damage was confined to lost compute.The doctor is off for the rest of this campaign.