Stop reporting an idle task as CPU-starved, and stop blaming its node

DESCRIPTION

@@ -1,5 +1,5 @@
 Package: EpiModelHPC
-Version: 2.8.2
+Version: 2.8.3
 Date: 2026-07-28
 Title: EpiModel Extensions for High-Performance Computing
 Description: Extension package to EpiModel to run large-scale stochastic network

NEWS.md

@@ -1,3 +1,10 @@
+# EpiModelHPC 2.8.3
+
+## BUG FIXES
+
+- `degen_watch.sh` no longer reports an idle task as CPU-starved, and no longer excludes the node it was running on. `dstate == 0` defaulted to `cpustarv` at any CPU level, but losing a share of the cores to a competitor reads near 50% and a task with no competitor and no I/O wait reads near zero. On a 64-task `swfcalib` array, 100 of 127 confirmed events read 0-3% against 23 at the genuine 51% signature; the nodes blamed were idle and process-free minutes later, and the requeued tasks had never begun work, all stopping inside package loading. `dstate == 0` with CPU at or below `HUNG_CPU` is now classified `hung`: still requeued promptly, but the node is not implicated, matching what the classifier already did for the same condition when one process happened to be in D-state.
+- The confirmation line reports the classification (`CONFIRMED HUNG`, `CONFIRMED CPU-STARVED`, `CONFIRMED IO-STALLED`) instead of a blanket `CONFIRMED STARVED`. The label is what sends an operator looking for a competitor that may never have been there. The `CONFIRMED` token is unchanged, so existing log greps still match.
+
 # EpiModelHPC 2.8.2
 
 ## OTHER

inst/hpc_doctor/README.md

@@ -101,11 +101,12 @@ That was the substantive finding, and it is now fixed. The classifier previously
 
 | condition | verdict | action |
 |---|---|---|
-| `dstate == 0`, CPU below floor | CPU starvation | requeue; exclude the node after `NODE_OFFENSE_LIMIT` repeats |
+| `dstate == 0`, CPU between `HUNG_CPU` and the floor | CPU starvation | requeue; exclude the node after `NODE_OFFENSE_LIMIT` repeats |
+| `dstate == 0`, CPU `<= HUNG_CPU` | hung task | requeue promptly, never exclude (added in the fourth campaign, below) |
 | `dstate >= IO_MIN_FRAC%` of `nproc` | filesystem stall | wait up to `IO_MAX_CYCLES` x `IO_RECHECK`, requeue only if still stalled, never exclude |
 | `dstate < IO_MIN_FRAC%` and CPU `<= HUNG_CPU` | hung task | requeue promptly, skip the wait, never exclude |
 
-`IO_MIN_FRAC` defaults to 50 and `HUNG_CPU` to 5. The ambiguous middle (few processes blocked but CPU well above `HUNG_CPU`) deliberately keeps the old behaviour and takes the wait path, so the change only affects the clearly-hung case. A hang does not exclude the node, because a hang is not evidence of contention; `dstate == 0` is what detects that.
+`IO_MIN_FRAC` defaults to 50 and `HUNG_CPU` to 5. The ambiguous middle (few processes blocked but CPU well above `HUNG_CPU`) deliberately keeps the old behaviour and takes the wait path, so the change only affects the clearly-hung case. A hang does not exclude the node, because a hang is not evidence of contention.
 
 **Requeue cost scales with queue depth.** A requeued array task receives a new `SubmitTime` and re-enters the queue behind everything submitted earlier. On this campaign that put both tasks behind a 3,072-task array, so two stragglers that would have finished in 20 minutes instead gated their family's merge step for the whole run. Requeuing is close to free on an empty cluster and expensive on a full one. Being conservative is particularly warranted when a family is nearly complete and a single task gates a barrier or a merge.
 
@@ -152,6 +153,30 @@ Verified by inverting the scope against two live campaigns sharing nodes. Pointe
 
 The match is also now against the job name field alone rather than the whole `squeue` line. `deploy_doctor.sh` counts against `squeue -o "%j"` while `degen_watch.sh` grepped `%N %j`, so an anchored pattern such as `^doxy-` matched in the wrapper and silently found nothing in the worker: the doctor reported tasks to sweep and then swept nothing, on every sweep, for as long as it ran. Both now match on the name, so anchored and unanchored patterns behave identically in the two scripts.
 
+## Field evidence, fourth campaign (swfcalib, 2026-07-29)
+
+The same `swfcalib` calibration, two days on. 186 sweeps, 127 confirmed events since the array-task fix, 120 requeues across 79 distinct tasks out of 192 run. Two in five tasks needed intervention, which is one to two orders of magnitude above every prior campaign and was the signal that the doctor was diagnosing the wrong thing.
+
+**Most of it was not CPU starvation.** Splitting the confirmed events by their CPU reading:
+
+| reading | events | reads as |
+|---|---|---|
+| 0-3% | 100 | nine processes alive, nothing blocked, no work |
+| 51% | 23 | the orphan-contention signature this tooling was built for |
+| other | 4 | 12, 27, 35, 61% |
+
+Every one of the 310 probe readings had `dstate=0`, so the filesystem branch never fired once.
+
+The 51% group is real and is what the doctor is for. The 0-3% group is a different failure wearing the same label, because the classifier defaulted `dstate == 0` to `cpustarv` regardless of level. That default is wrong on its own terms: losing a share of the cores to a competitor reads near 50%, and a task with no competitor and no I/O wait reads near zero. There is no reading at which "starved" means "idle".
+
+Two independent checks confirmed the misdiagnosis. The nodes blamed most (node18, node19, node20, node21, node22) were IDLE with zero processes when checked afterwards, so the exclusions those events wrote rested on nothing. And the requeued tasks had never started work: measuring how much output each had produced when the doctor killed it gave exactly four values across 56 tasks, 59, 126, 193 and 260 lines, which is the same startup banner repeating once per requeue cycle, with 260 the end of package loading. One inspected directly had sat at `Attaching package: 'dplyr'` for 93 minutes.
+
+So these were tasks wedged during R startup, loading a package library off shared storage, being requeued into starting the same load again. Requeue was still the right action and did eventually clear them, one task completing in 1.6 hours after its restart. Blaming the node was not, and it is actively harmful: excluding healthy nodes on false evidence shrinks the pool and concentrates the next attempt onto fewer machines, which makes the real contention worse.
+
+`degen_watch.sh` now separates them. `dstate == 0` with CPU at or below `HUNG_CPU` is classified `hung`, requeued promptly and does not implicate the node, matching what the classifier already did for the same condition when one process happened to be in D-state. `dstate == 0` above `HUNG_CPU` and below the floor stays `cpustarv` and still escalates to exclusion on repeats.
+
+Worth stating plainly, because the doctor cannot fix it: a startup stall is a storage problem. The doctor's job here is to stop the task burning its walltime and to report honestly what it saw. Requeue volume at this level is a symptom to escalate, not a thing to tune away.
+
 ## Node-level view (open)
 
 Every overnight stall event had two tasks on the same node go D-state simultaneously, which is a node-level filesystem event rather than independent per-task failures, yet the doctor judges each task in isolation. Aggregating D-state across a node's tasks (most/all blocked at once = node event to wait out; a lone blocked task judged on its own) could implement the patience and the ledger gating more directly. This needs more cross-cluster data before committing to a heuristic and is tracked as a GitHub issue.

inst/hpc_doctor/degen_watch.sh

@@ -207,7 +207,24 @@ for s in $suspects; do
   #      (Observed 2026-07-19: nine procs, median 0%, one in D-state, requeued
   #      correctly but only after being routed down the filesystem branch.)
   exclude_node=1; kind=cpustarv
-  if [ "${dst:-0}" -gt 0 ]; then
+  if [ "${dst:-0}" -eq 0 ] && [ "${med:-100}" -le "$HUNG_CPU" ]; then
+    # Nothing blocked and nothing running. CPU starvation means losing a SHARE of
+    # the cores to a competitor, which reads near 50%, not near zero; a task at
+    # 0-3% with no process in D-state has no competitor and no I/O wait, it is
+    # simply not working. Before this branch existed the `dstate == 0` default
+    # swept both cases into `cpustarv` and excluded the node for the second one.
+    #
+    # Measured 2026-07-29 on a 64-task swfcalib array: 100 of 127 confirmed
+    # events read 0-3% with `dstate=0` against 23 at the genuine 51% signature,
+    # and every requeued task's log stopped inside package loading, none had
+    # begun simulating. The nodes blamed were idle and process-free minutes
+    # later, so the exclusions rested on nothing. Requeue is still right, and did
+    # eventually clear them; blaming the node was not.
+    kind=hung
+    exclude_node=0
+    echo "  $tid: HUNG ${med}% with ${npr} proc(s) and none in D-state (idle, not contended)"
+    echo "     no competitor and no I/O wait; requeuing without implicating $node"
+  elif [ "${dst:-0}" -gt 0 ]; then
     exclude_node=0
     [ "${npr:-0}" -gt 0 ] || npr=1          # never divide by zero on a partial probe
     io_pct=$(( dst * 100 / npr ))
@@ -245,7 +262,17 @@ for s in $suspects; do
 
   restarts=$(scontrol show job "$tid" 2>/dev/null | grep -oE "Restarts=[0-9]+" | head -1 | cut -d= -f2)
   restarts=${restarts:-0}
-  echo "  $tid: CONFIRMED STARVED ${med}% on $node (age ${age}m, restarts $restarts)"
+  # Report the CLASSIFICATION, not a blanket "STARVED". Three quarters of one
+  # campaign's events were idle tasks labelled starved, and the label is what
+  # sent the operator looking for a competitor that was never there. The
+  # "CONFIRMED" token is unchanged so existing log greps still match.
+  case "$kind" in
+    cpustarv) verdict="CPU-STARVED";;
+    hung)     verdict="HUNG";;
+    iostall)  verdict="IO-STALLED";;
+    *)        verdict="$kind";;
+  esac
+  echo "  $tid: CONFIRMED $verdict ${med}% on $node (age ${age}m, restarts $restarts)"
 
   # Node offense ledger, gated by CLASSIFICATION. Escalation-to-exclusion only
   # counts CPU-STARVATION events (dstate==0), where the node genuinely harbours