Attribute a task's work by SLURM_JOB_ID, and decline to judge threaded tasks

DESCRIPTION

@@ -1,5 +1,5 @@
 Package: EpiModelHPC
-Version: 2.8.3
+Version: 2.9.0
 Date: 2026-07-28
 Title: EpiModel Extensions for High-Performance Computing
 Description: Extension package to EpiModel to run large-scale stochastic network

NEWS.md

@@ -1,3 +1,16 @@
+# EpiModelHPC 2.9.0
+
+## NEW FEATURES
+
+- The doctor no longer assumes a task's work happens in R. Task membership is decided by `SLURM_JOB_ID` from `/proc/<pid>/environ` rather than by process name, so a compiled child such as an `rstan`/`cmdstanr` `model_<hash>` binary is attributed to the task that launched it. `pgrep -x R` saw only the idle R wrapper and reported `nproc=1 med_cpu=0` for a task consuming four cores. The batch script wrapper is excluded by name so `nproc` still reflects the workload and a starting task keeps its `MIN_PROC` grace period.
+- `probe_node_cpu.sh` reports `top_cpu=`, the busiest single process in a task, and `degen_watch.sh` declines to judge any task whose `top_cpu` exceeds `MULTICORE_CPU` (default 150), reporting it as `threaded`. The whole detector rests on one simulation per core at about 100%, which is what makes a median a starvation signal; a threaded binary at several hundred percent is outside that model, and a healthy 4-thread chain cannot be told from a contended 8-thread one from outside the process. Under-flagging is the correct failure, since the alternative is requeuing a healthy Bayesian fit.
+- `probe_node_cpu.sh` reports `nodeinfo=1 cores= load1=` per node, printed for any node that produced a suspect. The probe only sees the invoking user's processes, so another account's job can starve a task with nothing in the per-task view to show for it. Load against core count separates a suspect on a node at 30 of 32, where contention is real and excluding the node may be justified, from the same suspect on a node at 4 of 32, where it is not.
+
+## BUG FIXES
+
+- `degen_watch.sh` warns when the probe reached its nodes but attributed no process to any running task. The existing guard only covered a probe that returned nothing at all, which the new per-node line makes impossible.
+- `probe_node_cpu.sh` no longer leaks "Permission denied" to stderr when scanning processes it cannot read. `2>/dev/null` after an input redirect is set up only after the failing open, so it never caught the shell's own message; it now precedes the redirect. Only reachable since the scan widened beyond the user's own R processes.
+
 # EpiModelHPC 2.8.3
 
 ## BUG FIXES

inst/hpc_doctor/README.md

@@ -8,7 +8,7 @@ The detector measures CPU utilisation rather than elapsed time. Healthy workers
 
 | script | role |
 |---|---|
-| `probe_node_cpu.sh` | node-side; samples `/proc/<pid>/stat` twice and differences it. One ssh per node covers every task on it. Prints `jobid= taskid= nproc= med_cpu= dstate=` per task. |
+| `probe_node_cpu.sh` | node-side; samples `/proc/<pid>/stat` twice and differences it. One ssh per node covers every task on it. Prints a `nodeinfo=` line, then `jobid= taskid= nproc= med_cpu= top_cpu= dstate=` per task. |
 | `degen_watch.sh` | per-task verdict; report-only by default, `--requeue` acts. `PATTERN` scopes both the nodes probed and the tasks judged. |
 | `deploy_doctor.sh` | entry point, `sbatch` this. Sweep loop for the life of a campaign, self-terminating. Carries its own `#SBATCH` defaults. |
 | `term_orphans.sh` | SIGTERMs genuinely orphaned workers, safe on shared nodes |
@@ -85,8 +85,22 @@ Counting now uses `squeue -r`, so a collapsed pending range such as `41746874_[4
 - Requeue, never cancel. A requeued task re-runs its own unit and leaves no gap.
 - `ExcNodeList` can only be set on a pending task. The working sequence is requeue, hold, update, release.
 - Address array tasks as `<ArrayJobId>_<TaskId>`. A bare `ArrayJobId` given to `scontrol requeue` restarts every task in the array.
+- Decide task membership by `SLURM_JOB_ID`, not by process name. A task's work is not always in R, and children inherit the variable.
+- Judge only what the one-core-per-simulation model describes. A process above `MULTICORE_CPU` is threaded and its median is not a starvation signal.
 - Apply `PATTERN` to tasks, not only to nodes. The probe reports every R process the user owns on a probed node, so a node shared with another of the user's campaigns yields that campaign's tasks too.
 
+## What this detector can and cannot see
+
+The method rests on one assumption: **one simulation per core, each pegged near 100%**. That is what makes a median across a task's processes a starvation signal, and why healthy at ~99% against starved at ~50% is a wide, unambiguous gap. Everything else follows from it.
+
+Two kinds of workload break the assumption, and the doctor handles them differently.
+
+**Multi-threaded tasks are declined, not guessed at.** An `rstan`/`cmdstanr` task is an idle R wrapper plus a compiled `model_<hash>` binary running the chains at several hundred percent. There is no median to speak of, the ~50% signature does not apply, and nothing on the node says how many threads the chain asked for, so a healthy 4-thread run and a contended 8-thread one are indistinguishable from outside. A task whose busiest process exceeds `MULTICORE_CPU` (default 150, above any single-threaded R process including threaded-BLAS bursts) is reported `threaded` and skipped. Under-flagging is the right failure here: the alternative is requeuing someone's healthy Bayesian fit.
+
+Membership in a task is therefore decided by `SLURM_JOB_ID` from `/proc/<pid>/environ` rather than by process name. `pgrep -x R` saw only the idle wrapper and reported `nproc=1 med_cpu=0`, which read as a task doing nothing; the compiled binary inherits the variable and is now attributed correctly. Verified on a live cmdstan array, where the model binary carries both `SLURM_JOB_ID` and `SLURM_ARRAY_TASK_ID`. The batch script wrapper is excluded by name, since counting it would inflate `nproc` past `MIN_PROC` and cost a starting task its grace period.
+
+**Other users are invisible, so the node is reported instead.** The probe only ever sees processes owned by the invoking user, which means a co-tenant from another account can starve one of our tasks with nothing in the per-task view to show for it. `nodeinfo=1 cores= load1=` is now reported per node and printed for any node that produced a suspect. One number separates the two readings that matter: a suspect on a node at load 30 of 32 is contention and excluding the node may be justified; the same suspect on a node at load 4 of 32 is not, and the node is not at fault. Measured while writing this: nodes running a 16-CPU-per-task Bayesian campaign sat at load 8 of 32, holding cores they were not using, which creates queue pressure for everyone else but does not starve a co-tenant.
+
 ## Field evidence, first production campaign
 
 An 5,856-task, eight-family campaign on RSPH, roughly 143 concurrent tasks over a saturated cluster. Several requeues over the run, all but one on a single node (node4). Most were filesystem stalls, and one was a hung task (the case that motivated the classifier fix, below). One, at roughly the six-hour mark, was **the orphan-contention mode this tooling was built for**: a full nine-process footprint at 51% CPU with `dstate=0`. The doctor requeued it and added node4 to its `ExcNodeList`, which is the designed response. That was the first in-the-wild sighting here; the mode is real but, at least on this cluster in this window, rarer than the filesystem stalls that dominate the log.

inst/hpc_doctor/degen_watch.sh

@@ -68,6 +68,11 @@ IO_RECHECK=${IO_RECHECK:-300}   # seconds per D-state (filesystem) stall recheck
 IO_MAX_CYCLES=${IO_MAX_CYCLES:-3}  # recheck cycles a filesystem stall gets before requeue (they usually clear)
 IO_MIN_FRAC=${IO_MIN_FRAC:-50}  # pct of a task's procs in D-state to call it a filesystem stall
 HUNG_CPU=${HUNG_CPU:-5}         # median cpu% at or below this, with few procs blocked, = hung
+# A single process above this is multi-threaded, so the one-sim-one-core model
+# this detector is built on does not describe it and the median is not a
+# starvation signal. 150 rather than 100 leaves headroom for the brief bursts a
+# nominally single-threaded R process gets from a threaded BLAS.
+MULTICORE_CPU=${MULTICORE_CPU:-150}
 NODE_OFFENSE_LIMIT=${NODE_OFFENSE_LIMIT:-3}   # confirmed CPU-STARVATION events on one node before it is a repeat offender
 # Campaign-scoped node offense ledger. Keyed to the doctor's own SLURM job so a
 # new campaign starts a clean ledger and concurrent doctors (e.g. two projects)
@@ -105,7 +110,7 @@ probe_all() {  # -> "<node> jobid=.. nproc=.. med_cpu=.. dstate=.."
   for n in $nodes; do
     ssh -o StrictHostKeyChecking=no -o ConnectTimeout=8 "$n" \
         "bash $ROOT/probe_node_cpu.sh $INT" 2>/dev/null \
-      | grep -E '^jobid=' | sed "s|^|$n |"
+      | grep -E '^(jobid|nodeinfo)=' | sed "s|^|$n |"
   done
 }
 
@@ -124,8 +129,15 @@ if [ -z "$probe_out" ]; then
 fi
 
 suspects=""
+declare -A NODEINFO
 while read -r node rest; do
   [ -n "${node:-}" ] || continue
+  # Node-level context, kept aside and printed only for nodes that produce a
+  # suspect. It is the difference between "the node is oversubscribed" and "the
+  # node is idle and my task is stuck on something else", and the per-task CPU
+  # cannot distinguish those.
+  case "$rest" in nodeinfo=*) NODEINFO[$node]="${rest#nodeinfo=1 }"; continue;; esac
+  seen_tasks=$(( ${seen_tasks:-0} + 1 ))
   jid=$(sed -n 's/.*jobid=\([0-9]*\).*/\1/p' <<< "$rest")
   med=$(sed -n 's/.*med_cpu=\([0-9]*\).*/\1/p' <<< "$rest")
   [ -n "${jid:-}" ] && [ -n "${med:-}" ] || continue
@@ -147,6 +159,20 @@ while read -r node rest; do
   if [ "${np:-0}" -lt "$MIN_PROC" ]; then
     printf "  startup   %-10s %-8s %s (nproc<$MIN_PROC: master-only, not judged)\n" "$jid" "$node" "$rest"; continue
   fi
+  # Multi-threaded workloads are outside this detector's model, so decline to
+  # judge them rather than guessing. The whole method rests on one simulation
+  # per core at about 100%, which makes a median across the task's processes a
+  # starvation signal. An rstan/cmdstanr task is an idle R wrapper plus a
+  # compiled binary at several hundred percent: the median is meaningless, the
+  # ~50% starvation signature does not apply, and there is no way to tell a
+  # healthy 4-thread chain from a contended 8-thread one without knowing how
+  # many threads it asked for. Under-flagging is the right failure here, because
+  # the alternative is requeuing someone's healthy Bayesian fit.
+  top=$(sed -n 's/.*top_cpu=\([0-9]*\).*/\1/p' <<< "$rest")
+  if [ "${top:-0}" -ge "$MULTICORE_CPU" ]; then
+    printf "  threaded  %-10s %-8s %s (top_cpu>=${MULTICORE_CPU}%%: not the one-core-per-sim model, not judged)\n" \
+           "$jid" "$node" "$rest"; continue
+  fi
   if [ "$med" -ge "$CPU_FLOOR" ]; then
     printf "  ok        %-10s %-8s %s\n" "$jid" "$node" "$rest"
   else
@@ -156,8 +182,26 @@ while read -r node rest; do
 done <<< "$probe_out"
 
 suspects=$(echo $suspects)
+
+# The probe now always returns a node line, so "nothing came back" no longer
+# covers the case where the probe RAN and attributed no process to any task.
+# SLURM says these nodes are running our work; if none of it is visible, the
+# attribution is broken, not the campaign. A warning rather than an error,
+# because a task genuinely between steps can be process-free for a moment.
+if [ "${seen_tasks:-0}" -eq 0 ]; then
+  echo "!! probe reached $(wc -w <<< "$nodes") node(s) but attributed no process to any of $(wc -l <<< "$mine") running task(s)" >&2
+  echo "!! expected at least one process per task; check that the tasks are really running and that SLURM_JOB_ID is set in their environment" >&2
+fi
 [ -n "$suspects" ] || { echo "---"; echo "all tasks healthy (median CPU >= ${CPU_FLOOR}%)"; exit 0; }
 
+# Node context for the nodes that produced a suspect. A starved task on a node
+# at load 30 of 32 is contention; the same reading on a node at load 8 of 32 is
+# not, and the difference decides whether excluding the node is justified. Other
+# users' processes never appear in the per-task view, so this is the only place
+# their weight shows up at all.
+for sn in $(tr ' ' '\n' <<< "$suspects" | awk -F: 'NF{print $2}' | sort -u); do
+  [ -n "${NODEINFO[$sn]:-}" ] && echo "  node      $sn  ${NODEINFO[$sn]}"
+done
 echo "--- strike 1 done; re-checking suspects in ${RECHECK}s (transient dips must not trigger) ---"
 sleep "$RECHECK"
 

inst/hpc_doctor/probe_node_cpu.sh

@@ -23,19 +23,57 @@
 ## bare ArrayJobId. For a non-array job the two are equal.
 ##
 ## Usage:  probe_node_cpu.sh [interval_secs]
-## Output: one line per SLURM task ->
-##         jobid=<numeric> taskid=<numeric|arrayjob_task> nproc=<n> med_cpu=<pct> dstate=<n>
+## Output: one node line, then one line per SLURM task ->
+##         nodeinfo=1 cores=<n> load1=<x>
+##         jobid=<numeric> taskid=<numeric|arrayjob_task> nproc=<n> med_cpu=<pct>
+##           top_cpu=<pct> dstate=<n>
+##
+## `med_cpu` is the median across the task's processes and is the starvation
+## signal for the one-sim-one-core model. `top_cpu` is the busiest single
+## process and is what identifies a workload that model does not describe: an
+## rstan/cmdstanr task is one idle R wrapper plus a threaded binary at several
+## hundred percent, where a median is meaningless.
 set -u
 INT="${1:-5}"
 HZ=$(getconf CLK_TCK 2>/dev/null || echo 100)
 tmp=$(mktemp) || exit 1
 trap 'rm -f "$tmp"' EXIT
 
+# Node-level context, reported whether or not this user has anything here. A
+# task's own CPU says nothing about WHY it is low, and the per-task view cannot
+# see the other tenants: processes owned by other users are invisible to this
+# probe by construction. Load against core count is the one cheap number that
+# distinguishes "this node is oversubscribed" from "this node is idle and my
+# task is stuck on something else", which is exactly the question a starved
+# reading raises.
+printf 'nodeinfo=1 cores=%s load1=%s\n' \
+  "$(nproc 2>/dev/null || echo '?')" \
+  "$(awk '{print $1}' /proc/loadavg 2>/dev/null || echo '?')"
+
 # Timestamp EACH sample from /proc/uptime rather than assuming the nominal
 # interval: reading ~36 procs takes real time, so dividing by INT alone inflates
 # every reading (measured ~1.27x, giving impossible >100% for 1-thread workers).
-for p in $(pgrep -u "$USER" -x R 2>/dev/null); do
-  env_kv=$(tr '\0' '\n' < "/proc/$p/environ" 2>/dev/null)
+# Every process the user owns, not just those named `R`, filtered to those that
+# carry a SLURM_JOB_ID. Membership of a task is decided by that variable rather
+# than by the process name, because a task's real work is not always in R: an
+# rstan/cmdstanr job is an idle R wrapper plus a compiled `model_<hash>` binary
+# doing everything at several hundred percent, and `pgrep -x R` saw only the
+# wrapper. Children inherit the variable, so the binary is attributed correctly
+# (verified on a live cmdstan array: the model binary carries both
+# SLURM_JOB_ID and SLURM_ARRAY_TASK_ID). Login shells, sshd and the probe's own
+# processes have no SLURM_JOB_ID and drop out for free.
+for p in $(pgrep -u "$USER" 2>/dev/null); do
+  # The batch script wrapper is always present, always idle, and is never the
+  # workload. Counting it would inflate `nproc` past MIN_PROC during startup and
+  # cost the master-only grace period that keeps a starting task from being
+  # judged.
+  case "$(cat "/proc/$p/comm" 2>/dev/null)" in slurm_script) continue;; esac
+  # `2>/dev/null` BEFORE the input redirect, not after. Redirections apply left
+  # to right, so a trailing one is set up after the open has already failed and
+  # the shell has already written "Permission denied" to the real stderr. Only
+  # matters now that the scan is every process rather than our own R ones:
+  # setuid processes are unreadable and short-lived ones vanish mid-scan.
+  env_kv=$(tr '\0' '\n' 2>/dev/null < "/proc/$p/environ")
   jid=$(sed -n 's/^SLURM_JOB_ID=//p' <<< "$env_kv" | head -1)
   [ -n "${jid:-}" ] || continue
   # The array-qualified id, taken from the task's own environment rather than
@@ -73,6 +111,12 @@ done < "$tmp" | awk '
       c  = m - lo + 1
       med = (c % 2) ? a[lo + int((c-1)/2)] \
                     : int((a[lo + int(c/2) - 1] + a[lo + int(c/2)]) / 2)
-      printf "jobid=%s taskid=%s nproc=%d med_cpu=%d dstate=%d\n", j, tid[j], n[j], med, (j in d ? d[j] : 0)
+      # The busiest single process, which is what tells a MULTI-THREADED workload
+      # apart from a starved single-threaded one. Nothing in the one-sim-one-core
+      # model ever exceeds ~100%; a reading well above it means threads, and the
+      # median stops being interpretable. `degen_watch.sh` uses this to decline
+      # to judge rather than to guess.
+      printf "jobid=%s taskid=%s nproc=%d med_cpu=%d top_cpu=%d dstate=%d\n", \
+             j, tid[j], n[j], med, a[m], (j in d ? d[j] : 0)
     }
   }'