From eeec62e1bbbec86ff6f61f710c95acceffa948a2 Mon Sep 17 00:00:00 2001 From: Samuel Jenness Date: Wed, 29 Jul 2026 07:31:41 -0400 Subject: [PATCH] Attribute a task's work by SLURM_JOB_ID, and decline to judge threaded tasks `pgrep -x R` decided which processes belonged to a task, which assumes the work happens in R. An rstan/cmdstanr task is an idle R wrapper plus a compiled `model_` binary running the chains, so the probe reported `nproc=1 med_cpu=0` for a task consuming four cores. It survived only by accident, spared by the `nproc < MIN_PROC` startup rule; a second R process, or a moment between phases, and the doctor would have requeued a healthy Bayesian fit. Observed on a live 23-task campaign sharing this cluster. Membership is now decided by SLURM_JOB_ID from /proc//environ. Children inherit it, so the binary is attributed correctly; verified against a running cmdstan array where the model binary carries both SLURM_JOB_ID and SLURM_ARRAY_TASK_ID. Login shells and the probe's own processes have no such variable and drop out for free. The batch script wrapper is excluded by name, so `nproc` keeps meaning the workload and a starting task keeps its grace period. Seeing the work is not the same as being able to judge it. The detector rests on one simulation per core at about 100%, which is what makes a median a starvation signal and gives the 99-against-51 gap its width. A threaded binary at several hundred percent has no meaningful median, does not show the ~50% signature, and carries nothing on the node to say how many threads it asked for, so a healthy 4-thread chain and a contended 8-thread one are indistinguishable from outside. Tasks whose busiest process exceeds MULTICORE_CPU (default 150, clear of any single-threaded R process including threaded-BLAS bursts) are reported `threaded` and skipped. Under-flagging is the right failure here. Third piece: the probe only ever sees the invoking user's processes, so a co-tenant on another account can starve one of our tasks with nothing in the per-task view to show for it. Each node now reports `cores=` and `load1=`, and that line is printed for any node producing a suspect. It separates the two readings that actually differ: a suspect on a node at load 30 of 32 is contention and exclusion may be justified, the same suspect at 4 of 32 is not. Measured while writing this, the Bayesian campaign's nodes sat at load 8 of 32, holding cores they were not using, which is queue pressure rather than starvation. Verified in report-only mode against that live campaign. All 21 chain tasks now report their real footprint (nproc 2-5, ~397%) and classify `threaded` instead of the old `nproc=1 med_cpu=0`; the two EpiModel-shaped tasks on the same campaign still read nproc=5 med_cpu=99-100 and judge normally; forcing the floor to 100 makes one a suspect and prints `node node1 cores=32 load1=4.17` beside it; raising MULTICORE_CPU past the chains drops them back into the normal path, so the boundary sits where it is documented. Also fixes a stderr leak the wider scan exposed: `2>/dev/null` placed after an input redirect is set up only after the failing open, so it never caught the shell's own "Permission denied" on unreadable processes. --- DESCRIPTION | 2 +- NEWS.md | 13 ++++++++ inst/hpc_doctor/README.md | 16 ++++++++- inst/hpc_doctor/degen_watch.sh | 46 +++++++++++++++++++++++++- inst/hpc_doctor/probe_node_cpu.sh | 54 ++++++++++++++++++++++++++++--- 5 files changed, 123 insertions(+), 8 deletions(-) diff --git a/DESCRIPTION b/DESCRIPTION index 2a2f7ae..addccb0 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,5 +1,5 @@ Package: EpiModelHPC -Version: 2.8.2 +Version: 2.9.0 Date: 2026-07-28 Title: EpiModel Extensions for High-Performance Computing Description: Extension package to EpiModel to run large-scale stochastic network diff --git a/NEWS.md b/NEWS.md index 7ff8297..66ffc23 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,16 @@ +# EpiModelHPC 2.9.0 + +## NEW FEATURES + +- The doctor no longer assumes a task's work happens in R. Task membership is decided by `SLURM_JOB_ID` from `/proc//environ` rather than by process name, so a compiled child such as an `rstan`/`cmdstanr` `model_` binary is attributed to the task that launched it. `pgrep -x R` saw only the idle R wrapper and reported `nproc=1 med_cpu=0` for a task consuming four cores. The batch script wrapper is excluded by name so `nproc` still reflects the workload and a starting task keeps its `MIN_PROC` grace period. +- `probe_node_cpu.sh` reports `top_cpu=`, the busiest single process in a task, and `degen_watch.sh` declines to judge any task whose `top_cpu` exceeds `MULTICORE_CPU` (default 150), reporting it as `threaded`. The whole detector rests on one simulation per core at about 100%, which is what makes a median a starvation signal; a threaded binary at several hundred percent is outside that model, and a healthy 4-thread chain cannot be told from a contended 8-thread one from outside the process. Under-flagging is the correct failure, since the alternative is requeuing a healthy Bayesian fit. +- `probe_node_cpu.sh` reports `nodeinfo=1 cores= load1=` per node, printed for any node that produced a suspect. The probe only sees the invoking user's processes, so another account's job can starve a task with nothing in the per-task view to show for it. Load against core count separates a suspect on a node at 30 of 32, where contention is real and excluding the node may be justified, from the same suspect on a node at 4 of 32, where it is not. + +## BUG FIXES + +- `degen_watch.sh` warns when the probe reached its nodes but attributed no process to any running task. The existing guard only covered a probe that returned nothing at all, which the new per-node line makes impossible. +- `probe_node_cpu.sh` no longer leaks "Permission denied" to stderr when scanning processes it cannot read. `2>/dev/null` after an input redirect is set up only after the failing open, so it never caught the shell's own message; it now precedes the redirect. Only reachable since the scan widened beyond the user's own R processes. + # EpiModelHPC 2.8.2 ## BUG FIXES diff --git a/inst/hpc_doctor/README.md b/inst/hpc_doctor/README.md index 86e6169..fe2e3bf 100644 --- a/inst/hpc_doctor/README.md +++ b/inst/hpc_doctor/README.md @@ -8,7 +8,7 @@ The detector measures CPU utilisation rather than elapsed time. Healthy workers | script | role | |---|---| -| `probe_node_cpu.sh` | node-side; samples `/proc//stat` twice and differences it. One ssh per node covers every task on it. Prints `jobid= taskid= nproc= med_cpu= dstate=` per task. | +| `probe_node_cpu.sh` | node-side; samples `/proc//stat` twice and differences it. One ssh per node covers every task on it. Prints a `nodeinfo=` line, then `jobid= taskid= nproc= med_cpu= top_cpu= dstate=` per task. | | `degen_watch.sh` | per-task verdict; report-only by default, `--requeue` acts. `PATTERN` scopes both the nodes probed and the tasks judged. | | `deploy_doctor.sh` | entry point, `sbatch` this. Sweep loop for the life of a campaign, self-terminating. Carries its own `#SBATCH` defaults. | | `term_orphans.sh` | SIGTERMs genuinely orphaned workers, safe on shared nodes | @@ -85,8 +85,22 @@ Counting now uses `squeue -r`, so a collapsed pending range such as `41746874_[4 - Requeue, never cancel. A requeued task re-runs its own unit and leaves no gap. - `ExcNodeList` can only be set on a pending task. The working sequence is requeue, hold, update, release. - Address array tasks as `_`. A bare `ArrayJobId` given to `scontrol requeue` restarts every task in the array. +- Decide task membership by `SLURM_JOB_ID`, not by process name. A task's work is not always in R, and children inherit the variable. +- Judge only what the one-core-per-simulation model describes. A process above `MULTICORE_CPU` is threaded and its median is not a starvation signal. - Apply `PATTERN` to tasks, not only to nodes. The probe reports every R process the user owns on a probed node, so a node shared with another of the user's campaigns yields that campaign's tasks too. +## What this detector can and cannot see + +The method rests on one assumption: **one simulation per core, each pegged near 100%**. That is what makes a median across a task's processes a starvation signal, and why healthy at ~99% against starved at ~50% is a wide, unambiguous gap. Everything else follows from it. + +Two kinds of workload break the assumption, and the doctor handles them differently. + +**Multi-threaded tasks are declined, not guessed at.** An `rstan`/`cmdstanr` task is an idle R wrapper plus a compiled `model_` binary running the chains at several hundred percent. There is no median to speak of, the ~50% signature does not apply, and nothing on the node says how many threads the chain asked for, so a healthy 4-thread run and a contended 8-thread one are indistinguishable from outside. A task whose busiest process exceeds `MULTICORE_CPU` (default 150, above any single-threaded R process including threaded-BLAS bursts) is reported `threaded` and skipped. Under-flagging is the right failure here: the alternative is requeuing someone's healthy Bayesian fit. + +Membership in a task is therefore decided by `SLURM_JOB_ID` from `/proc//environ` rather than by process name. `pgrep -x R` saw only the idle wrapper and reported `nproc=1 med_cpu=0`, which read as a task doing nothing; the compiled binary inherits the variable and is now attributed correctly. Verified on a live cmdstan array, where the model binary carries both `SLURM_JOB_ID` and `SLURM_ARRAY_TASK_ID`. The batch script wrapper is excluded by name, since counting it would inflate `nproc` past `MIN_PROC` and cost a starting task its grace period. + +**Other users are invisible, so the node is reported instead.** The probe only ever sees processes owned by the invoking user, which means a co-tenant from another account can starve one of our tasks with nothing in the per-task view to show for it. `nodeinfo=1 cores= load1=` is now reported per node and printed for any node that produced a suspect. One number separates the two readings that matter: a suspect on a node at load 30 of 32 is contention and excluding the node may be justified; the same suspect on a node at load 4 of 32 is not, and the node is not at fault. Measured while writing this: nodes running a 16-CPU-per-task Bayesian campaign sat at load 8 of 32, holding cores they were not using, which creates queue pressure for everyone else but does not starve a co-tenant. + ## Field evidence, first production campaign An 5,856-task, eight-family campaign on RSPH, roughly 143 concurrent tasks over a saturated cluster. Several requeues over the run, all but one on a single node (node4). Most were filesystem stalls, and one was a hung task (the case that motivated the classifier fix, below). One, at roughly the six-hour mark, was **the orphan-contention mode this tooling was built for**: a full nine-process footprint at 51% CPU with `dstate=0`. The doctor requeued it and added node4 to its `ExcNodeList`, which is the designed response. That was the first in-the-wild sighting here; the mode is real but, at least on this cluster in this window, rarer than the filesystem stalls that dominate the log. diff --git a/inst/hpc_doctor/degen_watch.sh b/inst/hpc_doctor/degen_watch.sh index f109626..4294e19 100755 --- a/inst/hpc_doctor/degen_watch.sh +++ b/inst/hpc_doctor/degen_watch.sh @@ -68,6 +68,11 @@ IO_RECHECK=${IO_RECHECK:-300} # seconds per D-state (filesystem) stall recheck IO_MAX_CYCLES=${IO_MAX_CYCLES:-3} # recheck cycles a filesystem stall gets before requeue (they usually clear) IO_MIN_FRAC=${IO_MIN_FRAC:-50} # pct of a task's procs in D-state to call it a filesystem stall HUNG_CPU=${HUNG_CPU:-5} # median cpu% at or below this, with few procs blocked, = hung +# A single process above this is multi-threaded, so the one-sim-one-core model +# this detector is built on does not describe it and the median is not a +# starvation signal. 150 rather than 100 leaves headroom for the brief bursts a +# nominally single-threaded R process gets from a threaded BLAS. +MULTICORE_CPU=${MULTICORE_CPU:-150} NODE_OFFENSE_LIMIT=${NODE_OFFENSE_LIMIT:-3} # confirmed CPU-STARVATION events on one node before it is a repeat offender # Campaign-scoped node offense ledger. Keyed to the doctor's own SLURM job so a # new campaign starts a clean ledger and concurrent doctors (e.g. two projects) @@ -105,7 +110,7 @@ probe_all() { # -> " jobid=.. nproc=.. med_cpu=.. dstate=.." for n in $nodes; do ssh -o StrictHostKeyChecking=no -o ConnectTimeout=8 "$n" \ "bash $ROOT/probe_node_cpu.sh $INT" 2>/dev/null \ - | grep -E '^jobid=' | sed "s|^|$n |" + | grep -E '^(jobid|nodeinfo)=' | sed "s|^|$n |" done } @@ -124,8 +129,15 @@ if [ -z "$probe_out" ]; then fi suspects="" +declare -A NODEINFO while read -r node rest; do [ -n "${node:-}" ] || continue + # Node-level context, kept aside and printed only for nodes that produce a + # suspect. It is the difference between "the node is oversubscribed" and "the + # node is idle and my task is stuck on something else", and the per-task CPU + # cannot distinguish those. + case "$rest" in nodeinfo=*) NODEINFO[$node]="${rest#nodeinfo=1 }"; continue;; esac + seen_tasks=$(( ${seen_tasks:-0} + 1 )) jid=$(sed -n 's/.*jobid=\([0-9]*\).*/\1/p' <<< "$rest") med=$(sed -n 's/.*med_cpu=\([0-9]*\).*/\1/p' <<< "$rest") [ -n "${jid:-}" ] && [ -n "${med:-}" ] || continue @@ -147,6 +159,20 @@ while read -r node rest; do if [ "${np:-0}" -lt "$MIN_PROC" ]; then printf " startup %-10s %-8s %s (nproc<$MIN_PROC: master-only, not judged)\n" "$jid" "$node" "$rest"; continue fi + # Multi-threaded workloads are outside this detector's model, so decline to + # judge them rather than guessing. The whole method rests on one simulation + # per core at about 100%, which makes a median across the task's processes a + # starvation signal. An rstan/cmdstanr task is an idle R wrapper plus a + # compiled binary at several hundred percent: the median is meaningless, the + # ~50% starvation signature does not apply, and there is no way to tell a + # healthy 4-thread chain from a contended 8-thread one without knowing how + # many threads it asked for. Under-flagging is the right failure here, because + # the alternative is requeuing someone's healthy Bayesian fit. + top=$(sed -n 's/.*top_cpu=\([0-9]*\).*/\1/p' <<< "$rest") + if [ "${top:-0}" -ge "$MULTICORE_CPU" ]; then + printf " threaded %-10s %-8s %s (top_cpu>=${MULTICORE_CPU}%%: not the one-core-per-sim model, not judged)\n" \ + "$jid" "$node" "$rest"; continue + fi if [ "$med" -ge "$CPU_FLOOR" ]; then printf " ok %-10s %-8s %s\n" "$jid" "$node" "$rest" else @@ -156,8 +182,26 @@ while read -r node rest; do done <<< "$probe_out" suspects=$(echo $suspects) + +# The probe now always returns a node line, so "nothing came back" no longer +# covers the case where the probe RAN and attributed no process to any task. +# SLURM says these nodes are running our work; if none of it is visible, the +# attribution is broken, not the campaign. A warning rather than an error, +# because a task genuinely between steps can be process-free for a moment. +if [ "${seen_tasks:-0}" -eq 0 ]; then + echo "!! probe reached $(wc -w <<< "$nodes") node(s) but attributed no process to any of $(wc -l <<< "$mine") running task(s)" >&2 + echo "!! expected at least one process per task; check that the tasks are really running and that SLURM_JOB_ID is set in their environment" >&2 +fi [ -n "$suspects" ] || { echo "---"; echo "all tasks healthy (median CPU >= ${CPU_FLOOR}%)"; exit 0; } +# Node context for the nodes that produced a suspect. A starved task on a node +# at load 30 of 32 is contention; the same reading on a node at load 8 of 32 is +# not, and the difference decides whether excluding the node is justified. Other +# users' processes never appear in the per-task view, so this is the only place +# their weight shows up at all. +for sn in $(tr ' ' '\n' <<< "$suspects" | awk -F: 'NF{print $2}' | sort -u); do + [ -n "${NODEINFO[$sn]:-}" ] && echo " node $sn ${NODEINFO[$sn]}" +done echo "--- strike 1 done; re-checking suspects in ${RECHECK}s (transient dips must not trigger) ---" sleep "$RECHECK" diff --git a/inst/hpc_doctor/probe_node_cpu.sh b/inst/hpc_doctor/probe_node_cpu.sh index 2a68ff3..7b6fc7e 100755 --- a/inst/hpc_doctor/probe_node_cpu.sh +++ b/inst/hpc_doctor/probe_node_cpu.sh @@ -23,19 +23,57 @@ ## bare ArrayJobId. For a non-array job the two are equal. ## ## Usage: probe_node_cpu.sh [interval_secs] -## Output: one line per SLURM task -> -## jobid= taskid= nproc= med_cpu= dstate= +## Output: one node line, then one line per SLURM task -> +## nodeinfo=1 cores= load1= +## jobid= taskid= nproc= med_cpu= +## top_cpu= dstate= +## +## `med_cpu` is the median across the task's processes and is the starvation +## signal for the one-sim-one-core model. `top_cpu` is the busiest single +## process and is what identifies a workload that model does not describe: an +## rstan/cmdstanr task is one idle R wrapper plus a threaded binary at several +## hundred percent, where a median is meaningless. set -u INT="${1:-5}" HZ=$(getconf CLK_TCK 2>/dev/null || echo 100) tmp=$(mktemp) || exit 1 trap 'rm -f "$tmp"' EXIT +# Node-level context, reported whether or not this user has anything here. A +# task's own CPU says nothing about WHY it is low, and the per-task view cannot +# see the other tenants: processes owned by other users are invisible to this +# probe by construction. Load against core count is the one cheap number that +# distinguishes "this node is oversubscribed" from "this node is idle and my +# task is stuck on something else", which is exactly the question a starved +# reading raises. +printf 'nodeinfo=1 cores=%s load1=%s\n' \ + "$(nproc 2>/dev/null || echo '?')" \ + "$(awk '{print $1}' /proc/loadavg 2>/dev/null || echo '?')" + # Timestamp EACH sample from /proc/uptime rather than assuming the nominal # interval: reading ~36 procs takes real time, so dividing by INT alone inflates # every reading (measured ~1.27x, giving impossible >100% for 1-thread workers). -for p in $(pgrep -u "$USER" -x R 2>/dev/null); do - env_kv=$(tr '\0' '\n' < "/proc/$p/environ" 2>/dev/null) +# Every process the user owns, not just those named `R`, filtered to those that +# carry a SLURM_JOB_ID. Membership of a task is decided by that variable rather +# than by the process name, because a task's real work is not always in R: an +# rstan/cmdstanr job is an idle R wrapper plus a compiled `model_` binary +# doing everything at several hundred percent, and `pgrep -x R` saw only the +# wrapper. Children inherit the variable, so the binary is attributed correctly +# (verified on a live cmdstan array: the model binary carries both +# SLURM_JOB_ID and SLURM_ARRAY_TASK_ID). Login shells, sshd and the probe's own +# processes have no SLURM_JOB_ID and drop out for free. +for p in $(pgrep -u "$USER" 2>/dev/null); do + # The batch script wrapper is always present, always idle, and is never the + # workload. Counting it would inflate `nproc` past MIN_PROC during startup and + # cost the master-only grace period that keeps a starting task from being + # judged. + case "$(cat "/proc/$p/comm" 2>/dev/null)" in slurm_script) continue;; esac + # `2>/dev/null` BEFORE the input redirect, not after. Redirections apply left + # to right, so a trailing one is set up after the open has already failed and + # the shell has already written "Permission denied" to the real stderr. Only + # matters now that the scan is every process rather than our own R ones: + # setuid processes are unreadable and short-lived ones vanish mid-scan. + env_kv=$(tr '\0' '\n' 2>/dev/null < "/proc/$p/environ") jid=$(sed -n 's/^SLURM_JOB_ID=//p' <<< "$env_kv" | head -1) [ -n "${jid:-}" ] || continue # The array-qualified id, taken from the task's own environment rather than @@ -73,6 +111,12 @@ done < "$tmp" | awk ' c = m - lo + 1 med = (c % 2) ? a[lo + int((c-1)/2)] \ : int((a[lo + int(c/2) - 1] + a[lo + int(c/2)]) / 2) - printf "jobid=%s taskid=%s nproc=%d med_cpu=%d dstate=%d\n", j, tid[j], n[j], med, (j in d ? d[j] : 0) + # The busiest single process, which is what tells a MULTI-THREADED workload + # apart from a starved single-threaded one. Nothing in the one-sim-one-core + # model ever exceeds ~100%; a reading well above it means threads, and the + # median stops being interpretable. `degen_watch.sh` uses this to decline + # to judge rather than to guess. + printf "jobid=%s taskid=%s nproc=%d med_cpu=%d top_cpu=%d dstate=%d\n", \ + j, tid[j], n[j], med, a[m], (j in d ? d[j] : 0) } }'