Doctor reports its own just-started tasks as foreign co-tenants

[smjenness]

On a 11,168-task campaign running 2.9.1, the doctor logged four of our own healthy tasks as starving co-tenants belonging to another project:

co-tenant 41756344  node6  jobid=41756344 taskid=41756066_277 nproc=1 med_cpu=22 dstate=0 (outside /doxy-/: not judged)
co-tenant 41756552  node3  jobid=41756552 taskid=41756066_479 nproc=1 med_cpu=53 dstate=0 (outside /doxy-/: not judged)
co-tenant 41759251  node7  jobid=41759251 taskid=41758479_771 nproc=1 med_cpu=38 dstate=0 (outside /doxy-/: not judged)
co-tenant 41759259  node9  jobid=41759259 taskid=41758479_779 nproc=1 med_cpu=54 dstate=0 (outside /doxy-/: not judged)

41756066 and 41758479 are the array job IDs of our own doxy-interventions_step3, which is what PATTERN=doxy- scopes to. All four are ours.

Nothing was requeued incorrectly and nothing was missed, so this is a log-correctness bug rather than a safety one. It matters because the co-tenant line exists specifically as evidence that a node is oversubscribed by another campaign, so a false entry is misleading in the one place the operator goes to interpret a starving task.

Not starvation, which is the point

The obvious reading is that the doctor went blind to four starved tasks. It did not. All four completed normally:

task	runtime
41756066_277	13.7 min
41756066_479	12.8 min
41758479_771	13.7 min
41758479_779	19.4 min

Normal for this campaign is about 14 minutes. These were caught during startup (package loading, reading a 6 MB restart file, reinit_msm), which is legitimately below the CPU floor and is exactly what MIN_AGE exists to excuse.

Cause

owned is a single snapshot taken before the probe (degen_watch.sh:101-106):

mine=$(squeue -u "$USER" -h -r -t RUNNING -o "%i|%N|%j" 2>/dev/null \
       | awk -F'|' -v pat="$PATTERN" '$3 ~ pat { print $1" "$2 }')
owned=" $(awk '{print $1}' <<< "$mine" | tr '\n' ' ')"

probe_all() then walks the node list serially over ssh, and each node sleeps INT seconds between its two /proc samples. At 32 nodes that is a probe window of minutes, not seconds. Any task that enters RUNNING during that window is seen by the probe but is absent from owned, and because it has just started it is necessarily below CPU_FLOOR, so it takes the co-tenant branch and gets printed.

Both sides agree on the key, so this is not a %i versus %A mismatch: probe_node_cpu.sh:82-84 builds taskid from SLURM_ARRAY_JOB_ID and SLURM_ARRAY_TASK_ID, and owned holds %i. The set is simply stale by the width of the probe.

The MIN_AGE guard that would have excused these tasks is at degen_watch.sh:235-237, after the ownership test at :161-168, so an unowned young task is reported rather than skipped.

Suggested fix

Cheapest and closest to the existing style: before printing the co-tenant line, confirm the task really is foreign, since this path is only reached for tasks already below the floor and is therefore rare. The file already does a per-task squeue -j "$tid" for the age check.

*)  if [ "$med" -lt "$CPU_FLOOR" ]; then
      # `owned` is a pre-probe snapshot and the probe window is minutes wide, so
      # a task that started during it looks foreign. Confirm before reporting.
      nm=$(squeue -j "$tid" -h -o "%j" 2>/dev/null | head -1)
      case "$nm" in
        *) if [ -n "$nm" ] && expr "$nm" : ".*$PATTERN" >/dev/null; then continue; fi ;;
      esac
      printf "  co-tenant %-10s ..." ...
    fi
    continue;;

Alternatives: re-read owned after the probe and use the union of the two snapshots, or move the MIN_AGE check ahead of the ownership classification so young tasks are skipped whoever owns them.

Happy to open the PR.