Follow-up from the deploy-doctor field data (PR #52), noting an improvement that needs more cross-cluster data before we commit to a heuristic.
Observation
Across two campaigns the dominant degeneration mode is PanFS filesystem stalls (D-state), not the orphaned-worker CPU contention the tool was built for (that has been seen once). In the overnight campaign, every one of the four stall events had two tasks on the same node go D-state at the same time. That is a node-level filesystem event, not independent per-task failures, but the doctor judges each task in isolation.
What's already in
PR #52 now gives a filesystem stall IO_MAX_CYCLES recheck cycles before requeuing (they usually clear: siblings recovered every time overnight, one suspect finished during the wait), and gates exclusion-to-node so only repeated CPU-starvation escalates, never filesystem stalls.
Proposal (needs data)
Add a node-level view to the sweep: aggregate D-state across all of a node's matching tasks.
- Most/all of a node's tasks D-state at once, then it is a node or cluster filesystem event: wait it out, do not requeue tasks individually.
- A lone task D-state while its siblings read ~99%: more likely task-specific, judge it as today.
This would implement the patience and the ledger gating more directly than the current per-task-plus-sibling-recovery inference.
Open question the data has not answered
We can tell transient cluster-wide stalls (wait) from the orphan mode (exclude), but not yet a persistently bad PanFS client on one node (which would justify exclusion at a higher, filesystem-specific threshold) from a node that just happened to catch several transient stalls. node3 stalled twice overnight but its siblings recovered each time, so it read transient. Before adding any filesystem-stall exclusion path we should collect: per-node stall frequency and duration across several clusters/campaigns, and whether stalls on a given node correlate with that node specifically or track cluster-wide load.
Not urgent; the current behavior (patient recheck, no filesystem exclusion) is safe. Filing so the node-level heuristic and the data it needs are not lost. See the field-evidence section of inst/hpc_doctor/README.md.
Follow-up from the deploy-doctor field data (PR #52), noting an improvement that needs more cross-cluster data before we commit to a heuristic.
Observation
Across two campaigns the dominant degeneration mode is PanFS filesystem stalls (D-state), not the orphaned-worker CPU contention the tool was built for (that has been seen once). In the overnight campaign, every one of the four stall events had two tasks on the same node go D-state at the same time. That is a node-level filesystem event, not independent per-task failures, but the doctor judges each task in isolation.
What's already in
PR #52 now gives a filesystem stall
IO_MAX_CYCLESrecheck cycles before requeuing (they usually clear: siblings recovered every time overnight, one suspect finished during the wait), and gates exclusion-to-node so only repeated CPU-starvation escalates, never filesystem stalls.Proposal (needs data)
Add a node-level view to the sweep: aggregate D-state across all of a node's matching tasks.
This would implement the patience and the ledger gating more directly than the current per-task-plus-sibling-recovery inference.
Open question the data has not answered
We can tell transient cluster-wide stalls (wait) from the orphan mode (exclude), but not yet a persistently bad PanFS client on one node (which would justify exclusion at a higher, filesystem-specific threshold) from a node that just happened to catch several transient stalls. node3 stalled twice overnight but its siblings recovered each time, so it read transient. Before adding any filesystem-stall exclusion path we should collect: per-node stall frequency and duration across several clusters/campaigns, and whether stalls on a given node correlate with that node specifically or track cluster-wide load.
Not urgent; the current behavior (patient recheck, no filesystem exclusion) is safe. Filing so the node-level heuristic and the data it needs are not lost. See the field-evidence section of
inst/hpc_doctor/README.md.