[9-71]

Overview

Add checkpoint-aware graceful shutdown for Studio training on top of the merged Runtime layer (#312).

• Save a resumable weights/shutdown.pt at a safe epoch boundary, using atomic replacement and parent-side checkpoint validation and acknowledgment.
• Drain IPC during shutdown and wait for natural process-tree exit within a bounded grace period. Confirm cleanup before publishing failed / SERVICE_SHUTDOWN and releasing execution slots.
• Preserve cancellation precedence and cleanup guarantees. A saved checkpoint does not mark the job as completed, and unconfirmed checkpoints are not exposed as artifacts.
• Improve Linux process-tree and zombie reaping, retaining worker ownership when cleanup fails.
• Reduce repeated process-tree scans during high-volume IPC log draining and add regression coverage.

Validation

• Studio Core CI at c63bf92:
  • Ubuntu / Python 3.10: 342 passed.
  • Windows / Python 3.10: 335 passed, 7 skipped.
• Doctests: 4 passed on each platform.
• Targeted Ruff, formatting, and codespell checks passed.
• Windows and WSL CPU training: checkpoint saved, loaded, and resumed from epoch 0 through epoch 1 (workers=0).
• Windows RTX 4050, single GPU with dataloader workers enabled: cooperative shutdown and two fallback scenarios passed (3 passed). Valid checkpoints were successfully resumed through epoch 1.
• Targeted CPU regressions for process reaping, cancellation, and IPC draining passed.

The real GPU tests were performed on an earlier candidate and were not rerun at c63bf92. Studio Core CI does not execute opt-in real-model integration tests.

Limitations

• Cooperative checkpoint shutdown is not supported for DDP/multi-rank training.
• Linux GPU dataloader integration and AMP-enabled checkpoint recovery have not been fully verified.

Related to #299.

[isLinXu]

LGTM