fix: gate watchdog recovery on bounded runtime observation

[LJC-god]

Summary

Require a trustworthy, bounded observation of the slot's own Docker host before the kernel watchdog authorizes a destructive recovery. This preserves the restart/revival ladder while avoiding a blind reboot on permission/transport errors.

• Inspect the selected slot host (including KVM runner containers); classify running, missing, stopped, restarting, paused, and unknown.
• Treat HTTP/permission/transport/parse failures and an observation deadline as unknown; neither unknown nor deliberately paused containers spends recovery budget or authorizes restart/revival.
• Confirmed missing/stopped/restarting containers can be revived only for running intent and within the existing backoff/fault budget. A running container with a dead kernel still uses the existing kernel restart path.
• Re-observe container_unavailable before falling back to a full boot; preserve operator holds across awaits and gate healthy config-convergence restarts too.
• Isolate per-slot exceptions and prevent interval promise rejection from starving later checks.

Verification

Pinned upstream base: 7efb0058ef26fb9b9a1ccb3e905d8b1085eea0dd.

• New observation tests on the unmodified base: 18/18 fail as expected; fixed affected backend regression: 171/171 pass across 17 files. Affected Biome formatting passes.
• Coverage: unknown/paused budgets; missing/stopped/restarting with Docker/KVM/cluster-shaped fixtures; manual stopped/planned intent; permission-like failures; hold/overlapping-tick races; per-slot health/observation/ensure failures; selected-host requests; malformed replies and bounded stalled observations.
• Unit/regression tests ran with synthetic HOME/TMPDIR, loopback-only network namespaces and host Docker socket masked.
• Separate disposable local Docker probes exercised the actual default slot-host observation path: stopped → running → paused → running → stopped → running → missing. Both task-owned containers were removed; unrelated containers' lifecycle state was unchanged.

Limits

Remote/KVM recovery coverage is fixture-based, not a real remote-node/KVM guest/account end-to-end test. Full backend/Rust/ARM64 CI and production deployment are not claimed. The deadline bounds watchdog waiting, not cancellation of underlying SSH setup; existing transport cleanup remains responsible for that connection. Integration CI remains authoritative if upstream advances.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T16:34:25.845753Z	44a67d8	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44a67d84ff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Recover remote restart loops instead of accepting them as running

When observeRuntime returns restarting for a node-placed Docker slot, this branch spends an attempt and invokes the production revive callback (powerOnSlot). Docker reports State.Running=true during restart-policy backoff (as noted in vm-runtime.mjs), so startRemoteSlot at src/lib/cluster/remote-slot.mjs:681-683 returns already-running without restarting or recreating the container; every watchdog attempt is therefore a no-op until admitRestart faults the VM. This state needs an actual remote restart/recreate path rather than the ordinary revive path.

Useful? React with 👍 / 👎.

[LJC-god]

Fixed in 4f0b315.

startRemoteSlot now excludes State.Restarting from the reusable-container match, so revival removes/recreates/starts a restart-looping container rather than returning already-running. Healthy running/stopped/missing/config-mismatch behavior is preserved, and failed removal cannot claim successful recovery.

Regression: node --test test/unit/remote-slot-restart-loop.test.mjs — 10/10 pass, including both Running states, Docker/KVM synthetic fixtures, and a watchdog tick -> real remote lifecycle function-body test. Before the fix: 6 failed / 4 passed. Related selected unit tests: 252/252 pass; formatting/diff checks pass. Fresh independent read-only evidence review: PASS.

Tests ran in network/mount-isolated synthetic fixtures; no real remote host or KVM boot was tested. These local results are not a full-suite or current-head CI claim.