Repository navigation
Settle remote jobs whose managed VM is destroyed so resume/cancel are not blocked forever - #1387
Conversation
… not blocked forever Restart recovery retained every handle whose node config/token was missing, including jobs on managed VMs that were destroyed. Those missions answered REMOTE_JOB_STILL_RUNNING to resume forever and cancel could not clear them. A managed registry record in Destroyed/Failed (set only after the provider confirms the destroy or the VM vanished), or authoritative absence from a successfully loaded registry, proves no node process can still run. One settlement (settle_job_on_gone_node) finishes the run lease with remote_node_gone, fails an Active/Pending mission, and retires the ledger handle. Recovery, resume and cancel all use it. Static nodes and live or stopped managed VMs with missing tokens are still retained.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f13020a719
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| crate::remote_node::job_ledger::finalize(working_dir, job_id, "failed", None) | ||
| .await | ||
| .map_err(|e| e.to_string())?; |
There was a problem hiding this comment.
When restart recovery encounters a RemoteBuild handle on a destroyed VM, this settlement writes a terminal receipt but never calls deliver_pending_remote_build_wakes, unlike poll_recovered_remote_build. If this settles the last unresolved handle, spawn_remote_job_reconciler immediately returns at its !unresolved check, so its next wake-delivery pass never happens. Moreover, finish_remote_job_lease only releases remote-job:<id> owners, leaving a remote-build:<id> wait lease untouched. Consequently, a mission awaiting this validation can remain stuck in WaitingRemoteJob despite its handle being retired. Deliver the newly created wakes before allowing recovery to exit.
Useful? React with 👍 / 👎.
Restart recovery retained every handle whose node config/token was missing,
including jobs on managed VMs that were destroyed. Those missions answered
REMOTE_JOB_STILL_RUNNING to resume forever and cancel could not clear them.
A managed registry record in Destroyed/Failed (set only after the provider
confirms the destroy or the VM vanished), or authoritative absence from a
successfully loaded registry, proves no node process can still run. One
settlement (settle_job_on_gone_node) finishes the run lease with
remote_node_gone, fails an Active/Pending mission, and retires the ledger
handle. Recovery, resume and cancel all use it. Static nodes and live or
stopped managed VMs with missing tokens are still retained.