Repository navigation
fix(cloud): stop the zone retry when the owner cancels a launch - #39915
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: passing the cancel event into the one step that now loops fixes the cause, and only adds an optional keyword. Notes
[DESIGN-REVIEWED] f8dec0a Description read[DESCRIPTION-READ] 83ce81ffb0638e4bafa41e56760d8d16bd9de1a83fa9b5c172fe937b1d383e34 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Each item is part of stopping the post-cancel zone retry. The defect comes from #39621's retry loop, and tests that fail on base pin it. What this change shipsInventory (5 items) — 5 justifiedIntent: when the owner cancels a Remote Crew launch, it stops creating new billed EC2 stacks in other zones. FIX.
[FIRST-PRINCIPLES-REVIEWED] f8dec0a Description read[DESCRIPTION-READ] 9f221f1b4588a4537ae75e49650f6fe2acdc685515bc9bba06367da59e9a98fa That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
A dashboard cancel during provisioning was read only after engine.provision returned. With the capacity retry, ec2.deploy could delete the refused stack and launch a new, billed stack in every untried zone after the owner had already cancelled. ec2.deploy now takes a cancelled callback, asked before it deletes a refused stack and again after the delete; once it answers True no new stack is created and the refusal is raised. The EC2 engine forwards it, and the launch job passes its cancel event and reads a provision failure that follows a cancel as the cancel, so the job ends cancelled and its rollback removes what is left.
46878a0 to
f8dec0a
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This review checked head f8dec0a. Both cancel checks in ec2.deploy work: before the refused stack is deleted, and after the long delete. With no cancel, the default lambda: False keeps the CLI and other callers as they were. The cancel is handed only to engines that accept it, through the existing _accepts_keyword seam. After a cancel, the rollback still tears the stack down. Mutation runs showed that each ec2.py and launch_job.py hunk turns a new test red. The tests use no sleeps, threads, ports or real AWS. The spec update in cloud.md is in the same commit.
Done well: test_a_cancel_during_the_delete_launches_nothing_new asserts the exact check order [False, True] and that only one stack is created. That pins the second check on its own.
Optional:
- No test fails if the
cancelled=cancelledforward is deleted fromRealLaunchEngine.provision(launch_engine.py:264). A short test that stubsec2.deployand asserts it received the probe would pin it. - At launch_job.py:1255,
raise LaunchCancelled() from Nonedrops the original provision error, so neither the job record nor the log shows it. Alogger.info(..., exc_info=True)before the raise would keep a trace. This applies to every engine, not only EC2. - The release/0.13.0 pick #39904 still has the retry with no cancel stop, so it needs this fix too.
A dashboard cancel during provisioning was read only after engine.provision returned. With the capacity retry, ec2.deploy could delete the refused stack and launch a new, billed stack in every untried zone after the owner had already cancelled. ec2.deploy now takes a cancelled callback, asked before it deletes a refused stack and again after the delete; once it answers True no new stack is created and the refusal is raised. The EC2 engine forwards it, and the launch job passes its cancel event and reads a provision failure that follows a cancel as the cancel, so the job ends cancelled and its rollback removes what is left. (cherry picked from commit 6aada4f)
A dashboard cancel during provisioning was read only after engine.provision returned. With the capacity retry, ec2.deploy could delete the refused stack and launch a new, billed stack in every untried zone after the owner had already cancelled. ec2.deploy now takes a cancelled callback, asked before it deletes a refused stack and again after the delete; once it answers True no new stack is created and the refusal is raised. The EC2 engine forwards it, and the launch job passes its cancel event and reads a provision failure that follows a cancel as the cancel, so the job ends cancelled and its rollback removes what is left. (cherry picked from commit 6aada4f)
Problem / Motivation
Goal: A Remote Crew launch the owner cancels stops retrying in other zones and creates no new EC2 stack.
Found by the Opus review lane on the 0.13 pick of #39621 (#39904). The dashboard reads a cancel only after
engine.provisionreturns. Since #39621,ec2.deployretries a capacity refusal in another zone: it deletes the refused stack and launches a new one, in every untried zone. A cancel that arrives during that time is acted on only after every retry finishes.Why it matters
The owner pressed Cancel, and the launch keeps creating billed EC2 instances, one per zone, each for up to the deploy timeout. Before #39621 a cancel during provisioning cost at most the one deploy already in flight.
Not a goal
cloudformation deploycall already in flight; that was true before fix(cloud): retry a launch in another zone when ec2 has no capacity #39621 too.What changed (motivation → approach → change)
Cancel read only between steps → the new retry loop runs inside one step → hand the step its cancel signal.
ec2.deploytakescancelled: Callable[[], bool]. It is asked before a refused stack is deleted for a retry, and again after the delete (which takes minutes). Once it answers True, no new stack is created and the capacity refusal is raised as before.RealLaunchEngine.provisionforwards it.run_launchpassescancel.is_setwhen the engine acceptscancelled(the same_accepts_keywordcheckon_network_fallbackuses), and reads a provision failure that follows a cancel asLaunchCancelled. The job then endscancelled, and the existing cancel path rolls back whatever stack is left.docs/system-specs/modules/cloud.mdsays how a cancel stops the retries.Backwards compatibility
Compatible:
cancelledis optional ondeployand on the engine, an engine without it is called as before, and a launch nobody cancels runs the same path.Removes nothing: the change only adds a stop to the retry.
Tests
test_a_cancel_stops_the_retry_and_launches_nothing_new: cancelled before the retry, one deploy, no delete, no second stack.test_a_cancel_during_the_delete_launches_nothing_new: cancelled while the refused stack is deleted, still no second stack.test_a_provision_that_fails_after_a_cancel_is_a_cancel: the job hands the engine its cancel, and a provision failure after the cancel endscancelledwith a teardown.Proof: PROVEN test/test_cloud_ec2_capacity_retry.py::test_a_cancel_stops_the_retry_and_launches_nothing_new, test/test_cloud_ec2_capacity_retry.py::test_a_cancel_during_the_delete_launches_nothing_new, test/test_cloud_launch_job.py::TestCancelRollsBackTheStack::test_a_provision_that_fails_after_a_cancel_is_a_cancel (head 46878a0) -- all three fail with the production hunks reverted.
Every
test/test_cloud*,test/test_ec2*andtest/test_launch*file: 2546 passed, 10 skipped.Manual verification
N/A — the path runs only when EC2 refuses capacity during a launch the owner then cancels, which cannot be forced on demand; the AWS calls are faked in the tests.
Real-machine verification
N/A -- the changed code runs only against a live EC2 capacity shortage plus a cancel, which cannot be forced on demand; the tests fake the AWS calls and fail on main.
Related Issues
Follow-up to #39621; found on #39904.
Sibling sweep
Swept: git grep -n "engine.provision|ec2.deploy(" -- src/kiro_crew/
Sibling: src/kiro_crew/cloud/wizard.py -- not affected -- the CLI wizard runs deploy in the foreground; Ctrl+C stops the process and its proc_sink kills the child
Sibling: src/kiro_crew/cloud/launch_engine.py -- changed here
Pattern harvest
Not generalizable: a retry loop added inside one long step; the cancel contract is per step and this was its only loop.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.