Skip to content

fix(cloud): stop the zone retry when the owner cancels a launch - #39915

Merged
buluoray merged 1 commit into
mainfrom
crew/slack/cloud-zone-retry-honours-cancel
Nov 20, 2026
Merged

buluoray merged 1 commit into
mainfrom
crew/slack/cloud-zone-retry-honours-cancel

Conversation

@buluoray

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Remote Crew launch the owner cancels stops retrying in other zones and creates no new EC2 stack.

Found by the Opus review lane on the 0.13 pick of #39621 (#39904). The dashboard reads a cancel only after engine.provision returns. Since #39621, ec2.deploy retries a capacity refusal in another zone: it deletes the refused stack and launches a new one, in every untried zone. A cancel that arrives during that time is acted on only after every retry finishes.

Why it matters

The owner pressed Cancel, and the launch keeps creating billed EC2 instances, one per zone, each for up to the deploy timeout. Before #39621 a cancel during provisioning cost at most the one deploy already in flight.

Not a goal

What changed (motivation → approach → change)

Cancel read only between steps → the new retry loop runs inside one step → hand the step its cancel signal.

  • ec2.deploy takes cancelled: Callable[[], bool]. It is asked before a refused stack is deleted for a retry, and again after the delete (which takes minutes). Once it answers True, no new stack is created and the capacity refusal is raised as before.
  • RealLaunchEngine.provision forwards it.
  • run_launch passes cancel.is_set when the engine accepts cancelled (the same _accepts_keyword check on_network_fallback uses), and reads a provision failure that follows a cancel as LaunchCancelled. The job then ends cancelled, and the existing cancel path rolls back whatever stack is left.
  • docs/system-specs/modules/cloud.md says how a cancel stops the retries.

Backwards compatibility

Compatible: cancelled is optional on deploy and on the engine, an engine without it is called as before, and a launch nobody cancels runs the same path.

Removes nothing: the change only adds a stop to the retry.

Tests

  • test_a_cancel_stops_the_retry_and_launches_nothing_new: cancelled before the retry, one deploy, no delete, no second stack.
  • test_a_cancel_during_the_delete_launches_nothing_new: cancelled while the refused stack is deleted, still no second stack.
  • test_a_provision_that_fails_after_a_cancel_is_a_cancel: the job hands the engine its cancel, and a provision failure after the cancel ends cancelled with a teardown.

Proof: PROVEN test/test_cloud_ec2_capacity_retry.py::test_a_cancel_stops_the_retry_and_launches_nothing_new, test/test_cloud_ec2_capacity_retry.py::test_a_cancel_during_the_delete_launches_nothing_new, test/test_cloud_launch_job.py::TestCancelRollsBackTheStack::test_a_provision_that_fails_after_a_cancel_is_a_cancel (head 46878a0) -- all three fail with the production hunks reverted.

Every test/test_cloud*, test/test_ec2* and test/test_launch* file: 2546 passed, 10 skipped.

Manual verification

N/A — the path runs only when EC2 refuses capacity during a launch the owner then cancels, which cannot be forced on demand; the AWS calls are faked in the tests.

Real-machine verification

N/A -- the changed code runs only against a live EC2 capacity shortage plus a cancel, which cannot be forced on demand; the tests fake the AWS calls and fail on main.

Related Issues

Follow-up to #39621; found on #39904.

Sibling sweep

Swept: git grep -n "engine.provision|ec2.deploy(" -- src/kiro_crew/
Sibling: src/kiro_crew/cloud/wizard.py -- not affected -- the CLI wizard runs deploy in the foreground; Ctrl+C stops the process and its proc_sink kills the child
Sibling: src/kiro_crew/cloud/launch_engine.py -- changed here

Pattern harvest

Not generalizable: a retry loop added inside one long step; the cancel contract is per step and this was its only loop.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 20, 2026 09:37
@buluoray
buluoray requested a review from CrysisDeu November 20, 2026 09:37
@buluoray
buluoray enabled auto-merge (squash) November 20, 2026 09:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 20, 2026
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f8dec0adbf2a144ff1b3645d03249e8e900b8021 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f8dec0adbf2a144ff1b3645d03249e8e900b8021 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: passing the cancel event into the one step that now loops fixes the cause, and only adds an optional keyword.

Notes

  • Reading every provision failure after a cancel as LaunchCancelled also hides real errors (billing, IAM) that happen to land after Cancel. The job records cancelled rather than the real cause. That is acceptable, since the owner asked to stop, but the error text is lost from the job record.

[DESIGN-REVIEWED] f8dec0a

Description read

[DESCRIPTION-READ] 83ce81ffb0638e4bafa41e56760d8d16bd9de1a83fa9b5c172fe937b1d383e34

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of f8dec0adbf2a144ff1b3645d03249e8e900b8021 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each item is part of stopping the post-cancel zone retry. The defect comes from #39621's retry loop, and tests that fail on base pin it.

What this change ships

Inventory (5 items) — 5 justified

Intent: when the owner cancels a Remote Crew launch, it stops creating new billed EC2 stacks in other zones. FIX.

  1. A cancel during a capacity refusal stops the retry in another zone, both before the refused stack is deleted and after the delete. — justified
  2. The launch engine passes the dashboard's cancel signal to the deploy. — justified
  3. The dashboard launch job hands its cancel event to engines that accept it. — justified
  4. A provision failure after the owner cancelled now ends cancelled instead of failed; the rollback runs either way. — justified
  5. The cloud spec now says how a cancel stops the retries. — justified

[FIRST-PRINCIPLES-REVIEWED] f8dec0a

Description read

[DESCRIPTION-READ] 9f221f1b4588a4537ae75e49650f6fe2acdc685515bc9bba06367da59e9a98fa

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f8dec0adbf2a144ff1b3645d03249e8e900b8021 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f8dec0a

Verdict parsed from the review's SHA-scoped output markers for commit f8dec0adbf2a144ff1b3645d03249e8e900b8021.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f8dec0adbf2a144ff1b3645d03249e8e900b8021 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f8dec0adbf2a144ff1b3645d03249e8e900b8021 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f8dec0a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f8dec0adbf2a144ff1b3645d03249e8e900b8021 false-positive: <reason, 20 to 500 characters>

A dashboard cancel during provisioning was read only after
engine.provision returned. With the capacity retry, ec2.deploy could
delete the refused stack and launch a new, billed stack in every
untried zone after the owner had already cancelled.

ec2.deploy now takes a cancelled callback, asked before it deletes a
refused stack and again after the delete; once it answers True no new
stack is created and the refusal is raised. The EC2 engine forwards
it, and the launch job passes its cancel event and reads a provision
failure that follows a cancel as the cancel, so the job ends cancelled
and its rollback removes what is left.
@buluoray
buluoray force-pushed the crew/slack/cloud-zone-retry-honours-cancel branch from 46878a0 to f8dec0a Compare November 20, 2026 10:11
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 20, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review checked head f8dec0a. Both cancel checks in ec2.deploy work: before the refused stack is deleted, and after the long delete. With no cancel, the default lambda: False keeps the CLI and other callers as they were. The cancel is handed only to engines that accept it, through the existing _accepts_keyword seam. After a cancel, the rollback still tears the stack down. Mutation runs showed that each ec2.py and launch_job.py hunk turns a new test red. The tests use no sleeps, threads, ports or real AWS. The spec update in cloud.md is in the same commit.
Done well: test_a_cancel_during_the_delete_launches_nothing_new asserts the exact check order [False, True] and that only one stack is created. That pins the second check on its own.
Optional:

  • No test fails if the cancelled=cancelled forward is deleted from RealLaunchEngine.provision (launch_engine.py:264). A short test that stubs ec2.deploy and asserts it received the probe would pin it.
  • At launch_job.py:1255, raise LaunchCancelled() from None drops the original provision error, so neither the job record nor the log shows it. A logger.info(..., exc_info=True) before the raise would keep a trace. This applies to every engine, not only EC2.
  • The release/0.13.0 pick #39904 still has the retry with no cancel stop, so it needs this fix too.

@buluoray
buluoray merged commit 6aada4f into main Nov 20, 2026
157 of 161 checks passed
@buluoray
buluoray deleted the crew/slack/cloud-zone-retry-honours-cancel branch November 20, 2026 11:41
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 20, 2026
buluoray added a commit that referenced this pull request Nov 20, 2026
A dashboard cancel during provisioning was read only after
engine.provision returned. With the capacity retry, ec2.deploy could
delete the refused stack and launch a new, billed stack in every
untried zone after the owner had already cancelled.

ec2.deploy now takes a cancelled callback, asked before it deletes a
refused stack and again after the delete; once it answers True no new
stack is created and the refusal is raised. The EC2 engine forwards
it, and the launch job passes its cancel event and reads a provision
failure that follows a cancel as the cancel, so the job ends cancelled
and its rollback removes what is left.

(cherry picked from commit 6aada4f)
buluoray added a commit that referenced this pull request Nov 20, 2026
A dashboard cancel during provisioning was read only after
engine.provision returned. With the capacity retry, ec2.deploy could
delete the refused stack and launch a new, billed stack in every
untried zone after the owner had already cancelled.

ec2.deploy now takes a cancelled callback, asked before it deletes a
refused stack and again after the delete; once it answers True no new
stack is created and the refusal is raised. The EC2 engine forwards
it, and the launch job passes its cancel event and reads a provision
failure that follows a cancel as the cancel, so the job ends cancelled
and its rollback removes what is left.

(cherry picked from commit 6aada4f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants