Skip to content

fix(cloud): retry another zone when ec2 has no capacity (0.13) - #39904

Merged
buluoray merged 3 commits into
release/0.13.0from
crew/slack/cloud-launch-picks-013
Nov 20, 2026
Merged

buluoray merged 3 commits into
release/0.13.0from
crew/slack/cloud-launch-picks-013

Conversation

@buluoray

@buluoray buluoray commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Ship the Remote Crew launch fixes for "no EC2 capacity in this zone" and "a launch without git installs the main build" in 0.13.

Backport to release/0.13.0. Every commit in this PR is a pick from main (cherry-pick -x), in main's order. Please merge with Rebase and merge so each pick keeps its own commit.

Pick Main commit Main PR Why
fix(cloud): pin the release over https when this machine has no git 5c9bf25f3ee #36600 Prerequisite of #39621: it touches the same ec2.py imports and cloud.md file table, so #39621 conflicts without it. It is itself a launch fix 0.13 users hit.
fix(cloud): retry a launch in another zone when ec2 has no capacity 9b46c7064e2 #39621 The fix this PR is for.
fix(cloud): stop the zone retry when the owner cancels a launch 6aada4f9c3e #39915 Follow-up to #39621, found by review on this PR: a cancel during provisioning did not stop the zone retry.

Two Remote Crew launch failures 0.13 still has:

  • A launch whose zone has no capacity for the size fails with a raw CloudFormation error. The user has to guess that another zone, size or Region would work.
  • A packaged install on a machine with no trusted git (most Windows desktops, a Mac without the developer tools) cannot confirm its release tag. Its cloud crew then installs the build from main, which this Kiro Crew may refuse to pair with.

Why it matters

Both hit a person launching a Remote Crew from 0.13. The first is common for popular sizes in busy zones. The second hits most Windows and Mac users who have no developer tools.

Not a goal

What changed (motivation → approach → change)

Same changes as the main PRs:

  • fix(cloud): pin the release over https when this machine has no git #36600: with no trusted git and the default public repository, the release tag is confirmed with one HTTPS GET of GitHub's exact-ref API (cloud/github_tag_probe.py). git ls-remote still runs first whenever git is found. A miss now says plainly that the crew will install the build from main.

  • fix(cloud): retry a launch in another zone when ec2 has no capacity #39621: when CloudFormation fails on InsufficientInstanceCapacity, ec2.deploy deletes the failed stack and launches again in a subnet in another zone. It does this only on a new stack whose subnet it picked itself, never with a chosen subnet, a reused stack or --keep-on-failure. When every zone refuses, the error leads with the plain cause, names the zones tried, and suggests the x86 size or another Region (cloud/ec2_failure.py).

  • fix(cloud): stop the zone retry when the owner cancels a launch #39915: ec2.deploy asks a cancelled callback before it deletes a refused stack for a retry and again after the delete; once it answers True no new stack is created. The launch job passes its cancel event and reads a provision failure that follows a cancel as the cancel, so the job ends cancelled and its rollback removes what is left.

Prerequisites: #36600, as listed above. #36600 and #39621 applied cleanly. #39915 conflicted on context only: this branch's RealLaunchEngine.provision and run_launch do not have main's instance_type / on_progress arguments (#39658 and a later progress change, neither picked), so the pick adds only the cancelled lines to the release versions of those two functions. No other line differs from main.

Backwards compatibility

Compatible: a launch with capacity, a chosen subnet, or a machine with git works as before; the new paths run only on a capacity refusal or when git is missing.

Removes nothing: the picks add a retry and a second way to confirm a tag.

Tests

On this branch: test/test_cloud_ec2_capacity_retry.py and test/test_launch_engine_ref.py (42 passed), and every test/test_cloud*, test/test_ec2* and test/test_launch* file (2453 passed, 10 skipped with #39915 picked). isort, flake8 and black are clean on the changed Python files.

Proof: not provable -- other: backport of #36600, #39621 and #39915, whose proofs were recorded on main

Manual verification

N/A — backport of fixes already verified and merged on main.

Real-machine verification

N/A -- backport of #39621 and #36600; same code as main. #39621's capacity path cannot be forced on demand, and its tests fake the AWS calls.

Related Issues

Part of #36376 and #32319 (both closed by the main PRs).

Sibling sweep

Siblings: none -- the swept sites are the same as in #39621 and #36600 and unchanged on this branch

Pattern harvest

Not generalizable: a backport of #36600, #39621 and #39915; their harvest applies.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

bolichen97 and others added 2 commits November 20, 2026 09:11
…36600)

* fix(cloud): pin the release over https when this machine has no git

A packaged install on a machine with no trusted git (most Windows
desktops, a Mac without the developer tools, a minimal Linux image)
launched its cloud crew from main: release_tag_exists answered False as
soon as no trusted git was found, the same answer as "the tag is
missing", so resolve_public_ref kept the template default and the crew
ran a build the laptop's version gate may refuse to pair with.

With no trusted git and the default public repository, the tag is now
confirmed with one unauthenticated HTTPS GET of GitHub's exact-ref API
endpoint, in the new cloud/github_tag_probe.py. It spawns nothing, and
any error is a WARNING and a miss. The git ls-remote probe still runs
first whenever git is found, and a fork without git is answered as
before, with no network call.

The final miss warning now says in plain words that the cloud crew will
install the development build from main, which this Kiro Crew may
refuse to pair with. The cloud spec describes the fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cloud): mark the fixed-url tag probe for semgrep

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-10-71.ec2.internal>
(cherry picked from commit 5c9bf25)
…39621)

EC2 can refuse an instance size in one Availability Zone for lack of
capacity. Automatic network discovery always picked the first suitable
subnet, so every retry from the dashboard landed in the same zone and
failed the same way, with only the raw CloudFormation error.

A new stack on an automatically picked subnet now deletes the refused
stack and launches again in a subnet of a zone not tried yet, until one
works or the zones run out. The final error says in plain words that
the size has no capacity, names the zones tried, and suggests the
same-size tier on the other architecture or another Region. A pinned
subnet, a reused stack and --keep-on-failure do not move.

The deploy failure text moves to cloud/ec2_failure.py, which keeps
ec2.py under the file line cap.

Refs #36376

(cherry picked from commit 9b46c70)
@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 20, 2026 09:16
@buluoray
buluoray requested review from chenmingwei23 and removed request for a team November 20, 2026 09:16
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 11ca807bf0620a5852604b20ccb544560c1927f6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 11ca807bf0620a5852604b20ccb544560c1927f6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: these are cherry-picks of main's #36600, #39621 and #39915, and the only adapted lines are the cancelled hooks in the older provision/run_launch signatures.

Notes

  • The zone retry runs only against real AWS, which CI never reaches. The PR relies on proofs recorded on main, so it can only be trusted if the picks really are identical to main. The description says they are.

[DESIGN-REVIEWED] 11ca807

Description read

[DESCRIPTION-READ] aa5cddd53dd410e1117d0864357bccd72ecef3df328866e32c7ba74da07922ff

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 11ca807bf0620a5852604b20ccb544560c1927f6 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This backports three fixes already merged on main (#36600, #39621, #39915), each linked to an issue. Every item fixes a launch failure 0.13 users hit, and nothing rides along.

What this change ships

Inventory (6 items) — 6 justified

Intent: Let a person launching a Remote Crew from 0.13 get a running crew when a zone has no capacity or the machine has no git. This is a FIX.

  1. A launch refused for EC2 capacity now retries by itself in a subnet in another zone — justified
  2. When every zone refuses, the error names the zones tried and suggests the x86 size or another Region — justified
  3. With a pinned subnet, the capacity error also says to clear the Subnet ID — justified
  4. Cancelling a launch during the zone retry stops it, and the job ends cancelled, not failed — justified
  5. A machine with no git confirms the release tag with one HTTPS request to GitHub, so the crew no longer falls back to main — justified
  6. The warning shown when the tag can't be confirmed now says the crew will install the development build from main — justified

[FIRST-PRINCIPLES-REVIEWED] 11ca807

Description read

[DESCRIPTION-READ] 0a365e85050c4d35a373d143a96ed006ef99da28c8c5fee684712d431aba161f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 11ca807bf0620a5852604b20ccb544560c1927f6 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 11ca807

Verdict parsed from the review's SHA-scoped output markers for commit 11ca807bf0620a5852604b20ccb544560c1927f6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 11ca807bf0620a5852604b20ccb544560c1927f6 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 11ca807bf0620a5852604b20ccb544560c1927f6 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 11ca807

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 11ca807bf0620a5852604b20ccb544560c1927f6 false-positive: <reason, 20 to 500 characters>

@buluoray

Copy link
Copy Markdown
Contributor Author

The Opus finding on 18c3336 is real, and main has the same gap: a cancel during provisioning did not stop the zone retry. The fix is on main first, in #39915. Once it merges, I will cherry-pick it here so every commit stays a pick from main.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 20, 2026
A dashboard cancel during provisioning was read only after
engine.provision returned. With the capacity retry, ec2.deploy could
delete the refused stack and launch a new, billed stack in every
untried zone after the owner had already cancelled.

ec2.deploy now takes a cancelled callback, asked before it deletes a
refused stack and again after the delete; once it answers True no new
stack is created and the refusal is raised. The EC2 engine forwards
it, and the launch job passes its cancel event and reads a provision
failure that follows a cancel as the cancel, so the job ends cancelled
and its rollback removes what is left.

(cherry picked from commit 6aada4f)
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 20, 2026
@buluoray
buluoray merged commit 5c56fc5 into release/0.13.0 Nov 20, 2026
91 of 98 checks passed
@buluoray
buluoray deleted the crew/slack/cloud-launch-picks-013 branch November 20, 2026 12:52
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants