Skip to content

feat(cloud): support EC2 Spot Instances in kirocrew cloud launch (--spot) - #3355

Closed
so0k wants to merge 1 commit into
kirodotdev:mainfrom
so0k:feat/cloud-spot-launch
Closed

so0k wants to merge 1 commit into
kirodotdev:mainfrom
so0k:feat/cloud-spot-launch

Conversation

@so0k

@so0k so0k commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

kirocrew cloud launch always deploys on-demand pricing — there is no way to ask for a Spot Instance. For the intermittent personal-use case the cloud launcher targets, the instance-hours on top of the fixed NAT floor are real money: t4g.xlarge (Balanced) is ~$0.134/hr and m7g.2xlarge (Power) ~$0.326/hr on-demand, while Spot pricing for the same shapes typically runs 60–90% lower. Full write-up in #3184.

Why it matters

Spot changes the "how long can I leave this running" calculus without touching the NAT floor, and it compounds with (rather than replaces) the EventBridge scheduled stop/start some users already run. Without it, cost-conscious users either babysit their instance or don't use the cloud launcher at all.

What changed (motivation → approach → change)

Goal: an additive, opt-in --spot flag threaded like --subnet (CLI → wizard → ec2.deploy() → CloudFormation), with on-demand behaviour byte-for-byte unchanged by default.

Why not the design proposed in the issue: #3184 (which we filed) proposed attaching InstanceMarketOptions directly to the AWS::EC2::Instance. While implementing we found that property does not exist on that resource type — cfn-lint 1.55 rejects it (E3002), the CFN registry schema has no such property, and a stack create would hard-fail. The current CI cfn-lint job never noticed because it only lints the artifact-deploy templates, not cloud/templates/.

Approach actually built:

  • Template: a Spot parameter (default "false") plus an IsSpot condition gate a new SpotLaunchTemplate (AWS::EC2::LaunchTemplate, Condition: IsSpot) that carries only InstanceMarketOptions (MarketType: spot, SpotInstanceType: persistent, InstanceInterruptionBehavior: stop) and a spot-instances-request TagSpecification. The instance references it via LaunchTemplate: !If [IsSpot, {...}, !Ref AWS::NoValue]. persistent + stop is the only shape that preserves the root volume (it is DeleteOnTermination: true, so the default one-time/terminate would wipe ~/.kiro/crew on interruption), and EC2 auto-resumes a persistent-stopped instance when capacity returns.
  • ValidUntil is pinned far-future (2099-01-01): the LaunchTemplate API documents a 7-day default for persistent requests, and an expired/cancelled request auto-terminates its stopped instance — i.e. the exact data loss this design exists to prevent, on a delay. The explicit date removes that failure mode.
  • cloud destroy cancels the Spot request before delete-stack — and refuses to delete when it can't confirm the request is gone. Terminating a persistent-spot instance without cancelling first flips the request to open and EC2 launches a replacement outside the stack — a billing zombie invisible to kirocrew's tag discovery. So: a Spot=true stack whose cancel failed (or whose lookup went unanswered) is not deleted — the CLI exits 1 with the runnable aws remedies and touches no local state; the dashboard answers 409 spot_sweep_blocked_destroy. The stack's own instance is left for CloudFormation to terminate (so a rejected delete-stack can never strand a half-destroyed stack whose volume we already deleted); only orphan replacement instances are terminated directly. The sweep runs by tag, also fires when the stack is already gone (cleanup after a rolled-back --spot launch) — behind the same confirmation the stack path uses, since cancelling a disabled request auto-terminates its stopped instance. On-demand destroys pay one describe call and are otherwise unchanged; users on the previous launcher policy (who cannot have created a spot stack) keep a quiet, successful teardown.
  • Least-privilege launcher policy (iam.py): spot-instances-request/* on RunInstances (unconditioned — AWS documents aws:RequestTag conditions on that resource as not supported; the template always tags the request instead, which is what lets CancelSpotInstanceRequests stay aws:ResourceTag-gated), the launch-template lifecycle actions, spot/LT describes, and iam:CreateServiceLinkedRole scoped to spot.amazonaws.com (first Spot use in an account needs AWSServiceRoleForEC2Spot; the console auto-creates it, the CLI does not). The policy had to stay under IAM's hard 6,144-char managed-policy cap, so two groups of same-shape statements were merged (rationale documented in the module and pinned by a size-guard test).
  • Resume guard: --spot while resuming an existing stack mirrors the --subnet guard — warns interactively, hard-fails under -y — instead of silently keeping on-demand billing.
  • Honest semantics in help + docs: only EC2 can restart an interruption-stopped Spot instance (cloud start works after a manual cloud stop, not after an interruption — wait for auto-resume); an interruption mid-agent-run kills that run ungracefully, same as a host reboot; capacity varies by tier/region.
  • CI: the cfn-lint job's glob now covers src/kiro_crew/cloud/templates/ and the pin is bumped 1.22.3 → 1.55.0 (1.22.3's stale schema false-flags the template's pre-existing, valid MetadataOptions).
  • Interruption UX (from the AI UX review): a failed cloud start on a Spot=true stack now explains itself — "likely an interruption-stop; only EC2 can restart it; your data is intact; do not destroy it" — on both the CLI and the dashboard (fetched only on the failure path). The panel renders remedy commands as copyable <code>, and the softer notices get a neutral tone instead of the warning amber.
  • Dashboard parity (added after the AI design review flagged the gap): the sweep grading lives in ec2.grade_spot_sweep() so the CLI and the dashboard destroy route reach identical verdicts. The dashboard attaches warnings (and, for the no-stack orphan case, softer notices) to its 200 response, audits partial instead of success when work is left, sweeps orphaned requests on an already-absent stack exactly like the CLI, and RemoteCrewPanel renders those lines in a warn-toned notice (screenshots below). Whether a denied lookup may be quietly shrugged off is decided by the stack's own Spot parameter, never by inference about the destroying principal — and the launch template also tags instance/volume, so a replacement instance launched by a re-opened request is discoverable and terminable by the tag-gated policy.

Tests

Net +80 tests across test_cloud_ec2.py / test_cloud_cli.py / test_cloud_wizard.py / test_cloud_iam.py / test_cloud_handlers.py plus the frontend RemoteCrewPanel.test.tsx (targeted cloud suite: 612 passed; panel suite: 24 passed). Highlights of what they lock in:

  • build_deploy_argv/deploy include Spot=true only when set (dry-run path covered); on-demand argv unchanged.
  • Template structure is parsed, not grepped: a CFN-short-tag-tolerant loader asserts the LT exists only under IsSpot, its SpotOptions are persistent+stop+the exact ValidUntil, the instance's LaunchTemplate property has the AWS::NoValue fallback, and the request TagSpecification carries both kirocrew tags.
  • Destroy sweep: cancel-then-terminate ordering, find_stack before the mutating sweep, orphan sweep reachable from the CLI when no stack exists, terminal request states excluded client-side (the state filter value disabled is undocumented, so filtering is by exclusion), denied describe vs. denied cancel vs. denied terminate each graded correctly (exit codes + suppression of the billing claim pinned), agent-session chokepoint refusal caught, --help renders (a literal % in help text crashed argparse — regression-tested).
  • IAM: policy statements re-pinned, no aws:RequestTag condition on the spot-request ARN anywhere, policy length under the 6,144-char IAM cap.
  • Wizard: --spot threads to ec2.deploy, prints the tradeoff, defaults to on-demand, resume guard parity with --subnet.
  • Dashboard: destroy responses surface sweep warnings/notices with the runnable remedies; clean sweeps leave the response byte-identical; the panel renders them (and a clean destroy renders nothing new).

Manual verification

Live server-side validation against a real AWS account (ap-southeast-1): aws cloudformation create-change-set (no execute) for both Spot=false and Spot=true → both CREATE_COMPLETE; Spot=false produces exactly the pre-change 6-resource graph (no launch template), Spot=true adds only SpotLaunchTemplate (re-validated after the TagSpecifications were extended). Change sets and review-state stacks deleted afterwards; no resources were created. A full live --spot launch/interrupt/destroy cycle was not run — flagging the destroy-sweep behaviour on a real interruption as the one thing a maintainer may want to see exercised end-to-end before merge.

Screenshots

The one UI change: kirocrew cloud destroy's honesty contract, rendered by the dashboard. When the destroy succeeds but the Spot sweep leaves something live (or unprovable), the response's warnings/notices render as a warn-toned status block above the crew row — each line self-contained with the runnable aws remedy. A clean destroy (and every on-demand teardown) renders nothing new.

Light:

Destroy sweep warnings, light theme

A failed cloud start on an interruption-stopped Spot crew explains itself instead of showing a bare AWS error:

Start failure hint on an interrupted Spot crew, light theme

Dark variants

Destroy sweep warnings, dark theme

Start failure hint, dark theme

Screenshots are reproducible via the committed harness: node website/scripts/capture-cloud-spot-remedies.mjs.

Related Issues

Fixes #3184

Checklist

  • Single commit with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@so0k
so0k requested a review from a team August 13, 2026 18:09
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Aug 13, 2026
Comment thread docs/guides/remote-crew-on-ec2.md Resolved
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from 67d3cea to 893f43a Compare August 14, 2026 00:21
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Aug 14, 2026
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from 893f43a to 68992f6 Compare August 14, 2026 00:36
@so0k

so0k commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Note for maintainers: the red Fork workflow-change guard is the designed block for fork PRs touching .github/** — this PR deliberately edits ci.yml in one place: the CloudFormation Lint step's pin is bumped cfn-lint 1.22.3 → 1.55.0 (1.22.3's stale AWS::EC2::Instance schema false-flags the template's long-valid MetadataOptions) and its glob is widened to also cover src/kiro_crew/cloud/templates/ — the cloud launcher template was previously not linted at all, which is exactly how the invalid-property design originally proposed in #3184 would have sailed through CI. Both template directories pass cfn-lint 1.55.0 locally, and the Lint CloudFormation templates check in this very run is green. Everything else in the run is green (2026-08-14). Once the workflow diff is reviewed, the allow-fork-workflow-change label re-evaluates the guard.

@github-actions

github-actions Bot commented Aug 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Advisory design-level review of c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline — updated in place on each push; does not block merge.

Design-Verdict: CONCERNS

Sound spot design end-to-end; the one wart is structure flattened into prose at the API boundary and reverse-parsed client-side with sentinel heuristics.

Watch

  • The backend holds every remedy structured (grade_spot_sweep's {summary, details}, the SPOT_START_FAILURE_HINT tuple), then flattens to strings the panel re-derives via lastIndexOf('aws ec2 '), the failed: regex, and a sentence-split on /\.\s+(?=[A-Z])/ — three exported parsers whose contract is "the command is last" and "the hint is after the first newline". Cause → mechanism → consequence: a future remedy that isn't aws ec2 … (e.g. an aws cloudformation command), or a translated hint (the panel ships in 12 languages; the capital-letter split is Latin-only), silently degrades back into the wrapped-paragraph rendering this machinery exists to prevent — with nothing failing loudly. The wording comments say "reword freely," but the format IS load-bearing.

Suggestions

  • Return problems as structured JSON ({summary, aws_error, command}) on the 200/409 bodies and let the CLI and panel each flatten for display — same shared grader, same no-drift property, zero client-side sentinel parsing; keep the newline-in-error trick only for the start-hint path if touching friendlyErrText is out of scope.

[DESIGN-REVIEWED] c92d069

@github-actions

github-actions Bot commented Aug 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — 🔴 changes requested (blocking)

Reviewed c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline; updated in place on each push.

BLOCKING -- src/kiro_crew/cloud/ec2.py:1622 -- stopped Spot teardown can delete data before stack deletion
sweep = cancel_spot_requests(tag, profile, region, exclude_instance_id=...)
Stopped Spot stack -> cancellation auto-terminates the instance and its DeleteOnTermination volume -> a subsequent delete-stack failure leaves the stack but destroys ~/.kiro/crew.
Anchor: residual/crash-data-loss-corruption
Fix: Revert the Spot launch/destroy hunks until stopped-stack teardown preserves the volume when stack deletion fails.

BLOCKING -- src/kiro_crew/cloud/iam.py:321 -- launcher credentials can create uncancellable persistent requests
`"[REDACTED-ARN]
Compromised launcher credentials -> tagged instance plus untagged persistent Spot request -> this unconditional grant authorizes creation, while the resource-tagged cancel grant rejects cleanup -> replacements continue after termination.
Anchor: residual/security
Fix: Revert the unconditional Spot-request grant and Spot feature until every authorized request can be cancelled fail-closed.

[BLOCK-MERGE] c92d069
[GPT-REVIEWED] c92d069

@github-actions

github-actions Bot commented Aug 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline; updated in place on each push.

Review details

I've verified the full chain. Let me confirm the errMsg → splitSpotStartHint path once more and check the production join format is a single newline (already seen at patch line 2373: message = "\n".join([" ".join(message.split()), " ".join(hint)])).

The chain is confirmed:

  • errMsg returns e.message unchanged (RemoteCrewPanel.tsx:695).
  • For the fixture's 502 {error: START_ERROR}, friendlyErrText unwraps error → the raw START_ERROR string (client.ts:830-831).
  • START_ERROR (patch 5194-5205) is space-joined with no \n.
  • splitSpotStartHint (patch 5410-5412): indexOf('\n') = -1 → returns {error: whole message, hint: []}.
  • onError (patch 5467-5471): setActionErr(whole string), setActionNote([]).

Production inserts exactly one \n (patch 2373); the fixture omits it, so the screenshot renders the hint inside the red ErrorNotice banner rather than the neutral note block. The unit test's flattened (patch 5785) correctly uses ${awsError}\n, confirming the intended format. The fixture's own comment (patch 5192-5193) states it reproduces the gateway's error + appended hint, so the omission contradicts its stated contract.

This is a real, grounded defect in an added line, but it lives in a dev screenshot-capture script (website/scripts/) — not a production path. No security/crash/data-loss/corruption/removed-guard. It classifies as advisory FINDING.

Screenshot evidence fixture misrepresents the fix: START_ERROR omits the \n the gateway inserts, so start-interrupted-*.png shows the interruption hint jammed in the red error banner — the exact pre-fix UX the panel change eliminates.

FINDING — website/scripts/capture-cloud-spot-remedies.mjs:5194 — START_ERROR is space-joined with no newline, so splitSpotStartHint returns hint: [] and the "Do NOT destroy the instance…" hint renders inside the red ErrorNotice instead of the neutral note block, making the committed evidence screenshot demonstrate the pre-fix behavior → Fix: join the AWS error and hint sentences with '\n' exactly as handlers_cloud._mutate_instance does (`${awsError}\n` + hint.join(' ')), matching the unit test's flattened fixture.

[OPUS-REVIEWED] c92d069

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Aug 14, 2026
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from 68992f6 to 4123060 Compare August 14, 2026 02:00
@so0k
so0k requested a review from a team August 14, 2026 02:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision labels Aug 14, 2026
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from 4123060 to 2df50fb Compare August 14, 2026 02:12
@so0k

so0k commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up on the Design Review's 🟡 CONCERNS — all three points are addressed as of 2df50fbb2:

  1. Dashboard destroy ignored spot_sweep → the grading now lives at the seam, as suggested: ec2.grade_spot_sweep() is the single grader both surfaces render. The dashboard destroy route attaches warnings to its 200 (same wording and runnable aws remedies the CLI prints before exiting 1), audits partial instead of success when the sweep left work behind, and also performs the CLI's no-stack orphan sweep on an already_absent result. RemoteCrewPanel renders the lines in a warn-toned status block — screenshots now in the PR body.

  2. SWEEP_ERROR_ACCESS_DENIED assumed destroyer == launcher → the quiet path is no longer an inference about the principal: the decision comes from the destroyed stack's own Spot template parameter (read from the describe-stacks payload find_stack already fetched — zero extra calls). Denied lookup on a Spot=true stack is a sweep failure (warn + manual describe remedy + no billing reassurance + rc 1); on a Spot=false stack the quiet path survives, now justified by the stack; with no stack at all the CLI and dashboard both emit the honest "nothing proves it either way" line.

  3. Replacement instances inherit no tags → the launch template's TagSpecifications now covers instance and volume alongside spot-instances-request (verified against AWS docs that LT-level and request-level tag specs merge, request wins on duplicates — so the primary launch is a harmless duplicate). A replacement launched from a re-opened request is thereby discoverable by the sweep's tag-filtered describe and terminable by the tag-gated ec2:TerminateInstances. The template re-validated server-side via a fresh no-execute change set after this change.

@so0k
so0k force-pushed the feat/cloud-spot-launch branch 2 times, most recently from b53221e to efd4509 Compare August 16, 2026 06:17
@github-actions

github-actions Bot commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5, fork) — 🟡 CONCERNS

Advisory UX-level review of c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline — updated in place on each push; does not block merge.

UX-Verdict: CONCERNS

The honesty contract is right, but the "still billing" remedies live in the most perishable and most muted parts of the UI.

Watch

  • The consequence clause gets the muted treatment. splitSweepError mutes everything from the AWS-error mark to the end of the prose, which includes the panel-authored "Cancel them yourself or EC2 keeps launching replacements:" tail (the PR's own test pins awsError swallowing it). Every warned destroy with an AWS error — the common case — greys out the one sentence saying why to run the command. Fix: end the muted span at the AWS sentence, or move the instruction into the summary.
  • Warnings are one-shot component state. actionWarn dies on dismiss, tab switch, or reload; after that the dashboard has no trace that a Spot request is still billing, and destroy can't be re-run to resurface it. Rare × real money × unrecoverable. Smallest fix: persist sweep warnings with the launch/job record or a notification.
  • "Do NOT destroy the instance…" ships in the neutral info tone (actionNote, accent/Info) while a Delete affordance sits in the same panel — the diff's highest-stakes, data-loss-preventing line gets the softest treatment users are trained to skim. Promote that one line (not the whole hint) to warn tone.

Suggestions

  • Hint lines render backticks literally in the note block ("check `kirocrew cloud status`.") — strip them or render as <code> in SweepRemedy's verbatim path.

[UX-REVIEWED] c92d069

@github-actions

github-actions Bot commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Advisory premise-level review of c92d0697f3f6b5a755dfe20af01de1aba9ab7e90 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push; does not block merge.

All evidence gathered; here is the review.

First-Principles-Verdict: CONCERNS

Everything here is derived from AWS platform rules and earns its place — except the start-hint's newline-in-the-error-string protocol, which duplicates a body-key mechanism this same diff ships.

What this change ships

Intent: let cloud-launcher users pay Spot instead of on-demand rates for their EC2 crew (#3184) — an ADDITION.

  1. cloud launch --spot provisions the box on Spot pricing — justified
  2. Persistent + stop + never-expire Spot shape via conditional launch template — justified (E3002, DeleteOnTermination, 7-day-expiry rules)
  3. destroy cancels the request first; refuses delete when unconfirmed (exit 1 / 409) — justified
  4. destroy on a stackless tag probes leftover requests, asks before cancelling — justified
  5. Failed start on a Spot crew explains the interruption (CLI + dashboard) — justified
  6. Destroy 200 carries warnings/notices; panel renders copyable remedy commands — justified
  7. Start hint flattened into the 502 error string, re-split client-side — duplicate of ApiError.body key (this diff's own 409 path)
  8. IAM: spot/LT grants + Spot service-linked role; 3 statements merged under the 6,144-char cap — justified
  9. cfn-lint glob now covers cloud/templates/; pin 1.22.3 → 1.55.0 — justified, cause-level (the unlinted dir is how feat(cloud): support EC2 Spot Instances in kirocrew cloud launch for cost savings #3184's invalid design would have shipped; all template dirs now covered — 0 unfixed siblings)
  10. Destroy audits partial when sweep work is left — justified

Watch

The comment justifying item 7 — "a sibling key would be silently dropped by every existing caller" — is refuted by this same PR: sweepRemediesFromError (RemoteCrewPanel.tsx) already parses a sibling warnings key out of ApiError.body for the 409, and the only caller that renders the hint is the panel this PR edits. Callers that read only error would drop the hint — which is exactly today's behavior, so nobody is worse off.

Subtractions

  • Replace the in-band hint protocol with a hint: string[] sibling key on the 502 JSON body, read via ApiError.body like the 409's warnings: deletes splitSpotStartHint, the sentence-boundary regex (/\.\s+(?=[A-Z])/), and the whitespace-collapse step in handlers_cloud.py whose sole job is keeping the delimiter unique.

[FIRST-PRINCIPLES-REVIEWED] c92d069

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Aug 16, 2026
@so0k
so0k force-pushed the feat/cloud-spot-launch branch from efd4509 to d5ba747 Compare August 16, 2026 07:45
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Aug 16, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Aug 16, 2026
@iamwhatever iamwhatever added the needs-pr-triage PR scanner: awaiting automated triage label Aug 21, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: iamwhatever]: The automated drive-to-green pipeline can't clear the remaining blockers here — they need human/maintainer decisions, so I'm routing this out of the auto-triage lane:

  • Fork workflow-change guard (red, by design): this PR deliberately edits .github/ci.yml. That guard is a hard, maintainer-only control for fork PRs touching .github/**; a rebase/auto-push won't clear it, so a maintainer has to decide whether to accept the workflow edit.
  • GPT 5.6 blocking review is disputed: you've fixed several findings and rebutted the remaining two as "no code change warranted / revert not proportionate." Adjudicating a blocking-review-vs-author disagreement is a maintainer call — the pipeline won't side against your design or revert the feature.
  • Merge conflict: the branch is behind main; a rebase is needed, but on its own it won't get this green given the guard above.

No automated changes will be made. When a maintainer clears the workflow-change guard and adjudicates the blocking review, the PR can proceed.

@iamwhatever iamwhatever added needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Aug 21, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: iamwhatever]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • Data-loss risk in stopped-Spot teardown (src/kiro_crew/cloud/ec2.py:1622, GPT blocking): cancelling the Spot request auto-terminates the instance and its DeleteOnTermination volume before stack deletion, so a later delete-stack failure can destroy ~/.kiro/crew. The safe teardown ordering / volume-preservation approach is a design decision in a data-destroying path — not a mechanical fix.
  • IAM grant scoping (src/kiro_crew/cloud/iam.py:321, GPT blocking): compromised launcher credentials could create an untagged, uncancellable persistent Spot request (the cancel grant is resource-tagged but the create grant is unconditional), so replacements keep launching after termination. Making cancellation fail-closed is a policy decision.
  • Fork CI-workflow change: this PR edits .github/workflows/ci.yml (bumps cfn-lint 1.22.3→1.55.0 and widens the CFN lint glob). The Fork workflow-change guard blocks fork PRs that modify workflows by design — a maintainer must decide whether to accept it; the automated pipeline will not touch .github/**.
  • Merge conflict: the branch conflicts with main and needs a rebase, but resolving it depends on the two design decisions above.

When you've addressed these, the pipeline will re-assess on its next cycle.

@bolichen97
bolichen97 enabled auto-merge (squash) August 24, 2026 07:01
@bolichen97

Copy link
Copy Markdown
Contributor

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • PR #2188 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #2188: CONTINUE_DEVELOPMENT. Independent features in the same module; both can land, with only ordinary same-file rebase churn in the docs and cloud tests. Files: src/kiro_crew/cloud/templates/kirocrew-ec2.yaml. The two independent directions used different labels; the matrix conservatively retains OVERLAPPING for coordination.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

…pot)

Adds an opt-in --spot flag to kirocrew cloud launch that provisions the
instance as a persistent Spot request with stop-on-interruption, cutting
instance-hour cost 60-90% for the intermittent-use case the cloud
launcher targets.

Deviation from the proposal in kirodotdev#3184: InstanceMarketOptions is not a
valid AWS::EC2::Instance property (cfn-lint E3002; confirmed against
the CFN registry schema), so the spot options ride on a conditional
AWS::EC2::LaunchTemplate referenced by the instance only when Spot=true.
The on-demand resource graph is unchanged (verified via a live
create-change-set for both parameter values).

Because the root volume is DeleteOnTermination and a persistent request
relaunches a replacement instance whenever its instance is terminated
without cancelling the request first, this also ships the safety rails
the naive flag would have lacked:

- cloud destroy cancels the tagged Spot request before delete-stack,
  and REFUSES to delete a Spot stack whose request cannot be confirmed
  cancelled (CLI exits 1 with runnable remedies and touches nothing;
  the dashboard answers 409): deleting anyway would terminate the
  instance and let the un-cancelled request launch an untracked
  replacement. The stack's own instance is left for CloudFormation to
  terminate; only orphan replacement instances are terminated directly.
  Orphaned requests are swept even when the stack is already gone,
  behind the same confirmation the stack path uses (cancelling a
  disabled request auto-terminates its stopped instance).
- The Spot request never expires (explicit far-future ValidUntil): a
  request that expired or was cancelled while the box is stopped would
  auto-terminate it and take ~/.kiro/crew with it.
- The request is tagged at create via the launch template, so the
  least-privilege launcher policy can tag-gate CancelSpotInstanceRequests;
  the policy also gains the launch-template lifecycle actions and the
  scoped service-linked-role grant first Spot use requires.
- --spot on a resume mirrors the --subnet guard (warn interactively,
  hard-fail under -y) instead of silently billing on-demand.
- Docs/help state the real recovery semantics: only EC2 can restart an
  interruption-stopped Spot instance; manual cloud stop/start still work.

CI's cfn-lint job now actually lints this template (pin bumped to
1.55.0, glob widened) and template tests parse structure instead of
grepping strings.

Fixes kirodotdev#3184
@bolichen97
bolichen97 force-pushed the feat/cloud-spot-launch branch from c92d069 to d4ef011 Compare September 8, 2026 22:28
@bolichen97

Copy link
Copy Markdown
Contributor

Rebased onto main 6c24f116e by a maintainer as part of the 2026-09-08 open-PR audit (was 2909 commits behind, mergeable_state: dirty). Please review the resolutions:

  • src/kiro_crew/cloud/ec2.py — 5 hunks: dropped the dashboard_port plumbing this branch re-added (main removed it in chore(cloud): pin the stock-port invariant and drop the dead dashboard_port plumbing #5452) and kept only the spot parameter/override/pass-through.
  • src/kiro_crew/cloud/iam.py — kept main's new Route53DnsPreflight statement (fix(cloud): detect Q VPC DNS shadowing of kiro-cli download #7888) and this branch's rewritten request-tag-gate comment.
  • CHANGELOG.md — main released [0.6.0], so the Spot entry moved verbatim into a new [Unreleased] section.
  • docs/system-specs/modules/cloud.md — main's reworded aws.py row plus this branch's ec2.py row.
  • RemoteCrewPanel.tsx / RemoteCrewPanel.test.tsx — kept main's askAgent error notice and its comments, re-added the X lucide import the merge dropped (4 tests failed without it), merged the import lists.
  • black formatted 4 lines in test_cloud_cli.py / test_cloud_iam.py (main's new formatting gate does not baseline them).

Gates run locally: black/isort/flake8 on changed files, pytest on the 6 touched test files (386 passed), tsc --noEmit, and vitest on the 2 touched panel test files (93 passed). No behaviour changed.

A maintainer push makes the maintainer the last pusher, so a second approver is needed under the repo's last-push rule. Reply if anything looks wrong.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@iamwhatever iamwhatever added the drive-to-green PR claimed by drive-to-green pipeline label Sep 23, 2026
@iamwhatever iamwhatever self-assigned this Sep 23, 2026
iamwhatever added a commit that referenced this pull request Sep 23, 2026
The `kirocrew cloud iam-policy` output was 6467 characters whitespace-free,
over AWS's non-adjustable 6,144-char managed-policy quota, so applying the
printed JSON failed with LimitExceeded for any non-admin operator.

Shrink to 5818 whitespace-free by merging statements that share Effect +
Resource + Condition (permission-neutral by construction):

- Ec2SecurityGroupRulesTagged + Ec2DestructiveTagged + Ec2LifecycleTagged ->
  Ec2ManagedResourceMutateTagged (Resource "*", managed-tag condition)
- IamPutRolePolicyForInstance + IamTagRoleOnManaged ->
  IamPutRolePolicyAndTagRoleOnManaged (role ARN, managed-tag condition)
- CloudFormationRead + TagDiscovery + Ec2Discovery + Route53DnsPreflight ->
  CloudFormationAndResourceDiscovery (Resource "*", no condition)

The effective permission set is identical before and after: a new test
flattens the pre-shrink policy (kept as a fixture) and the current policy into
canonical (Effect, Action, Resource-form, Resource, Condition) tuple sets and
asserts equality. A size-gate test asserts the whitespace-free length is at
most 6144 (citing the AWS quota) and at most 5900 for headroom so EC2 Spot
(#3355, roughly +152 chars) still fits.

Sid-pinning tests updated to the merged statement names; no assertions dropped.
iamwhatever added a commit that referenced this pull request Sep 23, 2026
The `kirocrew cloud iam-policy` output was 6467 characters whitespace-free,
over AWS's non-adjustable 6,144-char managed-policy quota, so applying the
printed JSON failed with LimitExceeded for any non-admin operator.

Shrink to 5818 whitespace-free by merging statements that share Effect +
Resource + Condition (permission-neutral by construction):

- Ec2SecurityGroupRulesTagged + Ec2DestructiveTagged + Ec2LifecycleTagged ->
  Ec2ManagedResourceMutateTagged (Resource "*", managed-tag condition)
- IamPutRolePolicyForInstance + IamTagRoleOnManaged ->
  IamPutRolePolicyAndTagRoleOnManaged (role ARN, managed-tag condition)
- CloudFormationRead + TagDiscovery + Ec2Discovery + Route53DnsPreflight ->
  CloudFormationAndResourceDiscovery (Resource "*", no condition)

The effective permission set is identical before and after: a new test
flattens the pre-shrink policy (kept as a fixture) and the current policy into
canonical (Effect, Action, Resource-form, Resource, Condition) tuple sets and
asserts equality. A size-gate test asserts the whitespace-free length is at
most 6144 (citing the AWS quota) and at most 5900 for headroom so EC2 Spot
(#3355, roughly +152 chars) still fits.

Sid-pinning tests updated to the merged statement names; no assertions dropped.
chenmingwei23 pushed a commit that referenced this pull request Sep 24, 2026
)

* fix(cloud): shrink launcher IAM policy under IAM's 6144-char cap

The `kirocrew cloud iam-policy` output was 6467 characters whitespace-free,
over AWS's non-adjustable 6,144-char managed-policy quota, so applying the
printed JSON failed with LimitExceeded for any non-admin operator.

Shrink to 5818 whitespace-free by merging statements that share Effect +
Resource + Condition (permission-neutral by construction):

- Ec2SecurityGroupRulesTagged + Ec2DestructiveTagged + Ec2LifecycleTagged ->
  Ec2ManagedResourceMutateTagged (Resource "*", managed-tag condition)
- IamPutRolePolicyForInstance + IamTagRoleOnManaged ->
  IamPutRolePolicyAndTagRoleOnManaged (role ARN, managed-tag condition)
- CloudFormationRead + TagDiscovery + Ec2Discovery + Route53DnsPreflight ->
  CloudFormationAndResourceDiscovery (Resource "*", no condition)

The effective permission set is identical before and after: a new test
flattens the pre-shrink policy (kept as a fixture) and the current policy into
canonical (Effect, Action, Resource-form, Resource, Condition) tuple sets and
asserts equality. A size-gate test asserts the whitespace-free length is at
most 6144 (citing the AWS quota) and at most 5900 for headroom so EC2 Spot
(#3355, roughly +152 chars) still fits.

Sid-pinning tests updated to the merged statement names; no assertions dropped.

* chore: prune black baseline entries the gate flags

Remove test/test_knowledge.py from .github/black-baseline.txt: the file
graduated to black-clean on main, so the baselined-file gate fails until
the stale entry is pruned (0 new offenders, 1 graduated entry). Same
base-defect fix applied to the other wave-3 PRs.

Also tighten two assertions on the merged IamPutRolePolicyAndTagRoleOnManaged
statement to full set-equality (was subset `in`), per a-ratchet-may-only-tighten:
a future third verb on that role-ARN statement now reds the test instead of
landing un-caught. Test-only, permission-neutral.

---------

Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com>
@iamwhatever iamwhatever removed the drive-to-green PR claimed by drive-to-green pipeline label Sep 26, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks @so0k for the careful work here -- the teardown and ownership hardening across the review rounds was thorough. After re-evaluating it against how Kiro Crew is actually used, we're closing this PR rather than landing it.

Why Spot is a poor fit for a Kiro Crew host

  • kirocrew cloud launch provisions a long-lived gateway host, not a batch job. The gateway keeps agent turns, sub-agents, cron jobs, monitors, and channel listeners (Slack, Discord, etc.) running continuously.
  • Spot capacity can be reclaimed by AWS at any time with only a two-minute notice. Even with the stop interruption behavior this PR uses, every reclaim kills in-flight agent turns and leaves channels unanswered until capacity returns -- which can take minutes or hours, and at an unpredictable rate per region.
  • Kiro Crew work is typically long-running and stateful (multi-hour goal loops, PR drive-to-green, scheduled jobs). It is not restartable from scratch the way CI or rendering jobs are, so the savings come directly at the cost of lost or half-finished work.
  • Supporting it also costs real complexity: ~1.2k lines across the CFN template, ec2.py, the CLI and dashboard, plus launcher IAM policy growth that pushes past AWS's 6,144-character managed-policy limit, and a sensitive teardown path (Spot request cancellation vs. stack deletion and data volumes).

For cost savings without interruptions, stopping the host when idle or choosing a smaller instance type keeps the gateway's always-on guarantee intact.

If a concrete interruption-tolerant use case comes up later (e.g. short-lived, disposable worker hosts), we'd be happy to revisit it in a design discussion first.

auto-merge was automatically disabled September 26, 2026 07:24

Pull request was closed

wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…odotdev#13029)

* fix(cloud): shrink launcher IAM policy under IAM's 6144-char cap

The `kirocrew cloud iam-policy` output was 6467 characters whitespace-free,
over AWS's non-adjustable 6,144-char managed-policy quota, so applying the
printed JSON failed with LimitExceeded for any non-admin operator.

Shrink to 5818 whitespace-free by merging statements that share Effect +
Resource + Condition (permission-neutral by construction):

- Ec2SecurityGroupRulesTagged + Ec2DestructiveTagged + Ec2LifecycleTagged ->
  Ec2ManagedResourceMutateTagged (Resource "*", managed-tag condition)
- IamPutRolePolicyForInstance + IamTagRoleOnManaged ->
  IamPutRolePolicyAndTagRoleOnManaged (role ARN, managed-tag condition)
- CloudFormationRead + TagDiscovery + Ec2Discovery + Route53DnsPreflight ->
  CloudFormationAndResourceDiscovery (Resource "*", no condition)

The effective permission set is identical before and after: a new test
flattens the pre-shrink policy (kept as a fixture) and the current policy into
canonical (Effect, Action, Resource-form, Resource, Condition) tuple sets and
asserts equality. A size-gate test asserts the whitespace-free length is at
most 6144 (citing the AWS quota) and at most 5900 for headroom so EC2 Spot
(kirodotdev#3355, roughly +152 chars) still fits.

Sid-pinning tests updated to the merged statement names; no assertions dropped.

* chore: prune black baseline entries the gate flags

Remove test/test_knowledge.py from .github/black-baseline.txt: the file
graduated to black-clean on main, so the baselined-file gate fails until
the stale entry is pruned (0 new offenders, 1 graduated entry). Same
base-defect fix applied to the other wave-3 PRs.

Also tighten two assertions on the merged IamPutRolePolicyAndTagRoleOnManaged
statement to full set-equality (was subset `in`), per a-ratchet-may-only-tighten:
a future third verb on that role-ARN statement now reds the test instead of
landing un-caught. Test-only, permission-neutral.

---------

Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cloud): support EC2 Spot Instances in kirocrew cloud launch for cost savings

3 participants