Skip to content

feat: add AWS Fargate session environments - #21258

Open
kyleseaman wants to merge 2 commits into
mainfrom
feat/session-environments-fargate
Open

kyleseaman wants to merge 2 commits into
mainfrom
feat/session-environments-fargate

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Stack: merged bottom up, one PR at a time. On main already: #19833 (Coder backend), #22238 (its UI half), #21259 (health), #21256 (credentialed git) and #21257 (Sprites and boxd). This PR targets main; #21995 (remote Git panel) comes next.

Off by default: behind the existing developer flag KIROCREW_DEV_MODE, like the whole stack. With it off no Fargate profile is offered, sessions run here, and retention, archive and shutdown never build the Fargate environments, so no registry is read and no AWS call is made (test_with_the_developer_flag_off_no_fargate_path_is_reachable). No live AWS run has been done.

Problem / Motivation

Goal: Let a chat run its workspace in an AWS Fargate task while the agent engine stays on the gateway.

Session environments can run a chat's files, builds, tests and shells on a Coder workspace, which needs a Coder deployment. A team that already runs ECS has no way to give one session a disposable, isolated machine from its own AWS account.

Why it matters

A risky or heavy session can run in a throwaway Fargate task built from the team's own task definition, network and IAM roles, while memory, crons, subagents, approvals and chat history stay on the gateway. When the task stops, nothing is left to clean up.

Not a goal

What changed (motivation → approach → change)

A Fargate profile in Settings > Session Environments lets a chat run its workspace in an ECS task from the owner's own AWS account. The engine stays on the gateway; only files, builds, tests and shells move.

Part What it does
src/kiro_crew/fargate_session profile lookup, task launch and registry, the SSM port forward, the shell, and stop/keep-alive
src/kiro_crew/remote_session/fargate_bootstrap.py stdlib-only guest bootstrap, run as the task's PID 1 from a python3 -c container command
src/kiro_crew/cloud runtask.network_configuration, shared with Deploy my crew so both place tasks the same way
src/kiro_crew/session_environments Fargate's launch dispatch and its row in the move guard (routing.py), the Files panel via the bootstrap and the no-terminal refusal (browse.py), archive-time stop, keep-alive, stop retries, shutdown count and SSM forward cleanup (retention.py); src/kiro_crew/coder calls them under its lifecycle lock and sweeper
src/kiro_crew/config, config-baseline.json, src/kiro_crew/docs, src/kiro_crew/change_card_catalog_kinds.py the session_environments.fargate_profiles list (references only, no keys) as a row of the PROVIDERS table, under its cross-provider id check and config-merge safeguard (kept entry by entry, and a Fargate profile can be the default); settings registry, code-exec change class
src/kiro_crew/dashboard, src/kiro_crew/providers, src/kiro_crew/slack PATCH path and readiness check, task rows in the workspace list, start gate on Fargate profiles, config watch (channel_boot.py) and retention start on a persisted task registry (process_entry.py)
website/src/pages, website/src/components, website/src/api, website/src/assets, website/vite.config.ts an AWS Fargate tab beside Coder, Sprites and boxd on the Settings card with its own intro, check and add buttons, the Fargate line among the shutdown notes, the removed-profile status, the Fargate mark on the composer chip
website/src/i18n, website/src/test, test, docs strings in all 13 catalogs, tests, and the providers, config and feature-map docs

What happens to a session's task:

When The task
The chat stays unarchived and the gateway is up Keeps running and billing, by design: it holds the session's files. Each retention sweep sends it an authenticated keep-alive.
The chat is archived Stopped at once, files deleted, with an unsaved-work notice.
The gateway shuts down gracefully Left running whatever stop_on_shutdown says (counted skipped); rows and keys stay so a restart reconnects, and its SSM forwards are stopped.
The gateway is gone (crash, sleep, stopped for good) The bootstrap exits, stopping the task, about 15 min (900 s) after its last authenticated use with no pipe open and no command running.
The chat reopens after its task is gone A fresh task; chat, memory and approvals are intact.

Unlimited retention while the gateway is up is the intended contract, confirmed by @kyleseaman: the task is the workspace, so an idle stop would silently delete an unarchived chat's files. The composer picker and Settings state the cost; archiving ends it.

The gateway starts each task from the owner's task definition, tagged with a hash of the session key, and reaches it through one AWS-StartPortForwardingSession forward. Enrolment is pre-committed: the gateway draws a 32-byte secret before RunTask, puts only its SHA-256 in the container override, and the bootstrap enrols only a secret matching it. The secret goes only to a forward proven to be the gateway's own Session Manager child, and every later request is attested the same way before it is sent; the secret lives in the encrypted vault and every request and answer carries an HMAC. A lost enrolment reply is settled by an authenticated no-op; a command is retried only if its request never left the gateway; an unreadable vault refuses the start rather than replacing the task.

A pending stop is a signed marker written before StopTask; a tampered or foreign row never authorizes one. If the marker cannot be written (full disk), the task is still stopped and the stop held in memory: the keep-alive never renews it, a reopened chat never reuses it, and the sweep retries until the marker saves. A task whose first registry write fails after RunTask is stopped at once. A Fargate registry failure never blocks Coder's archive or sweep. Retention starts whenever a task registry exists, and the keep-alive runs on its own clock and probes every task at once, so nothing slow delays a reachable task past its idle exit.

The card explains each choice: its own intro on the Fargate tab, a Fargate line among the shutdown switch's per-provider notes, "Removed “”. Discard changes restores it and undoes any other unsaved edits.", and under Manage that a task bills while it runs. In the composer picker a Fargate row reads "Separate cloud machine in your AWS account", and below the rows an info callout says the task bills your AWS account while it runs, gives the same two stop conditions as the Manage hint and the shutdown note, and warns that archiving deletes the files, directly above the "See AWS Fargate pricing" link. Per @kyleseaman, it shows no price or estimate: Fargate prices vary by region and change over time.

Spawn audit: fargate_session/forward.py::_spawn_ssm (gateway, fixed SSM argv, deliberately not cloud.ssm.open_port_forward, whose human-action gate is for the two operator tunnels) and remote_session/fargate_bootstrap.py::_run (guest only) are allowlisted one by one; no module exemption, no baseline change.

Backwards compatibility

Compatible: fargate_profiles is a new optional list, empty by default. An older client that omits it keeps its Coder-only form, and an existing config holds no Fargate ids, so the new cross-provider id check cannot refuse a save that works today.

Removes nothing: Coder sessions, Deploy my crew and local sessions keep every path, route and setting; the shared network_configuration helper builds the same RunTask value Deploy my crew sent before.

Tests

  • Backend: test_fargate_session, test_fargate_binding, test_fargate_keepalive and test_fargate_stop_and_enrollment lock in the enrolment commitment (test_a_local_process_that_reaches_the_bootstrap_first_cannot_enrol, test_a_bootstrap_without_a_commitment_enrols_nothing, test_a_stranger_racing_the_launch_cannot_take_the_task), refusal of a foreign listener, the HMAC identity, vault-only key storage, tampered-row refusal, durable stop retries, shutdown keeping tasks, keep-alive bounds, test_a_damaged_fargate_registry_never_stalls_coder_archive_or_sweep, test_restart_with_no_profiles_maintains_persisted_tasks, test_a_vault_read_failure_never_stops_a_live_task, test_an_enrolment_whose_reply_was_lost_is_still_ours, test_a_reply_corrupted_in_the_tunnel_is_never_acted_on (the command ran once) test_a_call_that_never_reached_the_bootstrap_is_retried (refused, closed or reset before the greeting) test_a_slow_stop_retry_never_delays_the_live_tasks_heartbeat, test_no_request_is_sent_on_a_connection_that_is_not_the_tasks_forward and test_a_stop_whose_record_changed_is_refused_not_reported. Fargate cases in test_session_environments_config, test_coder_retention, test_coder_boot_activation, test_session_environments_start_gate (test_fargate_profiles_alone_resolve_placement) and test_spawn_audit. test_session_environments_import_boundary (test_fargate_imports_nothing_shared_from_coder) keeps fargate_session off coder/'s shared pieces, and test_a_saved_fargate_profile_can_be_the_default covers the default-profile check.
  • Frontend: SessionEnvironmentsCard (provider choice and its hint, Fargate-only check, private-network default, failed check, per-provider list, default profile, the shutdown note shown only with a Fargate profile, the removed-profile status, the billing sentence under Manage, each task row naming its chat and saved profile, typing a comma-separated list key by key), SessionEnvironmentChip (the Fargate row's AWS line, the cost callout with no figure directly above the pricing link, both only with a Fargate profile, the check on the picked row), settings coverage.
  • Each new commitment, registry-isolation, persisted-registry, vault-failure, lost-reply, no-resend and UI-copy test was run with its production change reverted and failed.
  • Developer flag: test_with_the_developer_flag_off_no_fargate_path_is_reachable (picker, launch, keep-alive, stop retries, archive, shutdown and boot), run with the new _fargate_environments guard removed and failed.
  • Run locally on this head, on main: the 88 test files that touch session environments, Fargate, Coder, Sprites, boxd, remote git, config patching, change cards or the spawn audit: 2,786 passed, 0 failed. Vitest for SessionEnvironmentsCard, SessionEnvironmentChip, RemoteFilesPanel, settings coverage and the i18n suites: 1,066 passed. Every Fast Gate script plus the static and flake-pattern ratchets, scoped to this PR's commits: 29 passed. Semgrep 1.78.0 with CI's configs: rule tests 22/22, diff scan 0 findings. tsc, ESLint (--max-warnings 0), i18n:check, lint:i18n, the phantom-class and restyle ratchets, jscpd, the ui-index freshness check, flake8, isort, mypy and the baselined black gate (scripts/check_black_formatting.py), all clean.
  • Not run here: the full backend and frontend suites (CI runs them), native Windows or macOS, guest Python 3.9, and live AWS infrastructure.

Manual verification

  • No live ECS task was started; the AWS calls, the SSM forward and the bootstrap protocol are covered by the unit tests above only.
  • Not verified on real infrastructure: whether a half-open SSM forward (gateway asleep, tunnel not torn down) keeps the pipe counted as in use, which would hold the idle exit off.
  • Still to do: one end-to-end session on a real Fargate task from a dev gateway.

Screenshots / video

Rendered from this head's components in a Vite dev server with every /api call answered by a stub (no gateway, no AWS). Each part of the card is its own image.

Composer picker on a Fargate profile: the picked row's check, the line saying it is a separate cloud machine in your AWS account, the cost callout, and the "See AWS Fargate pricing" link right after it:

Composer picker with the Fargate row checked, the cost callout and the pricing link, dark

The AWS Fargate tab beside Coder, Fly.io Sprites and boxd, and the note that this is not Deploy my crew:

Session Environments provider tabs on AWS Fargate, dark

Global defaults (after the three steps): the Fargate line among the shutdown switch's per-provider notes:

Global defaults with the Fargate shutdown note, dark

Setup instructions expanded on the Fargate tab:

Fargate setup instructions expanded, dark

A Fargate profile expanded: its fields, two comma-separated subnets, the Assign public IP switch and Remove profile:

Fargate profile expanded, dark

After removing a profile, the save bar names it and says what Discard does:

Removed-profile status in the save bar, dark

Readiness check with one ready and one failed task definition, and Manage with the billing sentence and the task row, which names its profile and the chat it belongs to:

Verify readiness and Manage, the task row naming its profile and chat, dark

Light theme

Composer picker with the Fargate row checked, the cost callout and the pricing link, light

Session Environments provider tabs on AWS Fargate, light

Global defaults with the Fargate shutdown note, light

Fargate setup instructions expanded, light

Fargate profile expanded, light

Removed-profile status in the save bar, light

Verify readiness and Manage, the task row naming its profile and chat, light

Related Issues

Part of #10433

no linked issue: this stacked provider PR closes nothing; the Session Environments tracking issue stays open until the whole stack lands.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@kyleseaman
kyleseaman requested review from a team and KiroCrewCaptain as code owners October 16, 2026 14:38
@kyleseaman
kyleseaman requested review from chenmingwei23 and removed request for a team October 16, 2026 14:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 16, 2026
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of fca5ccf3985f2c791f54341a30c10beb821f41f1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the feature is off unless KIROCREW_DEV_MODE is set, it reuses the existing session-environment seams, and the one shared helper builds the same RunTask value Deploy my crew sent before.

Notes

  • "No live AWS run has been done": the SSM forward, the PID-1 python3 -c bootstrap and enrolment have never run against real ECS. Do a live run before lifting the developer flag, and record it in the PR that lifts it.
  • Unlimited retention means an unarchived Fargate chat bills for as long as the gateway stays up. That is a stated, maintainer-confirmed contract, so it is no blocker. If users get surprise bills, an opt-in idle cap per profile can be added later.

[DESIGN-REVIEWED] fca5ccf

Description read

[DESCRIPTION-READ] aba80b348a79c9665a387e32b02d64bfe91c78b4764df7ccf2fa6f4d92b5ddab

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of fca5ccf3985f2c791f54341a30c10beb821f41f1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

The blind reader read every primary control correctly. Where they would not click, the reason was a true cost or data-loss warning, not a misreading.

Notes

  • Reader was unsure whether removing a profile that a task still uses stops that task. Add a clause to profileRemoved: "Chats already on it keep their task until archived."
  • The stopOnShutdown switch shows On while stopOnShutdownFargate says it "Has no effect". Start the note with the Fargate rule, e.g. "AWS Fargate tasks keep running at shutdown…".
  • fargateVsDeploy reads as noise to a cold reader ("no idea what 'headless crew'… means"). Shorten it to one sentence that links to Deploy my crew.

[UX-REVIEWED] fca5ccf

Description read

[DESCRIPTION-READ] bb287d0ba5091b00fe954319121d39e680773306366850ede512d407ab75be17

That sha256 names everything this verdict read: the PR description, plus the 17 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of fca5ccf3985f2c791f54341a30c10beb821f41f1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. This adds a new provider behind the existing KIROCREW_DEV_MODE flag, next to Coder, Sprites and boxd. With the flag off, Fargate is never reached. No default changes, and every shared seam it touches has a named Fargate consumer.

What this change ships

Inventory (10 items) — 10 justified

Intent: let a chat run its workspace in a disposable AWS Fargate task from the team's own account while the engine stays on the gateway. ADDITION.

  1. Settings > Session Environments gains an AWS Fargate tab with intro, check and add buttons — justified
  2. Composer picker offers Fargate rows with a billing callout and a pricing link — justified
  3. A chat bound to a Fargate profile runs files, builds and shells in an ECS task over an SSM forward — justified
  4. Archiving a Fargate chat stops its task at once, with a Fargate-worded unsaved-work notice — justified
  5. Retention sweeper runs a separate keep-alive heartbeat clock and retries pending stops — justified
  6. Graceful shutdown leaves Fargate tasks running (counted skipped) but ends the SSM forwards, even with stop-on-shutdown off — justified
  7. Fargate session refuses a terminal; Files works through the bootstrap — justified
  8. New session_environments.fargate_profiles config key, live-watched, under the cross-provider id check — justified
  9. Retention starts at boot when a Fargate task registry exists — justified
  10. runtask.network_configuration extracted and shared with Deploy my crew (2 consumers) — justified

[FIRST-PRINCIPLES-REVIEWED] fca5ccf

Description read

[DESCRIPTION-READ] 781fab0d896ef59fc67ef9c05ca3994d4d1b320b49ee9694ea3857aca7fff54b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fca5ccf3985f2c791f54341a30c10beb821f41f1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fca5ccf

Verdict parsed from the review's SHA-scoped output markers for commit fca5ccf3985f2c791f54341a30c10beb821f41f1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fca5ccf3985f2c791f54341a30c10beb821f41f1 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fca5ccf3985f2c791f54341a30c10beb821f41f1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fca5ccf

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fca5ccf3985f2c791f54341a30c10beb821f41f1 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 16, 2026
@kyleseaman
kyleseaman force-pushed the feat/session-environments-git branch from 6b91cb7 to 47b1810 Compare October 19, 2026 00:03
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 19, 2026
@kyleseaman
kyleseaman force-pushed the feat/session-environments-fargate branch from 88d00ac to 7309946 Compare October 19, 2026 00:25
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 7309946 (span=08229dc59207): the provider cards did not say switching keeps the filled-in profile.

A line under the Provider legend, wired as the fieldset's aria-describedby: "Each provider keeps its own profiles. Switching only changes which one you are editing; nothing is lost." (sessionEnvironments.providerHint, all 13 catalogs).
"until you save" dropped on purpose: switching never changes the saved config, so it would imply Save after a switch does something.
Test says switching providers keeps both providers' profiles fails with the component reverted.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted (span=1f258adaa6b2): dispatch ignores the recorded provider.

Maintainer ruling (Kyle): profile ids stay unique across providers, and launch-by-recorded-provider is rebutted. The default profile (session_environments.default_profile) and the composer picker name a profile by id alone, so the id must resolve to one provider regardless of what a choice row recorded.
The recorded value is also not trustworthy as a dispatch key: it is written from the id at pick time and records "coder" for an id not found then (stale config, profile removed between list and pick); dispatching on it would launch the wrong provider instead of refusing. Save-time cross-provider uniqueness (test_session_environments_config collision tests) makes id dispatch unambiguous.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 7309946 (span=350ff7d8ed3a): the provider cards did not say switching keeps the filled-in profile.

A line under the Provider legend, wired as the fieldset's aria-describedby: "Each provider keeps its own profiles. Switching only changes which one you are editing; nothing is lost." (sessionEnvironments.providerHint, all 13 catalogs).
"until you save" dropped on purpose: switching never changes the saved config, so it would imply Save after a switch does something.
Test says switching providers keeps both providers' profiles fails with the component reverted.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 7309946 (span=3c122ae0eb80): the provider cards did not say switching keeps the filled-in profile.

A line under the Provider legend, wired as the fieldset's aria-describedby: "Each provider keeps its own profiles. Switching only changes which one you are editing; nothing is lost." (sessionEnvironments.providerHint, all 13 catalogs).
"until you save" dropped on purpose: switching never changes the saved config, so it would imply Save after a switch does something.
Test says switching providers keeps both providers' profiles fails with the component reverted.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted (span=5515aea1d1bd): no limit on how long an unarchived task runs.

Maintainer ruling (Kyle): a chat's Fargate task bills until the chat is archived, by design. A Fargate task IS the workspace, so any idle stop silently deletes the session's files; Coder's --stop-after keeps the disk, Fargate cannot.
The PR states this in the lifecycle table ("Keeps running and billing, by design: it holds the session's files") and the providers spec; archive stops the task at once, and with no gateway reaching it the bootstrap idle-exits after 900 s.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 7309946 (span=bffbc26c32dd): empty configuration abandoned persisted Fargate sessions.

slack/gateway_runtime/process_entry.py: retention activates after readiness whenever a Fargate task registry persists, even with no provider configured; the stat runs off-loop and an unstatable registry also activates the sweeper so its fail-closed reader reports the error.
providers/acp_runtime_start.py: a persisted choices file activates placement resolution, so a removed profile or unreadable choices refuses startup instead of starting locally.
Tests: test_restart_with_no_profiles_maintains_persisted_tasks[False/True], test_removed_profile_never_turns_a_bound_session_local, test_unconfigured_gateway_refuses_unreadable_choices; 6 fail with both modules reverted.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 7309946 (span=c0275a82cd34): first local process to reach the forward could enrol its own secret.

The gateway now draws the 32-byte secret before RunTask and puts only sha256(secret) in the container override (KIROCREW_BOOTSTRAP_COMMIT, fargate_session/launch.py).
fargate_bootstrap.py _serve refuses an ENROLL whose secret does not hash to that commitment (constant-time compare) and refuses every ENROLL when no commitment is set.
A refused ENROLL does not consume the one-shot slot, so the gateway's own committed enrol still succeeds afterwards.
Tests: test_a_local_process_that_reaches_the_bootstrap_first_cannot_enrol, test_a_bootstrap_without_a_commitment_enrols_nothing, test_a_stranger_racing_the_launch_cannot_take_the_task; all three fail with the bootstrap reverted.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 7309946 (span=e045f2d446a7): the provider cards did not say switching keeps the filled-in profile.

A line under the Provider legend, wired as the fieldset's aria-describedby: "Each provider keeps its own profiles. Switching only changes which one you are editing; nothing is lost." (sessionEnvironments.providerHint, all 13 catalogs).
"until you save" dropped on purpose: switching never changes the saved config, so it would imply Save after a switch does something.
Test says switching providers keeps both providers' profiles fails with the component reverted.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 7309946 (span=e1c3681301aa): shutdown note read as contradicting its switch and showed to Coder-only users.

The note now shows only when the draft has a Fargate profile and opens with the exception: "Except AWS Fargate: its tasks keep running at shutdown, because stopping one deletes its files. Each one ends on its own about 15 minutes after the gateway goes away." (15 min = bootstrap IDLE_SECS 900).
Kept the switch itself per maintainer ruling (stop_on_shutdown stays for faster reconnect on dev boxes). Test updated; Coder-only case asserts no note.

@kyleseaman
kyleseaman force-pushed the feat/session-environments-fargate branch from 22e148d to d61d459 Compare October 19, 2026 08:42
@kyleseaman
kyleseaman force-pushed the feat/session-environments-git branch from c017d8c to 762f3dd Compare October 19, 2026 08:47
@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted (span=08453b70a647): the fargateVsDeploy paragraph.

It is there for the moment someone picks the Fargate provider and may take it for Deploy my crew, which also runs on Fargate; folded into the closed Setup instructions it would not be read before that mix-up.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted (span=2b4cf25a8327): "profile" means three things on one form.

"Profile ID" and "Profile name" are the base PR's shared environment-profile fields, the same on every provider. "AWS CLI profile" is AWS's own name for what the field holds (aws configure --profile), the term a user searching AWS's docs finds; Setup step 1 says it lives on the machine that runs Kiro Crew and that only its name is stored.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed (span=2b7e2f519eb5): an unreadable Fargate registry skipped close_fargate_forwards() at shutdown.

The registry read still fails closed, but it now sits in a try whose finally closes the gateway's SSM forwards whatever the read did. test_forwards_end_with_the_gateway_even_when_the_registry_cannot_be_read fails with the fix reverted. In local commit 3b451a9d8b, which ships with the stack restack (the PR is CONFLICTING until then, so no lanes run on a push).

@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted (span=2c3b751a5d25): the removed-profile status names the profile for Coder users too.

The UX lane asked for it in this PR's round 3 ("Profile removed" did not say which profile was removed). The message is the card's one save-bar status for a removed profile, shared by every provider; giving only Fargate the named text would leave two messages for one action, and naming the profile is the same fix for a Coder user who removes one of several.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed (span=42c5f94c4d4c): managementHintFargate said "session" where the composer says "chat".

It now reads "A chat runs on a profile once you choose it in the chat composer. Each chat gets its own Fargate task…", matching the base Manage hint's word, in all 12 catalogs. In local commit 3b451a9d8b, which ships with the stack restack (the PR is CONFLICTING until then, so no lanes run on a push).

@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted (span=4b429a74ab78): the shutdown switch reads as on while the note says it has no effect on Fargate.

The switch is the base PR's shared control and governs workspaces that survive a stop (Coder); a Fargate task's files die with it, so no switch setting may stop one. The note under it opens with that exception at an earlier UX round's request, and states how a Fargate task does stop, in the same words as the callout and the Manage hint. Hiding or disabling the switch would also remove it for the Coder profiles it does control.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed (span=6c26e66b79a0): addressed per Kyle: callout added, explicit pricing deliberately out of scope (prices change by region and over time).

The row now leads with what the chat gets ("Separate cloud machine in your AWS account"), and the callout under it states what bills and both things that stop it. "4 vCPU" is the sample profile's own name, chosen by the owner, not product copy.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed (span=c21bf0a763ee): RunTask tags the task, so the gateway also needs ecs:TagResource.

providers.md now lists ecs:TagResource beside ecs:RunTask, and Setup step 3 on the card says the machine needs permission to "run, tag, describe and stop tasks" (all 12 catalogs). In local commit 3b451a9d8b, which ships with the stack restack.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted (span=ee0945042710): split fargateSetup2 / fargateSetup3 into bullets.

Each is one item in the base PR's numbered Setup instructions list, which renders every provider's steps the same way; each already leads with what is needed (cluster and task definition; the plugin and permissions), and splitting them would make Fargate's steps a second list shape inside one shared component.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted for this PR, as a deliberate stack-wide deferral (span=f148083bb7ee): provider dispatch inside coder/, with no provider seam.

Stack ruling: the provider seam (one dispatch table for prepare/archive/sweep/shutdown) and the saved config shape are one refactor across #19833, #21257, #21258 and #21260, done after the stack lands and owned by the stack conductor. Doing it in one sibling alone leaves the other three on the old shape and conflicts on every merge. No issue is filed for it.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

rebutted (PASS punchline): a 900 s idle exit against a 60 s keep-alive.

A task idle-exits only after 900 s with no authenticated keep-alive, i.e. about fifteen missed beats in a row. The keep-alive runs on its own loop, apart from stop retries and the Coder sweep, and probes every task at once with its own 30 s timeout, so one stalled task or a slow ECS call cannot hold another's beat back (test_fargate_keepalive.py). Fifteen minutes with no beat means no gateway can reach the task, which is the case the exit exists for; a longer exit only lengthens billing after the gateway is gone.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 56419f9 (span=5604c2afad80): subagent provisioning leaves stale startup progress on its parent.

FargateSessionEnvironments.prepare, _prepare and _launch take progress_key, as persistent_vm.py does; prepare_for_session passes progress_key=session_key while the task stays keyed by the owner.
provisioning and connecting are now reported on the starting session (the subagent), the one whose runtime start clears them.
Tests: test_a_subagent_start_reports_its_stages_on_itself_not_its_parent and the extended test_a_fargate_choice_is_prepared_by_the_fargate_provider (subagent case); both fail on the old code.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 56419f9 (span=a62dfd9054c0): Fargate settings deep links leave the target hidden.

The Fargate panel now carries data-provider-panel="fargate" and the deep-link containment effect selects fargate like the other three tabs.
Fargate profile rows now pass rowId, so a link into a collapsed row opens the row as well as the tab.
Test: SessionEnvironmentsCard.test.tsx "opens the Fargate tab and row a settings deep link points into"; it fails with the effect change reverted.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 56419f9 (span=0f5cd9aa4764): keepalive never reconciles a Fargate task that stopped on its own.

A failed keepalive probe now runs one bounded tasks.describe, at most once per 5 minutes per task, under the session lifecycle lock and only after the row binding verifies (a forged row never reaches ECS).
If ECS reports the task STOPPED or has purged it, _stop forgets the secret, drops the forward and removes the row; _stop now skips StopTask for a task already STOPPED. A task that is RUNNING but unreachable is kept.
So no dead row is re-dialled each pass, listed as running in Manage, or counted at shutdown.
Tests: test_a_task_that_stopped_on_its_own_is_forgotten_after_one_failed_probe[STOPPED|purged|RUNNING] (fail on the old code) and test_a_forged_row_never_reaches_ecs_from_a_failed_probe. Documented in providers.md.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in 43e1796 (span=496b878cf102): an ephemeral-port collision hangs the pipe test.

The worker listener now binds first, and the mocked bootstrap port is set to a port that differs from it (listener port plus or minus one), so the own-port refusal can never be why the worker is not reached.
Every socket wait and join in test_fargate_bootstrap_protocol.py is bounded (60 s socket timeouts, join(60) followed by assert not is_alive()), and the listener is closed before the echo thread is joined, so any early exit unblocks accept() and fails the test instead of stalling the suite.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 43e1796.

The base is feat/session-environments-sprites-boxd, not main, so this PR cannot be approved for merge here. It also adds a new network route (an SSM port forward to an ECS task with HMAC enrolment) and new task-stop and process-kill paths, which the reviewing policy reserves for a human maintainer.

On this head all AI lanes pass, every earlier GPT and Opus finding has a fixed disposition naming its span, and CI is green. The Design lane notes that no live AWS RunTask, enrol and archive cycle has run yet.

Clears when: the lower PRs in the stack merge, this PR is retargeted to main and re-reviewed on its new head, and a maintainer reviews the network and task-lifecycle paths.

— crew-pr-reviewer (r124)

@kyleseaman

Copy link
Copy Markdown
Contributor Author

Updated screenshot for 42b287b: Manage now names each Fargate task's chat and profile.

Verify readiness and Manage, task row names its chat, dark

Verify readiness and Manage, task row names its chat, light

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in cf5a7b6 (span=fbe431abb2d2): the Fargate half of the workspaces inventory failed open.

GET /api/session-environments/workspaces now re-raises an unreadable Fargate task registry as _WorkspacesUnreadable, so it answers 500 workspaces_unreadable like the Coder, Sprites and boxd registries instead of an empty list.
Test: test_an_unreadable_fargate_registry_is_reported_not_listed_as_empty; it fails on the old code. The one other Fargate registry read in this handler (the choice lock) already fails closed.

@kyleseaman

Copy link
Copy Markdown
Contributor Author

fixed in cf5a7b6 (span=86880e645bcc): an unreadable Fargate inventory was reported as empty.

GET /api/session-environments/workspaces now re-raises an unreadable Fargate task registry as _WorkspacesUnreadable, so it answers 500 workspaces_unreadable like the Coder, Sprites and boxd registries instead of an empty list.
Test: test_an_unreadable_fargate_registry_is_reported_not_listed_as_empty; it fails on the old code. The one other Fargate registry read in this handler (the choice lock) already fails closed.

A session environment profile can run a chat's workspace as an ECS Fargate
task reached over SSM, reusing the whole-Crew Fargate lane's placement and
task-ARN helpers with its own task tag so neither lane can stop the other's
tasks.
- Keepalive: a failed probe asks ECS (rate-limited, authenticated rows only)
  whether the task is gone, and forgets one that stopped on its own or was
  purged, without a StopTask.
- Start stages: a subagent's Fargate start reports provisioning and
  connecting on the subagent, not its parent (progress_key).
- Settings: a deep link into the Fargate tab opens the tab and the row.
@kyleseaman

kyleseaman commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

fixed in fca5ccf (span=94a4064dda41): a fargate_profiles leaf write could drop the saved default profile.

session_environments.fargate_profiles is now in the leaf-path check beside the Sprites and boxd lists, so a write that removes the profile default_profile names is refused with unknown_profile.
Test: test_a_fargate_list_leaf_may_not_drop_the_default_profile (PATCH API); it fails on the old code. A sweep of every Sprites/boxd list in src/ and website/src found no other place missing Fargate.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes at fca5ccf.

The base is now main, so the r124 base blocker is gone. In the parts I read I found no defect: the bootstrap enrols only the secret whose hash the launch pre-committed, every request and answer is MAC'd over a per-connection nonce, the gateway ties the accepted connection to its own session-manager-plugin child before sending anything, and a failed launch stops the task it started.

async def _launch(
self, session_key: str, profile: dict[str, Any], progress_key: str
) -> tuple[SessionTask, bytes]:
report_start_stage(progress_key, "provisioning")
container = await tasks.task_container(profile)
# Committed before the task exists: its bootstrap enrols only this
# secret, so a local process that reaches the forward first cannot.
secret = os.urandom(32)
task = await tasks.run_task(
profile,
session_key,
container=container,
command=bootstrap.container_command(),
environment={
"KIROCREW_WORKDIR": str(profile.get("workdir") or ""),
"KIROCREW_BOOTSTRAP_PORT": str(bootstrap.BOOTSTRAP_PORT),
"KIROCREW_BOOTSTRAP_COMMIT": hashlib.sha256(secret).hexdigest(),
},
)
row = SessionTask(session_key, str(profile["id"]), task.arn, task.task_id, task.cluster)
# Recorded before waiting, so an archive (or a failure below) can stop it.
try:
row = await asyncio.to_thread(self._record_task, row, profile)
except BaseException:
# No row names the task, so no archive or sweep could find it later:
# stop the RunTask result now instead of leaving it billing.
await self._stop_unrecorded_launch(profile, task)
raise
try:
running = await tasks.wait_running(profile, task)
report_start_stage(progress_key, "connecting")
await self._enroll(self.forward(profile, running), secret)
await asyncio.to_thread(self._save_secret, running.task_id, secret)
row = await asyncio.to_thread(
self._record_task, replace(row, runtime_id=running.runtime_id), profile
)
return row, secret
except BaseException:
await self._stop(profile, row, "the session task did not start cleanly")

One item needs a tracked issue: no live ECS run has been done (your Manual verification says so, and the Design lane asks for one before the developer flag is lifted). Please open an issue for a live RunTask, enrol, keep-alive and archive/StopTask cycle as the precondition for un-gating Fargate, and link it in the description.

Separately, this adds a new network path to ECS tasks with its own enrolment protocol, task start/stop on the owner's AWS account, vault-held session secrets and a PID-1 process reaper. Those stay with a human maintainer, so even with the issue linked this needs a maintainer review of the network and task-lifecycle paths. CI is green and the branch merges cleanly.

The UX lane's copy notes (profileRemoved, the stop-on-shutdown note, fargateVsDeploy) are worth folding in but do not block.

— crew-pr-reviewer (r136)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants