Skip to content

fix(cloud): register a launched Fargate task in the Instances registry - #12963

Merged
iamwhatever merged 1 commit into
mainfrom
fix/fargate-register-task-12511
Sep 23, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/fargate-register-task-12511

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

1. What is the problem?

A Fargate crew launched through the cloud lane never appears in /instances.
FargateLaunchEngine.register() was a documented no-op, so the launch created a
task, reported success, and left the crew unreachable from the Instances surface.

The blocker its docstring named is gone. That docstring said
instances/registry.py closes its transport set to ("ssh", "ssm"); on main
today CONNECTION_METHODS is ("ssh", "ssm", "fargate") and
SSM_TRANSPORT_METHODS is {"ssm", "fargate"} (#12510). The registry could
accept the record; nothing offered it one.

The launcher also could not compose the record's address. A registry record for
this lane is keyed on ssm_target = ecs:<cluster>_<task-id>_<runtime-id>, and
provision() returns the task ARN straight from RunTask, which carries the
cluster and the task id but not the runtimeId. ECS assigns that when the
container starts and only DescribeTasks reports it, so the third field has to
be read rather than derived.

2. Why this issue matters to the user

The EC2 lane registers its instance at launch, so that crew is switchable the
moment setup finishes. A Fargate launch left the user with a running, billing
task and no path to it from the dashboard: they had to add it by hand in
Settings with the fargate connection method and paste an ECS target whose
runtime id they had no way to obtain except by running aws ecs describe-tasks
themselves.

3. How our fix solves it

register() reads a target, then writes the record. It polls
ecs:DescribeTasks until the crew container reports a runtimeId, composes
ecs:<cluster>_<task-id>_<runtime-id> from the task's own coordinates, reads
that back through split_ecs_target (the registry's own reader, so a value the
registry would reject is refused where the reason can name the parts), and calls
connect.register_instance(connection_method="fargate", remote_port=FRONT_PORT).

remote_port is passed explicitly because register_instance defaults to the
stock remote dashboard port, and nothing in the crew container listens there:
the task definition publishes FRONT_PORT (8080). Idempotency is
register_instance's own, matched on ssm_target, so registering the same task
twice updates that record in place and preserves its id, allocated local port,
TTL and was_connected.

The runtime id is read by container NAME, not by position.
sighting_from_task gains runtime_id from the container named
CREW_CONTAINER_NAME. containers[0] would work today and would silently point
every registration at a sidecar's channel the moment one is added to the task
definition.

The wait budget is read, not invented. REGISTER_TARGET_TIMEOUT_SECONDS
(180) and REGISTER_TARGET_POLL_SECONDS (6) are the numbers cloud/wizard.py
already uses to wait for a freshly started EC2 instance's SSM agent, which is
the same question asked of the other lane. The budget is a ceiling, not a delay:
the poll returns on the first read carrying a runtime id, so a task that starts
in twenty seconds costs twenty. Blocking there is consistent with the step --
run_launch is documented blocking on a worker thread, and the EC2 lane's
provision already blocks for minutes on CloudFormation.

Whether a registration failure fails the launch depends on the task. This is
the chain from the symptom: the fix adds an AWS read to a step that previously
could not fail, and that step's existing contract is
RealLaunchEngine.register's -- raise, and the whole launch is reported FAILED.
Applied unconditionally to this lane, a task that was started and is billing
would be reported as a failed launch, which is the one outcome with nothing for
its owner to act on. Applied to no failure at all, a task that is already dead
would be reported as a good launch, which is worse: the job sits DONE for a crew
nothing can reach.

So await_registration_target reports whether the task it read may still come
up, and register() turns that into the failure class. A task that is gone or
past running raises RuntimeError and fails the launch, because there is no
running crew behind it to add by hand and none to tear down. Every other failure
raises a new launch_job.RegistrationUnavailable, which run_launch records on
the connect step while leaving the job DONE with the reason on job.error: a
denied ecs:DescribeTasks, a container still starting at the budget, coordinates
that form no target, and a registry write that declined. A failed read counts as
may-come-up, so a missing permission never reports a live crew as dead.
Any other exception from register stays fatal, so the EC2 lane's behaviour is
unchanged
-- the distinction is not which lane it is, it is whether the engine
can say the launch itself is sound.

A target is composed only from a task that is actually serving. ECS runs a
task through PROVISIONING, PENDING, ACTIVATING, RUNNING, DEACTIVATING,
STOPPING, DEPROVISIONING, STOPPED. The three states after RUNNING are
billable while nothing is listening, because the ENI is being torn down, and the
container keeps its runtimeId through all of them. So neither the presence of a
runtime id nor TaskSighting.is_running can decide reachability: that property
answers a BILLING question and admits every state but STOPPED, which is right
for the teardown warning it serves and wrong here. is_serving answers the
reachability one, and is_past_running separates "not serving yet" from "never
serving again", so a PENDING task is polled while a STOPPING one is refused.
desiredStatus is read too: a task ECS has been told to stop is on that path even
while lastStatus still says RUNNING.

The non-fatal path writes the failed step and the terminal status in one save.
A failed connect step beside a non-terminal status is the combination
reap_orphans reads as an interrupted launch, and it rewrites the status to
FAILED. Persisting the step on its own left a window where a restart turned the
intended DONE into a red card over a running crew -- the outcome this path
exists to avoid. Both exits already save, so the intermediate save bought no
observable progress (the span between them holds only in-memory assignments, with
no I/O and no cancel check) and dropping it removes the window rather than
narrowing it.

connect.register_instance gains connection_method and provisioner_id
keywords, defaulting to "ssm" and the EC2 provisioner id, so every existing EC2
call site is unchanged. The Fargate lane passes "aws_fargate": provisioner_id
is persisted source metadata and selects nothing: the engine driving a launch is
resolved from the launch job's own provisioner id, never from a registry record.
What reads it is the dashboard's crew list, which captions a row by it and picks
the lifecycle guidance and Remove warning it shows, so a Fargate task left with
the EC2 default is presented as an EC2 instance and its owner pointed at the wrong
console. (Review caught an earlier version of this paragraph claiming the field
selects the engine; engine_for does take a provisioner id, but the value comes
from the launch job, so the registry stamp is a label.)

run_launch checks for cancellation after the register step. That step holds a
poll of up to the full readiness budget, which is long enough for a user to press
cancel, and the cancel route only sets the event -- so without the check the
request is read by nobody, the job still reports DONE, and the task keeps
billing. The check sits where provision already puts one, so a cancel during a
long AWS call is acted on when that call returns and the handler tears the task
down.

InstancesRegistry takes its lock from a module-level table keyed by the
registry's resolved path instead of owning one per object. Its mutations are
read-modify-write pairs and callers build a registry per request -- the cloud
connect helpers, the instances handlers and the dashboard server each construct
their own -- so a lock owned by the object serialises nothing between them: a
second object admitted between the first's read and its write persists a document
that predates the first's record, and that record is gone. Registering a launched
task adds a writer on the gateway's launch path, which is what makes the overlap
with a dashboard edit reachable. Locks are never evicted from the table, because
a lock is what a writer may be holding. The guarantee spans threads, not
processes: a separate kirocrew CLI edit still serialises only on
atomic_write's rename.

An absence is not a death until the task has been seen. RunTask and
ecs:DescribeTasks are eventually consistent, so a task accepted moments ago is
legitimately missing from the first read. Treating that as gone raised
RuntimeError and failed the launch, and because the failed step is CONNECT
rather than PROVISION the provision rollback does not run, so the task kept
running and billing while the job reported it as not running. The poll now waits
through an absence that precedes any sighting and ends only on a DISAPPEARANCE, an
absence after one. The expiry wording separates a task ECS never listed from one
that was listed and stayed pre-RUNNING, because those are different things to go
and look at.

Both of this engine's ecs:StopTask paths remove the stopped task's Instances
record, after ECS accepts the stop. register is this lane's last launch step, so
a cancel observed just after it unwinds through teardown with the row already
written, and teardown only issued ecs:StopTask. The result was a crew list
entry addressing a stopped task, with no automatic recovery: the diagnosis ladder
reports ssm_unreachable and the operator deletes the row by hand. The removal
lives with each stop rather than on the cancel branch, because that is where the
set of resources actually stopped is known, and it is keyed on the task, so a
sibling crew in the same cluster keeps its record.
connect.unregister_ecs_task finds the row by reading each ECS target back
through split_ecs_target and comparing cluster and task id: a teardown holds the
task ARN and never the runtime id, so it cannot match a whole target, and a
ecs:<cluster>_<task-id>_ prefix test would let cluster crews remove a row
belonging to cluster crews_eu. It removes EVERY matching row rather than the first.
The launcher's own writer cannot produce two rows for one task -- it dedupes on the
exact target, the target embeds the container's fixed runtime id, and a re-launch
gets a new task id -- but a hand-added Remote crew row naming the same task under a
different runtime id can, and a row left behind addresses a stopped task that no
sweep prunes, because a stopped task is not listed for teardown at all.

The second stop path is the TTL bound sweep, and it is the one that reaches a
registered task in ordinary operation: it runs on every provision, so a crew that
outlives the default lifetime is stopped by the NEXT launch. Review found it after
the first fix covered only teardown, and it is the likelier path of the two --
nothing about it requires a cancel. It logs rather than withholding, because it
answers with a plan and has no confirmation to withhold.

teardown withholds its confirmation when a removal failed. A confirmation says
nothing of ours may remain, and a listed crew addressing a stopped task is
something of ours, so returning True would report the cleanup as finished while a
row is still there. That needs three answers rather than a bool: "there was no row"
and "the row could not be removed" are both falsy and mean opposite things, so
connect.unregister_ecs_task reports removed, absent or failed. An absent row
still confirms, because a task nothing registered leaves nothing behind.

The poll budget is spent against a monotonic clock rather than a count of polls,
and the last sleep is cut to whatever the budget has left. Each DescribeTasks
round trip costs wall time that a per-iteration counter cannot see, so thirty of
them pushed the documented 180s past its own number. The poll count is kept as a
second bound, so a sleep that returns early cannot turn the budget into a tight
spin on ECS.

Prose that stated the old behaviour was corrected in the same commit: the
describe_task and api_cloud_launch_task docstrings (both justified their ECS
read by "this lane registers nothing" -- the real reason is that a record naming
ONE task cannot report that task's current state), the provisioner guide's lane
table and register contract, the instances spec sections 16.6 and the Fargate
registration seam (which stated every failure was non-fatal, and that
provisioner_id was stamped on EC2 registrations only), and the Fargate RFC's
sections 5, 8, 9 and 11, whose deferred registry visibility this closes, and the
registry's module and class docstrings, which both stated that its read-then-write
keeps a live gateway and an out-of-band CLI edit from clobbering each other -- a
guarantee a lock owned by the object never provided and a thread lock cannot make
across processes. This round adds the same sweep for its own changes: the
instances spec's registry row repeated that cross-process claim and now states the
guarantee that holds, its Fargate seam section records the elapsed-time budget, the
eventual-consistency rule and the teardown removal, the provisioner guide's
teardown contract row now tells a lane whose register writes a record to remove
it there, and _rollback_cancelled_stack's docstring no longer says the registry
holds nothing when a cancel unwinds.

4. What tests we did

New tests, all passing locally:

test/test_fargate_launch_engine.py -- registration composes the right target
and passes connection_method="fargate", FRONT_PORT and the Fargate
provisioner id; a container that started and then died yields no target even
though its runtime id is still readable, and fails the launch rather than
registering; each of DEACTIVATING, STOPPING and DEPROVISIONING is refused
rather than registered, with the same case asserting that the sighting is
simultaneously is_running (billable), carrying a runtime id, and NOT
is_serving; a task whose desiredStatus is STOPPED is refused while
lastStatus still says RUNNING; each of PROVISIONING, PENDING and
ACTIVATING is polled to the budget instead of refused, which is what keeps one
predicate from answering both questions; a STOPPED task fails the launch and is
asserted NOT to be RegistrationUnavailable; a live but unregistered task stays
non-fatal; a declining registry and an unresolvable target both raise
RegistrationUnavailable and never reach the registry; the runtime id is matched
by container name with a sidecar listed first; a task with no containers and a
started container with no runtime id both report none; the poll returns on the
first read that carries one (asserting the sleep count, so a task that starts
fast is not charged the budget); a stopped task quotes ECS; a task ECS does not
list is not rounded to a state; a denied read is reported as itself and is
asserted to leave the task may-come-up; the budget is bounded by the documented
number and its expiry names the remedy; a short ARN falls back to the spec's
cluster; coordinates that do not form a target are refused.

test/test_cloud_connect.py -- the caller's provisioner_id reaches the record
on BOTH the add and the idempotent-update path, and the two existing EC2 cases
still assert aws_ec2, so the default is pinned.

test/test_cloud_launch_job.py -- RegistrationUnavailable keeps the job DONE
with a FAILED connect step and the reason on both the step and the job, and does
not roll back; no save on that path ever persists a failed connect step beside a
non-terminal status (the test wraps store.save, reloads from disk after each
one, and asserts the reaper-trap pair never appears, so it checks what a restart
would find rather than what the in-memory object holds); a restart after the
terminal save is skipped by reap_orphans and keeps its DONE and its reason; the
connect step never keeps its "Added to Your crews" wording on a failed
registration; a refused sign-in still outranks a registration failure.

test/test_instances.py -- two registry objects over one file share a lock while
objects over different files do not; and a second registry admitted between the
first's read and its write cannot drop the first's record. That test holds one
writer inside its own _write, releases the other to add a different instance,
and reads both records back from disk, so it checks what a restart would find.
Both waits are bounded, so the serialised case times out and proceeds rather than
hanging.

This round adds, in the same three files: a task absent from the first reads is
polled and still registers once it appears; a task ECS never listed expires with
alive True and its own wording; a task seen and THEN gone is terminal (the case
that used to fire on the first read); the final sleep is cut to the remaining
budget and the budget is never overspent; teardown removes the stopped task's row
and leaves a sibling task's row alone; and connect.unregister_ecs_task matches the
right record, refuses a look-alike cluster (crews must not remove crews_eu), and
ignores an EC2 record whose target is not an ECS target.

Also added: the TTL sweep removes the expired task's row; teardown does not
confirm when the removal failed and does confirm when there was no row to remove;
and connect.unregister_ecs_task reports a failure apart from an absence, with the
row still present in that case, which is why the two cannot share one answer.

Also added: connect.unregister_ecs_task removes two duplicate rows naming one
task under distinct ids, rather than returning on the first.

Every fix is verified to be caught by its tests, not merely accompanied by them.
Reverting the check order made
test_a_container_that_started_then_died_yields_no_target fail with the composed
target (assert 'ecs:crews_...' == '') and
test_register_fails_the_launch_for_a_task_that_crashed_after_starting fail with
RegistrationUnavailable. Putting the reachability guard back to the billing
predicate made all three winding-down cases and the desiredStatus case fail,
each with the composed target. Restoring the intermediate save made the invariant
test fail with the exact trap pair it guards, [('running', 'failed')].
Re-hardcoding the EC2 provisioner id made the connect test fail with
assert 'aws_ec2' == 'aws_fargate', and dropping the engine's keyword made the
engine test fail with KeyError: 'provisioner_id'. Dropping the cancel check made the
cancel test fail with assert 'done' == 'cancelled'. Restoring the per-object
lock made the identity test fail and made the interleave test fail with
assert ['first-1'] == ['first-1', 'second-1'], which is the dropped record
itself. Making a first-read absence terminal again made the
eventual-consistency test fail with the gone tuple where the composed target was
required, and the never-listed test fail with assert False is True. Removing the
registry removal from teardown made both teardown tests fail with the surviving
row, assert ['doomed'] == []. Restoring the full-interval sleep with no deadline
made the budget test fail with assert 6 < 6. Removing the sweep's own removal made the
sweep test fail with the surviving row, assert ['expired'] == []. Dropping the
confirmation gate made the teardown test fail with assert True is False. Folding
"failed" back into "absent" made the tri-state test fail with
assert 'absent' == 'failed'. Returning on the first matching row made the
duplicate test fail with the surviving row still listed.

Commands run (this worktree on PYTHONPATH via setup.cfg's pythonpath = src,
with KIROCREW_HOME unset so the suite's live-home guard is satisfied):

  • pytest -n0 test/test_fargate_launch_engine.py test/test_cloud_launch_job.py test/test_cloud_connect.py -- 244 passed
  • pytest -n0 test/test_instances.py -k TestRegistry -- 8 passed
  • Gates, each on its own exit status: flake8, isort --check-only, mypy,
    black --check, scripts/check_black_formatting.py (baselined, diff-scoped),
    scripts/check_comment_history.py (selftest first, then scoped to this base),
    scripts/docs-lint.sh, scripts/check_feature_map.py, scripts/check_brand_name.py

Gates: flake8, isort --check-only, mypy on the three changed modules,
black --check --target-version py310, scripts/check_black_formatting.py
(baselined, in scope), scripts/check_comment_history.py (selftest first, then
scoped to this change's base), scripts/docs_lint.py,
scripts/check_feature_map.py with FEATURE_MAP_BASE_REF -- all pass. The full
suite is CI's job; scripts/local-gate.py --dry-run planned 624 backend files and
506 frontend files at -n auto, which is not run on a shared box.

The branch is rebased onto the merge commit of #13007, which is current main.
That rebase is curative rather than churn: three reds on the previous head were
inherited from the base, where test_crew_log_exc_info_sites.py's vetted list did
not name session_tree_projection.py, and that merge commit adds the six entries.
main did not touch any source file in this diff.

5. Any other suggestions on the work

The teardown gap this section used to disclose is closed. It was listed as a
follow-up on the reasoning that the destroy path was already shaped that way and
that before this change there was no row to leave behind; review was right that
this is not a follow-up, because register is the last launch step and so the row
exists exactly when a cancel unwinds. teardown now removes it, keyed on the task
it stopped, which closes the destroy path as well as the cancel one. Section 3 has
the mechanism.

One review finding is deliberately not taken, and this is the reasoning.
Review flagged that a cancel landing inside the EC2 lane's registration leaves a
stale crew row, because that lane's teardown is ec2.destroy plus
wait_for_delete with no unregister, while the Fargate lane added here does remove
its row. The finding is accurate. Its suggested fix -- unregister from
_rollback_cancelled_stack -- is the one this repository has already considered and
refused: the destroy route at dashboard/handlers_cloud.py derives the instance id
from the stack itself, BEFORE the delete, and deliberately will not accept it from a
caller, because connect.unregister_instance matches a needle against every
registered box with no cross-check against the tag, so a mismatched value silently
removes a DIFFERENT, still-running crew's registration, which is not recoverable
from the dashboard. That code also names the failure direction it chose: an empty id
skips the unregister and leaves "a stale registry row the user can see and remove.
That is the safe direction to fail." The residual this finding describes is exactly
that accepted residual, and the window it needs is a cancel arriving strictly inside
a single local registry write with no AWS round trip, since any earlier cancel is
raised before the step. The review's own adjudication pass reached the same
conclusion and flagged it as an extreme edge case. Closing it properly belongs to
the EC2 lane -- the destroy path would have to carry the instance id into teardown --
which is outside this PR's subject.

One follow-up remains, outside launch-time auto-registration:

connect.is_launched_instance does not recognise a Fargate crew. It compares
a registry ssm_target against instance_id in the launch job files, which hold
the task ARN, so it answers False for these records. Its job is to protect a
correlated instance's addressing fields from PATCH /api/instances/{id}, so the
consequence is that a launched Fargate crew's target can be rewritten through
that endpoint. The root shape is that the launch job and the registry identify the
same crew by two different strings: the launch job holds the task ARN, the registry
holds the ECS target. Closing it wants the same kind of comparison
unregister_ecs_task uses -- split the target and compare the task id -- rather
than another string equality.

Pattern harvest

Rule candidate: review-prompt
Pattern: a no-op stub whose docstring defers to a named blocker that has since been lifted, leaving both the stub and the stale reason in place.

This defect was exactly that shape. FargateLaunchEngine.register() returned
without doing anything, and its docstring blamed a registry whose transport
tuple no longer excluded this lane. Nothing raised, nothing failed a test, so
nothing surfaced it; the only signal was prose that had quietly become false.

A review prompt can catch the class: on any diff that widens a registry,
allow-list or capability tuple, ask which deferral stubs cited the old narrower
value. A linter cannot, because the link between the stub and the constant
exists only in the docstring's prose, not in any reference a checker can follow.

Fixes #12511

@chenmingwei23
chenmingwei23 requested review from a team and cixuuz September 23, 2026 01:34
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 6217a7d961ab7910b403c87f52d4a322fe323eca and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/cloud/connect.py:274 -- "dashboard resolves the right engine" contradicts production; engines resolve from launch jobs, while this metadata only controls presentation -> Fix: describe provisioner_id as source metadata preventing incorrect EC2 guidance.
[GPT-REVIEWED] 6217a7d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6217a7d961ab7910b403c87f52d4a322fe323eca: <one-sentence reason>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-register-task-12511 branch from 227046d to 99d618f Compare September 23, 2026 04:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 6217a7d961ab7910b403c87f52d4a322fe323eca — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify once against a real Fargate launch: every DescribeTasks shape here is mocked, and the runtimeId/target composition is only proven against fixtures.

Suggestions

  • The disclosed is_launched_instance gap is created by this PR (Fargate rows now exist but lack the PATCH addressing protection EC2 rows get) — land the task-id comparison follow-up promptly rather than letting it age as a known hole.

[DESIGN-REVIEWED] 6217a7d

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 6217a7d961ab7910b403c87f52d4a322fe323eca — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

unregister_ecs_task's three-value result is wider than its readers: both consumers ask only "did a row survive?", so removed-vs-absent has zero readers.

Not justified as shipped

  • Item 8 — one consumer, generalized: the docstring argues "there was no row" and "the row could not be removed" are both a falsy answer, but the only 2 consumers (grep UNREGISTER_ → fargate_engine.py:1698, :1765) test solely == connect.UNREGISTER_FAILED; nothing distinguishes UNREGISTER_REMOVED from UNREGISTER_ABSENT.
    Subtraction: collapse unregister_ecs_task to a bool ("a row of ours may remain") and delete the UNREGISTER_REMOVED / UNREGISTER_ABSENT / UNREGISTER_FAILED constants.
    Clears when: the tri-state is collapsed to the two-state answer, or a real consumer of the removed/absent distinction is counted.
  • Item 10 — rides along: declared in the description; fixes a pre-existing cross-object race that this PR's launch-time writer makes live, with no premise risk.

What this change ships

Inventory (10 items) — 8 justified

Intent: FIX — a Fargate crew launched through the cloud lane never appeared in /instances because register() was a no-op whose documented blocker (#12510 closed it) is gone; wiring was tracked in #12511 on the base docs.

  1. A launched Fargate crew now appears under "Your crews" automatically — justified
  2. The launch can block up to 180s polling ECS for the container's runtime id — justified
  3. A registration failure over a live task leaves the launch DONE with the reason on the connect step — justified
  4. A task seen dead or stopping during registration fails the launch — justified
  5. Cancel pressed during the registration wait now tears the task down — justified
  6. Fargate teardown removes the stopped task's crew-list row and withholds confirmation when it cannot — justified
  7. The TTL reap sweep removes an expired task's row too — justified
  8. New connect.unregister_ecs_task answering three result constants — one consumer, generalized: both call sites read only UNREGISTER_FAILED
  9. register_instance gains connection_method / provisioner_id keywords with EC2 defaults, so existing callers are unchanged — justified
  10. Registry objects over one file now share a path-keyed lock so gateway threads cannot drop each other's records — rides along: declared, fixes the pre-existing cross-object race the new launch-time writer makes live

[FIRST-PRINCIPLES-REVIEWED] 6217a7d

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 6217a7d961ab7910b403c87f52d4a322fe323eca — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6217a7d

Verdict parsed from the review's SHA-scoped output markers for commit 6217a7d961ab7910b403c87f52d4a322fe323eca.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 6217a7d961ab7910b403c87f52d4a322fe323eca: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-register-task-12511 branch from 99d618f to 2686c5f Compare September 23, 2026 05:31
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 99d618fff27defa50ae10def0029f13e64c18042; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 99d618fff27defa50ae10def0029f13e64c18042: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 2686c5f3e05e9ac41e24475c3440e57defa36aac; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2686c5f3e05e9ac41e24475c3440e57defa36aac: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-register-task-12511 branch from 2686c5f to 3513bfc Compare September 23, 2026 07:34
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 2686c5f3e05e9ac41e24475c3440e57defa36aac; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2686c5f3e05e9ac41e24475c3440e57defa36aac: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 3513bfc0f422c33f21f7ddfa7f5ebbf14d8647cb; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3513bfc0f422c33f21f7ddfa7f5ebbf14d8647cb: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-register-task-12511 branch from 3513bfc to b1b9911 Compare September 23, 2026 09:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging b1b9911ddfddd94acf8968c6ab856f52dccbb475. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/cloud/fargate_engine.py:1495 -- stopped tasks can leave stale registry records
registered = connect.register_instance( / connect.unregister_ecs_task(task_cluster or cluster, task_id)
registered task -> TTL reap stops it without unregistering, or teardown ignores failed removal -> dead ECS row persists
Anchor: residual/crash-data-loss-corruption
Fix: unregister every stopped task and prevent teardown confirmation when registry removal fails.

FINDING -- src/kiro_crew/cloud/fargate_engine.py:1456 -- provisioner_id is claimed to select engine_for, but engine resolution uses launch-job provider_id -> Fix: describe it only as persisted source metadata and remove the engine-selection claim.

[BLOCK-MERGE] b1b9911
[GPT-REVIEWED] b1b9911

Adjudication (Opus 5) — is blocking on each finding proportionate?

F1 — harm rung: MEDIUM (visible stale crew-list row, one dashboard DELETE clears it), but rung is not the fenced test.
Conditions confirmed, none extreme: register writes the row at src/kiro_crew/cloud/fargate_engine.py:1495; reap runs on every launch at fargate_engine.py:1152 with the on-by-default 6h TTL (fargate_engine.py:614, :652) and stops the task at fargate_engine.py:1682-1688 with no unregister_ecs_task — the only call site is teardown, fargate_engine.py:1746, whose bool is dropped before return plan.confirmed at :1754. Recovery: DELETE /api/instances/{id} (routes/connections.py:111 → handlers_instances.py:560-586); nothing auto-prunes.
Real fix is non-trivial (reap would need per-task registry writes on the provision path, and gating confirmation needs a tri-state out of the deliberately never-raising connect.py:399-433 helper, whose "no row" and "failed" answers are both False), but a registered task outliving 6h and being reaped by the next launch is ordinary operation, not a contradictory or unproducible combination — so FLAG's rarity bar is not met.

[ADJUDICATION] b1b9911ddfddd94acf8968c6ab856f52dccbb475 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] b1b9911ddfddd94acf8968c6ab856f52dccbb475

[ADJUDICATION-FENCED] b1b9911ddfddd94acf8968c6ab856f52dccbb475 fenced=1 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/cloud/fargate_engine.py:1495 -- The reap path at fargate_engine.py:1682-1688 runs on every provision (:1152) under the default 6h TTL (:614) and never unregisters, so a registered task orphaning its registry row is ordinary operation rather than an extreme condition combination.
[GPT-ADJUDICATED-FENCED] b1b9911ddfddd94acf8968c6ab856f52dccbb475

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b1b9911ddfddd94acf8968c6ab856f52dccbb475: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-register-task-12511 branch from b1b9911 to ed14211 Compare September 23, 2026 09:30
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for b1b9911ddfddd94acf8968c6ab856f52dccbb475; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b1b9911ddfddd94acf8968c6ab856f52dccbb475: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for ed14211a5d68d42e865220b6f04f5e3d4014b3fd; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ed14211a5d68d42e865220b6f04f5e3d4014b3fd: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 23, 2026
FargateLaunchEngine.register resolves the crew container's runtimeId with a
bounded ecs:DescribeTasks poll, composes ecs:<cluster>_<task-id>_<runtime-id>,
and registers it with connection_method="fargate" and the task definition's
published port. RunTask answers with a task ARN only, which the registry does
not address, so the target has to be read rather than derived.

The poll budget and cadence are the EC2 lane's own launch-time readiness wait
(180s at 6s), and the budget is a ceiling: the poll returns on the first read
carrying a runtime id. It is spent against a monotonic clock rather than a count
of polls, and the last sleep is cut to whatever the budget has left, because each
DescribeTasks round trip costs wall time a per-iteration counter cannot see and
thirty of them would push the documented number past itself. The poll count stays
as a second bound, so a sleep that returns early cannot turn the budget into a
tight spin on ECS.

A target is composed only from a task that is actually serving. ECS runs a task
through PROVISIONING, PENDING, ACTIVATING, RUNNING, DEACTIVATING, STOPPING,
DEPROVISIONING, STOPPED; the three states after RUNNING are billable while
nothing is listening, because the ENI is being torn down, and the container keeps
its runtimeId through all of them. So neither the presence of a runtime id nor
TaskSighting.is_running can decide reachability: that property answers a billing
question and admits every state but STOPPED, which is right for the teardown
warning it serves and wrong here. TaskSighting.is_serving answers the
reachability one and is_past_running separates "not serving yet" from "never
serving again", so a PENDING task is polled while a STOPPING one is refused.
desiredStatus counts too: a task ECS has been told to stop is on that path even
while lastStatus still says RUNNING.

An absence is not a death until the task has been seen. RunTask and DescribeTasks
are eventually consistent, so a task accepted moments ago is legitimately missing
from the first read; treating that as gone would fail the launch of a task that
goes on to start and bill, and the failed step is CONNECT rather than PROVISION,
so the provision rollback does not run and nothing stops it. The poll therefore
waits through an absence that precedes any sighting and ends only on a
DISAPPEARANCE, an absence after one. The expiry wording separates a task ECS
never listed from one that was listed and stayed pre-RUNNING, because those are
different things to go and look at.

await_registration_target returns whether the task it read may still come up, and
register turns that into the failure class: a task that has been seen and is now
gone or past running raises RuntimeError and fails the launch, because there is no
crew behind it to add by hand and none to tear down. Everything else raises
launch_job.RegistrationUnavailable, a new orchestration exception run_launch
records on the connect step while still reporting the task as launched -- the task
may be running and billing, so a red card there would leave its owner nothing to
act on. A failed DescribeTasks read counts as may-come-up, so a denied permission
never reports a live crew as dead.

That non-fatal path writes the failed connect step and the terminal status in one
save. A failed connect step beside a non-terminal status is the combination
reap_orphans reads as an interrupted launch, and it rewrites the status to FAILED,
so persisting the step on its own would leave a window where a restart turns the
intended DONE into a red card over a running crew.

run_launch checks for cancellation after the register step. That step holds a
poll of up to the full readiness budget, which is long enough for a user to press
cancel, and the cancel route only sets the event -- so without the check the
request is read by nobody, the job still reports DONE, and the task keeps
billing. The check is placed where provision already puts one, so a cancel during
a long AWS call is acted on when that call returns and the handler tears the task
down.

A stopped task does not keep its registry row. Both of this engine's
ecs:StopTask paths remove it: teardown, which a cancel or a failure unwinds
through, and the TTL bound sweep, which runs on every provision and is the path
that reaches a REGISTERED task in ordinary operation -- a crew outliving the
default lifetime is stopped by the NEXT launch. register is this lane's last launch
step, so the row exists exactly when those paths run, and leaving it offers a crew
list entry whose target resolves to nothing, with the diagnosis ladder reporting
only ssm_unreachable and nothing pruning it. teardown withholds its confirmation
when a removal failed, because a confirmation says nothing of ours may remain and
a listed crew addressing a stopped task is something of ours; the sweep answers
with a plan rather than a confirmation, so it logs instead. Telling a failed
removal from an absent row needs three answers, not a bool: "no row" and "could
not remove the row" are both falsy and mean opposite things, so
connect.unregister_ecs_task reports removed, absent or failed. It finds the row by
reading each ECS target back through split_ecs_target and comparing cluster and
task id, because a teardown holds the task ARN and never the runtime id, and a
prefix test would let cluster "crews" remove a row belonging to cluster "crews_eu".
It removes EVERY matching row rather than the first. The launcher's own writer
cannot produce two rows for one task -- it dedupes on the exact target, the target
embeds the container's fixed runtime id, and a re-launch gets a new task id -- but a
hand-added Remote crew row naming the same task under a different runtime id can,
and a row left behind addresses a stopped task that no sweep prunes, because a
stopped task is not listed for teardown at all.

connect.register_instance gains connection_method and provisioner_id keywords,
defaulting to "ssm" and the EC2 provisioner id, so the EC2 lane's calls are
unchanged and its ssm_target idempotency serves both methods. The Fargate lane
passes aws_fargate. That field is persisted source metadata and selects nothing:
the engine driving a launch is resolved from the launch job's own provisioner id,
never from a registry record. What reads it is the dashboard's crew list, which
captions a row by it and picks the lifecycle guidance and Remove warning it shows,
so a Fargate task left with the EC2 default is presented as an EC2 instance and
its owner pointed at the wrong console.

InstancesRegistry takes its lock from a module-level table keyed by the
registry's resolved path, rather than owning one per object. Its mutations are
read-modify-write pairs and callers construct a registry per request -- the cloud
connect helpers, the instances handlers and the dashboard server each build their
own -- so a lock owned by the object serialised nothing between them: a second
object admitted between the first's read and its write persists a document that
predates the first's record, and that record is gone. Registering a launched task
adds a writer on the gateway's launch path, which is what makes the overlap with
a dashboard edit reachable. Locks are never evicted from the table, because a
lock is what a writer may be holding. The guarantee is across threads, not
processes, and both docstrings claiming otherwise are corrected.

Fixes #12511
@chenmingwei23
chenmingwei23 force-pushed the fix/fargate-register-task-12511 branch from ed14211 to 6217a7d Compare September 23, 2026 12:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re the non-blocking finding on src/kiro_crew/cloud/connect.py:274: it is correct, and it is a line this PR added.

The docstring says provisioner_id is stored "so the dashboard resolves the right engine and lifecycle guidance for it". The engine half of that is false. The engine driving a launch is resolved from the launch job's own provisioner id, never from a registry record, so this stamp selects nothing. Its only reader is the dashboard crew list, which captions a row by it and picks the lifecycle guidance and Remove warning it shows.

This is the third occurrence of the same wrong claim. The other two -- FargateLaunchEngine.register's docstring and the instances spec's Fargate seam section -- were corrected earlier in this PR after review raised the same point; this one was missed because it sits in register_instance rather than in the Fargate lane.

Not amending the head for it. The blocking lanes are green on 6217a7d961ab7910b403c87f52d4a322fe323eca and this is a one-sentence docstring correction with no behaviour attached, so a new head would re-roll ninety lanes and cancel the twenty-five still in flight to change prose. Recording it here instead, to be corrected in the follow-up that closes the remaining is_launched_instance gap this PR already discloses -- that work touches the same identity question, so the two belong together.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@iamwhatever
iamwhatever merged commit 9e87ab6 into main Sep 23, 2026
93 checks passed
@iamwhatever
iamwhatever deleted the fix/fargate-register-task-12511 branch September 23, 2026 16:19
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 23, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
kirodotdev#12963)

FargateLaunchEngine.register resolves the crew container's runtimeId with a
bounded ecs:DescribeTasks poll, composes ecs:<cluster>_<task-id>_<runtime-id>,
and registers it with connection_method="fargate" and the task definition's
published port. RunTask answers with a task ARN only, which the registry does
not address, so the target has to be read rather than derived.

The poll budget and cadence are the EC2 lane's own launch-time readiness wait
(180s at 6s), and the budget is a ceiling: the poll returns on the first read
carrying a runtime id. It is spent against a monotonic clock rather than a count
of polls, and the last sleep is cut to whatever the budget has left, because each
DescribeTasks round trip costs wall time a per-iteration counter cannot see and
thirty of them would push the documented number past itself. The poll count stays
as a second bound, so a sleep that returns early cannot turn the budget into a
tight spin on ECS.

A target is composed only from a task that is actually serving. ECS runs a task
through PROVISIONING, PENDING, ACTIVATING, RUNNING, DEACTIVATING, STOPPING,
DEPROVISIONING, STOPPED; the three states after RUNNING are billable while
nothing is listening, because the ENI is being torn down, and the container keeps
its runtimeId through all of them. So neither the presence of a runtime id nor
TaskSighting.is_running can decide reachability: that property answers a billing
question and admits every state but STOPPED, which is right for the teardown
warning it serves and wrong here. TaskSighting.is_serving answers the
reachability one and is_past_running separates "not serving yet" from "never
serving again", so a PENDING task is polled while a STOPPING one is refused.
desiredStatus counts too: a task ECS has been told to stop is on that path even
while lastStatus still says RUNNING.

An absence is not a death until the task has been seen. RunTask and DescribeTasks
are eventually consistent, so a task accepted moments ago is legitimately missing
from the first read; treating that as gone would fail the launch of a task that
goes on to start and bill, and the failed step is CONNECT rather than PROVISION,
so the provision rollback does not run and nothing stops it. The poll therefore
waits through an absence that precedes any sighting and ends only on a
DISAPPEARANCE, an absence after one. The expiry wording separates a task ECS
never listed from one that was listed and stayed pre-RUNNING, because those are
different things to go and look at.

await_registration_target returns whether the task it read may still come up, and
register turns that into the failure class: a task that has been seen and is now
gone or past running raises RuntimeError and fails the launch, because there is no
crew behind it to add by hand and none to tear down. Everything else raises
launch_job.RegistrationUnavailable, a new orchestration exception run_launch
records on the connect step while still reporting the task as launched -- the task
may be running and billing, so a red card there would leave its owner nothing to
act on. A failed DescribeTasks read counts as may-come-up, so a denied permission
never reports a live crew as dead.

That non-fatal path writes the failed connect step and the terminal status in one
save. A failed connect step beside a non-terminal status is the combination
reap_orphans reads as an interrupted launch, and it rewrites the status to FAILED,
so persisting the step on its own would leave a window where a restart turns the
intended DONE into a red card over a running crew.

run_launch checks for cancellation after the register step. That step holds a
poll of up to the full readiness budget, which is long enough for a user to press
cancel, and the cancel route only sets the event -- so without the check the
request is read by nobody, the job still reports DONE, and the task keeps
billing. The check is placed where provision already puts one, so a cancel during
a long AWS call is acted on when that call returns and the handler tears the task
down.

A stopped task does not keep its registry row. Both of this engine's
ecs:StopTask paths remove it: teardown, which a cancel or a failure unwinds
through, and the TTL bound sweep, which runs on every provision and is the path
that reaches a REGISTERED task in ordinary operation -- a crew outliving the
default lifetime is stopped by the NEXT launch. register is this lane's last launch
step, so the row exists exactly when those paths run, and leaving it offers a crew
list entry whose target resolves to nothing, with the diagnosis ladder reporting
only ssm_unreachable and nothing pruning it. teardown withholds its confirmation
when a removal failed, because a confirmation says nothing of ours may remain and
a listed crew addressing a stopped task is something of ours; the sweep answers
with a plan rather than a confirmation, so it logs instead. Telling a failed
removal from an absent row needs three answers, not a bool: "no row" and "could
not remove the row" are both falsy and mean opposite things, so
connect.unregister_ecs_task reports removed, absent or failed. It finds the row by
reading each ECS target back through split_ecs_target and comparing cluster and
task id, because a teardown holds the task ARN and never the runtime id, and a
prefix test would let cluster "crews" remove a row belonging to cluster "crews_eu".
It removes EVERY matching row rather than the first. The launcher's own writer
cannot produce two rows for one task -- it dedupes on the exact target, the target
embeds the container's fixed runtime id, and a re-launch gets a new task id -- but a
hand-added Remote crew row naming the same task under a different runtime id can,
and a row left behind addresses a stopped task that no sweep prunes, because a
stopped task is not listed for teardown at all.

connect.register_instance gains connection_method and provisioner_id keywords,
defaulting to "ssm" and the EC2 provisioner id, so the EC2 lane's calls are
unchanged and its ssm_target idempotency serves both methods. The Fargate lane
passes aws_fargate. That field is persisted source metadata and selects nothing:
the engine driving a launch is resolved from the launch job's own provisioner id,
never from a registry record. What reads it is the dashboard's crew list, which
captions a row by it and picks the lifecycle guidance and Remove warning it shows,
so a Fargate task left with the EC2 default is presented as an EC2 instance and
its owner pointed at the wrong console.

InstancesRegistry takes its lock from a module-level table keyed by the
registry's resolved path, rather than owning one per object. Its mutations are
read-modify-write pairs and callers construct a registry per request -- the cloud
connect helpers, the instances handlers and the dashboard server each build their
own -- so a lock owned by the object serialised nothing between them: a second
object admitted between the first's read and its write persists a document that
predates the first's record, and that record is gone. Registering a launched task
adds a writer on the gateway's launch path, which is what makes the overlap with
a dashboard edit reachable. Locks are never evicted from the table, because a
lock is what a writer may be holding. The guarantee is across threads, not
processes, and both docstrings claiming otherwise are corrected.

Fixes kirodotdev#12511
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fargate lane: a launched task is not registered in /instances (register() is a no-op)

2 participants