Repository navigation
fix(cloud): register a launched Fargate task in the Instances registry - #12963
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/cloud/connect.py:274 -- False positive or not applicable? A repository writer can comment: |
227046d to
99d618f
Compare
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify once against a real Fargate launch: every Suggestions
[DESIGN-REVIEWED] 6217a7d |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (10 items) — 8 justifiedIntent: FIX — a Fargate crew launched through the cloud lane never appeared in
[FIRST-PRINCIPLES-REVIEWED] 6217a7d |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
99d618f to
2686c5f
Compare
GPT 5.6 Review —
|
GPT 5.6 Review —
|
2686c5f to
3513bfc
Compare
GPT 5.6 Review —
|
GPT 5.6 Review —
|
3513bfc to
b1b9911
Compare
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/cloud/fargate_engine.py:1495 -- stopped tasks can leave stale registry records FINDING -- src/kiro_crew/cloud/fargate_engine.py:1456 -- [BLOCK-MERGE] b1b9911 Adjudication (Opus 5) — is blocking on each finding proportionate?F1 — harm rung: MEDIUM (visible stale crew-list row, one dashboard DELETE clears it), but rung is not the fenced test. False positive or not applicable? A repository writer can comment: |
b1b9911 to
ed14211
Compare
GPT 5.6 Review —
|
GPT 5.6 Review —
|
FargateLaunchEngine.register resolves the crew container's runtimeId with a bounded ecs:DescribeTasks poll, composes ecs:<cluster>_<task-id>_<runtime-id>, and registers it with connection_method="fargate" and the task definition's published port. RunTask answers with a task ARN only, which the registry does not address, so the target has to be read rather than derived. The poll budget and cadence are the EC2 lane's own launch-time readiness wait (180s at 6s), and the budget is a ceiling: the poll returns on the first read carrying a runtime id. It is spent against a monotonic clock rather than a count of polls, and the last sleep is cut to whatever the budget has left, because each DescribeTasks round trip costs wall time a per-iteration counter cannot see and thirty of them would push the documented number past itself. The poll count stays as a second bound, so a sleep that returns early cannot turn the budget into a tight spin on ECS. A target is composed only from a task that is actually serving. ECS runs a task through PROVISIONING, PENDING, ACTIVATING, RUNNING, DEACTIVATING, STOPPING, DEPROVISIONING, STOPPED; the three states after RUNNING are billable while nothing is listening, because the ENI is being torn down, and the container keeps its runtimeId through all of them. So neither the presence of a runtime id nor TaskSighting.is_running can decide reachability: that property answers a billing question and admits every state but STOPPED, which is right for the teardown warning it serves and wrong here. TaskSighting.is_serving answers the reachability one and is_past_running separates "not serving yet" from "never serving again", so a PENDING task is polled while a STOPPING one is refused. desiredStatus counts too: a task ECS has been told to stop is on that path even while lastStatus still says RUNNING. An absence is not a death until the task has been seen. RunTask and DescribeTasks are eventually consistent, so a task accepted moments ago is legitimately missing from the first read; treating that as gone would fail the launch of a task that goes on to start and bill, and the failed step is CONNECT rather than PROVISION, so the provision rollback does not run and nothing stops it. The poll therefore waits through an absence that precedes any sighting and ends only on a DISAPPEARANCE, an absence after one. The expiry wording separates a task ECS never listed from one that was listed and stayed pre-RUNNING, because those are different things to go and look at. await_registration_target returns whether the task it read may still come up, and register turns that into the failure class: a task that has been seen and is now gone or past running raises RuntimeError and fails the launch, because there is no crew behind it to add by hand and none to tear down. Everything else raises launch_job.RegistrationUnavailable, a new orchestration exception run_launch records on the connect step while still reporting the task as launched -- the task may be running and billing, so a red card there would leave its owner nothing to act on. A failed DescribeTasks read counts as may-come-up, so a denied permission never reports a live crew as dead. That non-fatal path writes the failed connect step and the terminal status in one save. A failed connect step beside a non-terminal status is the combination reap_orphans reads as an interrupted launch, and it rewrites the status to FAILED, so persisting the step on its own would leave a window where a restart turns the intended DONE into a red card over a running crew. run_launch checks for cancellation after the register step. That step holds a poll of up to the full readiness budget, which is long enough for a user to press cancel, and the cancel route only sets the event -- so without the check the request is read by nobody, the job still reports DONE, and the task keeps billing. The check is placed where provision already puts one, so a cancel during a long AWS call is acted on when that call returns and the handler tears the task down. A stopped task does not keep its registry row. Both of this engine's ecs:StopTask paths remove it: teardown, which a cancel or a failure unwinds through, and the TTL bound sweep, which runs on every provision and is the path that reaches a REGISTERED task in ordinary operation -- a crew outliving the default lifetime is stopped by the NEXT launch. register is this lane's last launch step, so the row exists exactly when those paths run, and leaving it offers a crew list entry whose target resolves to nothing, with the diagnosis ladder reporting only ssm_unreachable and nothing pruning it. teardown withholds its confirmation when a removal failed, because a confirmation says nothing of ours may remain and a listed crew addressing a stopped task is something of ours; the sweep answers with a plan rather than a confirmation, so it logs instead. Telling a failed removal from an absent row needs three answers, not a bool: "no row" and "could not remove the row" are both falsy and mean opposite things, so connect.unregister_ecs_task reports removed, absent or failed. It finds the row by reading each ECS target back through split_ecs_target and comparing cluster and task id, because a teardown holds the task ARN and never the runtime id, and a prefix test would let cluster "crews" remove a row belonging to cluster "crews_eu". It removes EVERY matching row rather than the first. The launcher's own writer cannot produce two rows for one task -- it dedupes on the exact target, the target embeds the container's fixed runtime id, and a re-launch gets a new task id -- but a hand-added Remote crew row naming the same task under a different runtime id can, and a row left behind addresses a stopped task that no sweep prunes, because a stopped task is not listed for teardown at all. connect.register_instance gains connection_method and provisioner_id keywords, defaulting to "ssm" and the EC2 provisioner id, so the EC2 lane's calls are unchanged and its ssm_target idempotency serves both methods. The Fargate lane passes aws_fargate. That field is persisted source metadata and selects nothing: the engine driving a launch is resolved from the launch job's own provisioner id, never from a registry record. What reads it is the dashboard's crew list, which captions a row by it and picks the lifecycle guidance and Remove warning it shows, so a Fargate task left with the EC2 default is presented as an EC2 instance and its owner pointed at the wrong console. InstancesRegistry takes its lock from a module-level table keyed by the registry's resolved path, rather than owning one per object. Its mutations are read-modify-write pairs and callers construct a registry per request -- the cloud connect helpers, the instances handlers and the dashboard server each build their own -- so a lock owned by the object serialised nothing between them: a second object admitted between the first's read and its write persists a document that predates the first's record, and that record is gone. Registering a launched task adds a writer on the gateway's launch path, which is what makes the overlap with a dashboard edit reachable. Locks are never evicted from the table, because a lock is what a writer may be holding. The guarantee is across threads, not processes, and both docstrings claiming otherwise are corrected. Fixes #12511
ed14211 to
6217a7d
Compare
|
Re the non-blocking finding on The docstring says This is the third occurrence of the same wrong claim. The other two -- Not amending the head for it. The blocking lanes are green on |
kirodotdev#12963) FargateLaunchEngine.register resolves the crew container's runtimeId with a bounded ecs:DescribeTasks poll, composes ecs:<cluster>_<task-id>_<runtime-id>, and registers it with connection_method="fargate" and the task definition's published port. RunTask answers with a task ARN only, which the registry does not address, so the target has to be read rather than derived. The poll budget and cadence are the EC2 lane's own launch-time readiness wait (180s at 6s), and the budget is a ceiling: the poll returns on the first read carrying a runtime id. It is spent against a monotonic clock rather than a count of polls, and the last sleep is cut to whatever the budget has left, because each DescribeTasks round trip costs wall time a per-iteration counter cannot see and thirty of them would push the documented number past itself. The poll count stays as a second bound, so a sleep that returns early cannot turn the budget into a tight spin on ECS. A target is composed only from a task that is actually serving. ECS runs a task through PROVISIONING, PENDING, ACTIVATING, RUNNING, DEACTIVATING, STOPPING, DEPROVISIONING, STOPPED; the three states after RUNNING are billable while nothing is listening, because the ENI is being torn down, and the container keeps its runtimeId through all of them. So neither the presence of a runtime id nor TaskSighting.is_running can decide reachability: that property answers a billing question and admits every state but STOPPED, which is right for the teardown warning it serves and wrong here. TaskSighting.is_serving answers the reachability one and is_past_running separates "not serving yet" from "never serving again", so a PENDING task is polled while a STOPPING one is refused. desiredStatus counts too: a task ECS has been told to stop is on that path even while lastStatus still says RUNNING. An absence is not a death until the task has been seen. RunTask and DescribeTasks are eventually consistent, so a task accepted moments ago is legitimately missing from the first read; treating that as gone would fail the launch of a task that goes on to start and bill, and the failed step is CONNECT rather than PROVISION, so the provision rollback does not run and nothing stops it. The poll therefore waits through an absence that precedes any sighting and ends only on a DISAPPEARANCE, an absence after one. The expiry wording separates a task ECS never listed from one that was listed and stayed pre-RUNNING, because those are different things to go and look at. await_registration_target returns whether the task it read may still come up, and register turns that into the failure class: a task that has been seen and is now gone or past running raises RuntimeError and fails the launch, because there is no crew behind it to add by hand and none to tear down. Everything else raises launch_job.RegistrationUnavailable, a new orchestration exception run_launch records on the connect step while still reporting the task as launched -- the task may be running and billing, so a red card there would leave its owner nothing to act on. A failed DescribeTasks read counts as may-come-up, so a denied permission never reports a live crew as dead. That non-fatal path writes the failed connect step and the terminal status in one save. A failed connect step beside a non-terminal status is the combination reap_orphans reads as an interrupted launch, and it rewrites the status to FAILED, so persisting the step on its own would leave a window where a restart turns the intended DONE into a red card over a running crew. run_launch checks for cancellation after the register step. That step holds a poll of up to the full readiness budget, which is long enough for a user to press cancel, and the cancel route only sets the event -- so without the check the request is read by nobody, the job still reports DONE, and the task keeps billing. The check is placed where provision already puts one, so a cancel during a long AWS call is acted on when that call returns and the handler tears the task down. A stopped task does not keep its registry row. Both of this engine's ecs:StopTask paths remove it: teardown, which a cancel or a failure unwinds through, and the TTL bound sweep, which runs on every provision and is the path that reaches a REGISTERED task in ordinary operation -- a crew outliving the default lifetime is stopped by the NEXT launch. register is this lane's last launch step, so the row exists exactly when those paths run, and leaving it offers a crew list entry whose target resolves to nothing, with the diagnosis ladder reporting only ssm_unreachable and nothing pruning it. teardown withholds its confirmation when a removal failed, because a confirmation says nothing of ours may remain and a listed crew addressing a stopped task is something of ours; the sweep answers with a plan rather than a confirmation, so it logs instead. Telling a failed removal from an absent row needs three answers, not a bool: "no row" and "could not remove the row" are both falsy and mean opposite things, so connect.unregister_ecs_task reports removed, absent or failed. It finds the row by reading each ECS target back through split_ecs_target and comparing cluster and task id, because a teardown holds the task ARN and never the runtime id, and a prefix test would let cluster "crews" remove a row belonging to cluster "crews_eu". It removes EVERY matching row rather than the first. The launcher's own writer cannot produce two rows for one task -- it dedupes on the exact target, the target embeds the container's fixed runtime id, and a re-launch gets a new task id -- but a hand-added Remote crew row naming the same task under a different runtime id can, and a row left behind addresses a stopped task that no sweep prunes, because a stopped task is not listed for teardown at all. connect.register_instance gains connection_method and provisioner_id keywords, defaulting to "ssm" and the EC2 provisioner id, so the EC2 lane's calls are unchanged and its ssm_target idempotency serves both methods. The Fargate lane passes aws_fargate. That field is persisted source metadata and selects nothing: the engine driving a launch is resolved from the launch job's own provisioner id, never from a registry record. What reads it is the dashboard's crew list, which captions a row by it and picks the lifecycle guidance and Remove warning it shows, so a Fargate task left with the EC2 default is presented as an EC2 instance and its owner pointed at the wrong console. InstancesRegistry takes its lock from a module-level table keyed by the registry's resolved path, rather than owning one per object. Its mutations are read-modify-write pairs and callers construct a registry per request -- the cloud connect helpers, the instances handlers and the dashboard server each build their own -- so a lock owned by the object serialised nothing between them: a second object admitted between the first's read and its write persists a document that predates the first's record, and that record is gone. Registering a launched task adds a writer on the gateway's launch path, which is what makes the overlap with a dashboard edit reachable. Locks are never evicted from the table, because a lock is what a writer may be holding. The guarantee is across threads, not processes, and both docstrings claiming otherwise are corrected. Fixes kirodotdev#12511
1. What is the problem?
A Fargate crew launched through the cloud lane never appears in
/instances.FargateLaunchEngine.register()was a documented no-op, so the launch created atask, reported success, and left the crew unreachable from the Instances surface.
The blocker its docstring named is gone. That docstring said
instances/registry.pycloses its transport set to("ssh", "ssm"); onmaintoday
CONNECTION_METHODSis("ssh", "ssm", "fargate")andSSM_TRANSPORT_METHODSis{"ssm", "fargate"}(#12510). The registry couldaccept the record; nothing offered it one.
The launcher also could not compose the record's address. A registry record for
this lane is keyed on
ssm_target = ecs:<cluster>_<task-id>_<runtime-id>, andprovision()returns the task ARN straight fromRunTask, which carries thecluster and the task id but not the
runtimeId. ECS assigns that when thecontainer starts and only
DescribeTasksreports it, so the third field has tobe read rather than derived.
2. Why this issue matters to the user
The EC2 lane registers its instance at launch, so that crew is switchable the
moment setup finishes. A Fargate launch left the user with a running, billing
task and no path to it from the dashboard: they had to add it by hand in
Settings with the
fargateconnection method and paste an ECS target whoseruntime id they had no way to obtain except by running
aws ecs describe-tasksthemselves.
3. How our fix solves it
register()reads a target, then writes the record. It pollsecs:DescribeTasksuntil the crew container reports aruntimeId, composesecs:<cluster>_<task-id>_<runtime-id>from the task's own coordinates, readsthat back through
split_ecs_target(the registry's own reader, so a value theregistry would reject is refused where the reason can name the parts), and calls
connect.register_instance(connection_method="fargate", remote_port=FRONT_PORT).remote_portis passed explicitly becauseregister_instancedefaults to thestock remote dashboard port, and nothing in the crew container listens there:
the task definition publishes
FRONT_PORT(8080). Idempotency isregister_instance's own, matched onssm_target, so registering the same tasktwice updates that record in place and preserves its id, allocated local port,
TTL and
was_connected.The runtime id is read by container NAME, not by position.
sighting_from_taskgainsruntime_idfrom the container namedCREW_CONTAINER_NAME.containers[0]would work today and would silently pointevery registration at a sidecar's channel the moment one is added to the task
definition.
The wait budget is read, not invented.
REGISTER_TARGET_TIMEOUT_SECONDS(180) and
REGISTER_TARGET_POLL_SECONDS(6) are the numberscloud/wizard.pyalready uses to wait for a freshly started EC2 instance's SSM agent, which is
the same question asked of the other lane. The budget is a ceiling, not a delay:
the poll returns on the first read carrying a runtime id, so a task that starts
in twenty seconds costs twenty. Blocking there is consistent with the step --
run_launchis documented blocking on a worker thread, and the EC2 lane'sprovisionalready blocks for minutes on CloudFormation.Whether a registration failure fails the launch depends on the task. This is
the chain from the symptom: the fix adds an AWS read to a step that previously
could not fail, and that step's existing contract is
RealLaunchEngine.register's -- raise, and the whole launch is reported FAILED.Applied unconditionally to this lane, a task that was started and is billing
would be reported as a failed launch, which is the one outcome with nothing for
its owner to act on. Applied to no failure at all, a task that is already dead
would be reported as a good launch, which is worse: the job sits DONE for a crew
nothing can reach.
So
await_registration_targetreports whether the task it read may still comeup, and
register()turns that into the failure class. A task that is gone orpast running raises
RuntimeErrorand fails the launch, because there is norunning crew behind it to add by hand and none to tear down. Every other failure
raises a new
launch_job.RegistrationUnavailable, whichrun_launchrecords onthe connect step while leaving the job DONE with the reason on
job.error: adenied
ecs:DescribeTasks, a container still starting at the budget, coordinatesthat form no target, and a registry write that declined. A failed read counts as
may-come-up, so a missing permission never reports a live crew as dead.
Any other exception from
registerstays fatal, so the EC2 lane's behaviour isunchanged -- the distinction is not which lane it is, it is whether the engine
can say the launch itself is sound.
A target is composed only from a task that is actually serving. ECS runs a
task through
PROVISIONING,PENDING,ACTIVATING,RUNNING,DEACTIVATING,STOPPING,DEPROVISIONING,STOPPED. The three states afterRUNNINGarebillable while nothing is listening, because the ENI is being torn down, and the
container keeps its
runtimeIdthrough all of them. So neither the presence of aruntime id nor
TaskSighting.is_runningcan decide reachability: that propertyanswers a BILLING question and admits every state but
STOPPED, which is rightfor the teardown warning it serves and wrong here.
is_servinganswers thereachability one, and
is_past_runningseparates "not serving yet" from "neverserving again", so a
PENDINGtask is polled while aSTOPPINGone is refused.desiredStatusis read too: a task ECS has been told to stop is on that path evenwhile
lastStatusstill saysRUNNING.The non-fatal path writes the failed step and the terminal status in one save.
A failed connect step beside a non-terminal status is the combination
reap_orphansreads as an interrupted launch, and it rewrites the status toFAILED. Persisting the step on its own left a window where a restart turned theintended
DONEinto a red card over a running crew -- the outcome this pathexists to avoid. Both exits already save, so the intermediate save bought no
observable progress (the span between them holds only in-memory assignments, with
no I/O and no cancel check) and dropping it removes the window rather than
narrowing it.
connect.register_instancegainsconnection_methodandprovisioner_idkeywords, defaulting to
"ssm"and the EC2 provisioner id, so every existing EC2call site is unchanged. The Fargate lane passes
"aws_fargate":provisioner_idis persisted source metadata and selects nothing: the engine driving a launch is
resolved from the launch job's own provisioner id, never from a registry record.
What reads it is the dashboard's crew list, which captions a row by it and picks
the lifecycle guidance and Remove warning it shows, so a Fargate task left with
the EC2 default is presented as an EC2 instance and its owner pointed at the wrong
console. (Review caught an earlier version of this paragraph claiming the field
selects the engine;
engine_fordoes take a provisioner id, but the value comesfrom the launch job, so the registry stamp is a label.)
run_launchchecks for cancellation after the register step. That step holds apoll of up to the full readiness budget, which is long enough for a user to press
cancel, and the cancel route only sets the event -- so without the check the
request is read by nobody, the job still reports DONE, and the task keeps
billing. The check sits where
provisionalready puts one, so a cancel during along AWS call is acted on when that call returns and the handler tears the task
down.
InstancesRegistrytakes its lock from a module-level table keyed by theregistry's resolved path instead of owning one per object. Its mutations are
read-modify-write pairs and callers build a registry per request -- the cloud
connect helpers, the instances handlers and the dashboard server each construct
their own -- so a lock owned by the object serialises nothing between them: a
second object admitted between the first's read and its write persists a document
that predates the first's record, and that record is gone. Registering a launched
task adds a writer on the gateway's launch path, which is what makes the overlap
with a dashboard edit reachable. Locks are never evicted from the table, because
a lock is what a writer may be holding. The guarantee spans threads, not
processes: a separate
kirocrewCLI edit still serialises only onatomic_write's rename.An absence is not a death until the task has been seen.
RunTaskandecs:DescribeTasksare eventually consistent, so a task accepted moments ago islegitimately missing from the first read. Treating that as gone raised
RuntimeErrorand failed the launch, and because the failed step is CONNECTrather than PROVISION the provision rollback does not run, so the task kept
running and billing while the job reported it as not running. The poll now waits
through an absence that precedes any sighting and ends only on a DISAPPEARANCE, an
absence after one. The expiry wording separates a task ECS never listed from one
that was listed and stayed pre-
RUNNING, because those are different things to goand look at.
Both of this engine's
ecs:StopTaskpaths remove the stopped task's Instancesrecord, after ECS accepts the stop.
registeris this lane's last launch step, soa cancel observed just after it unwinds through
teardownwith the row alreadywritten, and
teardownonly issuedecs:StopTask. The result was a crew listentry addressing a stopped task, with no automatic recovery: the diagnosis ladder
reports
ssm_unreachableand the operator deletes the row by hand. The removallives with each stop rather than on the cancel branch, because that is where the
set of resources actually stopped is known, and it is keyed on the task, so a
sibling crew in the same cluster keeps its record.
connect.unregister_ecs_taskfinds the row by reading each ECS target backthrough
split_ecs_targetand comparing cluster and task id: a teardown holds thetask ARN and never the runtime id, so it cannot match a whole target, and a
ecs:<cluster>_<task-id>_prefix test would let clustercrewsremove a rowbelonging to cluster
crews_eu. It removes EVERY matching row rather than the first.The launcher's own writer cannot produce two rows for one task -- it dedupes on the
exact target, the target embeds the container's fixed runtime id, and a re-launch
gets a new task id -- but a hand-added Remote crew row naming the same task under a
different runtime id can, and a row left behind addresses a stopped task that no
sweep prunes, because a stopped task is not listed for teardown at all.
The second stop path is the TTL bound sweep, and it is the one that reaches a
registered task in ordinary operation: it runs on every
provision, so a crew thatoutlives the default lifetime is stopped by the NEXT launch. Review found it after
the first fix covered only
teardown, and it is the likelier path of the two --nothing about it requires a cancel. It logs rather than withholding, because it
answers with a plan and has no confirmation to withhold.
teardownwithholds its confirmation when a removal failed. A confirmation saysnothing of ours may remain, and a listed crew addressing a stopped task is
something of ours, so returning
Truewould report the cleanup as finished while arow is still there. That needs three answers rather than a bool: "there was no row"
and "the row could not be removed" are both falsy and mean opposite things, so
connect.unregister_ecs_taskreports removed, absent or failed. An absent rowstill confirms, because a task nothing registered leaves nothing behind.
The poll budget is spent against a monotonic clock rather than a count of polls,
and the last sleep is cut to whatever the budget has left. Each
DescribeTasksround trip costs wall time that a per-iteration counter cannot see, so thirty of
them pushed the documented 180s past its own number. The poll count is kept as a
second bound, so a sleep that returns early cannot turn the budget into a tight
spin on ECS.
Prose that stated the old behaviour was corrected in the same commit: the
describe_taskandapi_cloud_launch_taskdocstrings (both justified their ECSread by "this lane registers nothing" -- the real reason is that a record naming
ONE task cannot report that task's current state), the provisioner guide's lane
table and
registercontract, the instances spec sections 16.6 and the Fargateregistration seam (which stated every failure was non-fatal, and that
provisioner_idwas stamped on EC2 registrations only), and the Fargate RFC'ssections 5, 8, 9 and 11, whose deferred registry visibility this closes, and the
registry's module and class docstrings, which both stated that its read-then-write
keeps a live gateway and an out-of-band CLI edit from clobbering each other -- a
guarantee a lock owned by the object never provided and a thread lock cannot make
across processes. This round adds the same sweep for its own changes: the
instances spec's registry row repeated that cross-process claim and now states the
guarantee that holds, its Fargate seam section records the elapsed-time budget, the
eventual-consistency rule and the teardown removal, the provisioner guide's
teardowncontract row now tells a lane whoseregisterwrites a record to removeit there, and
_rollback_cancelled_stack's docstring no longer says the registryholds nothing when a cancel unwinds.
4. What tests we did
New tests, all passing locally:
test/test_fargate_launch_engine.py-- registration composes the right targetand passes
connection_method="fargate",FRONT_PORTand the Fargateprovisioner id; a container that started and then died yields no target even
though its runtime id is still readable, and fails the launch rather than
registering; each of
DEACTIVATING,STOPPINGandDEPROVISIONINGis refusedrather than registered, with the same case asserting that the sighting is
simultaneously
is_running(billable), carrying a runtime id, and NOTis_serving; a task whosedesiredStatusisSTOPPEDis refused whilelastStatusstill saysRUNNING; each ofPROVISIONING,PENDINGandACTIVATINGis polled to the budget instead of refused, which is what keeps onepredicate from answering both questions; a
STOPPEDtask fails the launch and isasserted NOT to be
RegistrationUnavailable; a live but unregistered task staysnon-fatal; a declining registry and an unresolvable target both raise
RegistrationUnavailableand never reach the registry; the runtime id is matchedby container name with a sidecar listed first; a task with no containers and a
started container with no runtime id both report none; the poll returns on the
first read that carries one (asserting the sleep count, so a task that starts
fast is not charged the budget); a stopped task quotes ECS; a task ECS does not
list is not rounded to a state; a denied read is reported as itself and is
asserted to leave the task may-come-up; the budget is bounded by the documented
number and its expiry names the remedy; a short ARN falls back to the spec's
cluster; coordinates that do not form a target are refused.
test/test_cloud_connect.py-- the caller'sprovisioner_idreaches the recordon BOTH the add and the idempotent-update path, and the two existing EC2 cases
still assert
aws_ec2, so the default is pinned.test/test_cloud_launch_job.py--RegistrationUnavailablekeeps the job DONEwith a FAILED connect step and the reason on both the step and the job, and does
not roll back; no save on that path ever persists a failed connect step beside a
non-terminal status (the test wraps
store.save, reloads from disk after eachone, and asserts the reaper-trap pair never appears, so it checks what a restart
would find rather than what the in-memory object holds); a restart after the
terminal save is skipped by
reap_orphansand keeps its DONE and its reason; theconnect step never keeps its "Added to Your crews" wording on a failed
registration; a refused sign-in still outranks a registration failure.
test/test_instances.py-- two registry objects over one file share a lock whileobjects over different files do not; and a second registry admitted between the
first's read and its write cannot drop the first's record. That test holds one
writer inside its own
_write, releases the other to add a different instance,and reads both records back from disk, so it checks what a restart would find.
Both waits are bounded, so the serialised case times out and proceeds rather than
hanging.
This round adds, in the same three files: a task absent from the first reads is
polled and still registers once it appears; a task ECS never listed expires with
aliveTrue and its own wording; a task seen and THEN gone is terminal (the casethat used to fire on the first read); the final sleep is cut to the remaining
budget and the budget is never overspent;
teardownremoves the stopped task's rowand leaves a sibling task's row alone; and
connect.unregister_ecs_taskmatches theright record, refuses a look-alike cluster (
crewsmust not removecrews_eu), andignores an EC2 record whose target is not an ECS target.
Also added: the TTL sweep removes the expired task's row;
teardowndoes notconfirm when the removal failed and does confirm when there was no row to remove;
and
connect.unregister_ecs_taskreports a failure apart from an absence, with therow still present in that case, which is why the two cannot share one answer.
Also added:
connect.unregister_ecs_taskremoves two duplicate rows naming onetask under distinct ids, rather than returning on the first.
Every fix is verified to be caught by its tests, not merely accompanied by them.
Reverting the check order made
test_a_container_that_started_then_died_yields_no_targetfail with the composedtarget (
assert 'ecs:crews_...' == '') andtest_register_fails_the_launch_for_a_task_that_crashed_after_startingfail withRegistrationUnavailable. Putting the reachability guard back to the billingpredicate made all three winding-down cases and the
desiredStatuscase fail,each with the composed target. Restoring the intermediate save made the invariant
test fail with the exact trap pair it guards,
[('running', 'failed')].Re-hardcoding the EC2 provisioner id made the connect test fail with
assert 'aws_ec2' == 'aws_fargate', and dropping the engine's keyword made theengine test fail with
KeyError: 'provisioner_id'. Dropping the cancel check made thecancel test fail with
assert 'done' == 'cancelled'. Restoring the per-objectlock made the identity test fail and made the interleave test fail with
assert ['first-1'] == ['first-1', 'second-1'], which is the dropped recorditself. Making a first-read absence terminal again made the
eventual-consistency test fail with the gone tuple where the composed target was
required, and the never-listed test fail with
assert False is True. Removing theregistry removal from
teardownmade both teardown tests fail with the survivingrow,
assert ['doomed'] == []. Restoring the full-interval sleep with no deadlinemade the budget test fail with
assert 6 < 6. Removing the sweep's own removal made thesweep test fail with the surviving row,
assert ['expired'] == []. Dropping theconfirmation gate made the teardown test fail with
assert True is False. Folding"failed" back into "absent" made the tri-state test fail with
assert 'absent' == 'failed'. Returning on the first matching row made theduplicate test fail with the surviving row still listed.
Commands run (this worktree on
PYTHONPATHviasetup.cfg'spythonpath = src,with
KIROCREW_HOMEunset so the suite's live-home guard is satisfied):pytest -n0 test/test_fargate_launch_engine.py test/test_cloud_launch_job.py test/test_cloud_connect.py-- 244 passedpytest -n0 test/test_instances.py -k TestRegistry-- 8 passedflake8,isort --check-only,mypy,black --check,scripts/check_black_formatting.py(baselined, diff-scoped),scripts/check_comment_history.py(selftest first, then scoped to this base),scripts/docs-lint.sh,scripts/check_feature_map.py,scripts/check_brand_name.pyGates:
flake8,isort --check-only,mypyon the three changed modules,black --check --target-version py310,scripts/check_black_formatting.py(baselined, in scope),
scripts/check_comment_history.py(selftest first, thenscoped to this change's base),
scripts/docs_lint.py,scripts/check_feature_map.pywithFEATURE_MAP_BASE_REF-- all pass. The fullsuite is CI's job;
scripts/local-gate.py --dry-runplanned 624 backend files and506 frontend files at
-n auto, which is not run on a shared box.The branch is rebased onto the merge commit of #13007, which is current
main.That rebase is curative rather than churn: three reds on the previous head were
inherited from the base, where
test_crew_log_exc_info_sites.py's vetted list didnot name
session_tree_projection.py, and that merge commit adds the six entries.maindid not touch any source file in this diff.5. Any other suggestions on the work
The teardown gap this section used to disclose is closed. It was listed as a
follow-up on the reasoning that the destroy path was already shaped that way and
that before this change there was no row to leave behind; review was right that
this is not a follow-up, because
registeris the last launch step and so the rowexists exactly when a cancel unwinds.
teardownnow removes it, keyed on the taskit stopped, which closes the destroy path as well as the cancel one. Section 3 has
the mechanism.
One review finding is deliberately not taken, and this is the reasoning.
Review flagged that a cancel landing inside the EC2 lane's registration leaves a
stale crew row, because that lane's
teardownisec2.destroypluswait_for_deletewith no unregister, while the Fargate lane added here does removeits row. The finding is accurate. Its suggested fix -- unregister from
_rollback_cancelled_stack-- is the one this repository has already considered andrefused: the destroy route at
dashboard/handlers_cloud.pyderives the instance idfrom the stack itself, BEFORE the delete, and deliberately will not accept it from a
caller, because
connect.unregister_instancematches a needle against everyregistered box with no cross-check against the tag, so a mismatched value silently
removes a DIFFERENT, still-running crew's registration, which is not recoverable
from the dashboard. That code also names the failure direction it chose: an empty id
skips the unregister and leaves "a stale registry row the user can see and remove.
That is the safe direction to fail." The residual this finding describes is exactly
that accepted residual, and the window it needs is a cancel arriving strictly inside
a single local registry write with no AWS round trip, since any earlier cancel is
raised before the step. The review's own adjudication pass reached the same
conclusion and flagged it as an extreme edge case. Closing it properly belongs to
the EC2 lane -- the destroy path would have to carry the instance id into teardown --
which is outside this PR's subject.
One follow-up remains, outside launch-time auto-registration:
connect.is_launched_instancedoes not recognise a Fargate crew. It comparesa registry
ssm_targetagainstinstance_idin the launch job files, which holdthe task ARN, so it answers False for these records. Its job is to protect a
correlated instance's addressing fields from
PATCH /api/instances/{id}, so theconsequence is that a launched Fargate crew's target can be rewritten through
that endpoint. The root shape is that the launch job and the registry identify the
same crew by two different strings: the launch job holds the task ARN, the registry
holds the ECS target. Closing it wants the same kind of comparison
unregister_ecs_taskuses -- split the target and compare the task id -- ratherthan another string equality.
Pattern harvest
Rule candidate: review-prompt
Pattern: a no-op stub whose docstring defers to a named blocker that has since been lifted, leaving both the stub and the stale reason in place.
This defect was exactly that shape.
FargateLaunchEngine.register()returnedwithout doing anything, and its docstring blamed a registry whose transport
tuple no longer excluded this lane. Nothing raised, nothing failed a test, so
nothing surfaced it; the only signal was prose that had quietly become false.
A review prompt can catch the class: on any diff that widens a registry,
allow-list or capability tuple, ask which deferral stubs cited the old narrower
value. A linter cannot, because the link between the stub and the constant
exists only in the docstring's prose, not in any reference a checker can follow.
Fixes #12511