Skip to content

feat(instances): show a crew chained behind another as its own tab - #13936

Merged
iamwhatever merged 1 commit into
mainfrom
feat/remote-crew-chaining-13744
Sep 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/remote-crew-chaining-13744

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Give a crew that only another crew can reach its own top-level tab, by chaining a second port forward through the parent's existing hop.

You cannot add a crew your own machine has no route to, even when a crew you are
already connected to can reach it.

Three machines, named this way throughout: A is the dashboard in front of you
(the hub). B is a crew A is already connected to. C is a machine only B
can reach -- B sits on its network, A does not. Today the only way to see C is to
open B's dashboard inside its pane and work there, one frame deep, because A can
neither dial C nor learn that C exists: A never sees B's registry.

A crew is another Kiro Crew gateway this dashboard connects to and shows in a
tab. A pane is the iframe a crew's dashboard is drawn in. The pane's own page
calls the gateway showing it its host. A crew's connection is an ssh -L local
port forward -- a hop -- so "reaching C through B" means a second forward aimed
at the port B's own forward already listens on.

Why it matters

Crews behind a bastion are the normal case, not an edge one, and today they cost
the user the whole tab bar: C is usable only while B's pane is the active tab, and
only through a dashboard drawn inside another dashboard. C gets no tab of its own,
no place in the switcher, and no independent state.

Not a goal

  • Reaching a crew through a non-SSH parent. A chained forward needs a port on the
    parent to aim at, which SSM and Fargate transports do not give it. chain_parent_not_ssh
    refuses it as a product limitation, not a safety rule.
  • Depth beyond 2 hops. The cap is what the switcher draws legibly, not a hard limit.
  • Closing the post-exit credential window. A gateway exit frees the lent hop port while
    its credential stays valid, and this PR BOUNDS that window (capped lent TTL) rather than
    closing it. Closing it is tracked in instances: a gateway exit frees a lent hop port while its chained credential stays live #14461.
  • Nesting a dashboard in a dashboard. The chained crew is a top-level tab, one iframe
    deep like every other pane.

What changed

B connects C; A shows it. C becomes a top-level tab of A, one iframe deep like
every other pane -- nothing is nested inside anything.

The symptom is that A cannot dial C. The root cause is that A has neither a route
nor a credential for C, and the design follows from supplying each without
inventing a third trust relationship.

The route. Three new registry fields. via_instance_id is the parent's id here
and via_remote_port is the port on that parent where its own forward to C
listens, so a chained forward dials B's host and B's loopback port -- C's
own coordinates describe it on its own machine and are never dialled from A, which
has no route to them. via_remote_id is C's id in B's registry, which is a
different fact from either: the first two say how to reach C, this one says what B
calls it. All three empty means a top-level crew, so an existing instances.json
loads unchanged.

The credential. A must not need C's key. So the token cannot be minted from
A, and the pane cannot be trusted to supply one. POST /api/instances/{id}/embed-token
asks B to mint it over the hop B already owns, carrying A's own dashboard port
so C's CSP admits A's page as the pane's frame ancestor. The token is scoped to A's
port, which is what makes it usable by A and useless as a general credential. The
generic chat proxy's refusal of a peer's api/instances plane is unchanged.

That request is addressed by via_remote_id, because B resolves it in B's registry.
An id derived from the crew's name on A's side equals B's only by luck -- a name
collision on B, a rename there, or an explicitly assigned id makes them differ, and
B then answers 404 for a crew it is holding. The id is re-checked against the
registry's own grammar and encoded as exactly one path segment before the request is
built: Instance.from_dict is deliberately tolerant, so a registry file written by
hand or by an agent can carry any string, and an id like victim/disconnect?x=
would otherwise address a different authenticated route on B and spend A's
credential for B there.

One invariant, swept mechanically rather than site by site. Four of the defects
found in review were one shape: a value read or checked before an await, then used
after it without being re-validated against the live object, or refused.
Patching the
site a reviewer names leaves the siblings, so every such site in the two files this
change owns was enumerated by an AST pass -- each local bound from live mutable state
(_tunnels, _tunnel_epoch, _tokens, _chained_hop_port, status(),
status_all(), the registry) with an await between the binding and a later use.
Fifteen candidates, disposed as follows:

Disposition Sites
Already re-validates -- the generation compare IS the re-read the parent-hop re-mint store, the refresh's store, the self-heal's tier-2 store, the embed mint
Already re-validates -- explicit re-read _mark_recovered (re-reads, compares object identity AND generation), diagnose (re-fetches after its probes), connect (holds the lock across the whole body and re-reads the row after the one mutation it makes)
Inert -- the post-await use cannot mislead _teardown_locked returning whether there had been a tunnel, shutdown counting its snapshot for a log line, the self-heal's attempt counter
Fixed here the connect handler attaching a token to a port frozen before the probe
Read-only, disclosed not fixed api_instances_search_sessions dispatches peer searches from a status_all() snapshot; a crew that drops in between makes its own peer call fail into the existing return_exceptions merge, so the cost is one search row

The one fixed site: POST /api/instances/{id}/connect freezes the response body from
the status, then awaits a token probe and possibly a re-mint, both seconds long, then
attaches the token. The status object stays live, and a teardown in that window pops
the tunnel without zeroing the port on it while the allocator hands that port to the
next connect first -- so the frozen port can name another crew's forward and the pane
would load it holding this crew's token. The live object is now re-read before the
token is attached, and a moved port or a state that left CONNECTED is refused with the
instance_token_unconfirmed the surrounding code already uses for "we cannot stand
behind this credential". Both halves are load-bearing: a teardown moves the port, and
a probe marking the forward ERROR moves only the state.

The chained adoption race belongs to this class too and is answered differently, at the
authority rather than the reader: the announcing pane's list is stale by design, so the
uniqueness invariant lives in the gateway inside the insert's own lock, and the pane
treats that refusal as benign.

One rule about credentials, enforced in one place. A dashboard token is a bearer
credential for ONE crew and it reaches that crew only over the forward it was minted
against; deliver it over a different forward and it reaches whoever is behind that one.
Minting is a multi-second round trip, so the forward can move inside it:

A chained credential may only be stored or returned after proving that the hop the
forward currently rides is the same hop it was minted for. Otherwise discard the mint.

Review found four separate exits where that was not proven, each in a different
function, so it is enforced ONCE instead of at a fifth site.
_credential_forward_moved answers why a credential must not be used, or nothing, and
_store_token -- which every credential store already passes through -- calls it and
DISCARDS. minted_at_epoch is a required argument there, so a caller cannot store
without saying which forward it minted against, and a store site added later cannot
forget the rule; an AST test pins both properties, because behaviour cannot see them.
The parent-side mint is the one exit that RETURNS a credential rather than storing it,
so it calls the same function and refuses instance_hop_changed.

Three readings, because the forward moves three ways. It is gone. Another forward took
its place -- membership reads true again after any reinstall, so the generation counter
is what tells one from the next. Or the crew holding the hop now serves this crew on a
different loopback port, which neither membership nor the generation can see because our
own tunnel was never touched. Both values compared were already recorded, so this is a
comparison and not new state: _chained_hop_port holds what the mint reply named, and a
chained tunnel's remote_port IS the hop it dials.

Both sides of that pairing pass through the same place. The mint returns the parent's
whole answer -- credential, hop and lifetime as one value -- and publishes none of it;
the store writes the hop and the lifetime behind the generation fence, and only then
compares. The ordering is the substance, not tidiness: the hop is the very value the
comparison reads, so a mint that published on its way out could drop a superseded port
where the check looks, leaving it comparing stale against stale and passing. Fencing one
side of a comparison does not fence the comparison. The structural test covers the write
side too, so a publish site added later cannot bypass it either.

That rule is also why the self-heal mints before it rebuilds and stores AFTER. Its mint
is where the parent names its current port, and until a rebuild has the forward on that
port there is nothing this credential may travel over -- so it is carried past tier 1
and stored once a rebuild rides the hop, on both the tier-1 and tier-2 paths. A heal
whose rebuilds never land stores nothing and the credential already held stays: a stale
token yields a 403 the client recovers from, where one delivered over another crew's
forward is a disclosure nothing downstream re-checks.

The comparison is on an IDENTITY, not on a port number. A port number cannot serve
as one: the parent's allocator hands a just-freed port to the next connect first, so the
same number names a different forward after ordinary churn, and a comparison on numbers
clears exactly the case it exists to refuse. So the parent states its own identity for
the hop alongside the token -- hop_id, the id it knows the crew by, and hop_gen, the
generation of its own forward to it -- and this gateway compares that against the
identity the forward HERE was built against. The port is read first as a cheap early-out
and cannot be what clears the check.

Both fields are REQUIRED, with the lifetime's strictness rather than the port's: an
unreadable port is dropped and the caller refuses to dial, while an unidentifiable hop
would let the dial succeed and weaken only the check guarding the credential.

Requiring them costs no compatibility, and that is measurable rather than argued.
The endpoint and the fields ship in the same change, so no build can answer this route
and omit them -- origin/main carries mint_embed_token, _mint_through_parent,
via_remote_id and adoptChainedCrew in zero files. A parent either predates the
feature, and has no route at all, which the existing 404 branch already names as "its
build has no endpoint for chaining through it"; or it has the route and the fields
together. There is no in-between for an optional field to serve. (embed_parent_port
does appear on main, but that is the non-chained embed parameter of the same name.)

Identifying the hop only decides whether to accept a NEW credential. The forward and
token ALREADY in place are the ones riding a hop that changed hands, and refusing a
replacement leaves them exactly where they were -- so one crew's bearer token keeps
arriving at another crew's gateway. Three things therefore act, and they are not
redundant -- but they do not all do the same KIND of work: two keep the credential from
reaching the wrong gateway, and the third retires a forward that cannot work any more.

Prevention -- a lent hop is withheld from allocation. Minting a chained credential
records the hop it named and when that credential expires, and _reserved_ports keeps
that port out of allocate until then.
While a hub's token is valid the port it forwards to is never handed to another crew by
this gateway
, so the moment the disclosure needs does not arise here.

That qualifier is load-bearing and an earlier revision of this description did not carry
it. The reservation is an allocator exclusion, which is not ownership: once the forward
serving the lent crew is torn down the port is free at the OS level, so any other local
process may bind it -- including a second gateway with its own registry, which cannot see
the reservation at all -- and the hub goes on dialling that port, handing a live bearer
token to whatever answers. Neither identity check catches it, because both run on a MINT
and an established pane does not re-mint per request; the proactive re-mint is scheduled
at ~80% of the token's lifetime, so the exposure lasts as long as the credential does
rather than being caught.

Ownership -- the lent port is held by the OS for the reservation's life.
HopPortGuard (hop_port_guard.py) binds the port itself, armed by the teardown that
frees it and by sync_hop_holds at startup. Startup is not belt-and-braces: the
reservation is persisted and a listening socket is not, so a restart would otherwise
arrive holding every reservation and owning nothing.

The socket listens, and that is forced rather than preferred. A socket that is bound
but not listening does not refuse a second SO_REUSEADDR bind -- which is exactly what
OpenSSH's forward listener and this module's own availability probe issue -- so the
tempting shape, occupy the port without listening and let a stale client be refused
before it transmits, does not hold the port at all. Measured both ways, and pinned, so a
later edit cannot quietly take the cheaper one.

The option differs by platform because its meaning inverts. On Windows
SO_REUSEADDR lets another local process steal an ACTIVE listener, and merely omitting
it is not enough either, since the other process can set it on its own socket; the
non-POSIX branch therefore sets SO_EXCLUSIVEADDRUSE. This is the same branch, for the
same reason, as browser_cli.view and the dashboard's own listener -- an earlier
revision of this PR set SO_REUSEADDR unconditionally, held nothing whatsoever on
Windows, and said in a comment that it did. That is the platform-specific no-op class
#9731 exists to catch, and it was CI on the Windows shards that caught it, not any case
that can run here. The branch is now pinned on its CONDITION rather than on the two
option names being present: a mutation that flattened the test to a constant left both
names in the source, one unreachable, and satisfied a presence-only assertion.

Because listening is mandatory the handshake completes and a stale client does transmit:
"the credential is never sent" is unreachable. What is reachable is that nothing reads
it. The guard accepts and closes at once with SO_LINGER 0 and never reads a byte, so a
stale pane sees its connection reset immediately -- not a hang on a queued connection
whose buffered credential any later accept would hand over, and not an orderly empty
reply a client could mistake for success. The refusal is deliberately blunt: answering
something courteous like 410 Gone would mean draining the request first, and the
request is what carries the credential.

Where the holds are taken is part of the mechanism, not an implementation detail.
Reading the reservation table is a file read plus a parse of every row, so it never runs
on the event loop -- the rule _reserved_ports already follows a few hundred lines
above, whose own comment says a synchronous version "would stall unrelated requests and
heartbeats". At startup the whole re-take runs in a thread behind the tracked task that
already backgrounds the revive, so it is off the gateway boot path, and it finishes
BEFORE the revive because a revive reconnects instances that allocate ports. In teardown
the read is taken AHEAD of the forward's stop rather than after it: awaiting anything
between the port's release and its bind would stretch the one gap this mechanism cannot
close from two adjacent statements to however long the shared default executor takes to
schedule, so only the bind -- one non-blocking loopback syscall, one port rather than one
per reservation -- stays in line. That ordering is pinned, because both orderings end
with the port held and differ only in how long it was free.

A hold that cannot be taken is remembered and retried, not logged and forgotten: a
reservation whose bind failed once and is never retried is a live reservation with no
socket, which is the exposure itself reached by another route. The usual cause is one of
this gateway's own leaked forwarders still occupying the port after a hard kill, and the
retry takes it the moment that process goes.

A port counts as in use only while a forward is actually LISTENING on it -- CONNECTED
state -- not while any tunnel object remembers the number.
An unexpected ssh exit
leaves its tunnel in _tunnels with local_port intact and the state ERROR, and past
max_recovery_attempts it stays there for good, so counting it as in use skipped the
hold for a port the OS had already freed while the credential naming it stayed valid.
Every site deriving in-use, reserved or occupied was checked against that sentence,
including ones this diff never touched: the allocator's _reserved_ports and the
reclaim's live_pids both count everything remembered and are RIGHT to, because they run
in the opposite safety direction -- over-counting costs the allocator a port it declines
to reuse, while over-counting the holds withholds a hold. Exactly one site was wrong.

Placement follows from it. The exit path takes the holds BEFORE its backoff, since the
child is already gone and waiting would leave the port unheld for the whole delay on every
flap and permanently once attempts run out. The recovery releases its own hold immediately
before the rebuild rebinds, or the hold blocks the self-heal; and it re-settles in a
finally, which one call covers both outcomes precisely because of the invariant -- a
rebuild that succeeded leaves the tunnel CONNECTED and its port in use, one that failed
leaves the port free and therefore held. Enumerating the recovery's failure exits instead
would let a later return skip it silently. Both orderings are pinned structurally,
because either one ends with the port held and they differ only in how long it was free.

Reclaiming such a forwarder is refused while its port carries a live reservation.
Reclaiming exists to hand a crew its recorded port back, and a lent port is one
allocate deliberately routes around, so freeing it recovers nothing and converts a port
safely occupied by this gateway's own dead child into a free one a stranger can bind
while a hub still forwards a bearer token to it. The orphan is the ownership in that
state, and a stronger one than a held socket; it is reclaimed by the next connect once
the reservation lapses, and lingering is already a tolerated outcome on that path.

Each hold carries its own deadline and expires itself, so it cannot outlive the
credential and squat a port that is legitimately free again; release runs before a
reconnect binds, so our own hold is never what refuses our own crew. Worst case is one
descriptor per live reservation, bounded by the same count that already bounds the
reservation table and far under both the 7778-upward range and the descriptor limit; a
hold that cannot be taken is reported at ERROR rather than passed over, because an
unheld reservation is precisely the exposure.

The window a gateway exit opens, and what bounds it. Two windows stay open by
construction and neither is claimed closed. The forward must release the port before the
guard can bind it, which is the gap between two adjacent statements rather than hours.
And shutdown releases every hold, because a socket cannot outlive the process holding it,
while the credential stays valid -- the remote crew issued it, so nothing on this side
can invalidate it and there is no revoke primitive in the tree to call. A holder that
outlived the process would have to be a separate deployable component, and revoking would
be a new contract with the remote crew; neither is within this mechanism.

What IS within it is the window's LENGTH, and that is why the lent lifetime is capped at
LENT_HOP_TTL_CAP (30m) rather than taken from the crew's row. The exposure after an
exit runs to the lease deadline and no further, so it is one lent TTL: 30m instead of the
20h a default row would give, and the next start re-takes every hold through
sync_hop_holds. The cap is a minimum against the row, so a row configured shorter keeps
its own figure, and it applies only to the credential that LEAVES -- this gateway's own
pane is not lent to anyone and keeps the row's lifetime.

One number reaches three places: the lifetime the remote crew issues the token for, the
deadline the lease is recorded under, and the ttl the answer reports. It is computed
once and passed to each rather than re-read from the row at each site, because a site
disagreeing with the others would either free the port under a live credential or have
the hub schedule its re-mint after the token is already dead. 30m rather than less
because the re-mint fires at DEFAULT_TOKEN_REFRESH_FRACTION of the lifetime, leaving
20% -- 360s -- as the margin a mint has to complete inside, against a worst case of
MINT_TIMEOUT_CEILING_SECS + 15 for a chained mint and 90s for an SSM one; a cap that
ate that margin would expire the token mid-mint and reload the hub's pane every cycle. A
test computes that margin from those same constants, so lowering the cap without
re-checking the arithmetic fails there rather than in the field.

It is not a lease. The deadline is the TTL this gateway itself issued, so nothing is
expected of the hub and no state crosses the boundary -- no notification, no ordering
contract, and the parent learns of its own crew's disconnect locally as it always did.
The allocator already took the exclusion (allocate(exclude=...)), and
_reserved_ports is its single builder feeding the single call site, so the new source
reaches every allocation by construction. Persisted rather than held in memory, because a
restart would otherwise drop a reservation while the credential it protects is still live.

The reservation is a PRECONDITION of handing the token over, not a note taken on the way
out, and three things make it one. It is written under the manager lock, in the same
critical section as the check that the hop is still ours -- validating outside the lock
and writing inside would leave a gap the width of the acquire, and that gap is the whole
hazard. The write propagates its error rather than going through the registry's
best-effort hint helper, whose documented behaviour is to swallow one: right for a hint
that only has to be usually-there, wrong for the record that keeps this port away from
another crew. And a reservation that cannot be written refuses the mint with 503
instance_hop_lease_failed, because a credential this gateway cannot protect is one it
does not issue.

The reservation does not live on the crew's row, and that placement is the point.
Held there, removing the crew deleted it -- while its credential was still valid -- and
the port went back to the allocator, which is the disclosure the reservation exists to
prevent, reached through the one path that deletes a row. So it is held on the registry
DOCUMENT, keyed by port: remove filters the instance list and structurally cannot touch
it. Same file, same lock, same atomic write; the only writer prunes lapsed entries on its
way through, so the record stays bounded without a timer. Every row-deleting path was
swept for this -- there is exactly one, and update keeps the row, so it was never a
second hole.

The parent answers that hub's next mint with instance_not_found, which is why that code
retires the forward -- and why the reply's code is read BEFORE any status branch. A
build with no chaining route answers 404 as well, and retiring over that would tear a
chain down because the parent is OLD rather than because the hop moved. Stated plainly:
once the record is at the document level, removal is a case prevention ANSWERS rather
than one detection bounds -- the port stays withheld through it, and an earlier revision
of this description claiming otherwise was describing the row placement this one replaced.

Detection on the parent's own answer -- retirement, not a second guard. With the
reservation at the document level this path is not what keeps the credential out of
another gateway; prevention is. What it does is retire a forward that cannot work again,
because the crew it reaches is gone from the parent. A re-mint that fails because the
parent ANSWERS that the crew is not connected there, or not there at all, is terminal for
the forward: retrying cannot recover a hop that is not ours, and the forward still points
at a port the parent is free to reassign once the reservation lapses. Read from the
reply's code, never its sentence,
and raised as HopRetiredError so the caller acts on which failure it saw instead of
inferring "gone" from "failed". Every other mint failure keeps the deliberate
non-terminal retry -- a timeout is a blip, and tearing a working chain down over one
would turn every hiccup into a disconnect the user has to undo.

Detection on a disagreeing identity -- the case prevention cannot reach in principle.
The parent still holds the crew and merely serves it elsewhere, so no port was reused at
all: no reservation ever applied to that new port, the old one is still withheld, and
there is nothing for prevention to hold that would help. The comparison runs, the
credential is refused,
and the forward riding the moved hop is retired with it. Only this branch retires. A
superseded generation is rejected by the store's own fence before the comparison is
reached; a port-only disagreement is also what the connect path produces while a forward
is being built, so retiring on either would tear down a healthy tunnel -- and a
port-only disagreement needs no retirement anyway, because the port the forward still
dials is one prevention is holding.

A chained row's own coordinates are records, never dial targets. For a row carrying
via_instance_id the dial target is the parent's host plus via_remote_port; the row's
own ssh_host and remote_port are never dialled from here. That matters because both
arrive in the announcing pane's payload, which is untrusted -- and an origin check cannot
help, since the sender is the remote gateway the owner deliberately connected to and so
holds the correct origin. What makes a forged value harmless is the branch rather than a
check on the value: _resolve_transport returns _resolve_chained_transport as its first
branch whenever via_instance_id is set, and that resolver builds its params with
ssh_host=validate_ssh_host(parent.ssh_host). The branch reading inst.ssh_host is the
unchained one, below the return.

Since they are records, the dashboard stops rendering them where a verified target goes: a
chained row shows its host marked as reported and omits the crew's own gateway port. The
value is kept rather than replaced with the parent's host, because only the parent knows
which machine the crew is actually on -- deriving it would state something false and make
two crews behind one parent indistinguishable, while changing nothing about the dial
target. What a forged notice can still influence is via_remote_port, a loopback port on
the parent, a machine this actor already holds and whose own gateway mints the token
regardless; the depth, width, duplicate and cycle guards all apply here against a registry
the pane never sees, so the bounded result is rows pointing back through the same parent,
visible in the Remote Crew list and removable with the cascade.

Retirement keeps the user's intent, so an ordinary reconnect brings the crew back on a
hop that is actually ours. Clearing was_connected would present a security teardown as
the user having turned the crew off.

The retirement is scheduled rather than awaited, and that is not a style choice: the
store runs both inside connect's lock and outside it from the refresh, asyncio.Lock
is not reentrant, and awaiting a teardown from under the lock would park forever and
wedge every later connect, disconnect and self-heal. It hands the work to a task that
acquires the lock itself -- the same shape the self-heal already uses.

The token and the port B answers with are ONE claim. B's reply names the token
and the loopback port B's own forward to C listens on, and A dials that port. B's
mint is a multi-second round trip taken without B's manager lock, status() hands
out the tunnel's live status object, and a teardown pops that tunnel without zeroing
the port on it -- while B's allocator gives a just-freed port to the next connect
first, because it takes the first free port above its base. So C disconnected inside
that window, plus any crew connected before the mint returns, would pair C's token
with the other crew's forward, and A would forward to whatever now answers there.
Nothing downstream re-checks it. B therefore reads the hop once before the mint,
confirms the generation, the connected state and the port are all unmoved after it,
and refuses instance_hop_changed rather than answering with a pair it cannot vouch
for. The generation is what catches a reconnect, which satisfies membership again.

The token's lifetime comes from B too. B issues it under B's record for C, which
A cannot know; A's own row defaults to 20h, so scheduling the refresh from it would
put the refresh after a shorter token had already expired. The reply carries the TTL
it issued and A refuses a mint whose reply cannot state a usable one -- A's own number
is a separate value with nothing holding it below B's, so it is not a safe stand-in,
and every gateway that answers this endpoint at all reports its TTL. A then stores the
SHORTER of B's answer and its own, which makes the schedule safe in both directions
however the two are configured: short only costs an early re-mint, long schedules
after the token is dead.

And the total is published, because the remaining is measured against it. Every
reader asking how far through its life a token is divides token_ttl_remaining by a
total, and the two have to be the same number. The row's TTL is not that number for a
chained crew, so the status now carries token_ttl_total -- the figure the remaining
counts down from -- and one shared helper picks it for all three readers. Measured
against the row instead, a 1h token from B sits permanently past a 20h row's refresh
threshold: A re-mints on every 60s poll, and each re-mint re-derives the iframe src,
so a chained crew's hidden pane is remounted every minute and quietly loses whatever
was unsaved in it. Nothing self-corrects, because the 10s rate window never engages
against a 60s poll.

And the refresh SCHEDULE is re-derived, not remembered. The stored lifetime is
rewritten by every store, and for a chained crew it is the shorter of B's answer and A's
row -- so lowering the TTL on either side changes it underneath a running refresh loop.
ttl is not a transport field, so that edit tears no tunnel down and restarts nothing.
The loop therefore reads the interval back from the lifetime the last refresh stored
rather than binding it once before the loop: bound once, it would keep sleeping the old,
longer gap while the shorter token died early, reloading the pane and discarding unsaved
work on every cycle from then on rather than once. A failed refresh stores no new
lifetime, so it reads back the same number and the deliberate retry-at-the-same-interval
behaviour is unchanged.

That desync is NOT introduced here -- main has the same shape, where every store
rewrites the lifetime while the loop derives its interval once, so lowering an unchained
crew's ttl desyncs it today. What this change adds is a second way to reach it, because
the effective lifetime is now the shorter of two records and the parent holds one of them.
The fix is at the one site either trigger passes through, and the derivation moved into a
single helper so a future reader cannot compute an interval the loop will not refresh.

These three -- the denominator, the published total, and now the interval -- are one
pattern rather than three bugs: a value's MEANING changed, and each time a consumer of it
was still computing from what it used to mean. So the round enumerated every reader of
the stored lifetime and every value derived from a TTL, on both sides of the wire, and
asked of each whether it re-reads or computes once. Ten sites; nine re-read at the moment
they need the value, and the loop was the only one that did not.

Six refusals, all decided server-side, because the request can originate inside a
pane whose code A does not control:

Refusal When
chain_too_deep More than MAX_VIA_HOPS (2) hops. Checked before anything is written or dialled, and again by B, which is the only side that can see it reaches C through a further hop.
chain_duplicate B already has a row for this remote crew. One row per remote crew is an invariant about STORED state, so A settles it inside the same lock as the insert rather than leaving it to the announcing pane: that pane decides from a list it refreshes only after the add and the connect the add triggers, so a second announcement inside that multi-second window reads a list without the first row. The registry would take it under a -2 suffixed id, leaving one remote crew with two rows, two forwards, two tabs and two charges against the width cap, removable only by hand. The pane treats this refusal as benign -- it refreshes and relays nothing, because the crew is present and nothing the user did failed.
chain_parent_full B already carries MAX_CHAINED_PER_PARENT (8) chained crews. Depth bounds how far a chain reaches; this bounds how wide it gets, and neither implies the other -- a two-hop chain is legal however many crews sit at the second level. It is counted per parent, so one parent's pane cannot crowd the others out, and it matters because these rows are created by that pane rather than by anyone at A's dashboard.
chain_parent_unknown The named parent is not a crew configured here at all.
chain_parent_not_ssh B is reached over a non-ssh transport, whose forwarder takes no second local forward from this gateway. SSM-reached crews therefore cannot be chained through.
cycle After the hop opens, A reads the far end's gateway_id off /api/health; equal to A's own or to any ancestor's, the forward is torn down and the connect refused.

Ids are compared, never host strings: one machine answers to localhost, an ssh
alias, an FQDN and an IP, so a string comparison would refuse unrelated crews and
still admit a real loop. gateway_id is a random per-home id, minted once and
memoized, revealing nothing about the machine.

Reading that id judges a descriptor, not a name. The id file lives in the
owner-writable data home, so a read that checks the path and then opens the path
resolves the name twice and the two answers need not describe the same object -- it is
whatever is there at the moment of the open that gets read. That window has two
consequences and the second is the worse one: a symlink redirects the read, and a FIFO
makes the open wait for a writer. The wait is what costs the gateway, because this read
runs in asyncio.to_thread on the SHARED default executor, so a held worker is lost
capacity until restart rather than one slow request -- and /api/health re-enters the
read on every poll, because a rejected file is deliberately not cached. So the open
itself goes through platform_compat.open_file_no_reparse, which refuses the redirect at
the final name on either platform and, asked to, declines to wait; fstat
judges the object the descriptor already names, which leaves no second resolution to
disagree with.

Borrowing that primitive rather than writing the flags out is the point, not a style
choice. getattr(os, "O_NOFOLLOW", 0) is 0 on Windows, so a hand-rolled
os.open with it refuses a link on POSIX and silently follows a reparse point on
Windows while reading as though it refused both -- the no-op catalogued in #9731,
where this site is now recorded as one more of the read-only opens that issue calls
convertible today. The primitive does NOT refuse a FIFO, though; nonblocking only
stops the open from waiting for one, so the S_ISREG check stays this reader's own
and is what rejects a pipe whose writer wins the race. A regression to the raw flag
behaves identically on POSIX, so it is pinned structurally instead, by an assertion
that is deliberately not skipped on any platform.

Considered and deliberately NOT done: giving gateway_id a sandbox disposition. It is a
trust-root file, so a policy entry for it has a blast radius well past this PR, and the
no-follow open closes the reachable mechanism whatever the path's write permissions are;
it belongs in its own change rather than riding this one.

What crosses the frame boundary, exactly. The pane posts
{type, v, id, name, sshHost, remotePort, port} -- no token and no credential. The
two directions are guarded differently:

  • Outbound uses targetOrigin: '*', because a cross-origin iframe cannot read
    parent.location and so cannot name its host's origin. That is safe only because
    the payload holds nothing secret; intercepting it yields a port number.
  • Inbound is where trust is established. The host resolves the sender's origin
    against its map of currently-warm pane ports and drops any message whose origin
    does not resolve, so the parent id is derived from the origin and never supplied
    by the sender. The payload is then validated by readChainedCrewNotice -- pure and
    exported so its rules are tested without a host -- and handed to the gateway, which
    owns the decisions no frame may make.

A re-announcement always connects. When the announced crew already has a row here,
A repoints it only if B is serving it on a new port, but connects it either way. An
announcement means the pane's user just connected that crew on B, while the tab here
can be down for reasons the row cannot show -- A restarted, or the hop dropped and
took the child with it. Deciding from the port alone left a dead tab beside a running
crew. The decision is a pure chainAdoptionPlan, exported so that contract is tested
rather than buried in a callback, and the backend's connect is idempotent.

A refusal is reported, not swallowed. When A declines an announced crew, only A
holds the reason: the depth cap and the cycle guard are its gateway's decisions,
taken against a registry the pane never sees. A relays that reason down to the
announcing pane, addressed to that pane's exact loopback origin, and the pane shows
it beside the connect it already reported -- the crew IS connected there, and only
the tab up here was declined. The pane accepts the message solely from its own
parent frame, which is the check available on that side.

The chain is also a lifecycle. Disconnecting a crew closes the forwards behind
it first, since theirs ride its hop. Removing one brings every captured crew DOWN
first, deepest first, and a stop that raises refuses the whole removal with nothing
deleted: rows deleted over a live forwarder strand it together with its minted
token, and the row that goes carries the forwarder_pid reclaim hint with it, so
nothing finds it again while the gateway runs. Each descendant is named explicitly
rather than reached through the parent's own cascade, which suppresses a child's
failure because a shutdown must not abort on one stuck crew. Only once everything
is down do the rows go, leaves first and the named crew last, so an interrupted
sweep can only leave a parent with fewer children -- a state that still connects --
never a row naming a parent that is gone.

A second pass follows the deletion over the same captured list. That list is the
point: disconnect finds a crew's children by reading the registry, the rows are
gone by then, and reg.remove is offloaded and therefore yields, so a connect
racing the removal can have re-established a child's forward -- sweeping the named
crew alone would leave a forward holding a port with no row to find it by. That
pass cannot refuse, because the removal has already happened and raising would
report failure for rows that are already gone; a forward it fails to stop is written
to the gateway's warning log naming the crew.

Being precise about who sees which of those two, because they are not the same: the
409 reaches the user, since it is an error response and the removal mutation
rejects on it, so the panel shows the gateway's own sentence. The residual does not
reach the screen, and it is deliberately NOT a field on the success body either. An
earlier revision published a stranded list there and nothing read it -- both
callers of removeInstance await without touching the body, and no component
references the name -- so it was a field on a public response with no reader, which
is a contract to keep with nothing depending on it and which let a real gap look
answered. The honest surface is a persistent warning about a forward with no row, a
notification affordance rather than a line in a panel that is about to lose the row
it would hang off; until that exists the log is where an operator reads it. Stated
in docs/system-specs/modules/instances.md section 17.4.

A refusal leaves nothing changed, which the sentence had to earn. The teardown
goes crew by crew BY NAME rather than through the manager's cascade, to get an
unambiguous per-crew failure -- and a by-name disconnect defaults to clearing the
sticky connect intent, which is what decides whether a crew keeps its tab
(was_connected || connected || warm). So with three levels the deepest descendant
came down cleanly, a shallower one then raised, and the reply said "Nothing was
removed" while that descendant had already lost its tab and a reconnect of the parent
would not bring it back. It now passes keep_intent=True, which is what the manager's
own cascade already did for the same reason: on the success path these rows are
deleted a moment later so the flag is moot, and on the abort path it is what makes the
sentence true.

What is refusable and what is not. An exception out of the teardown is
unambiguous -- the forward is still up -- so the first pass refuses on it. A STATUS
reading is not, and an earlier revision of this PR refused on one: status cannot
tell a stop that failed from a reconnect that landed after a stop succeeded, which
is the ordinary race the second pass exists for, so refusing on it turned that race
into a row nothing could delete until the gateway restarted. The repository's own
test_remove_sweeps_a_reconnect_that_raced_the_removal predates this branch and
states that case must remove successfully and sweep; its stub returns cleanly, so
refusing on a raising stop leaves it untouched.

The 404 is decided before anything is deleted. Reading it off the parent's own
removal meant the descendants were already gone by the time a missing parent
answered "not found", so a DELETE of an id another path had already removed -- an
orphan's former parent -- reported a failure having just deleted the crews under it
and dropped the list of which ones they were. Rows still go leaves first.

Editing a parent tears down the crews riding it. A child's forward was built from
the parent's coordinates, so moving the parent to another machine while the child's
ssh -L ... <old host> stays up leaves that child reporting CONNECTED and serving
the machine the user just left while its row names the new one. The cascade runs
inside reconfigure's own critical section, and unlike a shutdown it is not tolerant:
a child that will not stop aborts the edit, for the same reason the parent's own
failed stop does. The children keep their connect intent -- the user edited the
parent, and forgetting them would mean reconnecting it does not bring them back.

Self-heal mints through the parent BEFORE tier 1 rebuilds. A parent that came back
on a different loopback port is the commonest reason a chained crew needs healing, and
the mint reply is the only place the parent names its current port. ssh -L binds the
LOCAL side whatever answers remotely, and start() waits only for the local forward to
accept -- so a tier-1 rebuild against the row's stale hop starts successfully, marks the
crew recovered and resets the attempt counter without ever reaching the tier-2 mint:
healthy on the board, unreachable in fact, every cycle. The mint rides the parent's own
hop, which is already up, so it runs first -- the same inversion connect makes, for
the same reason. A chained crew's tier 2 is therefore the rebuild alone, the shape
fargate's already has. A parent that cannot be minted over, or that names no port,
stands the attempt down rather than dialling the row's.

The parent-hop re-mint is guarded by the tunnel generation, not by a held lock.
Its three callers differ: connect holds the manager lock across the mint, while the
proactive refresh and the self-heal's own mint deliberately do not -- that is what
keeps the 401 branch out of the re-entrant deadlock above. So membership is not
enough: a parent disconnected and reconnected inside a mint budget measured in tens
of seconds satisfies it with a NEW tunnel, and the fresh credential would be
overwritten by one minted against the generation before it. The generation is
captured before the mint and compared before the store, as its two sibling stores
already do.

Adding a chained crew and removing its parent are one critical section. The add
validates the parent and inserts the child in two separate awaits. A removal
landing between them takes its subtree snapshot before the child exists and then
deletes the parent under it, leaving a crew whose only route is a hop that is gone.
Both handlers take the same lock across that pair, so either the child lands before
the snapshot and goes with the cascade, or it lands after the parent is gone and is
refused.

Two limits are deliberate, both recorded in
docs/system-specs/modules/instances.md section 17.4. A crew reporting no
gateway_id is chained without a cycle check, because the loop it cannot rule out
is a nested pane rather than an escape from a boundary and the depth cap already
bounds the arrangement; note that "no id" covers both a build predating the field
and a caller the direct-local gate fenced off, and that A's own probe rides the
loopback end of a forward it just opened, which that gate treats as direct-local by
documented design. And a top-level crew is not cycle-checked at all, because one
pointing back at this gateway is what the product already allows.

The tab is the promise, so the crew is connected and not merely added. A row
written by POST /api/instances carries no connect intent, and the switcher admits
a crew only once it is connected, warm, or carries that intent -- so adopting
without connecting left the crew in the Remote Crew list and nowhere else. Adoption
connects it, which is the same intent the user expressed in the pane, arriving here.

A chained chip is two boxes, not one string. The chip is width-limited, and one
string ellipsises from the right, which eats the crew's own name and leaves a chip
reading as its parent's duplicate. The parent segment takes the squeeze and the
crew's name never shrinks. The path reads from the string catalog in all 13
locales, so a locale can change the separator or the order.

The name cap is applied where a name is written, not on every record read

INSTANCE_NAME_MAX is new here, and nothing truncated a name on the way in before it --
no length check at the handler, none in Instance.from_dict -- so a row stored before
this change can be longer than the cap. InstancesRegistry.update() re-runs
Instance.validate() against the WHOLE record, and disconnect's reset rides through
update() as a hint-only patch (was_connected=False plus the freed local_port) whose
failure is swallowed as best-effort. Capping inside validate() therefore lost that reset
silently on a pre-cap row: the next start revived a crew the user had disconnected, on a
port it no longer owned, and the only exit was a manual rename.

The cap now lives in validate_instance_name, called from add() and from update()
only when the patch carries name. That is the same write-site placement validate_ttl
already documents in this file, and for the same reason: a rule added later must not fail
a write that touches a different field.

A teardown stops reporting CONNECTED at its start, not at its end

Every decision to issue a credential for a forward re-reads the same live
TunnelStatus object: the token/port pair a pane is answered with
(handlers_instances.py), and the mint paths that require CONNECTED. _SshTunnel.stop()
awaits task cancellation, a stdout drain and _terminate(), whose wait_for allows 5
seconds per signal, so the teardown spans seconds.

With the state assigned after those awaits, the live object reports CONNECTED with an
unchanged local_port for that whole span. The handler holds no manager lock across
token_validates or the re-mint, so a concurrent disconnect runs inside the window and
the caller is answered with this crew's token paired with a port that is already being
released. Nothing revokes a token once the response is sent.

stop() now assigns the state before its first await. The guard that re-reads the live
object therefore sees the teardown and refuses the pair, which is what its own comment
always intended.

Scope is narrower than it looks. _reserved_ports counts a port by tunnel MEMBERSHIP
rather than by state, and the caller pops the tunnel only after stop() returns, so the
port stays withheld across the window and nothing becomes allocatable earlier.
_apply_hop_holds under-counts in_use during the window, which is the direction its own
contract calls cheap: the bind loses to a still-live forward and the port is recorded as
owed and retried. A teardown that RAISES restores the previous state, since the forward is
then still up; a CANCELLED one stays non-connected, because the child may already be
signalled and refusing to mint for a half-gone forward is the safe direction.

Rendered evidence

Five frames attached to this description. Captured against the real built SPA with
a stubbed instances API, so the switcher photographed here is the one the header
renders. The harness that produced them is deliberately not part of this diff, and
neither are the frames as files: the lanes admit an attachment, and a capture
harness is not product code. Generic fixture names throughout -- nothing here names
a real machine.

The switcher as a tree, depth 0/1/2. build-host at depth 0, fresh-desktop
under it, gpu-box under that, each subtitle naming the crew it goes through as
well as the machine.

Switcher showing the chain as an indented tree

A parent hop down greys out the crews behind it. build-host reads tunnel error; fresh-desktop and gpu-box dim with it because they ride its tunnel,
while the unrelated lab-host stays bright.

Switcher with the parent hop down, each chained row naming the ancestor that is actually down

The chip path under a parent name too long for it. The parent is
build-host-with-a-long-name; its own chip truncates, and the chained chip reads
bui... > fresh-desktop -- the parent segment gives way and the crew's name
survives whole, which is what stops the chip reading as a duplicate.

Header chips, the chained one showing a truncated parent and the full child name

Remote Crew inside a pane. The rail entry used to be filtered out of an
embedded pane; connecting a further crew from there is the declared feature.

The Remote Crew settings panel rendered inside an embedded pane

The relayed refusal, in the pane that acted. The depth-cap sentence is the
host gateway's own, rendered above the crew list the user was just looking at.

The pane's Remote Crew panel showing the relayed refusal as a whole sentence naming the crew

A chained row's machine is labelled as reported, and its gateway port is dropped.
The host's own Settings -> Remote Crew, with two direct crews and two chained ones in
the same list. build-host and lab-host read build-host-alias - remote port 7777,
because both are values this dashboard dials. fresh-desktop and gpu-box read
fresh-desktop-alias (reported) and gpu-box-alias (reported) with no port at all,
because neither their host nor their own gateway port is anything this dashboard
reaches -- the forward goes to the parent. The label states who the value came from
instead of asserting it, which is what keeps a forged payload from making a row read
like a machine it is not.

The host's Remote Crew settings showing chained rows labelled as reported with no gateway port, beside direct rows that show both

Tests

  • test/test_remote_crew_chaining.py -- the three registry fields and the chain
    walk; that a file written before chaining loads with all three defaulted and stays
    writable; all six refusals, the uniqueness guard shown to refuse the second add
    of one remote crew while admitting a different crew behind the same parent and
    the same remote id behind a different parent, the width cap asserted exactly at
    its boundary and
    shown not to bleed onto another parent or onto a top-level add; the cycle guard
    against A's own id and an ancestor's,
    and the fail-open when a crew reports none; the disconnect and remove cascades,
    including that the post-removal sweep names every captured crew while the registry
    is empty for the whole sweep -- which is what proves the ids cannot come from it --
    and that rows go leaves first; that a chained add and its parent's removal cannot
    interleave into an orphan whichever order they land in; and the mint through the
    parent.
  • The mint is exercised end to end with the two ids genuinely different -- c
    on the hub, c-2 on the parent -- asserting the request is addressed to the
    parent's id and never to ours, and that the stored TTL is the one the parent
    reported. Sibling tests cover the six shapes of an unstatable TTL, each refusing
    the mint rather than guessing, and the clamp in both directions -- a shorter
    reported TTL is taken, a longer one is not.
  • The published total is pinned on both sides of the wire: the manager reports the
    token's own total and not the row's, it is absent rather than zero when no token is
    held, and the status dict carries it beside the remaining or carries neither.
  • The hop identity gets the case a port-number comparison cannot see, which is why that
    comparison read as sufficient: the crew holding the hop rebuilds its forward and its
    allocator hands out the SAME number, every number agrees, and the credential is
    discarded on the identity. The other direction too -- a self-heal that rebuilds onto
    the reported hop rebinds the identity and its store lands, so the check is not simply
    a refusal. A reply that cannot identify the hop is refused for each of eight malformed
    shapes. And a round-trip drives the parent for real and feeds its own payload to the
    hub's parser, because each side had its own tests and neither could see a field the
    OTHER side stopped sending.
  • The credential rule gets its exits pinned one by one: a refresh whose parent has moved
    this crew to another port is discarded and the valid credential survives; the store
    itself refuses a moved hop and a moved generation at the unit and keeps what it held;
    a heal whose rebuilds never land stores nothing at all, which is the only case that
    can see the ORDER rather than the comparison; the real mint is driven at its own level
    to prove it REPORTS the hop and the lifetime and publishes neither; and a structural
    test asserts every store names the generation it minted against, that the decision is
    reached from exactly two functions, and that the two published values are written from
    the store alone.
  • The connect handler's token/port pair gets two cases: a forward whose port moves
    during the token probe is refused with no token in the body, and a forward that
    drops to ERROR with its port unchanged is refused too -- so neither half of the
    re-validation can be removed without a test going red.
  • The mint's hop window gets four cases: the happy answer names the port it verified;
    a crew disconnected inside the mint, with a second crew taking the freed port, is
    refused rather than answered, and the test asserts the other crew really does hold
    that port; a reconnect inside the mint is refused although membership and the
    connected state both read true again; and a probe marking the forward ERROR is
    refused with the tunnel never popped and the generation never moved.
  • The four repair paths get their own class: a self-heal whose tier-1 rebuild SUCCEEDS
    still targets the port the parent reported and not the row's, its tier 2 rebuilds
    without minting a second time, and a parent that names no port stands the attempt
    down;
    editing a parent tears down the crew riding it while leaving that crew's intent; a
    child that will not stop aborts the edit and the old host survives on the record; a
    re-mint whose parent's generation moved mid-flight is discarded and the live
    credential survives, while an undisturbed one still stores.
  • Removing an id with no row answers 404 with its orphaned descendants still present.
  • A refusal leaves the surviving rows' connect intent alone, and that needs THREE
    levels to see: with a parent and one child the loop fails on its first candidate, so
    nothing was torn down yet and no intent could be lost. With a grandchild the loop
    tears it down successfully, then fails on the child, and the test asserts the order
    and that both teardowns were asked to keep intent. The assertion stops at the flag on
    purpose -- a stubbed manager performs no registry write, so reading was_connected
    off those rows would pass whatever the handler passed; that the manager HONOURS the
    flag is _teardown_locked's own unit and is asserted where it lives.
  • A child that will not stop answers 409 remove_teardown_failed with every row
    still on the record, and the teardown stops AT the failure rather than carrying on
    to the parent, whose forward is the child's route out. A forward that reappears
    after the rows go is logged rather than raised, proved by two witnesses that are
    not the wire: the pass called disconnect a second time, and it said so where an
    operator reads it. The sweep test asserts both passes -- the first with the rows
    present, which is what lets it refuse, and the second with the registry empty,
    which is why its ids can only come from the captured list.
  • The refresh SCHEDULE gets its own class, driving the real loop with its sleeps recorded
    rather than waited out: a parent lowering its answer from 20h to 1h between cycles must
    shorten the next interval, and the same stated as the property rather than the numbers
    (each wake lands inside the life of the token the previous wake minted, across 20h ->
    1h -> 30m) so a changed refresh fraction cannot make it vacuously true. A third case
    holds the line the fix must not cross: a transport error stores no new lifetime, so the
    retry keeps the interval it had.
  • website/src/lib/tokenTtl.test.ts -- the shared denominator: the published total is
    preferred, a missing one falls back to the row, and a zero or non-finite one is not
    trusted. Two cases state the outcome rather than the input -- the reported 1h-token
    case is due for refresh when measured against a 20h row and not due when measured
    against the published total, and a chained token genuinely near its end still
    refreshes.
  • website/src/test/remoteCrewChaining.test.ts also pins the adoption plan: an
    unchanged hop port still connects, a moved one repoints and connects, and a row that
    never carried a port is repointed. And chainRefusalCode, which is how the pane
    tells the benign refusal from the rest: it reads the gateway's machine-readable
    code off the error body and answers empty for a non-JSON body, a missing or
    non-string code, and a code that appears only in the human message -- empty means
    "relay it", so every unreadable shape errs toward telling the user.
  • test/test_api_health.py -- gateway_id is present and 32 chars for a
    direct-local request and absent for an anonymous, rebound or forwarded one, so it
    rides the same gate as version; a repeat read does not touch the disk; and the
    read does not run on the event loop's own thread, asserted by thread identity with
    a guard that fails if the identity was never read at all.
  • test/test_gateway_identity.py -- the id read is judged as a descriptor. The case
    that matters is the RACE, and it is the only one a path-then-open read fails: a
    statically planted link or FIFO is refused by an lstat too, so the test plants a
    real file and swaps it for a symlink at the instant the path is stat-ed, then asserts
    the id that comes back is the real one and never the attacker's. Beside it, a FIFO
    must not hold the worker -- driven in a thread with a join deadline so a regression
    fails here instead of hanging the suite -- and ten repeated reads of a refused path
    must each stay inside that deadline, which is what makes the uncached re-read safe
    rather than a window per poll. The S_ISREG guard is pinned STRUCTURALLY, and the
    test says why: every shape a test can plant statically is refused by the open or the
    read regardless, and the one shape the guard uniquely refuses -- a FIFO whose writer
    wins the race and so chooses this gateway's identity -- cannot be forced, because a
    FIFO's writer only unblocks once a reader has opened. So the AST pin requires one
    path resolution, a check on the descriptor, and that check BEFORE any read.
  • The depth refusal states the distance in hops. The count includes the link to the
    refused crew itself, so reporting it as machines in between overstated the chain by
    one; the test asserts the sentence says hops and does not say machines. Changing the
    unit in one place is not the whole job: the feature map's own description of that
    refusal said MAX_VIA_HOPS machines, which is the same error one document over, so
    every place expressing the cap was swept and that was the only other one.
  • test/test_error_code_contract.py -- the ratchet passes, so both new error
    responses are readable to it: a literal body carrying code, including the one
    whose status is computed.
  • test/test_instances.py -- the pre-existing racing-removal test, which asserts a
    successful removal sweeps a reconnect that slipped in, passes unchanged. It is the
    repository's own statement of what this race should do and it is why the refusal
    went.
  • website/src/test/remoteCrewChaining.test.ts -- the switcher tree and depth; the
    announce is a no-op at top level, refuses an out-of-range port and carries no
    credential; readChainedCrewNotice drops a notice with no id, one whose id is
    outside the registry grammar, and one whose hop port is out of range, while
    tolerating an unusable remote port because that field is only a record; a refusal
    arriving with nothing subscribed is still there to read, is capped, notifies a
    subscriber, clears on dismiss, and ignores a message of another type; and a row
    that was only added gets no tab while the same row with the connect intent does.
  • Every new guard was mutation-checked: each was reverted in place and the test
    covering it confirmed to fail, so none passes vacuously. The harness refuses to
    score a mutation it could not actually apply, and requires every run to print its
    own passed/failed count -- a run that collects nothing exits non-zero too. One
    mutation stays GREEN and is reported rather than hidden: dropping the mint
    window's PORT comparison changes no test outcome, because nothing assigns
    status.local_port, so a different port means a different tunnel object and every
    site that installs one bumps the generation the same check already compares. That
    reading is kept as the redundant one, so an install site added later without the
    bump is refused there rather than trusted. A first pass on the credential rule also
    found a real gap rather than a weak pin -- no case observed the ORDER, so moving the
    self-heal's store back before its rebuild went undetected -- and the case that can
    see it was added.
  • A crew name is bounded where it is RETAINED. validate() refuses one past
    INSTANCE_NAME_MAX = 200, the same number the relay already slices an announced name
    to, and a test reads that constant out of the relay so the two cannot drift. A bound
    bounds every field it keeps: the name is persisted per row and echoed in every list
    reply, so leaving it unbounded made the row caps no bound at all.
  • The locale work is the repo's own output, not hand-edited: the pseudolocale was
    regenerated with i18n:pseudo, and the i18n gate (extraction, key references, plurals
    and the untranslated-string report) passes against this branch's literal base rather
    than a moved origin/main.
  • The OS-level hold on a lent port is pinned from a SEPARATE PROCESS, because the
    assertion that matters is not one this gateway can make about itself. A test that the
    allocator skips the port would re-pin the defect being fixed -- the exclusion was
    already there and already passed that assertion. So test_hop_port_guard.py shells
    out and binds with SO_REUSEADDR, the way OpenSSH's forward and this module's own
    probe do: refused with EADDRINUSE while the reservation stands, and bindable again
    once it lapses or is released. test_remote_crew_chaining.py does the same after a
    real teardown, end to end.
  • One case exists only to record WHY the hold listens: a socket bound without listening
    is shown to let a second SO_REUSEADDR bind through. If that ever stops being true
    the case fails, which is the signal to re-derive the shape rather than discover it
    from a report.
  • What a stale pane sees is pinned twice, because one case could not tell the two
    apart. Closing a socket that already has unread data sends a reset regardless, so the
    case that sends a request passes with or without SO_LINGER -- a mutation removing
    the option proved exactly that by going unnoticed. The case that discriminates is a
    client which connects and sends NOTHING: an orderly FIN there, a reset with the
    option. Both are asserted, and "nothing read the credential" is additionally pinned
    structurally, since no behavioural test can separate a guard that reads and discards
    from one that never reads.
  • A chained row's TOOLTIP is pinned to the same reading as its visible word. The two
    disagreeing is invisible to a test of either one, so the pin is over the source: no
    row title may be built from the raw state. Two sites had it -- the host-side menu row
    and the relayed switcher's twin -- and reverting either one alone is caught.
  • Run before pushing: 650 Python tests across the six files that double the interfaces
    this change touches -- chosen by grepping the suite for those symbols rather than by
    listing the files edited -- and 236 vitest across the eight frontend files that
    exercise this branch's own components, with tsc --noEmit and eslint src clean.
    Every gate the CI jobs run, each checked by its own exit status rather than by reading
    a summary line: the baselined black gate, isort, flake8, mypy src/kiro_crew/,
    the subprocess-encoding gate, the ACP-frame host-data gate, the agent-SDK boundary
    gate, the blocking-IO-in-async gate, the lockdown-before-publish gate, the
    comment-history gate against this branch's literal base, and both documentation lints.
    The gate list is taken from the CI workflow rather than chosen by which files changed,
    and the test scope from which files double the changed interfaces: selecting either by
    "files I edited" is what let a stale double stay red on CI while passing locally, and
    mypy -- which is not one of the formatters -- is what caught the rebase defect below.
  • The rebase onto the current main moved 219 commits under this branch, and one of them
    had made the peer credential-header builder async with a third argument. The new call
    site this change adds was written against the old shape, so the rebased tree assigned a
    coroutine into a header dict and would have failed the first real chained mint. Every
    behavioural case here passed anyway, because this file replaces that builder with a
    stand-in at nine sites -- so the double hid it and mypy is what found it. It is now
    also pinned in the suite, over the SOURCE rather than behaviourally: the builder must
    be a coroutine function, and no call site may read it without awaiting. The pin was
    proved by removing one await (1 failed) and restoring it (2 passed).
  • The four upstream fixes this branch's board depends on were content-checked against the
    rebased base by running the tests that name them, not by reading ancestry: the
    spawn-approval delivery case and the sandbox leaf ratchet (8 passed), and the Chat rail
    registry and settings-registry extraction cases (19 passed). The spawn-approval test
    had been RENAMED upstream, which is why the check runs the class rather than a stored
    node id -- a stale id collects nothing, and nothing collected reads exactly like
    nothing failing.
  • Eight of the ownership cases install a REAL port guard and let it bind the first port
    the fake allocator hands out. One fixed base port made that a resource shared between
    parallel test processes, and the loser reported "no hold was taken" -- a real-looking
    failure with an unrelated cause, seen once locally. Each worker now gets its own band,
    so the flake cannot reach a CI lane and be read as a defect.
  • The hold is mutation-proved 26/26 against a passing control of 565, each verdict
    printing its own counts. The socket's shape: binding without listening, reading the
    request before closing, dropping SO_LINGER, never accepting. The lease's lifetime:
    letting a lapsed hold persist, never retrying a failed hold, re-logging every retry at
    ERROR. The platform: flattening the branch that picks the Windows option. The seams:
    removing the teardown arm or the startup re-arm, reclaiming an orphan off a still-lent
    port, moving the reservation read into the window between a port's release and its
    hold, counting any remembered port as in use, skipping the arm on the exit path, never
    releasing before the rebuild rebinds, re-settling outside a finally. The cache: the
    exit seam reading the record instead of it, never writing it at the mint, and dropping
    the synchronous hold from the seam altogether. And the invariant predicate itself, two
    ways: accepting a non-CONNECTED entry as the owner, and reporting nothing at all.
  • The lent-lifetime cap adds five more, each caught: removing the cap so the row is lent
    as-is; making it a maximum so a row shorter than it is widened; having the lease
    deadline read the row while the mint used the cap; having the answer report the row
    while the lease and the mint used the cap; and reading the credential header without
    awaiting it. The middle two are the ones that matter most, because each leaves the cap
    visibly in place while breaking the property it exists for -- one frees the port under a
    live credential, the other has the hub re-mint after its token is dead. Each anchor was
    checked to occur exactly once in the source before the run, against a control string
    known to occur twice: a mutation whose anchor is ambiguous reports a verdict about a
    site it never touched.
  • FIVE of those were MISSES first, reported rather than scored, and each miss was a weak
    PIN rather than a weak fix. Dropping SO_LINGER: a close with unread data already
    resets, so a case that sent a request could not discriminate, and the case that does is
    a client which connects and sends nothing. Flattening the platform branch: both option
    names stayed in the source, one unreachable, satisfying a presence-only assertion, so
    the pin now checks the branch's CONDITION. Skipping the exit-path arm: the pin covered
    the finally but not the ordering against the backoff. Dropping the synchronous hold
    from the exit seam: every case for the cache called the helper DIRECTLY, so deleting
    the seam's call to it left them all green -- the pin now drives the real entry point
    the monitor uses, on both the self-healing and the reconfiguring path. And one was a
    HARNESS defect rather than a pin defect -- release(local_port) also appears in
    connect, so a bare anchor mutated that site and reported MISSED without touching the
    recovery at all. The harness also refuses to score when its control fails or an anchor
    has moved, which it did twice across this refactor rather than reporting false passes.
  • The four changes above are mutation-proved, each verdict printing its own counts
    against a passing control (98 Python, 34 vitest): dropping the unreachable label's
    guard, dropping the hop from it, stripping the refusals' subject, silencing the
    residual log and skipping the second pass are each caught. One mutation is expected
    NOT to be caught and is run for that reason -- putting stranded back on the
    success body changes nothing the suite can see, which is the evidence that nothing
    depended on it.

The name cap

test_a_pre_cap_name_still_lets_disconnect_persist_its_reset stores a row 50 characters
past the cap, sends exactly the hint set disconnect sends, and asserts the reset is
persisted, the freed port is released, and the untouched name is not rewritten -- then
asserts a name CHANGE past the cap is still refused, and that renaming to a short name
works, since that is the legacy row's exit.

Mutation-proved rather than asserted: restoring the cap inside Instance.validate()
(asserted to be exactly one substitution site) turns that run red at 1 failed 3 passed,
and the failure is the new test. The file was restored from a copy kept outside the
repository and the checksum matched.

Suites run for this change: test_remote_crew_chaining.py 145 passed,
test_instances.py 410 passed, and test_remote_crew_adopt.py +
test_remote_crew_peer_slots.py + test_instance_credential_pairing.py +
test_remote_crew_capabilities.py 201 passed.

The teardown state window

test_stop_leaves_connected_before_its_first_await parks a real _SshTunnel inside
_terminate() and asserts the live status has already left CONNECTED while the port is
still unreleased. test_a_failed_teardown_restores_the_previous_state covers both
failure shapes: a raising teardown restores CONNECTED, a cancelled one stays STOPPED.

Mutation-proved: moving the assignment back after the awaits (asserted to be exactly one
substitution site per edit) gives 2 failed 1 passed, and both new tests are the
failures, so each is load-bearing. Restored from a copy kept outside the repository with a
matching checksum.

Suites run for this change: test_instances.py + test_fargate_closed_forward_remedy.py +
test_remote_crew_chaining.py 589 passed, and test_remote_crew_execution.py +
test_instance_credential_pairing.py + test_remote_crew_peer_slots.py +
test_remote_crew_adopt.py 359 passed.

5. Any other suggestions on the work

  • A gateway EXIT leaves one lent-TTL window and it is not an ordering problem: shutdown
    releases every hold because a socket cannot outlive its process, and the credential
    stays valid because the remote crew issued it and no revoke primitive exists here, so
    the exposure runs to the lease deadline -- one capped lent TTL (30m), after which the
    credential is dead and the next start re-takes every hold. Closing it rather than
    bounding it needs either a holder that survives the process (a separate deployable
    component, with its own lifecycle for the case where the gateway never returns) or a
    revoke endpoint on the remote crew (a new cross-gateway contract that an older remote
    would not serve, so the window would silently return). Both are follow-up architecture
    items; capping the lifetime is the part that lives inside this mechanism. Tracked as
    instances: a gateway exit frees a lent hop port while its chained credential stays live #14461, which records a THIRD option this description did not have: _peer_gateway_id
    already reads the peer's gateway_id over the forward with no credential attached, so
    requiring it to match before any credential travels closes the window without a new
    component and without a cross-gateway contract.
  • token_validates must not be read as a mitigation for that window -- it is worse than
    neutral. Its contract sends GET /api/status?token=... through the forward, so against
    a process squatting on a freed port the probe is itself an extra disclosure, and a 2xx
    answer makes the hub hand the token to the browser as well. Any identity check has to
    run BEFORE it rather than rely on it.
  • One window remains open and is not an ordering problem: between the forwarder child
    dying and the monitor seam noticing, the lent port is free and nothing holds it, so no
    arrangement of the arm relative to the teardown can close it -- only a gateway-owned
    listener bound for the whole lease, with the forward proxied or swapped behind it, and
    that changes how every forward binds its port rather than how this one is protected.
    Left as a follow-up architecture item deliberately.
  • The depth cap is 2 hops because that is what the switcher can draw legibly, not
    because anything breaks at 3. Raising it is a one-constant change plus a tree that
    indents further; a test asserts the constant so the expectation moves with it.
  • chain_parent_not_ssh is a product limitation rather than a safety rule: a
    chained forward needs a port on the parent to aim at, which a non-ssh transport
    does not give it. Supporting SSM-reached parents is separate work.
  • The relayed refusal renders the gateway's own sentence rather than a translated
    frame around it, so it reads in English on a non-English dashboard exactly as the
    existing backend error notices beside it do. Giving it a translated frame means a
    new key in fourteen catalogs and is worth doing as part of a pass over those
    notices generally, not for this one alone. The chained path label DOES read from
    the catalog, because that string is ours rather than the gateway's.

Disposition of the advisory review findings on this revision

The First Principles lane returned CONCERNS on four surfaces it counted as having
no reader. Two now have one and two are kept deliberately:

  • ChainedCrewRefusal is now the module store's own value, so the type has a
    reader rather than only a definition.
  • The v: 1 field on both frame messages is kept unread on purpose. It is a wire
    version on a boundary between two INDEPENDENTLY deployed gateways, where the
    older side is the one that has to recognise a payload it does not understand; a
    version added after the first mismatch cannot help that side. It costs one
    integer.
  • removed_chained and stranded are REMOVED from the DELETE success body. The
    lane is right that a field on a public response with no reader is a defect rather
    than a courtesy, and the earlier argument for keeping removed_chained -- that a
    client cannot reconcile a cascade it was not told about -- describes a client that
    does not exist: neither removeInstance caller reads the body at all. What they
    were evidence OF is kept, through witnesses that are not the wire: the cascade is
    proved by the rows that are gone, and the residual by the gateway's warning log
    plus a counter showing the second pass ran. A mutation that puts the field back is
    invisible to the suite, which is the measurement that says nothing depended on it.
  • A chained row's tooltip no longer contradicts its own visible word. A chained crew can
    be connected in its own right while an ancestor hop is down, and the row then reads
    "unreachable" on screen -- while a title built from the raw state said "connected".
    Since title is also what a screen reader falls back to, the two audiences were told
    opposite things about one row, which is the same class of false label as the code
    comments corrected in this revision rather than a cosmetic slip. Both sites are fixed;
    the flagged one and its relayed twin.
  • The fargate/SSM announce mislabel is NOT fixed here and is tracked as its own issue.
    It is a pre-existing seam rather than part of this feature: a non-SSH crew is
    announced with its SSM target in the ssh_host slot, which the grammar refuses, so a
    connect that succeeded reports instance_invalid. Today's refusal is the right
    OUTCOME reached for the wrong reason -- a chained crew rides a loopback port on the
    parent and a fargate crew has none -- and the fix is to gate the announce on the
    transport, which is a change to the announce contract rather than to anything this PR
    introduces.
  • gateway_id(create=False) has no production caller and the parameter could go.
    It is left because the symmetry with beacon.install_id(create=False) is what
    makes the two identity readers obviously the same shape, and removing it is a
    separate tidy-up rather than part of this feature.

The reversal's decision record is issue #13744, this PR's own Fixes target.
The lane asked for the "single-level by design" reversal to rest on something other
than this description's say-so, and it does: Raymond filed #13744 as an
enhancement, and its first paragraph is the reversal in those words -- "Remote
Crew is single-level by design today: a connected instance cannot connect onward to
another instance. Allow chaining ... while refusing any connection back up the chain
and capping how deep it can go" -- with the reason above it, that B and not A is the
one which can reach C, "which fails when only B holds C's key". The spec now cites
it where the reversal is described.

One measurement belongs beside that citation, and it is in the spec too. No
sentence in the spec tree ever declared Remote Crew single-level.
Grepping
origin/main's docs for single-level and single level returns one hit, in
computer-use.md and about write access to a parent directory. The constraint lived
entirely in the frontend -- SettingsPage.tsx dropped the Remote Crew tab from
baseTabs when isEmbeddedPane() -- with no depth, cycle or width check on the
gateway side at all, and inside a pane the tab vanished with no explanation. So what
reads as a reversed design decision was a frontend side effect that presented as a
missing feature, which is what #13744 says. Chaining adds the server-side guards a
real single-level rule would have needed to begin with.

The UX lane's watch items are FIXED in this revision rather than deferred.

An unreachable chained row no longer keeps a green "connected" label. When an
ancestor's hop drops, the row is dimmed -- and it used to go on reporting its own
state beside the dimming, so two cues said opposite things for the whole outage over
a claim the caller had already contradicted. It now reads unreachable -- <hop> is down, naming the hop because the row the reader has to go fix is a different one.
The choice came out of the JSX into an exported rowStateLabel, which is what gives
it a test at all.

Every chain refusal is a whole sentence naming the crew and the action. Each one
opened lowercase and mid-clause -- a "that would put ..." fragment with no subject --
and the panel renders the gateway's text raw and can show it after
the connect already looked like it succeeded -- so the reader saw a fragment naming
neither what was refused nor what had been doing it. The announced crew's name
arrives as a new parameter, because only the caller holds it. Two earlier watch
items stay fixed: the chip keeps the crew's own name under a long parent name, and
the refusal survives the panel being closed. Both are photographed above.

  • via_remote_id is not PATCH-editable, for the same reason via_instance_id is
    not: re-pointing it addresses a different crew, whose depth was never checked.

Fixes #13744

@chenmingwei23
chenmingwei23 requested review from a team and Zedmor September 25, 2026 18:34
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of c20eb645f7f320f8883fa426317fd4a7948c80fb — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm follow-up #14461 stays prioritized: a parent gateway exit leaves chained bearer tokens live up to 30 minutes, bounded here, closed only there.

[DESIGN-REVIEWED] c20eb64

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — 🟡 CONCERNS

UX-level review of c20eb645f7f320f8883fa426317fd4a7948c80fb — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

"(reported)" label and the relayed refusal both baffled the cold reader, and unreachable rows still show green state dots.

Watch

  • "(reported)" is unreadable alone — reader: "no idea what it means — reported by whom, about what?" (shot-06). The only explanation lives in a hover-only title (reported_host_hint), so every chained row's host reads as a mystery, persistently. Promote the meaning into the label: reported_host → e.g. {{host}} (reported by {{parent}}) or reuse the chain_via relationship the switcher already teaches.
  • Unreachable rows keep a green dot: SwitcherRow mutes the text (rowStateLabel, text-muted) but the dot stays stateDotCls(entry.state) — reader: "the dot says fine, the words say down. I can't tell which to believe" (shot-02). Every ancestor-hop outage, on every chained row. Mute/grey the dot when reachable === false.
  • The relayed depth refusal reads wrong inside the pane: "would put it N hops from this dashboard" — but rendered in the pane, "this dashboard" is the wrong surface, and the crew's row below shows green "Connected"; reader: "Contradiction I can't resolve" (shot-05). Hit on every depth/cycle refusal. Smallest fix: have the pane frame the relay with one line stating the crew connected here and only the outer dashboard's tab was declined.

[UX-REVIEWED] c20eb64

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of c20eb645f7f320f8883fa426317fd4a7948c80fb — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

hop_ownership_violations() and gateway_id(create=False) ship as production surface with zero non-test callers — test scaffolding living in src/kiro_crew/.

Not justified as shipped

  • Item 10 — zero consumers: grep hop_ownership_violations finds 1 definition and 7 references, all in test_remote_crew_chaining.py, none in production; grep gateway_id( shows both production call sites (handlers/core.py, ssh_tunnel_manager.py) use the default, so the create=False mode exists only for test_gateway_identity.py.
    Subtraction: move the invariant predicate into test_remote_crew_chaining.py and drop gateway_id's create parameter.
    Clears when: a production caller lands for either, or both are deleted/moved out of src/.

What this change ships

Inventory (10 items) — 9 justified

Intent: give a crew that only an already-connected crew can reach its own top-level tab — an ADDITION (decision cited to issue #13744; no chaining RFC exists, but no default flips and nothing user-facing is removed, so this is not the lens-9 exit).

  1. A crew a pane connects becomes a top-level tab of the hub, its token minted by the parent via new POST /api/instances/{id}/embed-token — justified
  2. The Remote Crew tab now appears inside embedded panes; the deleted "single-level by design" hide is reversed, with the issue cited — justified
  3. The switcher draws chains as a tree; chained chips read "parent › child" and grey out with the parent — justified
  4. Removing, disconnecting or editing a crew cascades over crews chained behind it; DELETE gains 409 remove_teardown_failed and decides 404 first — justified
  5. A chained add is refused six ways server-side: unknown parent, duplicate, depth 2, width 8, non-ssh parent, cycle — justified
  6. A persistent random gateway id is minted per data home and shown on /api/health to direct-local callers only — justified
  7. A lent hop's port is leased in the registry and held at OS level; a lent token's TTL is capped at 30m — justified
  8. connect now refuses a token whose forward moved or dropped during the probe, for every crew — justified
  9. Status gains token_ttl_total (2 production consumers counted) and pane refresh re-derives its schedule from it — justified
  10. hop_ownership_violations() and gateway_id(create=False), callable production surface — zero consumers

More than 10 observable differences exist; omitted quiet riders: a 200-char instance-name cap (mandated by AUTOSDE a-bound-bounds-every-field-it-retains, so derived) and a black-baseline row removal.

[FIRST-PRINCIPLES-REVIEWED] c20eb64

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of c20eb645f7f320f8883fa426317fd4a7948c80fb and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c20eb64

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c20eb645f7f320f8883fa426317fd4a7948c80fb: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed c20eb645f7f320f8883fa426317fd4a7948c80fb — this comment is updated in place on each push.

Review details

No findings that block.

FINDING — website/src/components/InstanceTabBar.tsx:134 — present.has(i.via_instance_id) ? i.via_instance_id : '' re-roots a chained crew whose parent left rows (the ordinary case: the user disconnects parent B, clearing B's was_connected, while child C was torn down with keep_intent=True and keeps its own), so C is walked from the synthetic root with parentName: '' — detail falls back to bare target, pathParent/pathName are empty and depth is 0, so the row renders as a plain top-level crew and never names the hop, contradicting chainRows' own doc ("the row still names the crew it goes through in its subtitle") and leaving the user no cue that B is the row to reconnect → Fix: resolve the absent parent's name from the row's via_instance_id (passing the full instance list into chainRows, whose two call sites this diff already changes), or emit the re-rooted child as reachable: false so the row at least states it cannot be reached.

FINDING — docs/system-specs/modules/instances.md:2357 — "Instance gains two fields, which travel as a pair" with a two-row table omits via_remote_id, which registry.py:464 makes MANDATORY for a chained record ("a chained instance needs via_remote_id"), so a client built from §17.1 — the owning spec this change is required to keep true — sends via_instance_id + via_remote_port only and is refused with instance_invalid → Fix: add a via_remote_id row (this crew's id in the PARENT's registry, which the parent mints against) and change "two fields, which travel as a pair" to the trio.

[OPUS-REVIEWED] c20eb64

Verdict parsed from the review's SHA-scoped output markers for commit c20eb645f7f320f8883fa426317fd4a7948c80fb.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c20eb645f7f320f8883fa426317fd4a7948c80fb: <one-sentence reason>

@chenmingwei23 chenmingwei23 changed the title Remote Crew chaining: show a crew reached through another crew as its own tab feat(instances): show a crew chained behind another as its own tab Sep 25, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/remote-crew-chaining-13744 branch from af11c9e to e038d9b Compare September 25, 2026 19:53
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/remote-crew-chaining-13744 branch from e038d9b to 02c477c Compare September 25, 2026 20:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/remote-crew-chaining-13744 branch 2 times, most recently from 1bfc126 to 2282bd5 Compare September 25, 2026 21:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/remote-crew-chaining-13744 branch from 2282bd5 to 2da51f0 Compare September 25, 2026 21:59
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/remote-crew-chaining-13744 branch from 2da51f0 to 0b0777f Compare September 25, 2026 22:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 26, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/remote-crew-chaining-13744 branch 2 times, most recently from e312eb6 to 5117965 Compare September 26, 2026 03:38
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/remote-crew-chaining-13744 branch from 5117965 to 0981849 Compare September 26, 2026 04:18
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 26, 2026
@chenmingwei23

chenmingwei23 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Inherited frontend reds on this revision

Frontend Tests (2) and Frontend Tests (3) fail on this head, and Frontend Coverage Merge / Coverage Gate are downstream of them. Neither is attributable to this branch.

Shard Failing file Assertion
Frontend Tests (2) src/test/ChatPanel.railRegistry.test.tsx :90 — expected 0 to be greater than 0; also :75 — expected [ Array(1) ] to deeply equal [] (CHAT_ENTRIES is empty)
Frontend Tests (3) src/components/commandPalette/providers/settingsRegistry.test.ts checked-in registry matches live extraction (run npm run gen:settings if this fails)

Why they are not this branch's:

  • origin/main fails both of them on its own. Measured by checking origin/main out into a detached worktree and running exactly those two files against it: Test Files 2 failed (2), Tests 4 failed | 16 passed (20), with the same two assertions plus the generated-JSON companion. The entry count differs only because the two trees hold different numbers of settings rows (224 on main); the failure mode is identical.
  • ChatPanel.railRegistry.test.tsx does not exist on this branch at all -- not in the working tree and not at the base 6b4afd708ca582f20b6e98ef54613ea955e052ac. It arrives through the merge ref, so no revision of this branch can have changed what it tests.
  • settingsRegistry.test.ts is an anti-stale guard over a checked-in generated registry, and it passes on this branch alone (10 passed). It is red only in the merge, against main's own extraction.
  • This change touches the instances plane, the gateway id, and the dashboard's instances handlers. Its frontend files are InstanceTabBar.tsx, InstancesViewport.tsx, EmbeddedHostBridge.tsx, instancesSlice.ts, one test file and 13 locale files. None of them is read by the settings-registry extractor or by the Chat rail registry.

The fix is a regeneration on main (npm run gen:settings) plus whatever left CHAT_ENTRIES empty; neither belongs in this PR, and no baseline was extended or test edited here to make them pass.


Addendum (2026-09-28, head c20eb645f7f320f8883fa426317fd4a7948c80fb): every lane named above is now green. Main fixed both upstream (and #14578 dropped /api/models from the route sweep's fresh-home table); this branch rebased onto cd783d2235. The board on this head reads 96 lanes, 88 success, 8 skipped, zero failure and zero cancelled, with PR Readiness success -- so the table above is kept only as the record of why those reds were not this branch's, not as a live claim.

A crew this dashboard has no route to gets a top-level tab by riding a hop an
already-connected crew holds. The connected crew's own pane announces the crew it
just connected; this gateway adopts it, opens a second ssh forward to the port the
pane serves it on, and connects it, so it becomes a tab rather than a row in a
list. The parent mints the token, because only the parent holds a key for a
machine this gateway cannot address.

The row records the hop as three fields: `via_instance_id` (the parent, in THIS
registry), `via_remote_port` (the loopback port the parent serves it on) and
`via_remote_id` (the crew's id in the PARENT's registry). The third is what the
parent is asked to mint by: an id derived from the name here coincides with it
only by luck, and the TTL comes back from that reply rather than from this
registry's default, so a refresh is not scheduled after the token has expired.

Four refusals, each naming what the user has to change: an unknown parent, a
chain deeper than the cap, a parent reached over SSM whose forwarder takes no
second local forward, and a loop detected once the hop is open. A refusal the
host decides is posted back down to the pane that acted, because only the host
holds the reason, and it lives in a module store so closing Settings does not
lose it.

Removing a crew removes the crews chained behind it, leaves first, then sweeps
every crew it captured before the rows went, deepest first. The captured list is
the point: `disconnect` finds children by reading the registry, the rows are gone
by then, and a connect racing the removal can have re-established a child's
forward. A chained add and a parent removal take one lock across validation and
insertion, so a child cannot land after the cascade's snapshot and outlive its
parent.

A chained crew is minted for BEFORE its forward opens, which is the opposite
order from every other transport here and is what keeps the hop port honest. The
forward is `ssh -L <local>:127.0.0.1:<hop>` against the parent's machine, and the
row's copy of `hop` arrives over a pane's postMessage -- so a forged notice would
otherwise choose which loopback-only service on that machine this gateway
forwards, and the pane would render whatever answers as a crew's dashboard inside
the owner's page. The mint reply names the port the parent's own forward listens
on, over the credential this gateway already holds for it, and that call rides the
PARENT's hop rather than this crew's, so it can be made first. A parent that names
no port is refused rather than dialled at the row's value.

The gateway id is one value per data home. Repairing a corrupt file takes a
cross-process lock, re-reads inside it so a concurrent repair is adopted rather
than overwritten, replaces atomically, and caches what is on disk rather than what
this process wrote -- two ids for one home would leave the cycle guard comparing
two answers for one gateway.

The switcher draws the chain as an indented tree and greys a crew out with the
parent whose tunnel it rides. A chained chip carries the path as two boxes, so the
parent segment absorbs a squeeze the crew's own name does not. The path reads from
the string catalog in all 13 locales.

The liveness route reports a persisted random gateway id to a direct-local caller,
which is how the cycle guard tells this gateway from another one. The read hops
through a thread: the first one on a fresh data home creates the file, and this is
the most-polled route there is.

The rendered evidence is attached to the pull request rather than committed, and
the harness that captured it is not part of this diff.

Three fixes made after the feature commit are squashed into it here. The crew name
cap moved to the WRITE sites -- `add`, and `update` only when the patch carries
`name` -- rather than the record invariant, because `update` re-validates the whole
record and `disconnect`'s best-effort hint write would otherwise lose the
`was_connected` reset silently on a row stored before the cap, reviving at the next
start a crew the user had disconnected. A teardown now leaves `CONNECTED` at its
START rather than its end, so a port whose forward is already gone is not counted
as in use by the hold arming.

And `socket()` moved inside the `try` that implements `hold`'s documented refusal:
a descriptor limit is one of the two causes that method promises to report as
False, but raised from the creation it propagated instead -- through
`sync_hop_holds` into `_teardown_locked`, which aborts after popping the tunnel and
before persisting the hint, silently for a chained child whose teardown suppresses
it. It is now recorded as an owed port and retried. The reaper's `OSError` arm got
the same 0.5s wait its `Exception` arm has, because `select`'s own timeout is what
paces that loop and an error raised before it returns skipped the only delay,
spinning a daemon thread while every held port stopped being accepted on.

Fixes #13744
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Opus 5's two advisory findings on head c20eb645f7f320f8883fa426317fd4a7948c80fb are tracked in #14631 rather than folded into this PR: the board on that head is fully green with five clean review verdicts, and an advisory does not justify re-rolling them.

  • website/src/components/InstanceTabBar.tsx:134 -- a chained child whose parent is absent from rows is re-rooted with parentName: '', so every consumer's chain_via branch is skipped and the row names no hop, contradicting chainRows' own docstring ("the row still names the crew it goes through in its subtitle").
  • docs/system-specs/modules/instances.md:2357 -- says the new fields "travel as a pair" and tables two, while registry.py:465 refuses a chained record that omits via_remote_id, so a client built from that section alone is refused instance_invalid.

Each carries its own "clears when" line in the issue. Neither blocks this PR.

@bolichen97

Copy link
Copy Markdown
Contributor

Post-merge review notes (security focus: embed-token guard, hop TTL clamps, postMessage origin attribution, loopback-only peer URLs, port guard -- all held; 907 backend + 158 frontend tests pass locally on c20eb64). Two follow-ups worth a small PR:

  1. Low -- src/kiro_crew/instances/ssh_tunnel_manager.py:3778-3780 / :3802-3803 vs :1930-1931. _recover_after reads live_hop_lease_deadlines on a thread, then _apply_hop_holds REPLACES self._lent_hops with {**leases, **_pending_lends} without the manager lock. _answer_embed_mint can register a lease and pop its _pending_lends marker between the read and the apply, so the fresh lease drops out of the cache and _hold_lent_port_now misses that port on the next ssh exit until the async re-read re-seeds it. Fix: merge with max-wins per port (keep unexpired _lent_hops entries), or take self._lock around read+apply.
  2. Low -- unshipped surface (open First Principles CONCERNS): hop_ownership_violations() (ssh_tunnel_manager.py:1844) and gateway_id(create=False) (gateway_identity.py:117) have zero production callers. Move the predicate into the test module / drop the create parameter, or post a disposition.

Nits: registry.py:966 update() allowlist still admits via_instance_id although handlers_instances.py:527 says re-parenting is not editable; server.py:2708 mutates _lent_hops on a worker thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote Crew: allow chaining an already-connected instance onward, with a cycle guard and a depth cap

3 participants