Repository navigation
feat(instances): show a crew chained behind another as its own tab - #13936
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm follow-up #14461 stays prioritized: a parent gateway exit leaves chained bearer tokens live up to 30 minutes, bounded here, closed only there. [DESIGN-REVIEWED] c20eb64 |
UX Review (Fable 5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS "(reported)" label and the relayed refusal both baffled the cold reader, and unreachable rows still show green state dots. Watch
[UX-REVIEWED] c20eb64 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: give a crew that only an already-connected crew can reach its own top-level tab — an ADDITION (decision cited to issue #13744; no chaining RFC exists, but no default flips and nothing user-facing is removed, so this is not the lens-9 exit).
More than 10 observable differences exist; omitted quiet riders: a 200-char instance-name cap (mandated by AUTOSDE [FIRST-PRINCIPLES-REVIEWED] c20eb64 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNo findings that block. FINDING — website/src/components/InstanceTabBar.tsx:134 — FINDING — docs/system-specs/modules/instances.md:2357 — " [OPUS-REVIEWED] c20eb64 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
af11c9e to
e038d9b
Compare
e038d9b to
02c477c
Compare
1bfc126 to
2282bd5
Compare
2282bd5 to
2da51f0
Compare
2da51f0 to
0b0777f
Compare
e312eb6 to
5117965
Compare
5117965 to
0981849
Compare
Inherited frontend reds on this revision
Why they are not this branch's:
The fix is a regeneration on main ( Addendum (2026-09-28, head |
A crew this dashboard has no route to gets a top-level tab by riding a hop an already-connected crew holds. The connected crew's own pane announces the crew it just connected; this gateway adopts it, opens a second ssh forward to the port the pane serves it on, and connects it, so it becomes a tab rather than a row in a list. The parent mints the token, because only the parent holds a key for a machine this gateway cannot address. The row records the hop as three fields: `via_instance_id` (the parent, in THIS registry), `via_remote_port` (the loopback port the parent serves it on) and `via_remote_id` (the crew's id in the PARENT's registry). The third is what the parent is asked to mint by: an id derived from the name here coincides with it only by luck, and the TTL comes back from that reply rather than from this registry's default, so a refresh is not scheduled after the token has expired. Four refusals, each naming what the user has to change: an unknown parent, a chain deeper than the cap, a parent reached over SSM whose forwarder takes no second local forward, and a loop detected once the hop is open. A refusal the host decides is posted back down to the pane that acted, because only the host holds the reason, and it lives in a module store so closing Settings does not lose it. Removing a crew removes the crews chained behind it, leaves first, then sweeps every crew it captured before the rows went, deepest first. The captured list is the point: `disconnect` finds children by reading the registry, the rows are gone by then, and a connect racing the removal can have re-established a child's forward. A chained add and a parent removal take one lock across validation and insertion, so a child cannot land after the cascade's snapshot and outlive its parent. A chained crew is minted for BEFORE its forward opens, which is the opposite order from every other transport here and is what keeps the hop port honest. The forward is `ssh -L <local>:127.0.0.1:<hop>` against the parent's machine, and the row's copy of `hop` arrives over a pane's postMessage -- so a forged notice would otherwise choose which loopback-only service on that machine this gateway forwards, and the pane would render whatever answers as a crew's dashboard inside the owner's page. The mint reply names the port the parent's own forward listens on, over the credential this gateway already holds for it, and that call rides the PARENT's hop rather than this crew's, so it can be made first. A parent that names no port is refused rather than dialled at the row's value. The gateway id is one value per data home. Repairing a corrupt file takes a cross-process lock, re-reads inside it so a concurrent repair is adopted rather than overwritten, replaces atomically, and caches what is on disk rather than what this process wrote -- two ids for one home would leave the cycle guard comparing two answers for one gateway. The switcher draws the chain as an indented tree and greys a crew out with the parent whose tunnel it rides. A chained chip carries the path as two boxes, so the parent segment absorbs a squeeze the crew's own name does not. The path reads from the string catalog in all 13 locales. The liveness route reports a persisted random gateway id to a direct-local caller, which is how the cycle guard tells this gateway from another one. The read hops through a thread: the first one on a fresh data home creates the file, and this is the most-polled route there is. The rendered evidence is attached to the pull request rather than committed, and the harness that captured it is not part of this diff. Three fixes made after the feature commit are squashed into it here. The crew name cap moved to the WRITE sites -- `add`, and `update` only when the patch carries `name` -- rather than the record invariant, because `update` re-validates the whole record and `disconnect`'s best-effort hint write would otherwise lose the `was_connected` reset silently on a row stored before the cap, reviving at the next start a crew the user had disconnected. A teardown now leaves `CONNECTED` at its START rather than its end, so a port whose forward is already gone is not counted as in use by the hold arming. And `socket()` moved inside the `try` that implements `hold`'s documented refusal: a descriptor limit is one of the two causes that method promises to report as False, but raised from the creation it propagated instead -- through `sync_hop_holds` into `_teardown_locked`, which aborts after popping the tunnel and before persisting the hint, silently for a chained child whose teardown suppresses it. It is now recorded as an owed port and retried. The reaper's `OSError` arm got the same 0.5s wait its `Exception` arm has, because `select`'s own timeout is what paces that loop and an error raised before it returns skipped the only delay, spinning a daemon thread while every held port stopped being accepted on. Fixes #13744
|
Opus 5's two advisory findings on head
Each carries its own "clears when" line in the issue. Neither blocks this PR. |
|
Post-merge review notes (security focus: embed-token guard, hop TTL clamps, postMessage origin attribution, loopback-only peer URLs, port guard -- all held; 907 backend + 158 frontend tests pass locally on c20eb64). Two follow-ups worth a small PR:
Nits: |
Problem / Motivation
Goal: Give a crew that only another crew can reach its own top-level tab, by chaining a second port forward through the parent's existing hop.
You cannot add a crew your own machine has no route to, even when a crew you are
already connected to can reach it.
Three machines, named this way throughout: A is the dashboard in front of you
(the hub). B is a crew A is already connected to. C is a machine only B
can reach -- B sits on its network, A does not. Today the only way to see C is to
open B's dashboard inside its pane and work there, one frame deep, because A can
neither dial C nor learn that C exists: A never sees B's registry.
A crew is another Kiro Crew gateway this dashboard connects to and shows in a
tab. A pane is the iframe a crew's dashboard is drawn in. The pane's own page
calls the gateway showing it its host. A crew's connection is an
ssh -Llocalport forward -- a hop -- so "reaching C through B" means a second forward aimed
at the port B's own forward already listens on.
Why it matters
Crews behind a bastion are the normal case, not an edge one, and today they cost
the user the whole tab bar: C is usable only while B's pane is the active tab, and
only through a dashboard drawn inside another dashboard. C gets no tab of its own,
no place in the switcher, and no independent state.
Not a goal
parent to aim at, which SSM and Fargate transports do not give it.
chain_parent_not_sshrefuses it as a product limitation, not a safety rule.
its credential stays valid, and this PR BOUNDS that window (capped lent TTL) rather than
closing it. Closing it is tracked in instances: a gateway exit frees a lent hop port while its chained credential stays live #14461.
deep like every other pane.
What changed
B connects C; A shows it. C becomes a top-level tab of A, one iframe deep like
every other pane -- nothing is nested inside anything.
The symptom is that A cannot dial C. The root cause is that A has neither a route
nor a credential for C, and the design follows from supplying each without
inventing a third trust relationship.
The route. Three new registry fields.
via_instance_idis the parent's id hereand
via_remote_portis the port on that parent where its own forward to Clistens, so a chained forward dials B's host and B's loopback port -- C's
own coordinates describe it on its own machine and are never dialled from A, which
has no route to them.
via_remote_idis C's id in B's registry, which is adifferent fact from either: the first two say how to reach C, this one says what B
calls it. All three empty means a top-level crew, so an existing
instances.jsonloads unchanged.
The credential. A must not need C's key. So the token cannot be minted from
A, and the pane cannot be trusted to supply one.
POST /api/instances/{id}/embed-tokenasks B to mint it over the hop B already owns, carrying A's own dashboard port
so C's CSP admits A's page as the pane's frame ancestor. The token is scoped to A's
port, which is what makes it usable by A and useless as a general credential. The
generic chat proxy's refusal of a peer's
api/instancesplane is unchanged.That request is addressed by
via_remote_id, because B resolves it in B's registry.An id derived from the crew's name on A's side equals B's only by luck -- a name
collision on B, a rename there, or an explicitly assigned id makes them differ, and
B then answers 404 for a crew it is holding. The id is re-checked against the
registry's own grammar and encoded as exactly one path segment before the request is
built:
Instance.from_dictis deliberately tolerant, so a registry file written byhand or by an agent can carry any string, and an id like
victim/disconnect?x=would otherwise address a different authenticated route on B and spend A's
credential for B there.
One invariant, swept mechanically rather than site by site. Four of the defects
found in review were one shape: a value read or checked before an
await, then usedafter it without being re-validated against the live object, or refused. Patching the
site a reviewer names leaves the siblings, so every such site in the two files this
change owns was enumerated by an AST pass -- each local bound from live mutable state
(
_tunnels,_tunnel_epoch,_tokens,_chained_hop_port,status(),status_all(), the registry) with anawaitbetween the binding and a later use.Fifteen candidates, disposed as follows:
_mark_recovered(re-reads, compares object identity AND generation),diagnose(re-fetches after its probes),connect(holds the lock across the whole body and re-reads the row after the one mutation it makes)_teardown_lockedreturning whether there had been a tunnel,shutdowncounting its snapshot for a log line, the self-heal's attempt counterapi_instances_search_sessionsdispatches peer searches from astatus_all()snapshot; a crew that drops in between makes its own peer call fail into the existingreturn_exceptionsmerge, so the cost is one search rowThe one fixed site:
POST /api/instances/{id}/connectfreezes the response body fromthe status, then awaits a token probe and possibly a re-mint, both seconds long, then
attaches the token. The status object stays live, and a teardown in that window pops
the tunnel without zeroing the port on it while the allocator hands that port to the
next connect first -- so the frozen port can name another crew's forward and the pane
would load it holding this crew's token. The live object is now re-read before the
token is attached, and a moved port or a state that left CONNECTED is refused with the
instance_token_unconfirmedthe surrounding code already uses for "we cannot standbehind this credential". Both halves are load-bearing: a teardown moves the port, and
a probe marking the forward ERROR moves only the state.
The chained adoption race belongs to this class too and is answered differently, at the
authority rather than the reader: the announcing pane's list is stale by design, so the
uniqueness invariant lives in the gateway inside the insert's own lock, and the pane
treats that refusal as benign.
One rule about credentials, enforced in one place. A dashboard token is a bearer
credential for ONE crew and it reaches that crew only over the forward it was minted
against; deliver it over a different forward and it reaches whoever is behind that one.
Minting is a multi-second round trip, so the forward can move inside it:
Review found four separate exits where that was not proven, each in a different
function, so it is enforced ONCE instead of at a fifth site.
_credential_forward_movedanswers why a credential must not be used, or nothing, and_store_token-- which every credential store already passes through -- calls it andDISCARDS.
minted_at_epochis a required argument there, so a caller cannot storewithout saying which forward it minted against, and a store site added later cannot
forget the rule; an AST test pins both properties, because behaviour cannot see them.
The parent-side mint is the one exit that RETURNS a credential rather than storing it,
so it calls the same function and refuses
instance_hop_changed.Three readings, because the forward moves three ways. It is gone. Another forward took
its place -- membership reads true again after any reinstall, so the generation counter
is what tells one from the next. Or the crew holding the hop now serves this crew on a
different loopback port, which neither membership nor the generation can see because our
own tunnel was never touched. Both values compared were already recorded, so this is a
comparison and not new state:
_chained_hop_portholds what the mint reply named, and achained tunnel's
remote_portIS the hop it dials.Both sides of that pairing pass through the same place. The mint returns the parent's
whole answer -- credential, hop and lifetime as one value -- and publishes none of it;
the store writes the hop and the lifetime behind the generation fence, and only then
compares. The ordering is the substance, not tidiness: the hop is the very value the
comparison reads, so a mint that published on its way out could drop a superseded port
where the check looks, leaving it comparing stale against stale and passing. Fencing one
side of a comparison does not fence the comparison. The structural test covers the write
side too, so a publish site added later cannot bypass it either.
That rule is also why the self-heal mints before it rebuilds and stores AFTER. Its mint
is where the parent names its current port, and until a rebuild has the forward on that
port there is nothing this credential may travel over -- so it is carried past tier 1
and stored once a rebuild rides the hop, on both the tier-1 and tier-2 paths. A heal
whose rebuilds never land stores nothing and the credential already held stays: a stale
token yields a 403 the client recovers from, where one delivered over another crew's
forward is a disclosure nothing downstream re-checks.
The comparison is on an IDENTITY, not on a port number. A port number cannot serve
as one: the parent's allocator hands a just-freed port to the next connect first, so the
same number names a different forward after ordinary churn, and a comparison on numbers
clears exactly the case it exists to refuse. So the parent states its own identity for
the hop alongside the token --
hop_id, the id it knows the crew by, andhop_gen, thegeneration of its own forward to it -- and this gateway compares that against the
identity the forward HERE was built against. The port is read first as a cheap early-out
and cannot be what clears the check.
Both fields are REQUIRED, with the lifetime's strictness rather than the port's: an
unreadable port is dropped and the caller refuses to dial, while an unidentifiable hop
would let the dial succeed and weaken only the check guarding the credential.
Requiring them costs no compatibility, and that is measurable rather than argued.
The endpoint and the fields ship in the same change, so no build can answer this route
and omit them --
origin/maincarriesmint_embed_token,_mint_through_parent,via_remote_idandadoptChainedCrewin zero files. A parent either predates thefeature, and has no route at all, which the existing
404branch already names as "itsbuild has no endpoint for chaining through it"; or it has the route and the fields
together. There is no in-between for an optional field to serve. (
embed_parent_portdoes appear on main, but that is the non-chained embed parameter of the same name.)
Identifying the hop only decides whether to accept a NEW credential. The forward and
token ALREADY in place are the ones riding a hop that changed hands, and refusing a
replacement leaves them exactly where they were -- so one crew's bearer token keeps
arriving at another crew's gateway. Three things therefore act, and they are not
redundant -- but they do not all do the same KIND of work: two keep the credential from
reaching the wrong gateway, and the third retires a forward that cannot work any more.
Prevention -- a lent hop is withheld from allocation. Minting a chained credential
records the hop it named and when that credential expires, and
_reserved_portskeepsthat port out of
allocateuntil then.While a hub's token is valid the port it forwards to is never handed to another crew by
this gateway, so the moment the disclosure needs does not arise here.
That qualifier is load-bearing and an earlier revision of this description did not carry
it. The reservation is an allocator exclusion, which is not ownership: once the forward
serving the lent crew is torn down the port is free at the OS level, so any other local
process may bind it -- including a second gateway with its own registry, which cannot see
the reservation at all -- and the hub goes on dialling that port, handing a live bearer
token to whatever answers. Neither identity check catches it, because both run on a MINT
and an established pane does not re-mint per request; the proactive re-mint is scheduled
at ~80% of the token's lifetime, so the exposure lasts as long as the credential does
rather than being caught.
Ownership -- the lent port is held by the OS for the reservation's life.
HopPortGuard(hop_port_guard.py) binds the port itself, armed by the teardown thatfrees it and by
sync_hop_holdsat startup. Startup is not belt-and-braces: thereservation is persisted and a listening socket is not, so a restart would otherwise
arrive holding every reservation and owning nothing.
The socket listens, and that is forced rather than preferred. A socket that is bound
but not listening does not refuse a second
SO_REUSEADDRbind -- which is exactly whatOpenSSH's forward listener and this module's own availability probe issue -- so the
tempting shape, occupy the port without listening and let a stale client be refused
before it transmits, does not hold the port at all. Measured both ways, and pinned, so a
later edit cannot quietly take the cheaper one.
The option differs by platform because its meaning inverts. On Windows
SO_REUSEADDRlets another local process steal an ACTIVE listener, and merely omittingit is not enough either, since the other process can set it on its own socket; the
non-POSIX branch therefore sets
SO_EXCLUSIVEADDRUSE. This is the same branch, for thesame reason, as
browser_cli.viewand the dashboard's own listener -- an earlierrevision of this PR set
SO_REUSEADDRunconditionally, held nothing whatsoever onWindows, and said in a comment that it did. That is the platform-specific no-op class
#9731 exists to catch, and it was CI on the Windows shards that caught it, not any case
that can run here. The branch is now pinned on its CONDITION rather than on the two
option names being present: a mutation that flattened the test to a constant left both
names in the source, one unreachable, and satisfied a presence-only assertion.
Because listening is mandatory the handshake completes and a stale client does transmit:
"the credential is never sent" is unreachable. What is reachable is that nothing reads
it. The guard accepts and closes at once with
SO_LINGER0 and never reads a byte, so astale pane sees its connection reset immediately -- not a hang on a queued connection
whose buffered credential any later accept would hand over, and not an orderly empty
reply a client could mistake for success. The refusal is deliberately blunt: answering
something courteous like
410 Gonewould mean draining the request first, and therequest is what carries the credential.
Where the holds are taken is part of the mechanism, not an implementation detail.
Reading the reservation table is a file read plus a parse of every row, so it never runs
on the event loop -- the rule
_reserved_portsalready follows a few hundred linesabove, whose own comment says a synchronous version "would stall unrelated requests and
heartbeats". At startup the whole re-take runs in a thread behind the tracked task that
already backgrounds the revive, so it is off the gateway boot path, and it finishes
BEFORE the revive because a revive reconnects instances that allocate ports. In teardown
the read is taken AHEAD of the forward's stop rather than after it: awaiting anything
between the port's release and its bind would stretch the one gap this mechanism cannot
close from two adjacent statements to however long the shared default executor takes to
schedule, so only the bind -- one non-blocking loopback syscall, one port rather than one
per reservation -- stays in line. That ordering is pinned, because both orderings end
with the port held and differ only in how long it was free.
A hold that cannot be taken is remembered and retried, not logged and forgotten: a
reservation whose bind failed once and is never retried is a live reservation with no
socket, which is the exposure itself reached by another route. The usual cause is one of
this gateway's own leaked forwarders still occupying the port after a hard kill, and the
retry takes it the moment that process goes.
A port counts as in use only while a forward is actually LISTENING on it -- CONNECTED
state -- not while any tunnel object remembers the number. An unexpected
sshexitleaves its tunnel in
_tunnelswithlocal_portintact and the state ERROR, and pastmax_recovery_attemptsit stays there for good, so counting it as in use skipped thehold for a port the OS had already freed while the credential naming it stayed valid.
Every site deriving in-use, reserved or occupied was checked against that sentence,
including ones this diff never touched: the allocator's
_reserved_portsand thereclaim's
live_pidsboth count everything remembered and are RIGHT to, because they runin the opposite safety direction -- over-counting costs the allocator a port it declines
to reuse, while over-counting the holds withholds a hold. Exactly one site was wrong.
Placement follows from it. The exit path takes the holds BEFORE its backoff, since the
child is already gone and waiting would leave the port unheld for the whole delay on every
flap and permanently once attempts run out. The recovery releases its own hold immediately
before the rebuild rebinds, or the hold blocks the self-heal; and it re-settles in a
finally, which one call covers both outcomes precisely because of the invariant -- arebuild that succeeded leaves the tunnel CONNECTED and its port in use, one that failed
leaves the port free and therefore held. Enumerating the recovery's failure exits instead
would let a later
returnskip it silently. Both orderings are pinned structurally,because either one ends with the port held and they differ only in how long it was free.
Reclaiming such a forwarder is refused while its port carries a live reservation.
Reclaiming exists to hand a crew its recorded port back, and a lent port is one
allocatedeliberately routes around, so freeing it recovers nothing and converts a portsafely occupied by this gateway's own dead child into a free one a stranger can bind
while a hub still forwards a bearer token to it. The orphan is the ownership in that
state, and a stronger one than a held socket; it is reclaimed by the next connect once
the reservation lapses, and lingering is already a tolerated outcome on that path.
Each hold carries its own deadline and expires itself, so it cannot outlive the
credential and squat a port that is legitimately free again;
releaseruns before areconnect binds, so our own hold is never what refuses our own crew. Worst case is one
descriptor per live reservation, bounded by the same count that already bounds the
reservation table and far under both the 7778-upward range and the descriptor limit; a
hold that cannot be taken is reported at ERROR rather than passed over, because an
unheld reservation is precisely the exposure.
The window a gateway exit opens, and what bounds it. Two windows stay open by
construction and neither is claimed closed. The forward must release the port before the
guard can bind it, which is the gap between two adjacent statements rather than hours.
And shutdown releases every hold, because a socket cannot outlive the process holding it,
while the credential stays valid -- the remote crew issued it, so nothing on this side
can invalidate it and there is no revoke primitive in the tree to call. A holder that
outlived the process would have to be a separate deployable component, and revoking would
be a new contract with the remote crew; neither is within this mechanism.
What IS within it is the window's LENGTH, and that is why the lent lifetime is capped at
LENT_HOP_TTL_CAP(30m) rather than taken from the crew's row. The exposure after anexit runs to the lease deadline and no further, so it is one lent TTL: 30m instead of the
20h a default row would give, and the next start re-takes every hold through
sync_hop_holds. The cap is a minimum against the row, so a row configured shorter keepsits own figure, and it applies only to the credential that LEAVES -- this gateway's own
pane is not lent to anyone and keeps the row's lifetime.
One number reaches three places: the lifetime the remote crew issues the token for, the
deadline the lease is recorded under, and the
ttlthe answer reports. It is computedonce and passed to each rather than re-read from the row at each site, because a site
disagreeing with the others would either free the port under a live credential or have
the hub schedule its re-mint after the token is already dead. 30m rather than less
because the re-mint fires at
DEFAULT_TOKEN_REFRESH_FRACTIONof the lifetime, leaving20% -- 360s -- as the margin a mint has to complete inside, against a worst case of
MINT_TIMEOUT_CEILING_SECS + 15for a chained mint and 90s for an SSM one; a cap thatate that margin would expire the token mid-mint and reload the hub's pane every cycle. A
test computes that margin from those same constants, so lowering the cap without
re-checking the arithmetic fails there rather than in the field.
It is not a lease. The deadline is the TTL this gateway itself issued, so nothing is
expected of the hub and no state crosses the boundary -- no notification, no ordering
contract, and the parent learns of its own crew's disconnect locally as it always did.
The allocator already took the exclusion (
allocate(exclude=...)), and_reserved_portsis its single builder feeding the single call site, so the new sourcereaches every allocation by construction. Persisted rather than held in memory, because a
restart would otherwise drop a reservation while the credential it protects is still live.
The reservation is a PRECONDITION of handing the token over, not a note taken on the way
out, and three things make it one. It is written under the manager lock, in the same
critical section as the check that the hop is still ours -- validating outside the lock
and writing inside would leave a gap the width of the acquire, and that gap is the whole
hazard. The write propagates its error rather than going through the registry's
best-effort hint helper, whose documented behaviour is to swallow one: right for a hint
that only has to be usually-there, wrong for the record that keeps this port away from
another crew. And a reservation that cannot be written refuses the mint with
503instance_hop_lease_failed, because a credential this gateway cannot protect is one itdoes not issue.
The reservation does not live on the crew's row, and that placement is the point.
Held there, removing the crew deleted it -- while its credential was still valid -- and
the port went back to the allocator, which is the disclosure the reservation exists to
prevent, reached through the one path that deletes a row. So it is held on the registry
DOCUMENT, keyed by port:
removefilters the instance list and structurally cannot touchit. Same file, same lock, same atomic write; the only writer prunes lapsed entries on its
way through, so the record stays bounded without a timer. Every row-deleting path was
swept for this -- there is exactly one, and
updatekeeps the row, so it was never asecond hole.
The parent answers that hub's next mint with
instance_not_found, which is why that coderetires the forward -- and why the reply's
codeis read BEFORE any status branch. Abuild with no chaining route answers 404 as well, and retiring over that would tear a
chain down because the parent is OLD rather than because the hop moved. Stated plainly:
once the record is at the document level, removal is a case prevention ANSWERS rather
than one detection bounds -- the port stays withheld through it, and an earlier revision
of this description claiming otherwise was describing the row placement this one replaced.
Detection on the parent's own answer -- retirement, not a second guard. With the
reservation at the document level this path is not what keeps the credential out of
another gateway; prevention is. What it does is retire a forward that cannot work again,
because the crew it reaches is gone from the parent. A re-mint that fails because the
parent ANSWERS that the crew is not connected there, or not there at all, is terminal for
the forward: retrying cannot recover a hop that is not ours, and the forward still points
at a port the parent is free to reassign once the reservation lapses. Read from the
reply's
code, never its sentence,and raised as
HopRetiredErrorso the caller acts on which failure it saw instead ofinferring "gone" from "failed". Every other mint failure keeps the deliberate
non-terminal retry -- a timeout is a blip, and tearing a working chain down over one
would turn every hiccup into a disconnect the user has to undo.
Detection on a disagreeing identity -- the case prevention cannot reach in principle.
The parent still holds the crew and merely serves it elsewhere, so no port was reused at
all: no reservation ever applied to that new port, the old one is still withheld, and
there is nothing for prevention to hold that would help. The comparison runs, the
credential is refused,
and the forward riding the moved hop is retired with it. Only this branch retires. A
superseded generation is rejected by the store's own fence before the comparison is
reached; a port-only disagreement is also what the connect path produces while a forward
is being built, so retiring on either would tear down a healthy tunnel -- and a
port-only disagreement needs no retirement anyway, because the port the forward still
dials is one prevention is holding.
A chained row's own coordinates are records, never dial targets. For a row carrying
via_instance_idthe dial target is the parent's host plusvia_remote_port; the row'sown
ssh_hostandremote_portare never dialled from here. That matters because botharrive in the announcing pane's payload, which is untrusted -- and an origin check cannot
help, since the sender is the remote gateway the owner deliberately connected to and so
holds the correct origin. What makes a forged value harmless is the branch rather than a
check on the value:
_resolve_transportreturns_resolve_chained_transportas its firstbranch whenever
via_instance_idis set, and that resolver builds its params withssh_host=validate_ssh_host(parent.ssh_host). The branch readinginst.ssh_hostis theunchained one, below the return.
Since they are records, the dashboard stops rendering them where a verified target goes: a
chained row shows its host marked as reported and omits the crew's own gateway port. The
value is kept rather than replaced with the parent's host, because only the parent knows
which machine the crew is actually on -- deriving it would state something false and make
two crews behind one parent indistinguishable, while changing nothing about the dial
target. What a forged notice can still influence is
via_remote_port, a loopback port onthe parent, a machine this actor already holds and whose own gateway mints the token
regardless; the depth, width, duplicate and cycle guards all apply here against a registry
the pane never sees, so the bounded result is rows pointing back through the same parent,
visible in the Remote Crew list and removable with the cascade.
Retirement keeps the user's intent, so an ordinary reconnect brings the crew back on a
hop that is actually ours. Clearing
was_connectedwould present a security teardown asthe user having turned the crew off.
The retirement is scheduled rather than awaited, and that is not a style choice: the
store runs both inside
connect's lock and outside it from the refresh,asyncio.Lockis not reentrant, and awaiting a teardown from under the lock would park forever and
wedge every later connect, disconnect and self-heal. It hands the work to a task that
acquires the lock itself -- the same shape the self-heal already uses.
The token and the port B answers with are ONE claim. B's reply names the token
and the loopback port B's own forward to C listens on, and A dials that port. B's
mint is a multi-second round trip taken without B's manager lock,
status()handsout the tunnel's live status object, and a teardown pops that tunnel without zeroing
the port on it -- while B's allocator gives a just-freed port to the next connect
first, because it takes the first free port above its base. So C disconnected inside
that window, plus any crew connected before the mint returns, would pair C's token
with the other crew's forward, and A would forward to whatever now answers there.
Nothing downstream re-checks it. B therefore reads the hop once before the mint,
confirms the generation, the connected state and the port are all unmoved after it,
and refuses
instance_hop_changedrather than answering with a pair it cannot vouchfor. The generation is what catches a reconnect, which satisfies membership again.
The token's lifetime comes from B too. B issues it under B's record for C, which
A cannot know; A's own row defaults to 20h, so scheduling the refresh from it would
put the refresh after a shorter token had already expired. The reply carries the TTL
it issued and A refuses a mint whose reply cannot state a usable one -- A's own number
is a separate value with nothing holding it below B's, so it is not a safe stand-in,
and every gateway that answers this endpoint at all reports its TTL. A then stores the
SHORTER of B's answer and its own, which makes the schedule safe in both directions
however the two are configured: short only costs an early re-mint, long schedules
after the token is dead.
And the total is published, because the remaining is measured against it. Every
reader asking how far through its life a token is divides
token_ttl_remainingby atotal, and the two have to be the same number. The row's TTL is not that number for a
chained crew, so the status now carries
token_ttl_total-- the figure the remainingcounts down from -- and one shared helper picks it for all three readers. Measured
against the row instead, a 1h token from B sits permanently past a 20h row's refresh
threshold: A re-mints on every 60s poll, and each re-mint re-derives the iframe
src,so a chained crew's hidden pane is remounted every minute and quietly loses whatever
was unsaved in it. Nothing self-corrects, because the 10s rate window never engages
against a 60s poll.
And the refresh SCHEDULE is re-derived, not remembered. The stored lifetime is
rewritten by every store, and for a chained crew it is the shorter of B's answer and A's
row -- so lowering the TTL on either side changes it underneath a running refresh loop.
ttlis not a transport field, so that edit tears no tunnel down and restarts nothing.The loop therefore reads the interval back from the lifetime the last refresh stored
rather than binding it once before the loop: bound once, it would keep sleeping the old,
longer gap while the shorter token died early, reloading the pane and discarding unsaved
work on every cycle from then on rather than once. A failed refresh stores no new
lifetime, so it reads back the same number and the deliberate retry-at-the-same-interval
behaviour is unchanged.
That desync is NOT introduced here --
mainhas the same shape, where every storerewrites the lifetime while the loop derives its interval once, so lowering an unchained
crew's
ttldesyncs it today. What this change adds is a second way to reach it, becausethe effective lifetime is now the shorter of two records and the parent holds one of them.
The fix is at the one site either trigger passes through, and the derivation moved into a
single helper so a future reader cannot compute an interval the loop will not refresh.
These three -- the denominator, the published total, and now the interval -- are one
pattern rather than three bugs: a value's MEANING changed, and each time a consumer of it
was still computing from what it used to mean. So the round enumerated every reader of
the stored lifetime and every value derived from a TTL, on both sides of the wire, and
asked of each whether it re-reads or computes once. Ten sites; nine re-read at the moment
they need the value, and the loop was the only one that did not.
Six refusals, all decided server-side, because the request can originate inside a
pane whose code A does not control:
chain_too_deepMAX_VIA_HOPS(2) hops. Checked before anything is written or dialled, and again by B, which is the only side that can see it reaches C through a further hop.chain_duplicate-2suffixed id, leaving one remote crew with two rows, two forwards, two tabs and two charges against the width cap, removable only by hand. The pane treats this refusal as benign -- it refreshes and relays nothing, because the crew is present and nothing the user did failed.chain_parent_fullMAX_CHAINED_PER_PARENT(8) chained crews. Depth bounds how far a chain reaches; this bounds how wide it gets, and neither implies the other -- a two-hop chain is legal however many crews sit at the second level. It is counted per parent, so one parent's pane cannot crowd the others out, and it matters because these rows are created by that pane rather than by anyone at A's dashboard.chain_parent_unknownchain_parent_not_sshgateway_idoff/api/health; equal to A's own or to any ancestor's, the forward is torn down and the connect refused.Ids are compared, never host strings: one machine answers to
localhost, an sshalias, an FQDN and an IP, so a string comparison would refuse unrelated crews and
still admit a real loop.
gateway_idis a random per-home id, minted once andmemoized, revealing nothing about the machine.
Reading that id judges a descriptor, not a name. The id file lives in the
owner-writable data home, so a read that checks the path and then opens the path
resolves the name twice and the two answers need not describe the same object -- it is
whatever is there at the moment of the open that gets read. That window has two
consequences and the second is the worse one: a symlink redirects the read, and a FIFO
makes the open wait for a writer. The wait is what costs the gateway, because this read
runs in
asyncio.to_threadon the SHARED default executor, so a held worker is lostcapacity until restart rather than one slow request -- and
/api/healthre-enters theread on every poll, because a rejected file is deliberately not cached. So the open
itself goes through
platform_compat.open_file_no_reparse, which refuses the redirect atthe final name on either platform and, asked to, declines to wait;
fstatjudges the object the descriptor already names, which leaves no second resolution to
disagree with.
Borrowing that primitive rather than writing the flags out is the point, not a style
choice.
getattr(os, "O_NOFOLLOW", 0)is 0 on Windows, so a hand-rolledos.openwith it refuses a link on POSIX and silently follows a reparse point onWindows while reading as though it refused both -- the no-op catalogued in #9731,
where this site is now recorded as one more of the read-only opens that issue calls
convertible today. The primitive does NOT refuse a FIFO, though;
nonblockingonlystops the open from waiting for one, so the
S_ISREGcheck stays this reader's ownand is what rejects a pipe whose writer wins the race. A regression to the raw flag
behaves identically on POSIX, so it is pinned structurally instead, by an assertion
that is deliberately not skipped on any platform.
Considered and deliberately NOT done: giving
gateway_ida sandbox disposition. It is atrust-root file, so a policy entry for it has a blast radius well past this PR, and the
no-follow open closes the reachable mechanism whatever the path's write permissions are;
it belongs in its own change rather than riding this one.
What crosses the frame boundary, exactly. The pane posts
{type, v, id, name, sshHost, remotePort, port}-- no token and no credential. Thetwo directions are guarded differently:
targetOrigin: '*', because a cross-origin iframe cannot readparent.locationand so cannot name its host's origin. That is safe only becausethe payload holds nothing secret; intercepting it yields a port number.
against its map of currently-warm pane ports and drops any message whose origin
does not resolve, so the parent id is derived from the origin and never supplied
by the sender. The payload is then validated by
readChainedCrewNotice-- pure andexported so its rules are tested without a host -- and handed to the gateway, which
owns the decisions no frame may make.
A re-announcement always connects. When the announced crew already has a row here,
A repoints it only if B is serving it on a new port, but connects it either way. An
announcement means the pane's user just connected that crew on B, while the tab here
can be down for reasons the row cannot show -- A restarted, or the hop dropped and
took the child with it. Deciding from the port alone left a dead tab beside a running
crew. The decision is a pure
chainAdoptionPlan, exported so that contract is testedrather than buried in a callback, and the backend's connect is idempotent.
A refusal is reported, not swallowed. When A declines an announced crew, only A
holds the reason: the depth cap and the cycle guard are its gateway's decisions,
taken against a registry the pane never sees. A relays that reason down to the
announcing pane, addressed to that pane's exact loopback origin, and the pane shows
it beside the connect it already reported -- the crew IS connected there, and only
the tab up here was declined. The pane accepts the message solely from its own
parent frame, which is the check available on that side.
The chain is also a lifecycle. Disconnecting a crew closes the forwards behind
it first, since theirs ride its hop. Removing one brings every captured crew DOWN
first, deepest first, and a stop that raises refuses the whole removal with nothing
deleted: rows deleted over a live forwarder strand it together with its minted
token, and the row that goes carries the
forwarder_pidreclaim hint with it, sonothing finds it again while the gateway runs. Each descendant is named explicitly
rather than reached through the parent's own cascade, which suppresses a child's
failure because a shutdown must not abort on one stuck crew. Only once everything
is down do the rows go, leaves first and the named crew last, so an interrupted
sweep can only leave a parent with fewer children -- a state that still connects --
never a row naming a parent that is gone.
A second pass follows the deletion over the same captured list. That list is the
point:
disconnectfinds a crew's children by reading the registry, the rows aregone by then, and
reg.removeis offloaded and therefore yields, so a connectracing the removal can have re-established a child's forward -- sweeping the named
crew alone would leave a forward holding a port with no row to find it by. That
pass cannot refuse, because the removal has already happened and raising would
report failure for rows that are already gone; a forward it fails to stop is written
to the gateway's warning log naming the crew.
Being precise about who sees which of those two, because they are not the same: the
409reaches the user, since it is an error response and the removal mutationrejects on it, so the panel shows the gateway's own sentence. The residual does not
reach the screen, and it is deliberately NOT a field on the success body either. An
earlier revision published a
strandedlist there and nothing read it -- bothcallers of
removeInstanceawait without touching the body, and no componentreferences the name -- so it was a field on a public response with no reader, which
is a contract to keep with nothing depending on it and which let a real gap look
answered. The honest surface is a persistent warning about a forward with no row, a
notification affordance rather than a line in a panel that is about to lose the row
it would hang off; until that exists the log is where an operator reads it. Stated
in
docs/system-specs/modules/instances.mdsection 17.4.A refusal leaves nothing changed, which the sentence had to earn. The teardown
goes crew by crew BY NAME rather than through the manager's cascade, to get an
unambiguous per-crew failure -- and a by-name
disconnectdefaults to clearing thesticky connect intent, which is what decides whether a crew keeps its tab
(
was_connected || connected || warm). So with three levels the deepest descendantcame down cleanly, a shallower one then raised, and the reply said "Nothing was
removed" while that descendant had already lost its tab and a reconnect of the parent
would not bring it back. It now passes
keep_intent=True, which is what the manager'sown cascade already did for the same reason: on the success path these rows are
deleted a moment later so the flag is moot, and on the abort path it is what makes the
sentence true.
What is refusable and what is not. An exception out of the teardown is
unambiguous -- the forward is still up -- so the first pass refuses on it. A STATUS
reading is not, and an earlier revision of this PR refused on one:
statuscannottell a stop that failed from a reconnect that landed after a stop succeeded, which
is the ordinary race the second pass exists for, so refusing on it turned that race
into a row nothing could delete until the gateway restarted. The repository's own
test_remove_sweeps_a_reconnect_that_raced_the_removalpredates this branch andstates that case must remove successfully and sweep; its stub returns cleanly, so
refusing on a raising stop leaves it untouched.
The 404 is decided before anything is deleted. Reading it off the parent's own
removal meant the descendants were already gone by the time a missing parent
answered "not found", so a DELETE of an id another path had already removed -- an
orphan's former parent -- reported a failure having just deleted the crews under it
and dropped the list of which ones they were. Rows still go leaves first.
Editing a parent tears down the crews riding it. A child's forward was built from
the parent's coordinates, so moving the parent to another machine while the child's
ssh -L ... <old host>stays up leaves that child reporting CONNECTED and servingthe machine the user just left while its row names the new one. The cascade runs
inside reconfigure's own critical section, and unlike a shutdown it is not tolerant:
a child that will not stop aborts the edit, for the same reason the parent's own
failed stop does. The children keep their connect intent -- the user edited the
parent, and forgetting them would mean reconnecting it does not bring them back.
Self-heal mints through the parent BEFORE tier 1 rebuilds. A parent that came back
on a different loopback port is the commonest reason a chained crew needs healing, and
the mint reply is the only place the parent names its current port.
ssh -Lbinds theLOCAL side whatever answers remotely, and
start()waits only for the local forward toaccept -- so a tier-1 rebuild against the row's stale hop starts successfully, marks the
crew recovered and resets the attempt counter without ever reaching the tier-2 mint:
healthy on the board, unreachable in fact, every cycle. The mint rides the parent's own
hop, which is already up, so it runs first -- the same inversion
connectmakes, forthe same reason. A chained crew's tier 2 is therefore the rebuild alone, the shape
fargate's already has. A parent that cannot be minted over, or that names no port,
stands the attempt down rather than dialling the row's.
The parent-hop re-mint is guarded by the tunnel generation, not by a held lock.
Its three callers differ: connect holds the manager lock across the mint, while the
proactive refresh and the self-heal's own mint deliberately do not -- that is what
keeps the 401 branch out of the re-entrant deadlock above. So membership is not
enough: a parent disconnected and reconnected inside a mint budget measured in tens
of seconds satisfies it with a NEW tunnel, and the fresh credential would be
overwritten by one minted against the generation before it. The generation is
captured before the mint and compared before the store, as its two sibling stores
already do.
Adding a chained crew and removing its parent are one critical section. The add
validates the parent and inserts the child in two separate awaits. A removal
landing between them takes its subtree snapshot before the child exists and then
deletes the parent under it, leaving a crew whose only route is a hop that is gone.
Both handlers take the same lock across that pair, so either the child lands before
the snapshot and goes with the cascade, or it lands after the parent is gone and is
refused.
Two limits are deliberate, both recorded in
docs/system-specs/modules/instances.mdsection 17.4. A crew reporting nogateway_idis chained without a cycle check, because the loop it cannot rule outis a nested pane rather than an escape from a boundary and the depth cap already
bounds the arrangement; note that "no id" covers both a build predating the field
and a caller the direct-local gate fenced off, and that A's own probe rides the
loopback end of a forward it just opened, which that gate treats as direct-local by
documented design. And a top-level crew is not cycle-checked at all, because one
pointing back at this gateway is what the product already allows.
The tab is the promise, so the crew is connected and not merely added. A row
written by
POST /api/instancescarries no connect intent, and the switcher admitsa crew only once it is connected, warm, or carries that intent -- so adopting
without connecting left the crew in the Remote Crew list and nowhere else. Adoption
connects it, which is the same intent the user expressed in the pane, arriving here.
A chained chip is two boxes, not one string. The chip is width-limited, and one
string ellipsises from the right, which eats the crew's own name and leaves a chip
reading as its parent's duplicate. The parent segment takes the squeeze and the
crew's name never shrinks. The path reads from the string catalog in all 13
locales, so a locale can change the separator or the order.
The name cap is applied where a name is written, not on every record read
INSTANCE_NAME_MAXis new here, and nothing truncated a name on the way in before it --no length check at the handler, none in
Instance.from_dict-- so a row stored beforethis change can be longer than the cap.
InstancesRegistry.update()re-runsInstance.validate()against the WHOLE record, anddisconnect's reset rides throughupdate()as a hint-only patch (was_connected=Falseplus the freedlocal_port) whosefailure is swallowed as best-effort. Capping inside
validate()therefore lost that resetsilently on a pre-cap row: the next start revived a crew the user had disconnected, on a
port it no longer owned, and the only exit was a manual rename.
The cap now lives in
validate_instance_name, called fromadd()and fromupdate()only when the patch carries
name. That is the same write-site placementvalidate_ttlalready documents in this file, and for the same reason: a rule added later must not fail
a write that touches a different field.
A teardown stops reporting CONNECTED at its start, not at its end
Every decision to issue a credential for a forward re-reads the same live
TunnelStatusobject: the token/port pair a pane is answered with(
handlers_instances.py), and the mint paths that require CONNECTED._SshTunnel.stop()awaits task cancellation, a stdout drain and
_terminate(), whosewait_forallows 5seconds per signal, so the teardown spans seconds.
With the state assigned after those awaits, the live object reports CONNECTED with an
unchanged
local_portfor that whole span. The handler holds no manager lock acrosstoken_validatesor the re-mint, so a concurrent disconnect runs inside the window andthe caller is answered with this crew's token paired with a port that is already being
released. Nothing revokes a token once the response is sent.
stop()now assigns the state before its first await. The guard that re-reads the liveobject therefore sees the teardown and refuses the pair, which is what its own comment
always intended.
Scope is narrower than it looks.
_reserved_portscounts a port by tunnel MEMBERSHIPrather than by state, and the caller pops the tunnel only after
stop()returns, so theport stays withheld across the window and nothing becomes allocatable earlier.
_apply_hop_holdsunder-countsin_useduring the window, which is the direction its owncontract calls cheap: the bind loses to a still-live forward and the port is recorded as
owed and retried. A teardown that RAISES restores the previous state, since the forward is
then still up; a CANCELLED one stays non-connected, because the child may already be
signalled and refusing to mint for a half-gone forward is the safe direction.
Rendered evidence
Five frames attached to this description. Captured against the real built SPA with
a stubbed instances API, so the switcher photographed here is the one the header
renders. The harness that produced them is deliberately not part of this diff, and
neither are the frames as files: the lanes admit an attachment, and a capture
harness is not product code. Generic fixture names throughout -- nothing here names
a real machine.
The switcher as a tree, depth 0/1/2.
build-hostat depth 0,fresh-desktopunder it,
gpu-boxunder that, each subtitle naming the crew it goes through aswell as the machine.
A parent hop down greys out the crews behind it.
build-hostreadstunnel error;fresh-desktopandgpu-boxdim with it because they ride its tunnel,while the unrelated
lab-hoststays bright.The chip path under a parent name too long for it. The parent is
build-host-with-a-long-name; its own chip truncates, and the chained chip readsbui... > fresh-desktop-- the parent segment gives way and the crew's namesurvives whole, which is what stops the chip reading as a duplicate.
Remote Crew inside a pane. The rail entry used to be filtered out of an
embedded pane; connecting a further crew from there is the declared feature.
The relayed refusal, in the pane that acted. The depth-cap sentence is the
host gateway's own, rendered above the crew list the user was just looking at.
A chained row's machine is labelled as reported, and its gateway port is dropped.
The host's own Settings -> Remote Crew, with two direct crews and two chained ones in
the same list.
build-hostandlab-hostreadbuild-host-alias - remote port 7777,because both are values this dashboard dials.
fresh-desktopandgpu-boxreadfresh-desktop-alias (reported)andgpu-box-alias (reported)with no port at all,because neither their host nor their own gateway port is anything this dashboard
reaches -- the forward goes to the parent. The label states who the value came from
instead of asserting it, which is what keeps a forged payload from making a row read
like a machine it is not.
Tests
test/test_remote_crew_chaining.py-- the three registry fields and the chainwalk; that a file written before chaining loads with all three defaulted and stays
writable; all six refusals, the uniqueness guard shown to refuse the second add
of one remote crew while admitting a different crew behind the same parent and
the same remote id behind a different parent, the width cap asserted exactly at
its boundary and
shown not to bleed onto another parent or onto a top-level add; the cycle guard
against A's own id and an ancestor's,
and the fail-open when a crew reports none; the disconnect and remove cascades,
including that the post-removal sweep names every captured crew while the registry
is empty for the whole sweep -- which is what proves the ids cannot come from it --
and that rows go leaves first; that a chained add and its parent's removal cannot
interleave into an orphan whichever order they land in; and the mint through the
parent.
con the hub,
c-2on the parent -- asserting the request is addressed to theparent's id and never to ours, and that the stored TTL is the one the parent
reported. Sibling tests cover the six shapes of an unstatable TTL, each refusing
the mint rather than guessing, and the clamp in both directions -- a shorter
reported TTL is taken, a longer one is not.
token's own total and not the row's, it is absent rather than zero when no token is
held, and the status dict carries it beside the remaining or carries neither.
comparison read as sufficient: the crew holding the hop rebuilds its forward and its
allocator hands out the SAME number, every number agrees, and the credential is
discarded on the identity. The other direction too -- a self-heal that rebuilds onto
the reported hop rebinds the identity and its store lands, so the check is not simply
a refusal. A reply that cannot identify the hop is refused for each of eight malformed
shapes. And a round-trip drives the parent for real and feeds its own payload to the
hub's parser, because each side had its own tests and neither could see a field the
OTHER side stopped sending.
this crew to another port is discarded and the valid credential survives; the store
itself refuses a moved hop and a moved generation at the unit and keeps what it held;
a heal whose rebuilds never land stores nothing at all, which is the only case that
can see the ORDER rather than the comparison; the real mint is driven at its own level
to prove it REPORTS the hop and the lifetime and publishes neither; and a structural
test asserts every store names the generation it minted against, that the decision is
reached from exactly two functions, and that the two published values are written from
the store alone.
during the token probe is refused with no token in the body, and a forward that
drops to ERROR with its port unchanged is refused too -- so neither half of the
re-validation can be removed without a test going red.
a crew disconnected inside the mint, with a second crew taking the freed port, is
refused rather than answered, and the test asserts the other crew really does hold
that port; a reconnect inside the mint is refused although membership and the
connected state both read true again; and a probe marking the forward ERROR is
refused with the tunnel never popped and the generation never moved.
still targets the port the parent reported and not the row's, its tier 2 rebuilds
without minting a second time, and a parent that names no port stands the attempt
down;
editing a parent tears down the crew riding it while leaving that crew's intent; a
child that will not stop aborts the edit and the old host survives on the record; a
re-mint whose parent's generation moved mid-flight is discarded and the live
credential survives, while an undisturbed one still stores.
levels to see: with a parent and one child the loop fails on its first candidate, so
nothing was torn down yet and no intent could be lost. With a grandchild the loop
tears it down successfully, then fails on the child, and the test asserts the order
and that both teardowns were asked to keep intent. The assertion stops at the flag on
purpose -- a stubbed manager performs no registry write, so reading
was_connectedoff those rows would pass whatever the handler passed; that the manager HONOURS the
flag is
_teardown_locked's own unit and is asserted where it lives.remove_teardown_failedwith every rowstill on the record, and the teardown stops AT the failure rather than carrying on
to the parent, whose forward is the child's route out. A forward that reappears
after the rows go is logged rather than raised, proved by two witnesses that are
not the wire: the pass called
disconnecta second time, and it said so where anoperator reads it. The sweep test asserts both passes -- the first with the rows
present, which is what lets it refuse, and the second with the registry empty,
which is why its ids can only come from the captured list.
rather than waited out: a parent lowering its answer from 20h to 1h between cycles must
shorten the next interval, and the same stated as the property rather than the numbers
(each wake lands inside the life of the token the previous wake minted, across 20h ->
1h -> 30m) so a changed refresh fraction cannot make it vacuously true. A third case
holds the line the fix must not cross: a transport error stores no new lifetime, so the
retry keeps the interval it had.
website/src/lib/tokenTtl.test.ts-- the shared denominator: the published total ispreferred, a missing one falls back to the row, and a zero or non-finite one is not
trusted. Two cases state the outcome rather than the input -- the reported 1h-token
case is due for refresh when measured against a 20h row and not due when measured
against the published total, and a chained token genuinely near its end still
refreshes.
website/src/test/remoteCrewChaining.test.tsalso pins the adoption plan: anunchanged hop port still connects, a moved one repoints and connects, and a row that
never carried a port is repointed. And
chainRefusalCode, which is how the panetells the benign refusal from the rest: it reads the gateway's machine-readable
code off the error body and answers empty for a non-JSON body, a missing or
non-string code, and a code that appears only in the human message -- empty means
"relay it", so every unreadable shape errs toward telling the user.
test/test_api_health.py--gateway_idis present and 32 chars for adirect-local request and absent for an anonymous, rebound or forwarded one, so it
rides the same gate as
version; a repeat read does not touch the disk; and theread does not run on the event loop's own thread, asserted by thread identity with
a guard that fails if the identity was never read at all.
test/test_gateway_identity.py-- the id read is judged as a descriptor. The casethat matters is the RACE, and it is the only one a path-then-open read fails: a
statically planted link or FIFO is refused by an
lstattoo, so the test plants areal file and swaps it for a symlink at the instant the path is stat-ed, then asserts
the id that comes back is the real one and never the attacker's. Beside it, a FIFO
must not hold the worker -- driven in a thread with a join deadline so a regression
fails here instead of hanging the suite -- and ten repeated reads of a refused path
must each stay inside that deadline, which is what makes the uncached re-read safe
rather than a window per poll. The
S_ISREGguard is pinned STRUCTURALLY, and thetest says why: every shape a test can plant statically is refused by the open or the
read regardless, and the one shape the guard uniquely refuses -- a FIFO whose writer
wins the race and so chooses this gateway's identity -- cannot be forced, because a
FIFO's writer only unblocks once a reader has opened. So the AST pin requires one
path resolution, a check on the descriptor, and that check BEFORE any read.
refused crew itself, so reporting it as machines in between overstated the chain by
one; the test asserts the sentence says hops and does not say machines. Changing the
unit in one place is not the whole job: the feature map's own description of that
refusal said
MAX_VIA_HOPSmachines, which is the same error one document over, soevery place expressing the cap was swept and that was the only other one.
test/test_error_code_contract.py-- the ratchet passes, so both new errorresponses are readable to it: a literal body carrying
code, including the onewhose status is computed.
test/test_instances.py-- the pre-existing racing-removal test, which asserts asuccessful removal sweeps a reconnect that slipped in, passes unchanged. It is the
repository's own statement of what this race should do and it is why the refusal
went.
website/src/test/remoteCrewChaining.test.ts-- the switcher tree and depth; theannounce is a no-op at top level, refuses an out-of-range port and carries no
credential;
readChainedCrewNoticedrops a notice with no id, one whose id isoutside the registry grammar, and one whose hop port is out of range, while
tolerating an unusable remote port because that field is only a record; a refusal
arriving with nothing subscribed is still there to read, is capped, notifies a
subscriber, clears on dismiss, and ignores a message of another type; and a row
that was only added gets no tab while the same row with the connect intent does.
covering it confirmed to fail, so none passes vacuously. The harness refuses to
score a mutation it could not actually apply, and requires every run to print its
own passed/failed count -- a run that collects nothing exits non-zero too. One
mutation stays GREEN and is reported rather than hidden: dropping the mint
window's PORT comparison changes no test outcome, because nothing assigns
status.local_port, so a different port means a different tunnel object and everysite that installs one bumps the generation the same check already compares. That
reading is kept as the redundant one, so an install site added later without the
bump is refused there rather than trusted. A first pass on the credential rule also
found a real gap rather than a weak pin -- no case observed the ORDER, so moving the
self-heal's store back before its rebuild went undetected -- and the case that can
see it was added.
validate()refuses one pastINSTANCE_NAME_MAX = 200, the same number the relay already slices an announced nameto, and a test reads that constant out of the relay so the two cannot drift. A bound
bounds every field it keeps: the name is persisted per row and echoed in every list
reply, so leaving it unbounded made the row caps no bound at all.
regenerated with
i18n:pseudo, and the i18n gate (extraction, key references, pluralsand the untranslated-string report) passes against this branch's literal base rather
than a moved
origin/main.assertion that matters is not one this gateway can make about itself. A test that the
allocator skips the port would re-pin the defect being fixed -- the exclusion was
already there and already passed that assertion. So
test_hop_port_guard.pyshellsout and binds with
SO_REUSEADDR, the way OpenSSH's forward and this module's ownprobe do: refused with
EADDRINUSEwhile the reservation stands, and bindable againonce it lapses or is released.
test_remote_crew_chaining.pydoes the same after areal teardown, end to end.
is shown to let a second
SO_REUSEADDRbind through. If that ever stops being truethe case fails, which is the signal to re-derive the shape rather than discover it
from a report.
apart. Closing a socket that already has unread data sends a reset regardless, so the
case that sends a request passes with or without
SO_LINGER-- a mutation removingthe option proved exactly that by going unnoticed. The case that discriminates is a
client which connects and sends NOTHING: an orderly FIN there, a reset with the
option. Both are asserted, and "nothing read the credential" is additionally pinned
structurally, since no behavioural test can separate a guard that reads and discards
from one that never reads.
disagreeing is invisible to a test of either one, so the pin is over the source: no
row title may be built from the raw state. Two sites had it -- the host-side menu row
and the relayed switcher's twin -- and reverting either one alone is caught.
this change touches -- chosen by grepping the suite for those symbols rather than by
listing the files edited -- and 236 vitest across the eight frontend files that
exercise this branch's own components, with
tsc --noEmitandeslint srcclean.Every gate the CI jobs run, each checked by its own exit status rather than by reading
a summary line: the baselined
blackgate,isort,flake8,mypy src/kiro_crew/,the subprocess-encoding gate, the ACP-frame host-data gate, the agent-SDK boundary
gate, the blocking-IO-in-async gate, the lockdown-before-publish gate, the
comment-history gate against this branch's literal base, and both documentation lints.
The gate list is taken from the CI workflow rather than chosen by which files changed,
and the test scope from which files double the changed interfaces: selecting either by
"files I edited" is what let a stale double stay red on CI while passing locally, and
mypy-- which is not one of the formatters -- is what caught the rebase defect below.mainmoved 219 commits under this branch, and one of themhad made the peer credential-header builder
asyncwith a third argument. The new callsite this change adds was written against the old shape, so the rebased tree assigned a
coroutine into a header dict and would have failed the first real chained mint. Every
behavioural case here passed anyway, because this file replaces that builder with a
stand-in at nine sites -- so the double hid it and
mypyis what found it. It is nowalso pinned in the suite, over the SOURCE rather than behaviourally: the builder must
be a coroutine function, and no call site may read it without awaiting. The pin was
proved by removing one
await(1 failed) and restoring it (2 passed).rebased base by running the tests that name them, not by reading ancestry: the
spawn-approval delivery case and the sandbox leaf ratchet (8 passed), and the Chat rail
registry and settings-registry extraction cases (19 passed). The spawn-approval test
had been RENAMED upstream, which is why the check runs the class rather than a stored
node id -- a stale id collects nothing, and nothing collected reads exactly like
nothing failing.
the fake allocator hands out. One fixed base port made that a resource shared between
parallel test processes, and the loser reported "no hold was taken" -- a real-looking
failure with an unrelated cause, seen once locally. Each worker now gets its own band,
so the flake cannot reach a CI lane and be read as a defect.
printing its own counts. The socket's shape: binding without listening, reading the
request before closing, dropping
SO_LINGER, never accepting. The lease's lifetime:letting a lapsed hold persist, never retrying a failed hold, re-logging every retry at
ERROR. The platform: flattening the branch that picks the Windows option. The seams:
removing the teardown arm or the startup re-arm, reclaiming an orphan off a still-lent
port, moving the reservation read into the window between a port's release and its
hold, counting any remembered port as in use, skipping the arm on the exit path, never
releasing before the rebuild rebinds, re-settling outside a
finally. The cache: theexit seam reading the record instead of it, never writing it at the mint, and dropping
the synchronous hold from the seam altogether. And the invariant predicate itself, two
ways: accepting a non-CONNECTED entry as the owner, and reporting nothing at all.
as-is; making it a maximum so a row shorter than it is widened; having the lease
deadline read the row while the mint used the cap; having the answer report the row
while the lease and the mint used the cap; and reading the credential header without
awaiting it. The middle two are the ones that matter most, because each leaves the cap
visibly in place while breaking the property it exists for -- one frees the port under a
live credential, the other has the hub re-mint after its token is dead. Each anchor was
checked to occur exactly once in the source before the run, against a control string
known to occur twice: a mutation whose anchor is ambiguous reports a verdict about a
site it never touched.
PIN rather than a weak fix. Dropping
SO_LINGER: a close with unread data alreadyresets, so a case that sent a request could not discriminate, and the case that does is
a client which connects and sends nothing. Flattening the platform branch: both option
names stayed in the source, one unreachable, satisfying a presence-only assertion, so
the pin now checks the branch's CONDITION. Skipping the exit-path arm: the pin covered
the
finallybut not the ordering against the backoff. Dropping the synchronous holdfrom the exit seam: every case for the cache called the helper DIRECTLY, so deleting
the seam's call to it left them all green -- the pin now drives the real entry point
the monitor uses, on both the self-healing and the reconfiguring path. And one was a
HARNESS defect rather than a pin defect --
release(local_port)also appears inconnect, so a bare anchor mutated that site and reported MISSED without touching therecovery at all. The harness also refuses to score when its control fails or an anchor
has moved, which it did twice across this refactor rather than reporting false passes.
against a passing control (98 Python, 34 vitest): dropping the unreachable label's
guard, dropping the hop from it, stripping the refusals' subject, silencing the
residual log and skipping the second pass are each caught. One mutation is expected
NOT to be caught and is run for that reason -- putting
strandedback on thesuccess body changes nothing the suite can see, which is the evidence that nothing
depended on it.
The name cap
test_a_pre_cap_name_still_lets_disconnect_persist_its_resetstores a row 50 characterspast the cap, sends exactly the hint set
disconnectsends, and asserts the reset ispersisted, the freed port is released, and the untouched name is not rewritten -- then
asserts a name CHANGE past the cap is still refused, and that renaming to a short name
works, since that is the legacy row's exit.
Mutation-proved rather than asserted: restoring the cap inside
Instance.validate()(asserted to be exactly one substitution site) turns that run red at
1 failed 3 passed,and the failure is the new test. The file was restored from a copy kept outside the
repository and the checksum matched.
Suites run for this change:
test_remote_crew_chaining.py145 passed,test_instances.py410 passed, andtest_remote_crew_adopt.py+test_remote_crew_peer_slots.py+test_instance_credential_pairing.py+test_remote_crew_capabilities.py201 passed.The teardown state window
test_stop_leaves_connected_before_its_first_awaitparks a real_SshTunnelinside_terminate()and asserts the live status has already left CONNECTED while the port isstill unreleased.
test_a_failed_teardown_restores_the_previous_statecovers bothfailure shapes: a raising teardown restores CONNECTED, a cancelled one stays STOPPED.
Mutation-proved: moving the assignment back after the awaits (asserted to be exactly one
substitution site per edit) gives
2 failed 1 passed, and both new tests are thefailures, so each is load-bearing. Restored from a copy kept outside the repository with a
matching checksum.
Suites run for this change:
test_instances.py+test_fargate_closed_forward_remedy.py+test_remote_crew_chaining.py589 passed, andtest_remote_crew_execution.py+test_instance_credential_pairing.py+test_remote_crew_peer_slots.py+test_remote_crew_adopt.py359 passed.5. Any other suggestions on the work
releases every hold because a socket cannot outlive its process, and the credential
stays valid because the remote crew issued it and no revoke primitive exists here, so
the exposure runs to the lease deadline -- one capped lent TTL (30m), after which the
credential is dead and the next start re-takes every hold. Closing it rather than
bounding it needs either a holder that survives the process (a separate deployable
component, with its own lifecycle for the case where the gateway never returns) or a
revoke endpoint on the remote crew (a new cross-gateway contract that an older remote
would not serve, so the window would silently return). Both are follow-up architecture
items; capping the lifetime is the part that lives inside this mechanism. Tracked as
instances: a gateway exit frees a lent hop port while its chained credential stays live #14461, which records a THIRD option this description did not have:
_peer_gateway_idalready reads the peer's
gateway_idover the forward with no credential attached, sorequiring it to match before any credential travels closes the window without a new
component and without a cross-gateway contract.
token_validatesmust not be read as a mitigation for that window -- it is worse thanneutral. Its contract sends
GET /api/status?token=...through the forward, so againsta process squatting on a freed port the probe is itself an extra disclosure, and a 2xx
answer makes the hub hand the token to the browser as well. Any identity check has to
run BEFORE it rather than rely on it.
dying and the monitor seam noticing, the lent port is free and nothing holds it, so no
arrangement of the arm relative to the teardown can close it -- only a gateway-owned
listener bound for the whole lease, with the forward proxied or swapped behind it, and
that changes how every forward binds its port rather than how this one is protected.
Left as a follow-up architecture item deliberately.
because anything breaks at 3. Raising it is a one-constant change plus a tree that
indents further; a test asserts the constant so the expectation moves with it.
chain_parent_not_sshis a product limitation rather than a safety rule: achained forward needs a port on the parent to aim at, which a non-ssh transport
does not give it. Supporting SSM-reached parents is separate work.
frame around it, so it reads in English on a non-English dashboard exactly as the
existing backend error notices beside it do. Giving it a translated frame means a
new key in fourteen catalogs and is worth doing as part of a pass over those
notices generally, not for this one alone. The chained path label DOES read from
the catalog, because that string is ours rather than the gateway's.
Disposition of the advisory review findings on this revision
The First Principles lane returned CONCERNS on four surfaces it counted as having
no reader. Two now have one and two are kept deliberately:
ChainedCrewRefusalis now the module store's own value, so the type has areader rather than only a definition.
v: 1field on both frame messages is kept unread on purpose. It is a wireversion on a boundary between two INDEPENDENTLY deployed gateways, where the
older side is the one that has to recognise a payload it does not understand; a
version added after the first mismatch cannot help that side. It costs one
integer.
removed_chainedandstrandedare REMOVED from the DELETE success body. Thelane is right that a field on a public response with no reader is a defect rather
than a courtesy, and the earlier argument for keeping
removed_chained-- that aclient cannot reconcile a cascade it was not told about -- describes a client that
does not exist: neither
removeInstancecaller reads the body at all. What theywere evidence OF is kept, through witnesses that are not the wire: the cascade is
proved by the rows that are gone, and the residual by the gateway's warning log
plus a counter showing the second pass ran. A mutation that puts the field back is
invisible to the suite, which is the measurement that says nothing depended on it.
be
connectedin its own right while an ancestor hop is down, and the row then reads"unreachable" on screen -- while a
titlebuilt from the raw state said "connected".Since
titleis also what a screen reader falls back to, the two audiences were toldopposite things about one row, which is the same class of false label as the code
comments corrected in this revision rather than a cosmetic slip. Both sites are fixed;
the flagged one and its relayed twin.
It is a pre-existing seam rather than part of this feature: a non-SSH crew is
announced with its SSM target in the
ssh_hostslot, which the grammar refuses, so aconnect that succeeded reports
instance_invalid. Today's refusal is the rightOUTCOME reached for the wrong reason -- a chained crew rides a loopback port on the
parent and a fargate crew has none -- and the fix is to gate the announce on the
transport, which is a change to the announce contract rather than to anything this PR
introduces.
gateway_id(create=False)has no production caller and the parameter could go.It is left because the symmetry with
beacon.install_id(create=False)is whatmakes the two identity readers obviously the same shape, and removing it is a
separate tidy-up rather than part of this feature.
The reversal's decision record is issue #13744, this PR's own
Fixestarget.The lane asked for the "single-level by design" reversal to rest on something other
than this description's say-so, and it does: Raymond filed #13744 as an
enhancement, and its first paragraph is the reversal in those words -- "RemoteCrew is single-level by design today: a connected instance cannot connect onward to
another instance. Allow chaining ... while refusing any connection back up the chain
and capping how deep it can go" -- with the reason above it, that B and not A is the
one which can reach C, "which fails when only B holds C's key". The spec now cites
it where the reversal is described.
One measurement belongs beside that citation, and it is in the spec too. No
sentence in the spec tree ever declared Remote Crew single-level. Grepping
origin/main'sdocsforsingle-levelandsingle levelreturns one hit, incomputer-use.mdand about write access to a parent directory. The constraint livedentirely in the frontend --
SettingsPage.tsxdropped the Remote Crew tab frombaseTabswhenisEmbeddedPane()-- with no depth, cycle or width check on thegateway side at all, and inside a pane the tab vanished with no explanation. So what
reads as a reversed design decision was a frontend side effect that presented as a
missing feature, which is what #13744 says. Chaining adds the server-side guards a
real single-level rule would have needed to begin with.
The UX lane's watch items are FIXED in this revision rather than deferred.
An unreachable chained row no longer keeps a green "connected" label. When an
ancestor's hop drops, the row is dimmed -- and it used to go on reporting its own
state beside the dimming, so two cues said opposite things for the whole outage over
a claim the caller had already contradicted. It now reads
unreachable -- <hop> is down, naming the hop because the row the reader has to go fix is a different one.The choice came out of the JSX into an exported
rowStateLabel, which is what givesit a test at all.
Every chain refusal is a whole sentence naming the crew and the action. Each one
opened lowercase and mid-clause -- a "that would put ..." fragment with no subject --
and the panel renders the gateway's text raw and can show it after
the connect already looked like it succeeded -- so the reader saw a fragment naming
neither what was refused nor what had been doing it. The announced crew's name
arrives as a new parameter, because only the caller holds it. Two earlier watch
items stay fixed: the chip keeps the crew's own name under a long parent name, and
the refusal survives the panel being closed. Both are photographed above.
via_remote_idis not PATCH-editable, for the same reasonvia_instance_idisnot: re-pointing it addresses a different crew, whose depth was never checked.
Fixes #13744