Skip to content

fix(pod): pod up waits for the credential, not just for health - #14318

Merged
bolichen97 merged 1 commit into
mainfrom
fix/pod-up-waits-for-credential
Sep 28, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/pod-up-waits-for-credential

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: kirocrew pod up returns a token that actually works, instead of intermittently reporting the pod has no credential and blaming the worktree.

no linked issue: this was found by driving a CI failure on another PR, not from a filed report. The two numbers referenced below are PRs, not issues this change closes -- #14173 is the pull request whose Windows canary failure started the investigation, and #13551 is where a separate macOS race originates.

The Pod Boot Canary (Windows) job fails intermittently at
test/test_pod_windows_boot.py::test_a_real_pod_boots_under_task_scheduler_and_leaves_nothing_behind,
on assert up.returncode == 0, with:

pod up failed:

pod: no internal-API credential for pod 'KiroCrew' — is it running? Looked for
C:\kcpb590d0c\pods\KiroCrew\run\gateway-8743.secret, then
C:\kcpb590d0c\pods\KiroCrew\.local_secret.

assert 1 == 0

"is it running?" is the opposite of true. The pod was serving /api/health the
whole time — that is why the wait returned and the mint ran.

Why it matters

pod up is how every Dev Fleet workflow reaches a preview gateway, and the failure
is intermittent: the bind-to-credential gap measured 31 ms on an idle 16-core Windows
host, so the canary went red roughly 1.4% of the time (2 failures in 143 completed
jobs) and did so on unrelated branches. An intermittent red on a required check is
worse than a deterministic one -- it teaches everyone to rerun, which hides the next
real failure. The error message also blamed the wrong component, sending whoever hit
it to debug their worktree instead of the wait.

Not a goal

  • Gating readiness on the gateway's pid/run-marker write. That write is
    fire-and-forget and can also land after pod up returns, which on this host makes
    up take the PodOwnershipUnproven path and hand back an empty token. Depending on
    it at boot would contradict its own documented contract that nothing at boot does,
    so it wants its own decision rather than a ride here.
  • Fixing the code_review_sage layout_lock race that failed one macOS shard on an
    earlier head. It comes from fix(sage): serialize the data-layout self-heal across processes #13551, an ancestor of both this branch's base and
    main, and it wants its own issue.
  • A general-purpose non-destructive teardown API. runtime.halt_pod covers exactly
    what a failed up owes -- stop the service, keep the home -- and deliberately does
    not grow options for callers that do not exist yet.
  • Changing what a successful pod up does. Only the failure paths move.

Backwards compatibility

Compatible. The readiness wait becomes stricter about what it accepts as success, so
a pod up that previously returned a token it could not use now reports why instead;
no invocation that works today stops working. --wait-secs and
KIROCREW_POD_HEALTH_SECS keep their spellings, their resolution order and their
5..3600 clamp. One user-visible behaviour does change and is declared under Behaviour
changes beyond the race itself
: a failed pod up no longer reclaims the isolated home
of a pod it did not create. That is strictly less destructive than before, so nothing
that used to succeed now fails -- an operator who relied on a failed up deleting a
pre-existing home now runs kirocrew pod down, which the failure message names.

Root cause

pod up treated "/api/health answers" as "the pod is up", then minted
immediately. But the gateway publishes its internal-API credential only AFTER the
bind. In start_dashboard the order is:

  1. await _start_site(site, port) — the listener is accepting from here, and the
    health route answers
  2. _arm_listener_guard, _kick_crewmate_prune
  3. await run_in_executor(start_deferred_app_backends) — spawns app backend
    subprocesses
  4. await _start_unix_site(...)
  5. await run_in_executor(_write_instance_credentials, ...) — the credential

Steps 2-5 are all awaits, so the event loop is free to serve health throughout
them while the credential does not exist. _wait_healthy returned the instant it
saw 200/401/403, and _up minted on the next line. When step 5 landed after that
read, pod up exited 1.

Measured from inside the gateway on an idle 16-core Windows host (temporary probe,
not in this diff):

LISTENER_BOUND            +0.000s
BEFORE_CREDENTIAL_WRITE   +0.000s
CREDENTIAL_PUBLISHED      +0.031s

31 ms — which is why it essentially never reproduces locally and why the failure
rate on shared runners is low but nonzero.

Evidence it is not the PR that surfaced it

Found while investigating #14173, whose diff touches no pod, Windows, Task
Scheduler or gateway code.

  • Attempt 2 of the same run, same head, was green — job 108506136660
    succeeded where 108503209478 failed. Same commit, same workflow, different
    runner, so neither the diff nor the base is a deterministic cause.
  • The same failure exists on an unrelated branch. feat/app-update-cli head
    1f6ff7a842 (job 108498857158) failed with a byte-identical message and the
    same ~39.7s call duration.
  • Rate: of 143 completed Pod Boot Canary (Windows) jobs across a ~2.5 hour
    window of CI history, 2 failed — both with this message. ~1.4%.

The timing exposes a real ordering defect in the product, not a runner fault, so
it is fixable rather than something to accept.

What changed (motivation → approach → change)

Readiness means serving AND credentialled. _wait_healthy's success condition
includes the credential, sharing the one existing wall-clock budget rather than
adding a second timer — --wait-secs and KIROCREW_POD_HEALTH_SECS still bound
the whole wait, and no new budget number is introduced.

Presence is not publication, and not generation. clear_marker runs only on a
graceful shutdown and the stale-marker prune deliberately never removes the
credential, so a pod home that survived a crash still holds the previous
generation's run/gateway-<port>.secret. A presence test would call that ready on
the first poll and hand the mint a secret the new gateway never minted. So the
reader is published_credential, returning the VALUE, and pod up captures the
outgoing credential immediately before start_pod and requires the live one to
differ. Content rather than mtime, so no clock or filesystem timestamp granularity
is involved; the capture's position before start_pod is pinned by a test, because
reading it after the start would compare a credential against itself and silently
disable the guard. An already-active pod supersedes nothing — it is not a new
generation, so its current credential is accepted rather than waited out for the
whole budget.

One reader, one path pair. published_credential and _pod_mint_secret both
go through _read_pod_secret_file over _pod_secret_candidates, so they cannot
drift into reporting a pod ready whose credential the mint cannot find. That reader
pins utf-8 instead of the locale default — bytes that are invalid UTF-8 but valid
cp1252 would otherwise be a credential on a Windows runner and an error on a UTF-8
host — and treats OSError and ValueError alike, because UnicodeDecodeError is
a ValueError: a corrupt per-listener file then falls through to the shared
.local_secret instead of becoming a traceback out of a mint or a boot poll.

Exhaustion is attributed from the last observation, and only once serving has
persisted.
A gateway that answered 200 once inside the bind-to-credential window
and then died or served 5xx is not a credential problem. And a gateway that binds in
the final second of its budget is serving on the last poll while having had no
chance to publish at all — that is a slow boot whose remedy is a bigger budget, so
the credential verdict requires the serving state to have survived one contiguous
poll interval, which is the smallest span that carries the claim it makes. A lapse
resets the span. Below it the wait returns the slow-boot input rather than its own
serving code, because membership in (200, 401, 403) is the caller's whole success
test. For the same reason last_http records non-serving codes only.

HEALTH_NO_CREDENTIAL is the verdict for a pod that served without publishing, and
its message names BOTH remedies: the journal for a write error, and the
--wait-secs / env escalation for a late publish.

Two things this deliberately does NOT do:

  • It does not reorder the gateway's startup. Moving the credential write earlier
    would narrow the window; gating readiness on the credential removes it. The boot
    path is pinned by startup-order tests and is the wrong place to take risk for an
    effect this change already achieves completely.
  • It does not touch the canary. The test was correct; pod up was not. No skip, no
    retry loop, no change to the 3 passed pin.

Tests

Run on Windows Server 2025, Python 3.12.10.

Red-first, at the level of the failing assertion. A temporary probe added an
await asyncio.sleep(6.0) between the bind and the credential write — modelling a
runner whose intermediate startup work is slow, with the loop left free exactly as
the real awaits leave it. Before the fix:

HTTP -> credential : +6.016s
pod up rc=1  stderr=pod: no internal-API credential for pod 'kcpod1' — is it
running? Looked for C:\kcpb6de2d6\pods\kcpod1\run\gateway-8738.secret, ...

After the fix, same 6.0s window, same probe:

HTTP -> credential : +6.015s
pod up rc=0

The canary itself, under that same 6-second window (200x the natural one):
3 passed. With the probe removed entirely: 3 passed. CI's own count gate
grep -qE '(^|[^0-9])3 passed' matches.

Unit negative controls, one per behaviour. Reverting only the credential
condition fails 8 tests on behaviour, not on a missing symbol — assert 1 == 4
(returned on the first poll), assert 200 == -3 (budget exhaustion handed the
caller a success code), assert 200 == -1 (a crash-looping gateway behind a
serving predecessor was no longer detected). Removing the generation comparison
fails the stale-predecessor and fail-closed tests. Dropping the span requirement
fails the final-second and comeback tests. Removing the utf-8 pin fails the
cp1252-decodable-but-invalid-UTF-8 parametrization on a cp1252 host.

Suites: 1299 passed / 260 skipped / 6 xfailed across test_pod,
test_pod_health_wait, test_pod_api, test_pod_windows, test_pod_windows_run,
test_pod_launchd, test_pod_e2e_harness_paths, test_dev_fleet_server_coverage
and test_instances under PYTHONUTF8=0 (cp1252, the CI runner's locale); 531
passed re-run under PYTHONUTF8=1. black --check and flake8 clean.

Two defects the tests caught rather than review: HEALTH_NO_CREDENTIAL was first
written as the literal -2, which is rt.HEALTH_FOREIGN's value, so pod up told
the operator to choose a different port for a pod whose port was fine — it is now
derived from that constant and pinned distinct by a test. And the
undecodable-credential test passed locally while failing on CI purely because this
host runs PYTHONUTF8=1; that is what exposed the locale-dependent read.

What I could not verify

  • I could not reproduce the failure on this host without inducing the window.
    The natural gap here is 31 ms and the observed CI rate is ~1.4%, so an unforced
    local reproduction would need hundreds of boots. The induced-window experiment is
    what establishes causation; the 31 ms measurement is what explains the rate.
  • I have no gateway-side log from either failing CI run. The canary uploads only the
    pytest log, and pod up fails before the step that would print the pod's own
    journal. So those two failures are attributed by mechanism and by their verbatim
    message, not by a log from those runs.
  • Separately noticed and NOT addressed here: the gateway's pid/run-marker write is a
    fire-and-forget background task, so it can also land after pod up returns. On
    this host that makes up take the PodOwnershipUnproven path and return an empty
    token (stderr says so; exit stays 0). Gating readiness on the marker too would make
    something at boot depend on a write documented as "nothing at boot depends on it",
    so it wants its own decision.
  • Also unrelated and NOT fixed here: macOS Tests / Backend Tests (macOS) (1) failed
    once on an earlier head in code_review_sage's layout_lock — os.open relative
    to a pinned dir_fd returned ENOENT with O_CREAT set, meaning the pinned
    directory was replaced under it by one of six racing children. That comes from
    fix(sage): serialize the data-layout self-heal across processes #13551, an ancestor of both this PR's base and main, and it passed on rerun with
    no change. It is a real main-side race and wants its own issue.

Behaviour changes beyond the race itself

Three changes ride along with the fix. None is a side effect of it, so each is
declared here rather than left for a reader to infer from the diff.

A failed pod up no longer stops a pod this invocation did not create. The old
contract was "up means healthy", and any unsatisfied readiness wait called stop_pod
to avoid leaking a half-booted service. That is safe only while every pod reaching the
wait was started by this command. It is not: stop_pod reaches cleanup_home, which
rmtree's the pod's isolated HOME with its sessions and config, and nothing restores it.
The teardown is now conditioned on this invocation's own start_pod having succeeded
AND the home having held no state beforehand, and that condition is checked on all five
failure exits rather than on one verdict. Deliberately not keyed on is_active: that
probe shells systemctl is-active --quiet, which succeeds only for active, so a pod
in activating or in Restart=on-failure backoff reads as not running and a
pre-existing crash-looping pod would be treated as one this command created. Every
failure exit now also says the pod was not stopped and names kirocrew pod down as the
deliberate way to retire it — it states only that, never that the pod is running, since
a spared pod whose unit has genuinely failed is not.

The ceiling-aware wait hint also changes the pre-existing "still starting" message.
_health_wait_secs clamps to a 3600s ceiling, so at the ceiling the old advice to
double --wait-secs named a value the resolver reduces straight back to the one that
just failed. Both exhausted-budget messages now share one helper that reports the
ceiling instead. The slow-boot message was not part of the race, but it carried the
identical defect one line away, and fixing one while leaving its twin would have been
the less honest change.

The foreign-port latch now clears on proven handover. A predecessor seen on an
early poll used to be reported at the deadline even after this pod's own gateway had
won the port back. The latch clears only when port_owner answers OWNER_POD, because
a serving health code proves only "not provably foreign" — health returns the status
unchanged for OWNER_UNPROVEN, which is what an unavailable listener lookup leaves.
Reachable only through the new exhaustion path, so it is a rider rather than a
prerequisite.

Pattern harvest

Rule candidate: a liveness signal is not a readiness signal. When a process
publishes a resource AFTER it starts serving, a client must wait for the resource it
is about to use, not for the serve — otherwise the gap between the two becomes an
intermittent failure whose error message blames the wrong thing.

Rule candidate: when several verdicts share one integer return channel, derive new
sentinels from the existing ones and pin them mutually distinct in a test; a literal
silently routes the new verdict into an old branch.

Rule candidate: read a credential or any machine-written file with an explicit
encoding. read_text() without one decodes with the host locale, so the same bytes
are valid on a cp1252 runner and an error on a UTF-8 host, and the test that would
catch it passes or fails depending on which machine runs it.

@bolichen97
bolichen97 requested review from a team and Zedmor September 26, 2026 23:56
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 133e54e4e811d841b726681cbec16e8f49825997 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 133e54e

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 133e54e4e811d841b726681cbec16e8f49825997 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify before merge: the halt_pod platform premise — that launchd.stop, win_backend.stop, and the Linux stale-hook refresh stop the unit WITHOUT reclaiming the home; if wrong on any OS, the "keep the home" path deletes the state this change exists to protect.

What this change ships

Inventory (9 items) — 9 justified

Intent: make kirocrew pod up hand back a usable token instead of intermittently minting against an unpublished credential and blaming the worktree. FIX (provenance: measured 31ms bind-to-credential window, 2/143 canary failures, a reproduction test that fails on base).

  1. pod up readiness now requires a published credential, not just a serving /api/health — justified
  2. Failed up no longer reclaims a home it did not create; stops-only via halt_pod when it started the unit over pre-existing state — justified (coupled: the credential verdict newly lets a serving pod reach cleanup)
  3. New HEALTH_NO_CREDENTIAL / HEALTH_SERVING_TOO_BRIEFLY verdicts and messages — justified
  4. left_running teardown notice appended to every failure verdict — justified (teardown is now conditional, so the notice must be too)
  5. New runtime.halt_pod (stop, keep home), one consumer, deliberately un-generalized — justified
  6. published_credential reader + _pod_secret_candidates/_read_pod_secret_file shared with the mint (dedup) — justified
  7. _wait_escalation_hint caps escalation advice at the ceiling instead of naming wait_secs*2 — rides along (pre-existing advice defect; harm named, derived from the ceiling clamp — inventory only)
  8. utf-8 pin on the credential read (was locale-default) — justified (part of the Windows cp1252 failure)
  9. Foreign-latch clearing via port_owner while serving — justified (coupled: the wait now loops past the first serving poll)

[FIRST-PRINCIPLES-REVIEWED] 133e54e

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 133e54e4e811d841b726681cbec16e8f49825997 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 133e54e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 133e54e4e811d841b726681cbec16e8f49825997: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 133e54e4e811d841b726681cbec16e8f49825997 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 133e54e

Verdict parsed from the review's SHA-scoped output markers for commit 133e54e4e811d841b726681cbec16e8f49825997.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 133e54e4e811d841b726681cbec16e8f49825997: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision labels Sep 27, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 27, 2026 01:56
@bolichen97

bolichen97 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Accepted, fix staged locally: span=919acf5166fa

The argument holds and the consequence is worse than a wording problem. The bind-to-credential gap is seconds wide on a loaded host, so a gateway that binds at ~t=89s of a 90s budget is serving on the final poll while having had no chance at all to publish. Keying on that poll alone blamed a credential write that never ran, and because the credential branch is tested before still_starting, the only remedy that helps a late publish — a bigger budget — was withheld. The previous revision even pinned that suppression with assert "--wait-secs" not in err, so the test was protecting the defect.

Both halves of the suggested fix are taken, because each covers a case the other does not.

The verdict now requires the serving state to have survived one contiguous poll interval. That is the smallest span that can carry the claim the verdict makes: the pod was watched serving, a full poll was waited out, and no credential appeared. A lapse resets it, so a gateway that served, dropped and came back in the final second is timed from the comeback rather than credited with the earlier span. Below that span the wait returns the slow-boot input instead of its own serving code — necessary, because membership in (200, 401, 403) is the caller's whole success test and this pod has no credential, so returning 200 would report success into a mint that cannot work.

The message now carries both remedies: the journal for a write error, and the --wait-secs / KIROCREW_POD_HEALTH_SECS escalation for a late publish. The suppression assertion is inverted to assert the remedy is present.

Pinned red-first: dropping the span requirement fails test_a_gateway_that_binds_in_the_final_second_is_a_slow_boot and test_a_serving_state_that_lapsed_is_timed_from_the_comeback.

Stated plainly: this is not on head 292332a yet. Code Review / PR Hygiene hard-requires 1-2 commits and this branch exceeds that, so it must be squashed before any head can be green, and pushing this alone would spend a full matrix plus every lane on a head that still cannot pass the gate. The squash needs a force-push the repository owner has been asked to authorize; every staged fix ships in that one commit and I will re-answer against the resulting head.

@bolichen97

bolichen97 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Accepted, fix staged locally: span=eafee42a5123

Correct and entirely mine: the previous revision renamed the reader to published_credential and left _pod_secret_candidates' docstring pointing at credential_published, a name that exists nowhere. That docstring is the single place stating the reader/mint path-pair invariant, so the one cross-reference a maintainer would follow to check that invariant was dangling.

The reference now names published_credential, and the surrounding sentence is reworded to match what that function actually is — a reader that reports what a caller would find — rather than the boolean predicate the old name described.

Stated plainly: this is not on head 292332a yet, for the same reason as the sibling finding. Code Review / PR Hygiene hard-requires 1-2 commits, so this branch has to be squashed before it can be green, and the squash needs a force-push the repository owner has been asked to authorize. Every staged fix ships in that single commit and I will re-answer against the resulting head.

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision 133e54e4e811d841b726681cbec16e8f49825997 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Accepted, fix staged locally: span=c7d8344b8408

Correct, and it is a consequence of the previous round's own fix rather than an old wording slip. Once the attribution moved to the final poll, the verdict stopped checking that the gateway served throughout, so "served on :PORT for the whole Ns" asserted something the wait no longer establishes.

The message now reads that the gateway was still serving on that port when the readiness budget expired but had published no internal-API credential, and the test asserts the new wording and explicitly asserts the overclaim is absent, so the message and the verdict cannot drift apart again.

Stated plainly: this is NOT on head 292332a yet. It sits in a local commit. Code Review / PR Hygiene hard-requires 1-2 commits and this branch already exceeds that, so the branch has to be squashed before it can be green, and pushing this fix on its own would spend a full check matrix plus every review lane on a head that still cannot pass that gate. The squash needs a force-push, which I have asked the repository owner to authorize; every staged fix ships in that single commit and I will re-answer against the resulting head.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Accepted, scheduled with the squash: span=627f7d7e6fb3

The observation is right: the body still describes the second revision, so it names credential_published as a boolean and mentions neither the generation-supersede check nor the last-observation attribution. The record does not currently match what would ship.

I am deliberately not editing the body this minute, and the reason is mechanical rather than reluctant: a body edit re-triggers the same-name workflows and cancels the runs in flight, and one review lane is still re-reading this head. Editing now would discard that verdict and spend another full round on a head that cannot pass Code Review / PR Hygiene regardless, because that gate hard-requires 1-2 commits and this branch already exceeds it.

The body is rewritten in the same step as the squash, so the description, the single commit and the head all land together and the record matches exactly what ships. The squash needs a force-push, which the repository owner has been asked to authorize.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Agreed on the diagnosis, remedy pending authorization: span=afa5d8d8b279

This is the third head on which the rides-along artefact has been raised, under a third span id, and the finding's own reading is the correct one: the diff base is stale, so a diff taken against the fork point reproduces commits that landed on main afterwards. Two of the three clusters it names are exactly that — 53bc123 (feat(desktop): re-land the bundled kiro-cli, #13207) and 8ab7636 (ci(readiness): drop the fork lanes, #14164) are ancestors of origin/main and of neither this branch's base 962247d7a5 nor its head. The third, 5f06cdf, is this branch's own second commit rather than one of main's, which is itself a symptom of the same stale-base arithmetic.

Because the id keeps changing while the mechanism does not, I am treating this as one recurring cause and not patching instances: this PR's payload is six files — src/kiro_crew/pod/cli.py, src/kiro_crew/pod/runtime.py, docs/system-specs/modules/cli.md, test/test_pod.py, test/test_pod_health_wait.py, test/test_pod_launchd.py — and none of the cited work is in it, so there is nothing to justify or describe.

The remedy is a rebase onto current origin/main, which removes the artefact at its source and which I am about to need anyway: Code Review / PR Hygiene hard-requires 1-2 commits, so this branch must be squashed to go green. Both are one history rewrite and therefore one force-push, which the repository owner has been asked to authorize. It is the only step left that I cannot take myself.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Agreed on the diagnosis, remedy pending authorization: span=afa5d8d8b279

This is the third head on which the rides-along artefact has been raised, under a third span id, and the finding's own reading is the correct one: the diff base is stale, so a diff taken against the fork point reproduces commits that landed on main afterwards. Two of the three clusters it names are exactly that — 53bc123 (feat(desktop): re-land the bundled kiro-cli, #13207) and 8ab7636 (ci(readiness): drop the fork lanes, #14164) are ancestors of origin/main and of neither this branch's base 962247d7a5 nor its head. The third, 5f06cdf, is this branch's own second commit rather than one of main's, which is itself a symptom of the same stale-base arithmetic.

Because the id keeps changing while the mechanism does not, I am treating this as one recurring cause and not patching instances: this PR's payload is six files — src/kiro_crew/pod/cli.py, src/kiro_crew/pod/runtime.py, docs/system-specs/modules/cli.md, test/test_pod.py, test/test_pod_health_wait.py, test/test_pod_launchd.py — and none of the cited work is in it, so there is nothing to justify or describe.

The remedy is a rebase onto current origin/main, which removes the artefact at its source and which I am about to need anyway: Code Review / PR Hygiene hard-requires 1-2 commits, so this branch must be squashed to go green. Both are one history rewrite and therefore one force-push, which the repository owner has been asked to authorize. It is the only step left that I cannot take myself.

@bolichen97
bolichen97 force-pushed the fix/pod-up-waits-for-credential branch from 292332a to f03f1b5 Compare September 27, 2026 03:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@bolichen97
bolichen97 force-pushed the fix/pod-up-waits-for-credential branch from d8bfad8 to 6050883 Compare September 27, 2026 05:40
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@bolichen97
bolichen97 force-pushed the fix/pod-up-waits-for-credential branch from 6050883 to c4defd4 Compare September 27, 2026 05:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@bolichen97
bolichen97 force-pushed the fix/pod-up-waits-for-credential branch from c4defd4 to 8c982ec Compare September 27, 2026 06:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Already implemented: span=b1f5ed632860

The coupling is real and worth pinning: the supersede check can only tell a new
generation's credential from its predecessor's if a starting gateway MINTS a fresh
one, and if that ever changes to reuse-what-is-on-disk the failure is silent and
expensive -- every restart of a crashed pod would read the stale secret as "not yet
superseded" and burn the whole readiness budget before failing.

That pin is in this revision, on the gateway side as suggested rather than in the pod
package: test_the_gateway_mints_a_fresh_credential_per_generation reads
dashboard/server.py, collects EVERY _internal_secret = line, and requires each to
be os.urandom(16).hex(). Collecting all of them rather than asserting two known
offsets is deliberate -- a third entrypoint that mints differently is exactly the
drift the guard exists to catch, and a line-number assertion would go stale on the
next edit to that file.

Verified it actually catches drift: breaking one of the two mint sites to a derived
value fails the test, and server.py is byte-unchanged afterwards. This PR adds no
production code there -- the guard reads that file, it does not modify it.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Accepted as a declared rider: span=1fa41b7b8269

Correctly characterised, and the declaration the lane asks for is in the PR body
under "Behaviour changes beyond the race itself".

It rides along because it is not separable from the change that reaches it. Adding a
failure verdict a SERVING pod can reach is what first made up's teardown able to
delete a home that predated the command: before the credential joined readiness, a
serving pod returned success immediately and never reached that path at all. So the
rider is not an unrelated improvement bundled in -- it is the guard for a hazard this
change newly exposes, and shipping the verdict without it would introduce an
irreversible cleanup_home on a pod the caller only asked about.

Scope is held to that: stopping is owed whenever this invocation started the gateway,
because the unit is Restart=on-failure/RestartSec=5 with no StartLimit override
and walking away leaks a respawning service; deleting is permitted only when this
invocation also created the home. One consumer, no new abstraction, and a
parametrised test pins that the other verdicts keep their existing cleanup so the
carve-out cannot quietly widen.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Accepted as a declared rider: span=36d1be1a87e1

Agreed on all three counts -- micro-fix, derived from the resolver's own clamp,
harm-free -- and it is declared in the PR body alongside the other rider.

Kept in rather than split out because it is the same sentence the race fix rewrites.
The new verdicts hand the operator a budget-raise remedy, and that remedy is
generated by one helper shared with the pre-existing slow-boot message; advising a
value the resolver would clamp away tells someone at a 2000s budget they are "at the
3600s ceiling" and suppresses the raise that would have worked. Fixing the helper
while adding its second caller is smaller than shipping a new caller onto known-wrong
advice.

The test asks the resolver rather than re-deriving the cap: it asserts
_health_wait_secs returns the advised value unchanged, which is the property that
matters (typing the number back in produces that number) and cannot drift from the
resolver's own arithmetic.

A pod's gateway publishes its internal-API credential AFTER its listener is
bound, so from the bind until the end of the remaining post-bind startup work
the port already answers `/api/health` while the credential does not exist yet.
`_wait_healthy` returned on the HTTP status alone and `_up` mints immediately
after it returns, so on a slow runner the mint read a credential that was still
unwritten and `pod up` exited 1 with:

    pod: no internal-API credential for pod 'X' - is it running? Looked for
    .../run/gateway-<port>.secret, then .../.local_secret.

"is it running?" was the opposite of true: the pod was serving the whole time.
That is the intermittent red on the Pod Boot Canary (Windows) job.

Readiness now means serving AND credentialled, sharing the one existing
wall-clock budget rather than adding a second timer, so `--wait-secs` and
KIROCREW_POD_HEALTH_SECS still bound the whole wait and no new number is
introduced.

Presence is not enough: `clear_marker` runs only on a graceful shutdown and the
stale-marker prune deliberately never removes the credential, so a pod home that
survived a crash still holds the previous generation's secret. `pod up` therefore
captures the outgoing credential immediately before `start_pod` and requires the
live one to differ. Content rather than mtime, so no clock or filesystem
timestamp granularity is involved. An already-active pod supersedes nothing: it
is not a new generation, so its current credential is accepted rather than waited
out.

The reader and the mint go through one helper over one path pair, so they cannot
drift into reporting a pod ready whose credential the mint cannot find. That
helper pins `utf-8` instead of the locale default, because bytes that are invalid
UTF-8 but valid cp1252 would otherwise be a credential on a Windows runner and an
error on a UTF-8 host, and it treats `OSError` and `ValueError` alike so a corrupt
per-listener file falls through to the shared `.local_secret` rather than becoming
a traceback out of a mint or a boot poll.

Exhaustion is attributed from the last observation and only once the serving state
has survived a contiguous poll interval. A gateway that answered once inside the
bind-to-credential window and then died is not a credential problem, and a gateway
that binds in the final second of its budget has had no chance to publish at all --
that one is a slow boot whose remedy is a bigger budget. Below that span the wait
returns the slow-boot input rather than its own serving code, because membership in
`(200, 401, 403)` is the caller's whole success test. The message names both
remedies: the journal for a write error, the budget escalation for a late publish.

`last_http` records non-serving codes only, for the same reason: once a serving
code stops returning immediately it could otherwise be handed back at exhaustion
for a pod that has stopped serving and has no credential.

Also updates the owning `cli.md` spec row, which described the budget as waiting on
`/api/health` alone and named only the old exhaustion verdict.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants