Repository navigation
fix(pod): pod up waits for the credential, not just for health - #14318
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 133e54e |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify before merge: the What this change shipsInventory (9 items) — 9 justifiedIntent: make
[FIRST-PRINCIPLES-REVIEWED] 133e54e |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
Accepted, fix staged locally: span=919acf5166fa The argument holds and the consequence is worse than a wording problem. The bind-to-credential gap is seconds wide on a loaded host, so a gateway that binds at ~t=89s of a 90s budget is serving on the final poll while having had no chance at all to publish. Keying on that poll alone blamed a credential write that never ran, and because the credential branch is tested before Both halves of the suggested fix are taken, because each covers a case the other does not. The verdict now requires the serving state to have survived one contiguous poll interval. That is the smallest span that can carry the claim the verdict makes: the pod was watched serving, a full poll was waited out, and no credential appeared. A lapse resets it, so a gateway that served, dropped and came back in the final second is timed from the comeback rather than credited with the earlier span. Below that span the wait returns the slow-boot input instead of its own serving code — necessary, because membership in The message now carries both remedies: the journal for a write error, and the Pinned red-first: dropping the span requirement fails Stated plainly: this is not on head 292332a yet. |
|
Accepted, fix staged locally: span=eafee42a5123 Correct and entirely mine: the previous revision renamed the reader to The reference now names Stated plainly: this is not on head 292332a yet, for the same reason as the sibling finding. |
UX Review (Fable 5) — ⏭️ skippedRevision |
|
Accepted, fix staged locally: span=c7d8344b8408 Correct, and it is a consequence of the previous round's own fix rather than an old wording slip. Once the attribution moved to the final poll, the verdict stopped checking that the gateway served throughout, so "served on :PORT for the whole Ns" asserted something the wait no longer establishes. The message now reads that the gateway was still serving on that port when the readiness budget expired but had published no internal-API credential, and the test asserts the new wording and explicitly asserts the overclaim is absent, so the message and the verdict cannot drift apart again. Stated plainly: this is NOT on head 292332a yet. It sits in a local commit. |
|
Accepted, scheduled with the squash: span=627f7d7e6fb3 The observation is right: the body still describes the second revision, so it names I am deliberately not editing the body this minute, and the reason is mechanical rather than reluctant: a body edit re-triggers the same-name workflows and cancels the runs in flight, and one review lane is still re-reading this head. Editing now would discard that verdict and spend another full round on a head that cannot pass The body is rewritten in the same step as the squash, so the description, the single commit and the head all land together and the record matches exactly what ships. The squash needs a force-push, which the repository owner has been asked to authorize. |
|
Agreed on the diagnosis, remedy pending authorization: span=afa5d8d8b279 This is the third head on which the rides-along artefact has been raised, under a third span id, and the finding's own reading is the correct one: the diff base is stale, so a diff taken against the fork point reproduces commits that landed on Because the id keeps changing while the mechanism does not, I am treating this as one recurring cause and not patching instances: this PR's payload is six files — The remedy is a rebase onto current |
|
Agreed on the diagnosis, remedy pending authorization: span=afa5d8d8b279 This is the third head on which the rides-along artefact has been raised, under a third span id, and the finding's own reading is the correct one: the diff base is stale, so a diff taken against the fork point reproduces commits that landed on Because the id keeps changing while the mechanism does not, I am treating this as one recurring cause and not patching instances: this PR's payload is six files — The remedy is a rebase onto current |
292332a to
f03f1b5
Compare
d8bfad8 to
6050883
Compare
6050883 to
c4defd4
Compare
c4defd4 to
8c982ec
Compare
|
Already implemented: span=b1f5ed632860 The coupling is real and worth pinning: the supersede check can only tell a new That pin is in this revision, on the gateway side as suggested rather than in the pod Verified it actually catches drift: breaking one of the two mint sites to a derived |
|
Accepted as a declared rider: span=1fa41b7b8269 Correctly characterised, and the declaration the lane asks for is in the PR body It rides along because it is not separable from the change that reaches it. Adding a Scope is held to that: stopping is owed whenever this invocation started the gateway, |
|
Accepted as a declared rider: span=36d1be1a87e1 Agreed on all three counts -- micro-fix, derived from the resolver's own clamp, Kept in rather than split out because it is the same sentence the race fix rewrites. The test asks the resolver rather than re-deriving the cap: it asserts |
A pod's gateway publishes its internal-API credential AFTER its listener is
bound, so from the bind until the end of the remaining post-bind startup work
the port already answers `/api/health` while the credential does not exist yet.
`_wait_healthy` returned on the HTTP status alone and `_up` mints immediately
after it returns, so on a slow runner the mint read a credential that was still
unwritten and `pod up` exited 1 with:
pod: no internal-API credential for pod 'X' - is it running? Looked for
.../run/gateway-<port>.secret, then .../.local_secret.
"is it running?" was the opposite of true: the pod was serving the whole time.
That is the intermittent red on the Pod Boot Canary (Windows) job.
Readiness now means serving AND credentialled, sharing the one existing
wall-clock budget rather than adding a second timer, so `--wait-secs` and
KIROCREW_POD_HEALTH_SECS still bound the whole wait and no new number is
introduced.
Presence is not enough: `clear_marker` runs only on a graceful shutdown and the
stale-marker prune deliberately never removes the credential, so a pod home that
survived a crash still holds the previous generation's secret. `pod up` therefore
captures the outgoing credential immediately before `start_pod` and requires the
live one to differ. Content rather than mtime, so no clock or filesystem
timestamp granularity is involved. An already-active pod supersedes nothing: it
is not a new generation, so its current credential is accepted rather than waited
out.
The reader and the mint go through one helper over one path pair, so they cannot
drift into reporting a pod ready whose credential the mint cannot find. That
helper pins `utf-8` instead of the locale default, because bytes that are invalid
UTF-8 but valid cp1252 would otherwise be a credential on a Windows runner and an
error on a UTF-8 host, and it treats `OSError` and `ValueError` alike so a corrupt
per-listener file falls through to the shared `.local_secret` rather than becoming
a traceback out of a mint or a boot poll.
Exhaustion is attributed from the last observation and only once the serving state
has survived a contiguous poll interval. A gateway that answered once inside the
bind-to-credential window and then died is not a credential problem, and a gateway
that binds in the final second of its budget has had no chance to publish at all --
that one is a slow boot whose remedy is a bigger budget. Below that span the wait
returns the slow-boot input rather than its own serving code, because membership in
`(200, 401, 403)` is the caller's whole success test. The message names both
remedies: the journal for a write error, the budget escalation for a late publish.
`last_http` records non-serving codes only, for the same reason: once a serving
code stops returning immediately it could otherwise be handed back at exhaustion
for a pod that has stopped serving and has no credential.
Also updates the owning `cli.md` spec row, which described the budget as waiting on
`/api/health` alone and named only the old exhaustion verdict.
Problem / Motivation
Goal:
kirocrew pod upreturns a token that actually works, instead of intermittently reporting the pod has no credential and blaming the worktree.no linked issue: this was found by driving a CI failure on another PR, not from a filed report. The two numbers referenced below are PRs, not issues this change closes -- #14173 is the pull request whose Windows canary failure started the investigation, and #13551 is where a separate macOS race originates.
The
Pod Boot Canary (Windows)job fails intermittently attest/test_pod_windows_boot.py::test_a_real_pod_boots_under_task_scheduler_and_leaves_nothing_behind,on
assert up.returncode == 0, with:"is it running?" is the opposite of true. The pod was serving
/api/healththewhole time — that is why the wait returned and the mint ran.
Why it matters
pod upis how every Dev Fleet workflow reaches a preview gateway, and the failureis intermittent: the bind-to-credential gap measured 31 ms on an idle 16-core Windows
host, so the canary went red roughly 1.4% of the time (2 failures in 143 completed
jobs) and did so on unrelated branches. An intermittent red on a required check is
worse than a deterministic one -- it teaches everyone to rerun, which hides the next
real failure. The error message also blamed the wrong component, sending whoever hit
it to debug their worktree instead of the wait.
Not a goal
fire-and-forget and can also land after
pod upreturns, which on this host makesuptake thePodOwnershipUnprovenpath and hand back an empty token. Depending onit at boot would contradict its own documented contract that nothing at boot does,
so it wants its own decision rather than a ride here.
code_review_sagelayout_lockrace that failed one macOS shard on anearlier head. It comes from fix(sage): serialize the data-layout self-heal across processes #13551, an ancestor of both this branch's base and
main, and it wants its own issue.runtime.halt_podcovers exactlywhat a failed
upowes -- stop the service, keep the home -- and deliberately doesnot grow options for callers that do not exist yet.
pod updoes. Only the failure paths move.Backwards compatibility
Compatible. The readiness wait becomes stricter about what it accepts as success, so
a
pod upthat previously returned a token it could not use now reports why instead;no invocation that works today stops working.
--wait-secsandKIROCREW_POD_HEALTH_SECSkeep their spellings, their resolution order and their5..3600 clamp. One user-visible behaviour does change and is declared under Behaviour
changes beyond the race itself: a failed
pod upno longer reclaims the isolated homeof a pod it did not create. That is strictly less destructive than before, so nothing
that used to succeed now fails -- an operator who relied on a failed
updeleting apre-existing home now runs
kirocrew pod down, which the failure message names.Root cause
pod uptreated "/api/healthanswers" as "the pod is up", then mintedimmediately. But the gateway publishes its internal-API credential only AFTER the
bind. In
start_dashboardthe order is:await _start_site(site, port)— the listener is accepting from here, and thehealth route answers
_arm_listener_guard,_kick_crewmate_pruneawait run_in_executor(start_deferred_app_backends)— spawns app backendsubprocesses
await _start_unix_site(...)await run_in_executor(_write_instance_credentials, ...)— the credentialSteps 2-5 are all
awaits, so the event loop is free to serve health throughoutthem while the credential does not exist.
_wait_healthyreturned the instant itsaw 200/401/403, and
_upminted on the next line. When step 5 landed after thatread,
pod upexited 1.Measured from inside the gateway on an idle 16-core Windows host (temporary probe,
not in this diff):
31 ms — which is why it essentially never reproduces locally and why the failure
rate on shared runners is low but nonzero.
Evidence it is not the PR that surfaced it
Found while investigating #14173, whose diff touches no pod, Windows, Task
Scheduler or gateway code.
108506136660succeeded where
108503209478failed. Same commit, same workflow, differentrunner, so neither the diff nor the base is a deterministic cause.
feat/app-update-clihead1f6ff7a842(job108498857158) failed with a byte-identical message and thesame ~39.7s call duration.
Pod Boot Canary (Windows)jobs across a ~2.5 hourwindow of CI history, 2 failed — both with this message. ~1.4%.
The timing exposes a real ordering defect in the product, not a runner fault, so
it is fixable rather than something to accept.
What changed (motivation → approach → change)
Readiness means serving AND credentialled.
_wait_healthy's success conditionincludes the credential, sharing the one existing wall-clock budget rather than
adding a second timer —
--wait-secsandKIROCREW_POD_HEALTH_SECSstill boundthe whole wait, and no new budget number is introduced.
Presence is not publication, and not generation.
clear_markerruns only on agraceful shutdown and the stale-marker prune deliberately never removes the
credential, so a pod home that survived a crash still holds the previous
generation's
run/gateway-<port>.secret. A presence test would call that ready onthe first poll and hand the mint a secret the new gateway never minted. So the
reader is
published_credential, returning the VALUE, andpod upcaptures theoutgoing credential immediately before
start_podand requires the live one todiffer. Content rather than mtime, so no clock or filesystem timestamp granularity
is involved; the capture's position before
start_podis pinned by a test, becausereading it after the start would compare a credential against itself and silently
disable the guard. An already-active pod supersedes nothing — it is not a new
generation, so its current credential is accepted rather than waited out for the
whole budget.
One reader, one path pair.
published_credentialand_pod_mint_secretbothgo through
_read_pod_secret_fileover_pod_secret_candidates, so they cannotdrift into reporting a pod ready whose credential the mint cannot find. That reader
pins
utf-8instead of the locale default — bytes that are invalid UTF-8 but validcp1252 would otherwise be a credential on a Windows runner and an error on a UTF-8
host — and treats
OSErrorandValueErroralike, becauseUnicodeDecodeErrorisa
ValueError: a corrupt per-listener file then falls through to the shared.local_secretinstead of becoming a traceback out of a mint or a boot poll.Exhaustion is attributed from the last observation, and only once serving has
persisted. A gateway that answered 200 once inside the bind-to-credential window
and then died or served 5xx is not a credential problem. And a gateway that binds in
the final second of its budget is serving on the last poll while having had no
chance to publish at all — that is a slow boot whose remedy is a bigger budget, so
the credential verdict requires the serving state to have survived one contiguous
poll interval, which is the smallest span that carries the claim it makes. A lapse
resets the span. Below it the wait returns the slow-boot input rather than its own
serving code, because membership in
(200, 401, 403)is the caller's whole successtest. For the same reason
last_httprecords non-serving codes only.HEALTH_NO_CREDENTIALis the verdict for a pod that served without publishing, andits message names BOTH remedies: the journal for a write error, and the
--wait-secs/ env escalation for a late publish.Two things this deliberately does NOT do:
would narrow the window; gating readiness on the credential removes it. The boot
path is pinned by startup-order tests and is the wrong place to take risk for an
effect this change already achieves completely.
pod upwas not. No skip, noretry loop, no change to the
3 passedpin.Tests
Run on Windows Server 2025, Python 3.12.10.
Red-first, at the level of the failing assertion. A temporary probe added an
await asyncio.sleep(6.0)between the bind and the credential write — modelling arunner whose intermediate startup work is slow, with the loop left free exactly as
the real
awaits leave it. Before the fix:After the fix, same 6.0s window, same probe:
The canary itself, under that same 6-second window (200x the natural one):
3 passed. With the probe removed entirely:3 passed. CI's own count gategrep -qE '(^|[^0-9])3 passed'matches.Unit negative controls, one per behaviour. Reverting only the credential
condition fails 8 tests on behaviour, not on a missing symbol —
assert 1 == 4(returned on the first poll),
assert 200 == -3(budget exhaustion handed thecaller a success code),
assert 200 == -1(a crash-looping gateway behind aserving predecessor was no longer detected). Removing the generation comparison
fails the stale-predecessor and fail-closed tests. Dropping the span requirement
fails the final-second and comeback tests. Removing the
utf-8pin fails thecp1252-decodable-but-invalid-UTF-8 parametrization on a cp1252 host.
Suites: 1299 passed / 260 skipped / 6 xfailed across
test_pod,test_pod_health_wait,test_pod_api,test_pod_windows,test_pod_windows_run,test_pod_launchd,test_pod_e2e_harness_paths,test_dev_fleet_server_coverageand
test_instancesunderPYTHONUTF8=0(cp1252, the CI runner's locale); 531passed re-run under
PYTHONUTF8=1.black --checkandflake8clean.Two defects the tests caught rather than review:
HEALTH_NO_CREDENTIALwas firstwritten as the literal
-2, which isrt.HEALTH_FOREIGN's value, sopod uptoldthe operator to choose a different port for a pod whose port was fine — it is now
derived from that constant and pinned distinct by a test. And the
undecodable-credential test passed locally while failing on CI purely because this
host runs
PYTHONUTF8=1; that is what exposed the locale-dependent read.What I could not verify
The natural gap here is 31 ms and the observed CI rate is ~1.4%, so an unforced
local reproduction would need hundreds of boots. The induced-window experiment is
what establishes causation; the 31 ms measurement is what explains the rate.
pytest log, and
pod upfails before the step that would print the pod's ownjournal. So those two failures are attributed by mechanism and by their verbatim
message, not by a log from those runs.
fire-and-forget background task, so it can also land after
pod upreturns. Onthis host that makes
uptake thePodOwnershipUnprovenpath and return an emptytoken (stderr says so; exit stays 0). Gating readiness on the marker too would make
something at boot depend on a write documented as "nothing at boot depends on it",
so it wants its own decision.
macOS Tests / Backend Tests (macOS) (1)failedonce on an earlier head in
code_review_sage'slayout_lock—os.openrelativeto a pinned
dir_fdreturned ENOENT withO_CREATset, meaning the pinneddirectory was replaced under it by one of six racing children. That comes from
fix(sage): serialize the data-layout self-heal across processes #13551, an ancestor of both this PR's base and
main, and it passed on rerun withno change. It is a real main-side race and wants its own issue.
Behaviour changes beyond the race itself
Three changes ride along with the fix. None is a side effect of it, so each is
declared here rather than left for a reader to infer from the diff.
A failed
pod upno longer stops a pod this invocation did not create. The oldcontract was "up means healthy", and any unsatisfied readiness wait called
stop_podto avoid leaking a half-booted service. That is safe only while every pod reaching the
wait was started by this command. It is not:
stop_podreachescleanup_home, whichrmtree's the pod's isolated HOME with its sessions and config, and nothing restores it.
The teardown is now conditioned on this invocation's own
start_podhaving succeededAND the home having held no state beforehand, and that condition is checked on all five
failure exits rather than on one verdict. Deliberately not keyed on
is_active: thatprobe shells
systemctl is-active --quiet, which succeeds only foractive, so a podin
activatingor inRestart=on-failurebackoff reads as not running and apre-existing crash-looping pod would be treated as one this command created. Every
failure exit now also says the pod was not stopped and names
kirocrew pod downas thedeliberate way to retire it — it states only that, never that the pod is running, since
a spared pod whose unit has genuinely failed is not.
The ceiling-aware wait hint also changes the pre-existing "still starting" message.
_health_wait_secsclamps to a 3600s ceiling, so at the ceiling the old advice todouble
--wait-secsnamed a value the resolver reduces straight back to the one thatjust failed. Both exhausted-budget messages now share one helper that reports the
ceiling instead. The slow-boot message was not part of the race, but it carried the
identical defect one line away, and fixing one while leaving its twin would have been
the less honest change.
The foreign-port latch now clears on proven handover. A predecessor seen on an
early poll used to be reported at the deadline even after this pod's own gateway had
won the port back. The latch clears only when
port_owneranswersOWNER_POD, becausea serving health code proves only "not provably foreign" —
healthreturns the statusunchanged for
OWNER_UNPROVEN, which is what an unavailable listener lookup leaves.Reachable only through the new exhaustion path, so it is a rider rather than a
prerequisite.
Pattern harvest
Rule candidate: a liveness signal is not a readiness signal. When a process
publishes a resource AFTER it starts serving, a client must wait for the resource it
is about to use, not for the serve — otherwise the gap between the two becomes an
intermittent failure whose error message blames the wrong thing.
Rule candidate: when several verdicts share one integer return channel, derive new
sentinels from the existing ones and pin them mutually distinct in a test; a literal
silently routes the new verdict into an old branch.
Rule candidate: read a credential or any machine-written file with an explicit
encoding.
read_text()without one decodes with the host locale, so the same bytesare valid on a cp1252 runner and an error on a UTF-8 host, and the test that would
catch it passes or fails depending on which machine runs it.