Repository navigation
fix(token-secret): move a poisoned signing key aside and mint a fresh one - #13222
Pearcekieser wants to merge 1 commit into
Conversation
|
Intent: A gateway that finds a short (<32 byte) |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check one thing before merge: the in-place creator now fails whenever the heal lock can't be opened or taken. Confirm a link-less home (FAT/exFAT, Windows) still keeps its key across a restart. [DESIGN-REVIEWED] 88f6abe Description read[DESCRIPTION-READ] 3958d3e6ff60fd5586e536e13355c38bbbc71e01b74ff7e130dcc610a4574acb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: a gateway whose signing key file is too short should repair it on boot, so restarts stop logging every dashboard user out and quarantining tag grants. FIX.
[FIRST-PRINCIPLES-REVIEWED] 88f6abe Description read[DESCRIPTION-READ] 1a2e411ddcec9f5f8b0c05e480adb0488c91afcd21641973963bb0f3b52f0dc8 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/dashboard/token_secret.py:323 -- Successful reclamation executes |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
1698e2a to
bcf5893
Compare
|
self-added: yes
|
bcf5893 to
c1e5352
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
c1e5352 to
f4124e8
Compare
|
self-added: yes
|
|
self-added: yes
|
f4124e8 to
6acaf64
Compare
|
|
Pushed |
f43cfae to
f969ddf
Compare
|
|
Pushed |
f969ddf to
deca281
Compare
|
|
Pushed |
deca281 to
839b6a4
Compare
|
Retriggered: identical tree, |
|
Parked at Why parked. Across 18 review rounds the security lanes found 14 edge cases, all one class: a boot-time process moving, removing or adopting the key file while an operator restore, a sibling gateway or a sandboxed process may touch the same path. In order: a creator paused mid-create losing its empty file; a lock in the writable root a sandbox could swap; a quarantine name readable in the sandbox; a lock loser going ephemeral; a pre-planted symlink at the lock (POSIX, then Windows); an fsync outlasting the lock ceiling; a restore landing in the lstat-to-unlink gap; a read-only seal handing What #14457 does instead: on boot, a This PR stays open and unchanged unless a maintainer prefers the in-place heal; the reviewer`s final remedy for this round (keep moved candidates staged, never relink) is the one change it would still need. |
|
Rebased (conflict only, PR stays parked): |
|
Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable - I'll push fixes directly to this branch as a co-author. Assessment: Merge conflict with main (rebase). Windows lane fails because the new If you'd prefer I don't touch this PR, add the |
|
Rebased on main ( Main now refuses a sealable ceiling whose parent is missing, returns early for an absent data home, and records every file leaf it sees in 662 targeted tests pass ( @chenmingwei23 the head moved from |
|
Fixed the SAST (Semgrep) failure on
All 323 tests in The Sensitive Change Review failure is the gate that every fork PR touching |
|
GPT 6.1 F1 (upheld, Backend Tests (Windows) shard 2: the only failure was this PR's own test, Design Review concerns: the spec, the Code behaviour is unchanged. Only tests, docs and the docstring changed. All 637 tests in @chenmingwei23, this covers the Windows lock-mode test fix and the docstring fix you listed on 10-02. If you still have other changes in mind, please start from the new head. |
cruisercohen
left a comment
There was a problem hiding this comment.
Reviewed locally at head 88f6abe. The bug is still present on main: a short signing key falls back to an ephemeral secret on every boot. I found no weakening of key strength in this diff, and the targeted token-auth and sandbox suites pass locally (662 tests). Requesting changes for two reasons:
- The branch conflicts with main in
test/test_sandbox_protected_name_holds.py, because the ratchet counts moved again on main. - This PR has been parked in favour of the fail-closed alternative #14457, which refuses to start on a short key and adds a doctor check, with no boot-time file mutation. That approach avoids the class of repair races that kept coming up here. If #14457 is the path forward, please close this one so reviewers focus there.
|
note: closing in favour of #14457 @cruisercohen thanks for the local review. #14457 is the path forward. It refuses to start on a short signing key and adds a doctor check, so nothing mutates the key file at boot and the repair races this PR kept drawing cannot happen. I'm closing this PR so review goes there. On your first point, I had rebased this branch onto main |
Problem / Motivation
Goal: A gateway that finds a truncated
token_signing.keyon boot replaces it with a fresh persisted key instead of degrading to an ephemeral secret on every restart.A
token_signing.keyshorter than 32 bytes makes every gateway boot sign with a random in-memory secret. Each restart then asks every dashboard browser for a fresh login token and quarantines the agent tag-grants store, and no restart repairs it: the next boot reads the same short file, logs one WARNING, and does it again. Only deleting the file by hand ends it.Why it matters
The symptom looks like something else entirely (a stale browser, a bad token), so it sits unnoticed for days. Two consecutive gateway builds on one host booted this way; the second restart logged the browser out and quarantined
tag-grants/, which then had to be re-granted by hand. The file that causes it is a 0-byte key, which is what a kill between the old in-place create and its write leaves behind, what the in-place fallback on a link-less filesystem still leaves behind, and what a partialrsyncor a truncated restore produces.Not a goal
kirocrew doctoror the dashboard, and pointing the tag-grants quarantine message at the signing key as its cause: both are follow-ups from the same RCA and are deliberately not in this change.lstatand its rename; POSIX has no rename-by-inode, so the second look after the move narrows that window rather than removing it.What changed (motivation → approach → change)
_remove_poisoned_key(new)platform_compat.file_lockon.token_signing.key.heal.lock, serialises the healer's stat-then-rename, so two gateways booting on the same home cannot move each other's freshly published key away. The healer waits at most 30 s for a holder (the in-place creator holds the lock across its write and fsync) and then takes the extra pass; the in-place creator acquires single-shot and retries through its own loopsandbox.pyMASKS the lock file (_CREW_HIDDEN_LEAVES, besidetoken_signing.key) and pre-creates it (_CREW_PRECREATE_HIDDEN_FILE_LEAVES) so the mask always has a name to bind. Masked rather than sealed read-only because POSIXflockcan be taken through a read-only descriptor: a read-only seal would let a sandboxed process hold the lock and starve the healer into an ephemeral secret, while behind the mask it locks an empty tmpfs inode that reaches nothing. The masked-leaf alias pass refuses a symlink at the name. The gateway opens the lockO_NOFOLLOW, so a symlink planted at the lock path refuses the heal and the in-place create instead of locking through the referentO_EXCLcreate through its durable write, so its momentarily empty destination can never be mistaken for a poisoned key_move_aside_for_discardrenames it into the maskedauth-store-stagingdirectory and_reclaim_if_grownlooks at it once more there: one that grew to a full key (an in-placecpthat opened before the move keeps writing into the moved inode) is linked back for the extra pass to read; one still under 32 bytes is KEPT in staging and named in the WARNING, so a restore that lands later is recoverable there. Its partial bytes stay on disk, but only in a directory the sandbox masks. A refused or unusable staging directory leaves the file alone and degrades to the ephemeral secretfor/elsesetsexhausted; the heal runs only when the loop ran its whole budget AND read a short file present, so theAuthStoreStagingRefusedbreakafter one attempt never reaches it_load_or_create_secret(after_heal=True)runs the existing single-creator election once more against the now-empty name; that pass never heals againThe loader never rejects a short file in place. Truncate-and-rewrite was ruled out because two racing boots would each write their own key, the divergence the stage-then-link publish exists to prevent. Moving the file aside and re-running the election keeps that election as the only thing that mints a key.
_heal_verdictreads the file once and answers heal, retry or leave: a stale short regular file is healed; a missing path or a full key means a sibling got there first and earns the extra pass; a symlink or a fresh short file leaves the ephemeral fallback exactly as it was, with no lock file created.The remaining fallback WARNING says what the operator will observe (every restart logs every dashboard session out) and how long the key must be.
docs/system-specs/modules/dashboard-token-auth.mddescribes the heal, the lock and the 30 s gate.token_secret.pyis black-clean after this change and leaves.github/black-baseline.txt.Backwards compatibility
Compatible: a valid key is read verbatim as before; a short file is only ever moved aside into the masked
auth-store-stagingdirectory and kept there, never rewritten or unlinked, and only when older than 30 s. The one new sibling name (.heal.lock) falls under the existing.lockkeystone fence and is masked (not sealed) in the sandbox, for theflockreason above.Tests
test_signing_secret_poisoned_key_is_removed_and_replaced: a stale 0-byte key is moved aside into staging, a 32-byte key is published and returned, the heal is logged, and a second load reads that key verbatim.test_signing_secret_fresh_short_key_is_not_removed: a short file written just now is untouched and the loader degrades to ephemeral as before.test_remove_pre_check_asks_for_another_pass_when_a_sibling_healed: a full key or a missing path yields the extra pass with nothing moved; a symlink or a fresh short file yields nothing.test_remove_waits_for_holder_and_asks_for_another_passandtest_remove_bounds_wait_for_wedged_holder: a contender waits for a live holder, leaves the key the holder published and reports that one more pass should run; a holder stalled past the ceiling is treated as wedged and the file is left alone, which the test proves by the contender returning while the lock is still held and passing the patched timeout, not by measuring elapsed time.test_create_key_in_place_declines_a_held_heal_lock: under a held lock the in-place creator makes no file; after release it produces the key.test_signing_secret_heal_lock_is_masked_by_the_sandbox: the lock leaf is in the hidden and hidden-pre-create lists and not in the read-only list.test_signing_secret_heal_lock_symlink_is_not_followed: with a symlink at the lock path, the heal returns without removing anything, the in-place creator makes no file, and nothing is written through the link.test_remove_leaves_a_key_restored_after_the_verdictandtest_remove_leaves_a_key_written_in_place_after_the_verdict: a full key landing at the path after the under-lock verdict (new inode, or the same inode via an in-place write) survives and the extra pass is requested; both fail with the second look disabled.test_staging_refusal_break_does_not_enter_the_heal: a staging policy refusal that breaks the publish loop after one attempt does not enter the heal and leaves the persisted file untouched; fails without the exhaustion gate.Removing the heal call makes the first test fail, so it observes the fix rather than the constants.
Manual verification
No manual gateway boot was run for this head. What is covered, and how:
_load_or_create_secretin the tests above, on real files in a temp home (Linux, macOS and Windows CI shards).os.linkto fail (TestHiddenLockLeafIsCreatedWithoutHardLinks); no real FAT or network mount was used..heal.lockrun on every sandboxed agent spawn, so CI's hosted Linux E2E lanes on this head spawn agents through the real bwrap sandbox with the leaf in place. They do not boot a gateway against a stale 0-byte key.Not covered: a real gateway boot against a stale 0-byte key, and a spawn on a real link-less filesystem.
Related Issues
Fixes #13221
Pattern harvest
Not generalizable: the defect is one retry loop that treated "file exists but is unusable" as "wait for the writer" with no bound on how long that state may persist; no other keystone leaf has a size-gated fast path.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)