Skip to content

fix(token-secret): move a poisoned signing key aside and mint a fresh one - #13222

Closed
Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/signing-key-selfheal
Closed

Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/signing-key-selfheal

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway that finds a truncated token_signing.key on boot replaces it with a fresh persisted key instead of degrading to an ephemeral secret on every restart.

A token_signing.key shorter than 32 bytes makes every gateway boot sign with a random in-memory secret. Each restart then asks every dashboard browser for a fresh login token and quarantines the agent tag-grants store, and no restart repairs it: the next boot reads the same short file, logs one WARNING, and does it again. Only deleting the file by hand ends it.

Why it matters

The symptom looks like something else entirely (a stale browser, a bad token), so it sits unnoticed for days. Two consecutive gateway builds on one host booted this way; the second restart logged the browser out and quarantined tag-grants/, which then had to be re-granted by hand. The file that causes it is a 0-byte key, which is what a kill between the old in-place create and its write leaves behind, what the in-place fallback on a link-less filesystem still leaves behind, and what a partial rsync or a truncated restore produces.

Not a goal

  • Surfacing the ephemeral fallback in kirocrew doctor or the dashboard, and pointing the tag-grants quarantine message at the signing key as its cause: both are follow-ups from the same RCA and are deliberately not in this change.
  • Closing every race an external process that ignores the advisory lock can create between the healer's lstat and its rename; POSIX has no rename-by-inode, so the second look after the move narrows that window rather than removing it.
  • Changing how a healthy key is created or published; the stage-then-link path from fix(auth): publish the token signing key whole, never a 0-byte one #8230 is untouched.

What changed (motivation → approach → change)

Part What it does
_remove_poisoned_key (new) After the publish loop exhausts against a short key file, moves that file aside under a lock (it is kept, never unlinked) and asks for one more publish pass
Age check Only a short file older than 30 s is moved; a younger one may be an in-place creator between its create and its write, and is left alone
Advisory lock platform_compat.file_lock on .token_signing.key.heal.lock, serialises the healer's stat-then-rename, so two gateways booting on the same home cannot move each other's freshly published key away. The healer waits at most 30 s for a holder (the in-place creator holds the lock across its write and fsync) and then takes the extra pass; the in-place creator acquires single-shot and retries through its own loop
Sandbox mask sandbox.py MASKS the lock file (_CREW_HIDDEN_LEAVES, beside token_signing.key) and pre-creates it (_CREW_PRECREATE_HIDDEN_FILE_LEAVES) so the mask always has a name to bind. Masked rather than sealed read-only because POSIX flock can be taken through a read-only descriptor: a read-only seal would let a sandboxed process hold the lock and starve the healer into an ephemeral secret, while behind the mask it locks an empty tmpfs inode that reaches nothing. The masked-leaf alias pass refuses a symlink at the name. The gateway opens the lock O_NOFOLLOW, so a symlink planted at the lock path refuses the heal and the in-place create instead of locking through the referent
Creator under the same lock The in-place creator (link-less filesystems) holds that lock from its O_EXCL create through its durable write, so its momentarily empty destination can never be mistaken for a poisoned key
Move aside, then judge again The healer never unlinks the candidate, at the key path or anywhere else. _move_aside_for_discard renames it into the masked auth-store-staging directory and _reclaim_if_grown looks at it once more there: one that grew to a full key (an in-place cp that opened before the move keeps writing into the moved inode) is linked back for the extra pass to read; one still under 32 bytes is KEPT in staging and named in the WARNING, so a restore that lands later is recoverable there. Its partial bytes stay on disk, but only in a directory the sandbox masks. A refused or unusable staging directory leaves the file alone and degrades to the ephemeral secret
Heal only after exhaustion The publish loop's for/else sets exhausted; the heal runs only when the loop ran its whole budget AND read a short file present, so the AuthStoreStagingRefused break after one attempt never reaches it
One more pass _load_or_create_secret(after_heal=True) runs the existing single-creator election once more against the now-empty name; that pass never heals again

The loader never rejects a short file in place. Truncate-and-rewrite was ruled out because two racing boots would each write their own key, the divergence the stage-then-link publish exists to prevent. Moving the file aside and re-running the election keeps that election as the only thing that mints a key.

_heal_verdict reads the file once and answers heal, retry or leave: a stale short regular file is healed; a missing path or a full key means a sibling got there first and earns the extra pass; a symlink or a fresh short file leaves the ephemeral fallback exactly as it was, with no lock file created.

The remaining fallback WARNING says what the operator will observe (every restart logs every dashboard session out) and how long the key must be. docs/system-specs/modules/dashboard-token-auth.md describes the heal, the lock and the 30 s gate. token_secret.py is black-clean after this change and leaves .github/black-baseline.txt.

Backwards compatibility

Compatible: a valid key is read verbatim as before; a short file is only ever moved aside into the masked auth-store-staging directory and kept there, never rewritten or unlinked, and only when older than 30 s. The one new sibling name (.heal.lock) falls under the existing .lock keystone fence and is masked (not sealed) in the sandbox, for the flock reason above.

Tests

  • test_signing_secret_poisoned_key_is_removed_and_replaced: a stale 0-byte key is moved aside into staging, a 32-byte key is published and returned, the heal is logged, and a second load reads that key verbatim.
  • test_signing_secret_fresh_short_key_is_not_removed: a short file written just now is untouched and the loader degrades to ephemeral as before.
  • test_remove_pre_check_asks_for_another_pass_when_a_sibling_healed: a full key or a missing path yields the extra pass with nothing moved; a symlink or a fresh short file yields nothing.
  • test_remove_waits_for_holder_and_asks_for_another_pass and test_remove_bounds_wait_for_wedged_holder: a contender waits for a live holder, leaves the key the holder published and reports that one more pass should run; a holder stalled past the ceiling is treated as wedged and the file is left alone, which the test proves by the contender returning while the lock is still held and passing the patched timeout, not by measuring elapsed time.
  • test_create_key_in_place_declines_a_held_heal_lock: under a held lock the in-place creator makes no file; after release it produces the key.
  • test_signing_secret_heal_lock_is_masked_by_the_sandbox: the lock leaf is in the hidden and hidden-pre-create lists and not in the read-only list.
  • test_signing_secret_heal_lock_symlink_is_not_followed: with a symlink at the lock path, the heal returns without removing anything, the in-place creator makes no file, and nothing is written through the link.
  • test_remove_leaves_a_key_restored_after_the_verdict and test_remove_leaves_a_key_written_in_place_after_the_verdict: a full key landing at the path after the under-lock verdict (new inode, or the same inode via an in-place write) survives and the extra pass is requested; both fail with the second look disabled.
  • test_staging_refusal_break_does_not_enter_the_heal: a staging policy refusal that breaks the publish loop after one attempt does not enter the heal and leaves the persisted file untouched; fails without the exhaustion gate.

Removing the heal call makes the first test fail, so it observes the fix rather than the constants.

Manual verification

No manual gateway boot was run for this head. What is covered, and how:

  • The heal itself is a pure filesystem path, exercised end to end through _load_or_create_secret in the tests above, on real files in a temp home (Linux, macOS and Windows CI shards).
  • The link-less home is simulated by patching os.link to fail (TestHiddenLockLeafIsCreatedWithoutHardLinks); no real FAT or network mount was used.
  • The spawn-time pre-create and mask of .heal.lock run on every sandboxed agent spawn, so CI's hosted Linux E2E lanes on this head spawn agents through the real bwrap sandbox with the leaf in place. They do not boot a gateway against a stale 0-byte key.

Not covered: a real gateway boot against a stale 0-byte key, and a spawn on a real link-less filesystem.

Related Issues

Fixes #13221

Pattern harvest

Not generalizable: the defect is one retry loop that treated "file exists but is unusable" as "wait for the writer" with no bound on how long that state may persist; no other keystone leaf has a size-gated fast path.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and cixuuz September 24, 2026 00:16
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Intent: A gateway that finds a short (<32 byte) token_signing.key older than the in-place create window moves it aside and mints a fresh persisted key, so one restart ends the every-restart logout instead of repeating it forever.
Not a goal: surfacing the ephemeral fallback in kirocrew doctor or the dashboard, changing how a valid key is read or created, or repairing the tag-grants quarantine message.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 88f6abebc23bcae07e89d08066cfb104ad3bd6e0 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check one thing before merge: the in-place creator now fails whenever the heal lock can't be opened or taken. Confirm a link-less home (FAT/exFAT, Windows) still keeps its key across a restart.

[DESIGN-REVIEWED] 88f6abe

Description read

[DESCRIPTION-READ] 3958d3e6ff60fd5586e536e13355c38bbbc71e01b74ff7e130dcc610a4574acb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 88f6abebc23bcae07e89d08066cfb104ad3bd6e0 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

_reclaim_if_grown and its WARNING branches guard a microsecond race under the lock. The move-aside step already keeps the restored file, and the WARNING says where it went.

Not justified as shipped

  • Item 4 — oversized: the "second look" exists for "an in-place cp that opened before the move keeps writing into the moved inode", yet the description's own Not a goal says this "narrows that window rather than removing it". Without it, a key restored inside that window is still kept in auth-store-staging and named in the WARNING. Nothing is lost.
    Subtraction: delete _reclaim_if_grown, its two logger.warning branches, the relink/fsync_dir dance and the two ..._after_the_verdict tests. Log the kept path unconditionally.
    Clears when: the mechanism is removed, or a linked report shows a restore landing between the under-lock lstat and rename.

What this change ships

Inventory (7 items) — 6 justified

Intent: a gateway whose signing key file is too short should repair it on boot, so restarts stop logging every dashboard user out and quarantining tag grants. FIX.

  1. A short key file older than 30 s is moved into staging and a fresh persisted key is minted — justified
  2. A short file younger than 30 s is left alone (age gate) — justified
  3. The in-place creator and the healer now share the advisory lock .token_signing.key.heal.lock — justified
  4. A moved file that has grown back to a full key is linked back under the key's name — oversized
  5. The sandbox masks the lock leaf and creates it before every spawn (new hidden-file pre-create branch) — justified
  6. The ephemeral-fallback WARNING now says every restart logs sessions out — rides along
  7. token_secret.py is reformatted with black and removed from the baseline — rides along

[FIRST-PRINCIPLES-REVIEWED] 88f6abe

Description read

[DESCRIPTION-READ] 1a2e411ddcec9f5f8b0c05e480adb0488c91afcd21641973963bb0f3b52f0dc8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 88f6abebc23bcae07e89d08066cfb104ad3bd6e0 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/dashboard/token_secret.py:323 -- Successful reclamation executes os.unlink(aside), contradicting “Nothing is ever unlinked here” -> Fix: State that still-short candidates are retained, while a recovered full key’s staging name is removed after its relink is durable.
[GPT-REVIEWED] 88f6abe

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 88f6abebc23bcae07e89d08066cfb104ad3bd6e0 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 88f6abe

@Pearcekieser
Pearcekieser force-pushed the fix/signing-key-selfheal branch from 1698e2a to bcf5893 Compare September 24, 2026 01:06
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the in-place creator now holds the heal advisory lock from its O_EXCL create through its durable write

  • An active in-place writer can be quarantined span=e3f343692baa — fixed in bcf5893

The window was real: between O_EXCL create and first write the destination is a 0-byte file, and a healer stalled past 30 s of that pause would move it aside.
Fix: _create_key_in_place takes the same platform_compat.file_lock on .token_signing.key.heal.lock that _quarantine_poisoned_key takes, and holds it through os.fsync; a healer therefore sees either no file or a full key, never the creator's empty one.
A refused lock returns None into the existing bounded retry loop, so the boot path gains no new failure mode beyond a retry.
An identity check after the write was considered and rejected: a healer that stats the empty inode, pauses, and renames after the write lands is not caught by it; the lock is.
Regression test test_create_key_in_place_waits_for_the_heal_lock: while a holder keeps the lock the creator produces no file; after release it produces the key.

@Pearcekieser
Pearcekieser force-pushed the fix/signing-key-selfheal branch from bcf5893 to c1e5352 Compare September 24, 2026 01:59
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: .token_signing.key.heal.lock is read-only sealed and pre-created by sandbox.py, the same disposition as agent_model_state.json.lock

  • replaceable lock file does not serialize healers span=e3f343692baa — fixed in c1e5352 (second occurrence of this path-scoped span; the first was the creator window, this one the lock's identity)

The finding holds: the lock leaf sat in the sandbox-writable crew-home root with no sandbox disposition, so a sandboxed process could unlink and recreate it and two gateways would lock different inodes.
Fix follows the repo's own precedent for agent_model_state.json.lock: the leaf is added to _CREW_READONLY_LEAVES (the mount pins the inode; unlink of a mount point fails), to _CREW_PRECREATE_READONLY_FILE_LEAVES so an absent lock is materialised before every namespace spawn, and to _CREW_CHILD_READABLE_LEAVES for the completeness pin.
test_signing_secret_heal_lock_is_sealed_by_the_sandbox pins the lock name against both sandbox lists so a rename on either side fails the test.
Boundary, stated: a same-UID process OUTSIDE the sandbox can still replace the file; that is the trust boundary every keystone leaf in this module already has.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: _heal_verdict returns heal / retry / leave from one lstat; the unlocked pre-check and the locked re-check both consume it

  • unlocked pre-check maps a valid key or a missing path onto return False span=0dbac0e59b0f — fixed in c1e5352

The finding holds: a sibling that lost the heal race by milliseconds saw the winner's full key or the moved-aside name and skipped the extra pass, staying ephemeral.
Fix: _heal_verdict answers retry for a missing path or a full key (a sibling is ahead), leave only for a symlink or a short file younger than 30 s, heal for a stale short regular file; _quarantine_poisoned_key returns True on retry before and after taking the lock.
test_quarantine_pre_check_asks_for_another_pass_when_a_sibling_healed covers all four shapes.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: both platform_compat.file_lock acquires in token_secret.py are wait=False

  • wait=True lock acquire can poll _LOCK_TIMEOUT_SECS per attempt across the 50-attempt loop span=0dbac0e59b0f — fixed in c1e5352

The finding holds: off the event loop the acquire polled up to 300 s per call, and the in-place creator is called once per attempt, twice over with after_heal.
Fix: both sites pass wait=False; a held lock raises BlockingIOError at once, the creator returns None into its existing bounded loop and the healer returns True so the caller runs the extra pass. Comments describe that behaviour.
test_create_key_in_place_declines_a_held_heal_lock and test_quarantine_yields_to_the_holder_and_asks_for_another_pass assert the immediate no-wait outcome.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • owning spec does not describe the heal, its artifacts or the age gate span=c9e7bac1a6c3 — fixed in c1e5352

docs/system-specs/modules/dashboard-token-auth.md, section "State: in-memory vs. persisted", describes the heal condition, the .token_signing.key.poisoned-<unix time>.tmp quarantine and why it is kept, the .token_signing.key.heal.lock advisory lock shared with the in-place creator and its sandbox seal, the single-shot acquire, the 30 s gate and the WARNING, in the same commit as the code.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/signing-key-selfheal branch from c1e5352 to f4124e8 Compare September 24, 2026 02:33
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the poisoned file is unlinked under the heal lock; no sibling copy is created

  • quarantined key material escapes the OS sandbox span=e3f343692baa — fixed in f4124e8 (third occurrence of this path-scoped span; each occurrence was a different defect in the same file)

The finding holds: .token_signing.key.poisoned-<ts>.tmp sat under a name the sandbox does not mask, so a truncated prefix of a key still live on another host could be read from a sandboxed shell.
Fix by subtraction: _remove_poisoned_key unlinks the file instead of renaming it. A file under 32 bytes was never accepted as a key by any boot, so nothing usable is lost; the WARNING keeps the forensic facts (path, byte count, age). With no sibling copy there is nothing for the fence question to apply to; the quarantine name, its keystone-suffix reasoning and its fence test are removed.
Pre-existing and out of scope: the full-key staging leftover this module keeps on a failed directory sync sits at the same in-sandbox-readable disposition on main; that is a separate finding against main, not something this PR introduces.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the healer acquires the lock with wait=True and an explicit 5 s timeout (_HEAL_LOCK_WAIT_SECONDS); the in-place creator stays single-shot

  • lock contention can leave gateways using divergent signing keys span=e3f343692baa — fixed in f4124e8

The finding holds as a bounded case: a contender refused at once took its ~1 s extra pass and could end on an ephemeral secret while a stalled holder later published. Publish remains a single-creator election (os.link is non-clobbering, the in-place creator is O_EXCL), so two persisted keys cannot arise; the exposure was one gateway ephemeral until its next boot.
Fix between the two extremes the lanes objected to: the healer waits up to 5 s for a live holder (its critical section is an lstat plus an unlink, milliseconds), re-judges under the lock and reads what the holder published; a holder stalled past 5 s is treated as wedged and the contender takes the extra pass, so a wedged holder cannot stall the port bind for _LOCK_TIMEOUT_SECS per attempt (the Opus finding this replaces). On the event-loop thread file_lock is single-shot regardless and follows the same extra-pass path.
Tests: test_remove_waits_for_holder_and_asks_for_another_pass (holder publishes under the lock, contender waits and leaves the full key) and test_remove_bounds_wait_for_wedged_holder (ceiling monkeypatched low, holder stalls, contender returns within the ceiling and removes nothing).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/signing-key-selfheal branch from f4124e8 to 6acaf64 Compare September 24, 2026 03:14
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • ratchet bullet says sealed read-only (test/test_sandbox_protected_name_holds.py, TestLeafOnlyPopulationIsRecorded.EXPECTED comment): fixed; the bullet now says the lock is masked (empty tmpfs inode per namespace, because flock works through a read-only descriptor) and pre-created.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed fcc0d8ab4 -> f43cfae81 (rebased onto current main, one commit). Closes the GPT fenced finding with its own remedy (move the candidate into masked staging, judge it again, discard only if still short) and both Opus advisories (heal gated on loop exhaustion; ratchet bullet wording). Dispositions above name the tests. Inherited test reds (main-red prepare-pr skill-size test, frontend 3d ago, Windows timing) are unchanged on main.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/signing-key-selfheal branch from f43cfae to f969ddf Compare September 27, 2026 07:57
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • The discard still races an active in-place restore (src/kiro_crew/dashboard/token_secret.py, _discard_if_still_short): fixed, using the reviewers remedy. The moved-aside candidate is retained for _DISCARD_QUIESCENCE_SECONDS(2 s, under the heal lock and well inside the 30 s lock ceiling) before its second look; only a file still short after that is unlinked, and one that grew is relinked. Test:test_remove_waits_for_a_delayed_in_place_write_before_discarding` (a writer that opened before the heal lands its bytes 150 ms into a 500 ms wait; the key survives) fails with the wait removed.
  • A recovered key is returned before its directory entry is durable (_discard_if_still_short): fixed. The relink now follows the publish paths shape: os.link, strict fsync_dir(key_path.parent), and the recovery name is unlinked only after the sync succeeds; a refused sync keeps it as a second name and logs both paths. Test: test_remove_keeps_the_recovery_name_when_the_relink_cannot_be_synced` fails with the guard removed.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed f43cfae81 -> f969ddf32 (rebased onto current main, one commit). Both GPT findings on the previous head are closed as proposed: a 2 s quiescence before the discard`s second look, and a durable relink (link, fsync_dir, then drop the recovery name). Two new tests, each negative-controlled; 646 scoped tests, black, flake8, mypy pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/signing-key-selfheal branch from f969ddf to deca281 Compare September 27, 2026 08:25
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • fixed quiescence window can delete an active restore (src/kiro_crew/dashboard/token_secret.py): fixed, using the reviewers remedy. The moved candidate is never unlinked. _reclaim_if_grownreplaces_discard_if_still_short: a candidate that has grown to a full key is linked back under the keys name (link, strict fsync_dir, drop the recovery name only once durable); a candidate still short is KEPT in the masked auth-store-staging directory and its path is named in the heal WARNING, so a restore that lands at any later time lands in a file that still exists. The quiescence constant is removed. Tests: test_remove_keeps_a_late_in_place_write_in_staging (a writer that opened before the heal writes after it finished; the bytes are intact in staging), test_remove_leaves_a_key_written_in_place_after_the_verdict, test_remove_keeps_the_recovery_name_when_the_relink_cannot_be_synced. These three hold the candidate open across the rename, which Windows refuses by design (the heal then leaves the file alone and degrades), so they are POSIX-only; that is what Windows shard 6 reported on the previous head.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed f969ddf32 -> deca2814e (rebased onto current main, one commit). The healer no longer discards anything: still-short candidates stay in masked staging, grown ones are relinked durably. Windows shard 6 on the previous head was the three open-handle tests, now POSIX-only. Opus ended "review incomplete" (lane error, no verdict) and reruns here.

@Pearcekieser
Pearcekieser force-pushed the fix/signing-key-selfheal branch from deca281 to 839b6a4 Compare September 27, 2026 08:48
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 27, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Retriggered: identical tree, deca2814e -> 839b6a496 (main unmoved, timestamp-only amend), to rerun the fork review lanes the CI app rate limit aborted at 08:32Z. No code change.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Parked at 398bea6f4, and a simpler alternative is up as #14457 so maintainers can choose between the two by merging one.

Why parked. Across 18 review rounds the security lanes found 14 edge cases, all one class: a boot-time process moving, removing or adopting the key file while an operator restore, a sibling gateway or a sandboxed process may touch the same path. In order: a creator paused mid-create losing its empty file; a lock in the writable root a sandbox could swap; a quarantine name readable in the sandbox; a lock loser going ephemeral; a pre-planted symlink at the lock (POSIX, then Windows); an fsync outlasting the lock ceiling; a restore landing in the lstat-to-unlink gap; a read-only seal handing flock to the sandbox (upheld); a same-inode in-place restore beating a dev/ino check; a relink with no fsync_dir (upheld); a delayed first write beating an immediate second look; a fixed 2 s quiescence beating a stalled restore; a hard-link pre-create refusing link-less homes; and, this round, a size-based relink adopting a still-writing restore`s interim bytes. Each patch added a mechanism and each mechanism exposed the next seam; writer completion is not observable from the gateway, so no finite set of checks closes the class.

What #14457 does instead: on boot, a token_signing.key that exists and is shorter than 32 bytes refuses the start with a CRITICAL log line and the one-line move-aside remedy, and kirocrew doctor reports the same state. No boot-time mutation, so the class above does not apply; the cost is one operator command on a rare, already visible event. A dashboard banner and a kirocrew auth repair-signing-key command are proposed as follow-ups.

This PR stays open and unchanged unless a maintainer prefers the in-place heal; the reviewer`s final remedy for this round (keep moved candidates staged, never relink) is the one change it would still need.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased (conflict only, PR stays parked): 398bea6f4 -> 2478de1aa onto current main. Main added six leaf-only protected names (mcp-launch-approvals, mcp/resolved) to the test_sandbox_protected_name_holds.py ratchet; the resolution keeps both bullets and sums the counts (250/257/258 + 3 for the heal lock = 253/260/261). No behaviour change; token-auth, ratchet and seal suites pass locally.

@chenmingwei23

Copy link
Copy Markdown
Contributor

Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable - I'll push fixes directly to this branch as a co-author.

Assessment: Merge conflict with main (rebase). Windows lane fails because the new .token_signing.key.heal.lock leaf is created 0o666 instead of 0o600 (test_a_linkless_home_still_gets_the_leaf); fix the leaf's mode on Windows. Opus FINDING: the spec paragraph in docs/system-specs/modules/dashboard-token-auth.md:396 and the _remove_poisoned_key / _heal_verdict docstrings still describe the old unlink design; update them to the move-aside design. The test_fence_marker_separators timing failure looks like a Windows flake and will be re-checked after the rebase. No design change.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Rebased on main (9a6f9b94e), conflict only, no behaviour change. The PR stays parked next to #14457.

Main now refuses a sealable ceiling whose parent is missing, returns early for an absent data home, and records every file leaf it sees in _note_established. The resolution keeps all three, and the heal-lock leaf goes through the same _note_established call when it is created or already exists. In test_sandbox_protected_name_holds.py the counts are main's 265/272/273 plus this PR's 3, so 268/275/276. The spec keeps main's mode-0 mask paragraph followed by this PR's heal paragraphs.

662 targeted tests pass (test_sandbox_protected_name_holds.py, test_token_auth.py, test_sandbox_absent_ceiling_seal.py, the token-secret and ceiling suites), and the black check is clean.

@chenmingwei23 the head moved from 2478de1aa because of this rebase. If you still plan the Windows lock-mode and docstring fixes you listed, please start from the new head.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Fixed the SAST (Semgrep) failure on 388ecf7f6. The new test-determinism rules on main flagged six lines in this PR's tests in test/test_token_auth.py. Only tests changed in e89a50dca.

  • test-unseeded-random: the four os.urandom restore payloads now come from random.Random(13222).randbytes(...). The tests check that those bytes survive, not what they are.
  • test-sleep-as-barrier, in test_remove_waits_for_holder_and_asks_for_another_pass: the holder used to sleep 0.3 s and the test then required the contender to take at least 0.2 s. Now the holder keeps the lock until the contender reaches its file_lock call, and only then writes the full key. The test checks that the contender asked for a blocking acquire with a timeout (wait=True), and that it read the holder's key and not the short file.
  • test-sleep-as-barrier, in test_remove_bounds_wait_for_wedged_holder: the holder used to sleep 1.0 s. Now it holds the lock until the test releases it, after the contender has returned, so the contender's return can only come from the bounded wait. The upper bound went from 0.8 s to 10 s. It still separates the patched 0.2 s wait from the 30 s default, and a loaded runner can't break it.

All 323 tests in test_token_auth.py pass, and black is clean.

The Sensitive Change Review failure is the gate that every fork PR touching src/kiro_crew/sandbox.py fails. This PR can't clear it from a fork. It stays parked next to #14457.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

88f6abebc fixes three things on e89a50dca.

GPT 6.1 F1 (upheld, test/test_token_auth.py): test_remove_bounds_wait_for_wedged_holder no longer measures wall time. The holder keeps the lock until the test releases it. That release happens only after the contender has returned, and the holder records whether it was released by the test or by its 60 s backstop. The test now asserts two things. First, the holder was released by the test, which proves the contender returned while the lock was still held, so its own ceiling ended the wait. Second, the contender's single file_lock call carried the patched timeout=0.2. An unbounded wait fails the first assertion, and a wait that ignores the patched ceiling fails the second. No assertion depends on how fast the runner is.

Backend Tests (Windows) shard 2: the only failure was this PR's own test, test_sandbox_absent_ceiling_seal.py::TestHiddenLockLeafIsCreatedWithoutHardLinks::test_a_linkless_home_still_gets_the_leaf[.token_signing.key.heal.lock]. It asserted mode 0o600 on the pre-created heal lock. Windows has no POSIX permission bits, so os.stat reports 0o666 for any writable file, which is the 438 == 384 in the log. The mode check now runs only on POSIX, the same way main guards its other mode checks in that file (if os.name == "posix").

Design Review concerns: the spec, the _remove_poisoned_key docstring and the description still described the old removal design. All three now describe the code as written. The short file is renamed into the masked auth-store-staging directory and never unlinked. It is linked back if it grew to a full key and kept there otherwise, so its partial bytes stay on disk only inside a directory the sandbox masks, and the WARNING names the kept file. The lock is masked, not sealed. The description no longer mentions _discard_if_still_short, a function that doesn't exist. Manual verification now lists what CI covers and what it doesn't: no real gateway boot against a stale 0-byte key, and no spawn on a real link-less filesystem.

Code behaviour is unchanged. Only tests, docs and the docstring changed. All 637 tests in test_token_auth.py and test_sandbox_absent_ceiling_seal.py pass locally, and black is clean.

@chenmingwei23, this covers the Windows lock-mode test fix and the docstring fix you listed on 10-02. If you still have other changes in mind, please start from the new head.

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed locally at head 88f6abe. The bug is still present on main: a short signing key falls back to an ephemeral secret on every boot. I found no weakening of key strength in this diff, and the targeted token-auth and sandbox suites pass locally (662 tests). Requesting changes for two reasons:

  1. The branch conflicts with main in test/test_sandbox_protected_name_holds.py, because the ratchet counts moved again on main.
  2. This PR has been parked in favour of the fail-closed alternative #14457, which refuses to start on a short key and adds a doctor check, with no boot-time file mutation. That approach avoids the class of repair races that kept coming up here. If #14457 is the path forward, please close this one so reviewers focus there.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note: closing in favour of #14457

@cruisercohen thanks for the local review. #14457 is the path forward. It refuses to start on a short signing key and adds a doctor check, so nothing mutates the key file at boot and the repair races this PR kept drawing cannot happen. I'm closing this PR so review goes there.

On your first point, I had rebased this branch onto main c34eca2e86 locally (ratchet counts 271/278/279) but had not pushed it. That rebase is dropped with the close. If #14457 is rejected, this branch can be reopened from 88f6abebc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A short token_signing.key logs every dashboard session out on every restart and is never repaired

5 participants