Skip to content

fix(agent): ignore per-launch MCP env nonces in the worker spec fingerprint - #16879

Closed
paigebelliveau wants to merge 2 commits into
kirodotdev:mainfrom
paigebelliveau:fix/worker-spec-fingerprint-volatile-env
Closed

paigebelliveau wants to merge 2 commits into
kirodotdev:mainfrom
paigebelliveau:fix/worker-spec-fingerprint-volatile-env

Conversation

@paigebelliveau

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A kirocrew-worker spawn survives another session launching at the same moment.

On a host where a launcher re-stamps a per-launch nonce into every agent spec it manages (AIM_CREDS_AGENT_INJECTION in the creds-agent entry of ~/.kiro/agents/kirocrew.json, rewritten each time a session starts), a kirocrew-worker spawn that overlaps any other session launch dies before its first turn with:

kiro_crew.agent.DerivedSpecStale: the default agent spec changed during worker load, so this session may have started on a spec nobody verified; ending it (verified e40fcd81fcdb, now 28cd23ec06ed)

Nothing runs, nothing is written. With a dozen dashboard sessions open, a launch lands every few minutes, so most worker dispatches die.

Why it matters

Conductors cannot dispatch workers on a host with such a launcher installed: three of three kirocrew-worker spawns died this way on one host in one evening, each time with the user's task undone and reported as a runtime error. The gate is right to end a worker when a server is revoked under it; it is wrong to read a nonce as a revocation.

Not a goal

  • Changing how or when the launcher stamps the nonce.
  • Changing which keys the worker mirrors (tools, allowedTools, excludedTools, mcpServers, model).
  • Relaxing the gate for any env value that is not on the volatile list: a rotated credential is still a changed grant and still ends the load.

What changed (motivation → approach → change)

The worker gate takes a content fingerprint of the default spec before the spawn and again after kiro-cli's initialize, and ends the session when they differ. The fingerprint in agent_materialization/worker_agent.py hashed every mcpServers entry whole, env values included, so the nonce moved the hash on every launch. The repo already knew that key is a nonce: acp/skill_projection.py left volatile_env_keys() out of its skill-view identity digests. The gate simply was not using that list.

The fix hashes mcpServers with each volatile env value replaced by a fixed marker. The key itself stays in the hash, so a default that drops the key is still a changed spec; only the per-launch value is ignored. The volatile list (VOLATILE_ENV_KEYS_DEFAULT, the KIROCREW_SKILL_VIEW_VOLATILE_ENV extension and volatile_env_keys()) moves to agent_spec_format.py, beside the spec parser, because the gate may not import the ACP layer (check_agent_sdk_boundary). skill_projection.py re-exports the same names it had, so its callers and tests are unchanged.

Backwards compatibility

Compatible: fingerprints of specs with no volatile env key are unchanged; a spec with one hashes differently once, which the gate answers with a single re-derive of the worker mirror on the next spawn, exactly as it does after any default-spec edit.

Removes nothing: no key, path, symbol or kind is removed; skill_projection._VOLATILE_ENV_KEYS_DEFAULT, _VOLATILE_ENV_VAR and volatile_env_keys keep their names.

Tests

  • test_a_per_launch_env_nonce_does_not_move_the_fingerprint locks in that a changed AIM_CREDS_AGENT_INJECTION value leaves default_spec_fingerprint() unchanged, while any other env value in the same entry still moves it, and so does removing the key.
  • test_a_nonce_restamp_during_the_load_does_not_end_the_session reproduces the production shape: capture, a re-stamp that changes the file identity, then require_unchanged_derived_spec returns instead of raising.

Both fail on main and pass with this change.

Manual verification

Reproduced on the affected host: two captures of ~/.kiro/agents/kirocrew.json seven minutes apart differed only in the nonce and fingerprinted differently on main; with this change both fingerprint the same, a removed key still changes the hash, and so does an edited non-volatile env value.

Related Issues

no linked issue: found on a host, not filed first.

Pattern harvest

Rule candidate: review-prompt
Pattern: two digests of the same structure (skill-view identity and worker-spec fingerprint) with different exclusion lists; any new hash over mcpServers must consume volatile_env_keys().

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@paigebelliveau
paigebelliveau requested a review from a team as a code owner October 5, 2026 01:34
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 5, 2026
@paigebelliveau
paigebelliveau force-pushed the fix/worker-spec-fingerprint-volatile-env branch 2 times, most recently from a1e85fe to 2acbbc7 Compare October 5, 2026 01:55
@paigebelliveau

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in files this PR also touches since this head went green (f449c8a -> 595ad42); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of 33c0a5bbed0f95e95f5de40d55795ba21c10c9da via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The worker mirror now keeps a stale AIM_CREDS_AGENT_INJECTION value indefinitely. Nothing shows the creds-agent server accepts an old per-launch nonce.

Watch

  • The mirror stops picking up new nonce values. _spec_fingerprint is also what set_mirrored_from records (worker_agent.py:781). _derived_spec_matches_default compares against the same value (:977). On main, every re-stamp produced a mismatch, and the pre-spawn check re-derived kirocrew-worker.json with the current nonce. With _without_volatile_mcp_env, a re-stamp no longer triggers a re-derive. Each worker therefore launches creds-agent with the nonce from the last time something else in the spec changed, which could be days old. If the launcher's credential injection checks that nonce, workers stop dying at the gate and instead lose credentials partway through a task, which is harder to diagnose. The skill-view alias digest already reuses copies that carry a stale nonce, so there is precedent. The description doesn't say that changes the worker, though: it only claims "a single re-derive … on the next spawn".
    Clears when: the PR states, with evidence from the affected host, that a worker started with an older AIM_CREDS_AGENT_INJECTION value still gets working creds-agent credentials. Alternatively, the gate ignores the nonce while the derivation still copies the live value.

Suggestions

  • KIROCREW_SKILL_VIEW_VOLATILE_ENV now also widens what the worker freshness gate exempts. Mention that in agent_spec_format.py and in the worker-gate spec, so an operator who extends the list for skill views knows it also relaxes the worker revocation check.

[DESIGN-REVIEWED] 33c0a5b

Description read

[DESCRIPTION-READ] 25c14d52de6b2993c04c3f668482562361b9d15790b3980b9c97b8e61b5a2dc3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 33c0a5bbed0f95e95f5de40d55795ba21c10c9da via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The fix moves the volatile-key list but not the normalizer, so _without_volatile_mcp_env becomes a second copy of skill_projection._alias_identity.

Not justified as shipped

  • Item 1 — duplicate of src/kiro_crew/acp/skill_projection.py:_alias_identity: _alias_identity already maps mcpServers to "every key, and … each value but the volatile ones" through _env_identity, keeping the keys. The new helper does the same job with its own marker ("<volatile>" where _env_identity uses None). The description's own harvest names the cause: "two digests of the same structure … with different exclusion lists". Sharing the list fixes the list, but the logic that applies it still lives in two places.
    Subtraction: move _alias_identity/_env_identity into agent_spec_format beside the list. Then delete _without_volatile_mcp_env and _VOLATILE_VALUE_SENTINEL, and have _spec_fingerprint call the moved helper.
    Clears when: one function normalizes volatile mcpServers env for both digests.
  • Item 3 — zero consumers: the description says the re-exported _VOLATILE_ENV_KEYS_DEFAULT = VOLATILE_ENV_KEYS_DEFAULT and _VOLATILE_ENV_VAR are kept so callers and tests are unchanged. I grepped src/ and test/ for both names and found 0 readers once volatile_env_keys delegates. The pass-through volatile_env_keys() wrapper has 2 internal callers, and a direct import would serve them.
    Subtraction: delete both aliases and the wrapper, and import volatile_env_keys straight from agent_spec_format.
    Clears when: the aliases and the wrapper are deleted.

What this change ships

Inventory (3 items) — 1 justified

Intent: a worker spawn should survive another session re-stamping a launcher nonce at the same moment. This is a FIX.

  1. Worker spawns no longer die when a per-launch MCP env nonce changes during load — duplicate of skill_projection._alias_identity
  2. The volatile-key list and its KIROCREW_SKILL_VIEW_VOLATILE_ENV extension move to agent_spec_format — justified
  3. skill_projection keeps re-export aliases and a pass-through wrapper — zero consumers, rides along

[FIRST-PRINCIPLES-REVIEWED] 33c0a5b

Description read

[DESCRIPTION-READ] 4ac2e62013e8d30b3c74490173a091803092ef012eaf61f434c918db94920e38

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 33c0a5bbed0f95e95f5de40d55795ba21c10c9da via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 33c0a5b

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ human override accepted

@buluoray overrode this lane for 33c0a5bbed0f95e95f5de40d55795ba21c10c9da (the recorded decision, this commit only); the model review was skipped.

[GPT-OVERRIDE] 33c0a5b

@paigebelliveau
paigebelliveau force-pushed the fix/worker-spec-fingerprint-volatile-env branch from 2acbbc7 to a51f1dc Compare October 5, 2026 02:38
@paigebelliveau

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/acp/skill_projection.py (and siblings this PR imports) since this head went green (595ad42 -> f481b88); CI re-running. The previous head's Backend Lint & Type Check shard hit its 25-minute job timeout before mypy finished (runner stall, not a type error); the new head re-runs it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@paigebelliveau

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted: the worker spec does not depend on a re-derive to receive the per-launch value (worker_agent.py pre-spawn freshness check, advisory FINDING)

The launcher that stamps the volatile key rewrites every agent spec in the agents directory in one pass, the worker spec included, and assigns each file its own value: observed on a live host, kirocrew.json and kirocrew-worker.json carry identical mtimes to the nanosecond and different values for that key after one launch.
The worker's MCP server is started from the worker's own spec, so it runs with the value the launcher stamped into that file, never with one copied from the default spec. A re-derive would overwrite that per-file value with the default's copy until the next launch re-stamps it; skipping the re-derive leaves the launcher as the sole owner of the key, which is the documented contract for VOLATILE_ENV_KEYS_DEFAULT ("a per-launch nonce, not a credential the agent's grants depend on").
Keeping an exact fingerprint for the pre-spawn check would re-derive the mirror on every launch for a key no consumer reads from the mirror, and the finding names no consumer that does; that is the same class of nonce-driven churn this PR removes from the post-load gate.
Every other mcpServers env value still moves the fingerprint and still re-derives: test_a_per_launch_env_nonce_does_not_move_the_fingerprint locks in that a non-volatile value change and a volatile key removal both change it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 5, 2026
@paigebelliveau
paigebelliveau force-pushed the fix/worker-spec-fingerprint-volatile-env branch from a51f1dc to f70ddd2 Compare October 5, 2026 03:13
@paigebelliveau

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in sibling modules this PR imports (src/kiro_crew/sandbox*.py, cron_script.py, slack/gateway.py) since this head went green (f481b88 -> 161f731); CI re-running. Note for maintainers: Backend Lint & Type Check has now hit its 25-minute job timeout twice on this PR with the Check formatting (black) step alone taking ~15 minutes on the fork runner; mypy itself never finished. The diff does not touch formatting or the lint configuration, and the same job passed on main at f481b88 in 18.5 minutes. If it times out again on this head a maintainer rerun of that job is the only lever; a fork contributor cannot rerun it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 5, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 5, 2026
@paigebelliveau

Copy link
Copy Markdown
Contributor Author

Maintainer ask: the Backend Lint & Type Check (3.12) job has hit its 25-minute timeout on three consecutive heads of this PR (2acbbc7, a51f1dc, f70ddd2), each time at the Type check (mypy) step after the Check formatting (black) step consumed roughly 15 minutes. Every other check is green (87 pass, 0 fail on f70ddd2). The same job passes on main in about 18 minutes (f481b88), and this diff touches no formatting or lint configuration, so this looks like fork-runner slowness rather than a defect. A fork contributor cannot rerun the job; could a maintainer rerun just that job on f70ddd2 (run 37258560676)? I am holding off on further rebases so the rerun lands on a stable head.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@paigebelliveau

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted: the worker spec never holds a copied nonce, so there is no stale value to keep (Watch: "The worker spec keeps an old launcher nonce")

The stamping is done by the launcher, outside this codebase: on every launch it rewrites every spec in the agents directory in one pass, kirocrew-worker.json included, and gives each file its own value. Observed on a live host after one launch: kirocrew.json and kirocrew-worker.json have identical mtimes to the nanosecond and different values for AIM_CREDS_AGENT_INJECTION.
The worker's creds-agent server is started from the worker's own spec, so it runs with the value the launcher stamped into that file. It never ran on a value copied from the default spec; before this PR the re-derive copied the default's value in and the next launch overwrote it again.
So the stated consequence (every worker hands creds-agent an old nonce) does not arise: the launcher is the sole writer of the key in both files, and skipping the re-derive leaves its per-file value untouched. That is the documented contract for VOLATILE_ENV_KEYS_DEFAULT: a per-launch nonce, not a credential the agent's grants depend on.
The "clears when" test cannot be written as stated because the mirror does not carry "the default's current nonce" in either the old or the new code; the consequence of the mirror keeping a stale copy is nil because no consumer reads the key from the mirror. Stated and accepted.
Every other mcpServers env value still moves both fingerprints and still re-derives: test_a_per_launch_env_nonce_does_not_move_the_fingerprint locks in that a non-volatile value change and a volatile key removal both change it.

@paigebelliveau

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted: splitting the fingerprint would re-derive the mirror on every launch for a key nothing reads from the mirror (Suggestion: nonce-blind only in the post-load comparison)

The suggestion's benefit is "the worker picks up the current nonce at the next spawn". The launcher already stamps kirocrew-worker.json directly with its own per-file value on every launch (see the Watch disposition), so a re-derive gains nothing and briefly overwrites the launcher's value with the default's copy until the next launch.
Its cost is one full re-derive of the worker spec per launch for a change in a key that is, by contract, a per-launch nonce; that is the same class of nonce-driven churn this PR removes from the post-load gate, moved one step earlier.
Two fingerprint flavours for one spec also means set_mirrored_from and require_unchanged_derived_spec can disagree about whether the same two files match, which is a new way for the post-load check to fire on a mirror the pre-spawn check just accepted.
One fingerprint, blind to the volatile keys at both sites, keeps the two checks consistent and keeps the launcher as the sole owner of the key. Disproportional for the goal; the code stays as-is.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 6, 2026
@buluoray

buluoray commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

/ai-review override gpt 33c0a5b: agent: buluoray ruled this test-only (equal-length nonce rewrite leans on mtime; runtime fix unaffected); the one-line nonce-length fix is tracked in #17616 after merge.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded

@buluoray marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 33c0a5bbed0f95e95f5de40d55795ba21c10c9da.

agent: buluoray ruled this test-only (equal-length nonce rewrite leans on mtime; runtime fix unaffected); the one-line nonce-length fix is tracked in #17616 after merge.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@peterhieuvu

Copy link
Copy Markdown
Contributor

Verified this live on Linux under a sandbox launcher that stamps a fresh env value into every agent spec on each start: on head 33c0a5b, one kirocrew-worker and three concurrent ones all reached a tool call with no DerivedSpecStale (main fails both). Since this now needs a rebase and it's blocking kirocrew-worker for several people, I've opened #17895 carrying your commit unchanged plus the rebase and the test-determinism fix, with you as co-author. Happy to close mine if you'd rather land this one.

@paigebelliveau

Copy link
Copy Markdown
Contributor Author

Hey Peter! Thank you so much for carrying it forward and for the live verification 😊 Please land #17895, no need to close yours. Closing this one in its favor.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants