Skip to content

fix(agent): ignore per-launch MCP env nonces in the worker spec fingerprint - #17895

Merged
buluoray merged 2 commits into
kirodotdev:mainfrom
peterhieuvu:fix/worker-spec-fingerprint-volatile-env
Oct 8, 2026
Merged

buluoray merged 2 commits into
kirodotdev:mainfrom
peterhieuvu:fix/worker-spec-fingerprint-volatile-env

Conversation

@peterhieuvu

@peterhieuvu peterhieuvu commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #16879 by @paigebelliveau, who closed it in favour of this PR. This PR carries their commit rebased onto current main, plus one co-authored commit. The fix and the credit are theirs.

Review requested from @chenmingwei23.

Problem / Motivation

Goal: A kirocrew-worker spawn survives another session launching at the same moment.

A sandbox launcher that stamps a fresh env value into every agent spec each time a session starts (#15119) changes the worker spec fingerprint, so a kirocrew-worker whose load overlaps any launch, its own included, dies with DerivedSpecStale before its first turn (full write-up in #16879).

Why it matters

On a host with such a launcher, every kirocrew-worker dispatch dies, because the worker's own launch re-stamps the default spec too. A conductor cannot hand work to a worker, and each failure is reported as a runtime error with the task undone. The gate is right to end a worker when a grant changes under it, but a per-launch nonce is not a grant.

Not a goal

  • Changing how or when the launcher stamps its value.
  • Changing which keys the worker mirrors (tools, allowedTools, excludedTools, mcpServers, model).
  • Ignoring any env value that is not on the volatile list: a rotated credential still changes the fingerprint.

What changed (motivation → approach → change)

The worker gate fingerprints the default spec's mirrored keys before the spawn and again after initialize, and ends the session when the two differ. The same fingerprint decides whether the worker mirror needs a re-derive. It hashed every mcpServers env value, so the launcher's per-launch value moved it on every launch. Skill-view identity already skips those keys through volatile_env_keys(); the gate did not.

Part Commit Change
The fix @paigebelliveau, carried rebased agent_materialization/worker_agent.py hashes each volatile env value as a fixed marker. The key, every other env value, command, args, tools, allow and deny lists, and model still move the hash. The volatile list moves to agent_spec_format.py, so the gate does not import the ACP layer. The rebase merged one import: main put SKILL_VIEW_PROJECTION_CEILING on the same skill_projection.py line, so the commit's grouped import names it too.
One import source co-authored acp/skill_projection.py imports volatile_env_keys straight from agent_spec_format.py, with no alias constants or forwarding wrapper: nothing read them. worker_agent.py takes its two agent_spec_format names on one line, where the carried commit keeps main's parse_markdown_spec import beside its own.
Docs co-authored _without_volatile_mcp_env names both checks that read the fingerprint, and why a value-only change may skip the mirror re-derive. docs/system-specs/modules/acp-client.md and the comment beside the list say a key added through KIROCREW_SKILL_VIEW_VOLATILE_ENV must be one the launcher writes into every agent spec it manages, the worker's included, because the worker gate ignores its value too.
Test determinism co-authored The nonce-restamp test's second nonce is longer, so the file size always changes. With equal lengths, its identity assertion rested on st_mtime_ns alone and could fail on a coarse-timestamp filesystem. The nonce-fingerprint test clears KIROCREW_SKILL_VIEW_VOLATILE_ENV, as its sibling tests do, so an extended volatile list in the environment cannot change its result.

Backwards compatibility

Compatible: a spec with no volatile env key fingerprints exactly as before. A spec with one hashes differently once, and the gate answers that with one re-derive of the worker mirror on the next spawn. skill_projection.volatile_env_keys stays importable under the same name, now the function imported from agent_spec_format.py. The private _VOLATILE_ENV_KEYS_DEFAULT and _VOLATILE_ENV_VAR move to agent_spec_format.py and stay private; nothing on main reads them from outside the module.

  • Reader: src/kiro_crew/acp/skill_projection.py:_note_env_values -- chat -- calls volatile_env_keys(), the same function imported from agent_spec_format; every ACP spawn reaches it (chat, subagent, cron); test_native_skill_projection.py
  • Reader: src/kiro_crew/acp/skill_projection.py:_alias_identity -- chat -- the same import; test_native_skill_projection.py, test_skill_view_relaunch_harness.py

Tests

  • test_a_per_launch_env_nonce_does_not_move_the_fingerprint: a changed volatile value leaves the fingerprint unchanged, while another env value in the same entry, or removing the key, still moves it.
  • test_a_nonce_restamp_during_the_load_does_not_end_the_session: capture, a re-stamp that changes the file identity, then require_unchanged_derived_spec returns instead of raising.

Both fail on main with the production DerivedSpecStale and pass here. test_worker_agent.py, test_native_skill_projection.py and test_skill_view_relaunch_harness.py: 386 passed, which covers both skill_projection readers above.

Manual verification

Verified live on Linux under a sandbox launcher that stamps a per-launch env value into every agent spec, on #16879's head 33c0a5b, whose behaviour matches this PR. One kirocrew-worker and three concurrent ones each reached a tool call (pwd) and replied, with no DerivedSpecStale. The launcher re-stamped the default spec mid-run, inside the window main rejects; main fails both cases.

On the same host, two read-only snapshots of the agents directory, with values hashed and never printed, show each launch rewriting every spec in one pass, kirocrew-worker.json included, each with its own new value. So skipping the re-derive on a value-only change never leaves the worker on a copied value.

Related Issues

Closes #17616

Pattern harvest

Rule candidate: review-prompt
Pattern: two digests of the same structure (skill-view identity and worker-spec fingerprint) with different exclusion lists; any new hash over mcpServers must consume volatile_env_keys().

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@peterhieuvu
peterhieuvu requested a review from a team as a code owner October 7, 2026 21:58
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision fafea925842e11c43e0eb52ad1e255cf49ddbb6e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of fafea925842e11c43e0eb52ad1e255cf49ddbb6e via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merge, confirm that the launcher re-stamps kirocrew-worker.json itself; the mirror re-derive skip relies on that, and only one host's live run shows it.

[DESIGN-REVIEWED] fafea92

Description read

[DESCRIPTION-READ] 9aa5b061281b96f6658ccd56d5eb343c6f10529bf1d23fa54ccf25cab0a3064c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of fafea925842e11c43e0eb52ad1e255cf49ddbb6e via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check one claim before merge: the launcher writes its own nonce into kirocrew-worker.json. If it doesn't, skipping the mirror re-derive leaves the worker on a stale value.

What this change ships

Inventory (7 items) — 7 justified

Intent: let a kirocrew-worker dispatch survive a session launching at the same moment, without DerivedSpecStale (#17616). This is a FIX.

  1. A worker no longer dies when a concurrent launch re-stamps a volatile MCP env value in the default spec — justified
  2. Removing a volatile key, or changing any other env value, still moves the worker fingerprint — justified
  3. A change to a volatile value alone no longer triggers a re-derive of the worker mirror — justified
  4. Keys added through KIROCREW_SKILL_VIEW_VOLATILE_ENV now also widen what the worker gate ignores — justified
  5. volatile_env_keys and its private constants move from acp/skill_projection.py to agent_spec_format.py (0 kiro_crew.acp imports under agent_materialization/) — justified
  6. The acp-client spec now says the worker gate reads the volatile list — justified
  7. Test determinism: a longer second nonce, and the env var cleared in the fingerprint test — justified

[FIRST-PRINCIPLES-REVIEWED] fafea92

Description read

[DESCRIPTION-READ] f4ff7b75f86d71537c92e287f50e0769b189af22ba93afdac5836e58e2cffb86

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed fafea925842e11c43e0eb52ad1e255cf49ddbb6e via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fafea92

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed fafea925842e11c43e0eb52ad1e255cf49ddbb6e via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fafea92

@peterhieuvu

Copy link
Copy Markdown
Contributor Author

self-added: no

  • rebutted: the launcher writes the worker spec itself, so the mirror never runs on a copied nonce (Watch: "The mirror can keep an old nonce")

This meets the first Clears-when, observed read-only on the affected host with the values hashed and never printed. One launch rewrote all 86 agent specs in ~/.kiro/agents in one pass of about 50 ms. kirocrew-worker.json and kirocrew.json have the same st_mtime_ns, and all 86 volatile values are distinct.
A second snapshot after a later launch: all 86 files were rewritten again, kirocrew-worker.json included, each with a new value.
A gateway re-derive writes only kirocrew-worker.json and copies the default's value, so it cannot produce either pattern. The launcher is the writer of this key in every spec, the worker's included, each time it starts kiro-cli.
So a value-only change that skips the re-derive leaves the worker on the value the launcher stamped for that launch. Any non-volatile change still moves both checks and still re-derives.

@peterhieuvu

Copy link
Copy Markdown
Contributor Author

The four red backend shards on 721ec75 are main's, not this diff's. Two tests fail, and both fail the same way on a pristine upstream/main checkout (828dd0b) with no PR code. Neither test file, nor anything it reads, is in this diff:

  • test/test_session_control_doc.py::test_every_tabulated_refusal_code_is_one_the_source_raises (Linux shard 6, Windows shard 6)
  • test/test_http_server_bind_ratchet.py::test_bare_http_server_constructions_match_the_recorded_count (Linux shard 2, Windows shard 2)

Tracked in #17900, with the fix in flight in #17907. This PR will rebase onto main once that lands, rather than push around the red.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@peterhieuvu
peterhieuvu force-pushed the fix/worker-spec-fingerprint-volatile-env branch from 721ec75 to ac79a74 Compare October 8, 2026 00:16
@peterhieuvu

Copy link
Copy Markdown
Contributor Author

self-added: no

  • fixed in ac79a74: the alias constants and the forwarding wrapper are removed (Item 3, zero consumers)

skill_projection.py imports volatile_env_keys straight from agent_spec_format.py. _VOLATILE_ENV_KEYS_DEFAULT, _VOLATILE_ENV_VAR, the wrapper def and the comment that claimed callers and tests used them are deleted.
git grep across src/ and test/ finds no reader of any of them outside the module, and no test patches them. Both in-module callers (_note_env_values, _alias_identity) resolve to the imported function; the PR body lists them as Reader: lines.

@peterhieuvu

Copy link
Copy Markdown
Contributor Author

self-added: no

  • fixed in ac79a74: the _without_volatile_mcp_env docstring names both readers of the fingerprint (Suggestion: name the two uses of the fingerprint)

It names the post-load bracket (require_unchanged_derived_spec) and the mirror-freshness check (_derived_spec_matches_default, which compares against the fingerprint set_mirrored_from records at derive time).
It states that a change to a volatile value alone also skips the mirror re-derive, and why that is safe: the launcher writes every agent spec it manages, the worker's own included, each with its own per-launch value.

@peterhieuvu
peterhieuvu force-pushed the fix/worker-spec-fingerprint-volatile-env branch from ac79a74 to ac5bcb6 Compare October 8, 2026 01:02
@peterhieuvu

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: removing the skill_projection re-export aliases left the public constants in agent_spec_format.py with no outside reader

  • fixed in ac5bcb6: the volatile-list constants are private again (Item 4, zero consumers)

agent_spec_format.py names them _VOLATILE_ENV_KEYS_DEFAULT and _VOLATILE_ENV_VAR, the names main used, and exports only volatile_env_keys(), which both callers import.
git grep -E '\bVOLATILE_ENV_KEYS_DEFAULT\b|\bVOLATILE_ENV_VAR\b' across the whole repo returns nothing at the new head.

@peterhieuvu

Copy link
Copy Markdown
Contributor Author

self-added: no

  • fixed in ac5bcb6: the docs say a key declared volatile must be one the launcher writes into every agent spec, the worker's included (Watch: "One env var now drives two decisions")

This meets the first Clears-when. docs/system-specs/modules/acp-client.md now says the same list governs the worker-spec freshness gate: a change to a volatile value alone neither ends a worker load (require_unchanged_derived_spec) nor re-derives the kirocrew-worker.json mirror.
It follows that a key declared volatile through KIROCREW_SKILL_VIEW_VOLATILE_ENV must be one the launcher writes into every agent spec it manages, the worker's included. The comment beside the list in agent_spec_format.py says the same.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@peterhieuvu
peterhieuvu force-pushed the fix/worker-spec-fingerprint-volatile-env branch from ac5bcb6 to 4d4e7b5 Compare October 8, 2026 01:59
@peterhieuvu

Copy link
Copy Markdown
Contributor Author

Rebased onto main 5dbde27 to clear a merge conflict with #17870, which rewrote the same agent_spec_format import line in src/kiro_crew/acp/skill_projection.py. In the carried commit, the grouped import now also names SKILL_VIEW_PROJECTION_CEILING; nothing else in that commit changed, and its author and message are as they were. The co-authored commit resolves to one import of NATIVE_SKILL_ALIAS_PREFIX, SKILL_VIEW_PROJECTION_CEILING and volatile_env_keys. Locally the three related test modules pass (391), including #17870's ceiling tests. CI re-running.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 8, 2026
@peterhieuvu

Copy link
Copy Markdown
Contributor Author

Round 4 on 4d4e7b5 finished with one red: Backend Tests (Windows) (8), failing test/test_subagent_memory_wake.py::TestOneEventStartsAWaitingSpawn::test_a_sampler_tick_starts_it_and_runs_only_while_a_start_waits with AssertionError: 3600.0000000000005. That is the Windows float round-trip flake tracked in #17876, also seen on unrelated PRs; this diff touches neither that test nor anything it imports. Every other check and all five AI review lanes are green.

A fork account cannot re-run jobs. Could a maintainer re-run the failed job? https://github.com/kirodotdev/KiroCrew/actions/runs/37715637871/job/113112328799

@peterhieuvu
peterhieuvu force-pushed the fix/worker-spec-fingerprint-volatile-env branch from 4d4e7b5 to c7e66e3 Compare October 8, 2026 06:42
@peterhieuvu

Copy link
Copy Markdown
Contributor Author

Rebased onto main 372915e to pick up the flake fix #17957 (closes #17876), which failed Backend Tests (Windows) (8) on the previous head. Both commits' patches are unchanged by the rebase. One test-only line was added in the co-authored commit: test_a_per_launch_env_nonce_does_not_move_the_fingerprint now clears KIROCREW_SKILL_VIEW_VOLATILE_ENV first, as its sibling tests do, so an extended volatile list in the environment cannot change its result. Locally the related modules pass (416), including test_subagent_memory_wake.py. CI re-running.

@peterhieuvu
peterhieuvu force-pushed the fix/worker-spec-fingerprint-volatile-env branch from c7e66e3 to 5cf5f45 Compare October 8, 2026 07:18
@peterhieuvu

Copy link
Copy Markdown
Contributor Author

Rebased onto main 1c5a71d; both commits' patches are unchanged. On the previous head (c7e66e3), none of the reds came from this diff:

Locally the related modules pass (434), including test_crew_recency.py. CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@peterhieuvu

Copy link
Copy Markdown
Contributor Author

Round 6 on 5cf5f45: all five AI review lanes are clean (GPT 6.1 and Opus 5.5 found nothing; Design and First Principles PASS; UX skipped), and 86 checks pass. None of the three reds comes from this diff:

A fork account cannot re-run jobs. Could a maintainer re-run the failed jobs in this run? https://github.com/kirodotdev/KiroCrew/actions/runs/37742642107

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
paigebelliveau and others added 2 commits October 8, 2026 16:04
Small changes on top of the carried fix, none of them behavioural:

- The nonce-restamp test rewrote the default spec with a second nonce
  of the same length, so its identity assertion rested on st_mtime_ns
  alone; on a coarse-timestamp filesystem both writes can land in one
  tick. A longer nonce changes the size, so the identity always moves.
  The nonce-fingerprint test clears KIROCREW_SKILL_VIEW_VOLATILE_ENV,
  as its sibling tests do, so an extended volatile list in the
  environment cannot change its result.
- worker_agent.py imports from agent_spec_format on one line (the
  rebase left two side by side, and isort wants one), and the
  _without_volatile_mcp_env docstring names both checks that read the
  fingerprint and why a value-only change may skip the re-derive.
- skill_projection.py imports volatile_env_keys straight from
  agent_spec_format, without alias constants or a forwarding wrapper,
  and the list's constants stay private there: nothing else read them.
- The acp-client system spec and the comment beside the list say a key
  declared volatile must be one the launcher writes into every agent
  spec it manages, the worker's included, since the worker gate
  ignores its value too.

Co-authored-by: paigebelliveau <40582209+paigebelliveau@users.noreply.github.com>
@peterhieuvu
peterhieuvu force-pushed the fix/worker-spec-fingerprint-volatile-env branch from 5cf5f45 to fafea92 Compare October 8, 2026 16:07
@peterhieuvu

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/chat_runner.py (#18137; it imports agent_spec_format) and same-directory files since this head's CI ran (1c5a71d -> 8625baa). Both commits' patches are unchanged, and the related tests pass locally, including the chat_runner suites (1030). CI re-running.

This replaces my re-run request for 5cf5f45 above. That head's two reds, Integration test_owner_only_boot (#17964) and the test_platform_compat argv-fingerprint test, are main-side and may recur here.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 8, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head fafea92. PR Readiness success on this head, no CHANGES_REQUESTED, no open blocking review request, merges cleanly into current main.

Reviewed by Kiro (agent) on behalf of bolichen97 as part of a repo-wide ready-PR sweep.

@buluoray
buluoray merged commit 8af084c into kirodotdev:main Oct 8, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 8, 2026
@peterhieuvu
peterhieuvu deleted the fix/worker-spec-fingerprint-volatile-env branch October 9, 2026 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: make the nonce-rewrite worker spec test deterministic

4 participants