Skip to content

fix(windows): fail the probe, not the gateway, on a bad tree scan - #35992

Merged
buluoray merged 1 commit into
mainfrom
crew/slack/windows-probe-tracker-failure
Nov 12, 2026
Merged

buluoray merged 1 commit into
mainfrom
crew/slack/windows-probe-tracker-failure

Conversation

@buluoray

@buluoray buluoray commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Windows gateway keeps starting, and keeps its kiro-cli version checks working, when one scan of a kiro-cli probe's process tree fails.

On Windows the gateway sometimes exits while it boots. The log ends with:

File "...\kiro_crew\kiro_prerequisite_process.py", line 645, in _wait_for_completion
asyncio.exceptions.CancelledError

The gateway reads the kiro-cli version at boot through kiro_cli._sandboxed_probe. That function promises that it never raises. On Windows the probe tracks its child's process tree. When a later scan of that tree fails, the tracker cancels the future that the probe run is waiting on. The run reads that as a cancel of itself and raises CancelledError. CancelledError is not an Exception, so the probe's guard does not catch it, and the gateway exits before it is ready.

Why it matters

A Windows user whose gateway hits this sees Kiro Crew fail to start. In CI, 8 of the 261 E2E Gateway Boot (windows-latest) legs created after 09:06Z on 2026-11-12 failed this way, counted at about 16:20Z (3.1%). The probe path came in with #35731, which merged at 09:06Z that day. The red legs hit unrelated pull requests (#31604, #34811, #35831, #32983, #34766).

Not a goal

  • This PR does not make the tree scan fail less often. descendant_termination_handles_async is untouched.
  • This PR does not change which processes a probe ends, or how the probe is wrapped in the sandbox.
  • This PR does not change how a timed-out probe is cached. That stays as on main.
  • This PR adds no retry bound to the three probe readers that main already re-runs after every failed read (kas_relay_help, kas_relay_start_output, kiro_acp_engine_args).
  • POSIX runs are untouched.

What changed (motivation → approach → change)

The tracker's own error now reaches the probe run as that error, not as a cancel. The run ends the process tree as before, logs the reason, and returns a failed result. The gateway keeps booting. The next reader of the version probes again, and after 3 such failed runs in a row on one binary the answer is kept as unknown.

Part Before After
_track_windows_descendants (tracker) cancels the future the run awaits sets its error on that future, then re-raises
_run_process (run) raises CancelledError ends the tree, logs the reason, returns ok=False with tree_unverified=True
kiro_cli._probe_ran (reader) no such case reads that result as unknown, never as the child's answer
kiro_cli_version_at, chat_agent_engines_at (caches) no such case leave that answer out of the cache, up to 3 runs
A cancel from outside the run propagates propagates

How the two cancels are told apart. The run does not catch CancelledError at all. The tracker is the only inside source of one, and it now hands over its real error. So any CancelledError the run still sees came from outside, and it propagates after the same teardown.

What happens to the probe's process tree. Lines are in src/kiro_crew/kiro_prerequisite_process.py. "Before" is 8d04c52d2213.

before: tracker fails -> finally :332 cancels the waiter
        -> gather :645 raises CancelledError
        -> :668 _terminate_process(proc, windows_descendants)
        -> re-raise; handles closed at :691 and :693

after:  tracker fails -> except :336 sets its error on the waiter (:343)
        -> gather :659 raises that error
        -> :702 _terminate_process(proc, windows_descendants)
        -> :712 ok=False, tree_unverified=True
        -> reason logged at :721; handles closed at :727 and :729

The teardown call is the same one in both paths. It terminates the leader and calls terminate_process_handle on every retained handle. The run never reports success, so containment stays fail-closed.

How often a failed read is retried. A run that ended its tree unverified says nothing about the binary, like a sandbox that could not be built. Unlike that case, it did spawn kiro-cli. So kiro_cli._retry_unverified_probe counts such runs per binary and per probe. The first two stay out of the cache, and the next reader probes again at once. The third is cached as unknown, the way main caches a timed-out probe. There is no clock and no backoff: the count is the bound, three spawns at most.

What the gateway does when the version probe fails. It keeps running. While the version is unknown, spec writers leave out the permissions block (agent_materialization/auto_approve.py), each chat keeps its own process (acp/chat_runtime_sharing.py), Crew's MCP servers are not deferred, and kirocrew doctor prints "kiro-cli version unknown". With the retry, one failed scan costs that for one read only.

Security (builder-security skill). This PR opens no listener, port, route, storage grant or sign-in path, and it weakens none. The probe still goes through _prepare_sandboxed_spawn with first_party_fixed_argv; that code is not in the diff. The diff adds no bind or host argument.

docs/system-specs/modules/security.md describes the tracker hand-over, tree_unverified and the retry count.

Backwards compatibility

Compatible: ProcessResult.tree_unverified is a new field that defaults to False, and a run that used to raise CancelledError now returns a failed result that every reader already treats as a failure.

Removes nothing: no path, key, record or public function is hidden, deleted or tightened.

Tests

Proof: PROVEN test/test_kiro_prerequisite_process.py::TestWindowsTrackerFailureDuringARun::test_a_tracker_that_fails_under_a_timeout_is_still_logged (head 7c947f3)

All tests are in test/test_kiro_prerequisite_process.py.

Test What it locks in
test_an_unexpected_failure_reaches_the_waiting_run_as_its_own_error, test_a_failed_later_snapshot_reaches_the_waiting_run_as_its_own_error the tracker hands its error to the waiter
test_a_cancelled_tracker_still_cancels_its_waiters a cancelled tracker still cancels
test_the_run_ends_the_tree_and_returns_a_failed_result (4 cases) failed result, tree ended, handles closed, reason logged
test_a_cancel_from_outside_still_cancels_the_run an outside cancel is still a cancel
test_a_run_that_times_out_is_still_a_timeout, test_a_tracker_that_fails_under_a_timeout_is_still_logged the timeout path is unchanged and the tracker error is still logged
test_the_version_probe_fails_and_the_spec_gate_still_answers the boot path survives: _sandboxed_probe returns a result and the spec gate answers
test_a_run_that_was_cut_short_is_not_read_as_the_childs_answer partial help with exit 0 is unknown
test_the_next_read_probes_again_and_gets_the_real_version, test_a_tree_that_never_verifies_is_probed_a_bounded_number_of_times one failed scan is retried; three in a row end the retries

How the Windows branch runs. On Linux the tests set platform_compat.IS_WINDOWS and replace only the Windows handle calls (duplicate_asyncio_process_handle, process_handle_active, close_process_handle, terminate_process_handle, descendant_termination_handles_async) with recorded fakes. The real _run_process, _track_windows_descendants and _run_on_private_loop code runs. The file is not in test/windows-collect-ignore.txt, so the Windows shards run it too. No test here calls the real Win32 handle functions.

Red before the change. On unmodified source the new tests failed with asyncio.exceptions.CancelledError at kiro_prerequisite_process.py:645, the line in the CI log. The two retry tests failed with assert None == (2, 23, 0) and assert 1 == 3 before the retry was added.

Mutations, each run alone, source restored after each.

Mutation Result
Tracker does not hand its error to the waiter killed, 6 tests red
tree_unverified is always False killed, 6 red
_probe_ran ignores tree_unverified killed, 1 red
The warning is not logged killed, 6 red
Every cancel becomes a failed result killed, 2 red
The tracker task is not read as an error source killed, 1 red
A failed scan is cached at once (no retry) killed, 3 red
A failed scan is never cached (no bound) killed, 2 red

Determinism. 20 of 20 repeats at -n0. 10 of 10 repeats at -n 4 --dist loadgroup with test_kiro_probe_spawn_off_loop.py and test_kiro_prerequisite_supervisor.py beside it. One shuffled order, seed 25379: 37 passed. No test sleeps or reads a clock; the poll interval is set to 0 and every wait ends on an event or a cancel.

Local gate. scripts/local-gate.py on this patch before it was rebased (f109c49529d2): 10279 passed, 5 failed. The same 5 fail on the base 8d04c52d2213 on this host and are host-dependent: test_kas_cli_floor.py::test_a_pinned_link_flipped_after_the_check_never_runs_unsandboxed, test_kas_engine_refusal.py::test_the_probe_reads_the_refusal_from_a_real_start, test_kiro_cli_probe_sandbox.py::test_the_version_probe_child_runs_under_the_sandbox_launcher, test_materialization_restamp_probe.py::test_permissions_removed_for_an_old_cli_is_vouched, test_native_skill_projection.py::test_launch_record_resources_match_the_alias_file_kiro_cli_loads.

Manual verification

Not run on a real Windows host. One green E2E Gateway Boot (windows-latest) leg proves little here: the failure shows in about 3 of 100 legs, so a green leg is the likely result with or without the fix. The deterministic tests above are the evidence.

Related Issues

Closes #35849

Sibling sweep

Swept: git grep -nE "_run_process(" -- src/
Sibling: src/kiro_crew/agent_sdk/drivers/acp.py:854 -- not affected -- it has no tracker of its own; it awaits the shared _run_process that this PR fixes
Swept: git grep -nE "_cache[key] = (version|engines)" -- src/kiro_crew/kiro_cli.py
Sibling: src/kiro_crew/kiro_cli.py:525 -- changed here

Pattern harvest

Rule candidate: review-prompt
Pattern: a helper task cancels the future its caller awaits when the helper itself fails, so the caller reads the helper's failure as its own cancellation

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

On Windows a kiro-cli probe tracks its child's process tree. When a
scan of that tree failed after the first one, the tracker cancelled
the future the run was waiting on. The run read that as a cancel of
itself and raised CancelledError, which the probe's "never raises"
guard does not catch, so a gateway that reads the kiro-cli version
while it boots exited before it was ready.

The tracker now hands its own error to the waiting run. The run ends
the leader and every retained handle as before, logs the reason, and
returns a failed result marked tree_unverified. A probe reader treats
that result as unknown. A cancel of the run itself still propagates.

Closes #35849
@buluoray
buluoray force-pushed the crew/slack/windows-probe-tracker-failure branch from 874a46f to 7c947f3 Compare November 12, 2026 19:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 7c947f36dc66f754167b5c79d462645e9bca5f1a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7c947f36dc66f754167b5c79d462645e9bca5f1a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The CI traceback and the red-before tests show the cause, the fix sits at the tracker that turned its own failure into a cancel, and the retry bound is inside the stated Goal.

What this change ships

Inventory (6 items) — 6 justified

Intent: keep a Windows gateway booting, with working kiro-cli version checks, when one scan of a probe's process tree fails. FIX.

  1. A Windows gateway no longer exits at boot when a probe's tree scan fails (the tracker passes its error to the run instead of cancelling it) — justified
  2. A probe run whose tree scan failed now ends the tree and returns a failed result marked tree_unverified (2 src consumers: _probe_ran, _retry_unverified_probe) — justified
  3. A warning naming the tracker's error is logged when that happens — justified
  4. Help-text readers (kas_relay_help, kas_relay_start_output, login_target) treat a cut-short run as unknown, not as the child's answer — justified
  5. The version and chat --help probes skip the cache for an unverified run and probe again, up to 3 runs per binary — justified
  6. The security spec section describes the hand-over, tree_unverified and the retry count — justified

[FIRST-PRINCIPLES-REVIEWED] 7c947f3

Description read

[DESCRIPTION-READ] 37999b1599710c93277c257168aca25af43e45f6773d0c341df91095b15e96a8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7c947f36dc66f754167b5c79d462645e9bca5f1a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the tracker's error now reaches the run as that error instead of a cancel, which fixes the actual cause, and outside cancels still propagate.

Notes

  • _unverified_probe_runs is only ever incremented. A verified run never resets it, so "3 runs in a row" really means 3 in total per (probe, binary, key). This only matters if the key's int can repeat across reads. Either reset the count on a verified result or say "in total" in the spec.

[DESIGN-REVIEWED] 7c947f3

Description read

[DESCRIPTION-READ] 76f42ff2abdffa4440e6cb0e99b1f7fd6a72bc9ea4b403edd626edb31eaa253f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7c947f36dc66f754167b5c79d462645e9bca5f1a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7c947f3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7c947f36dc66f754167b5c79d462645e9bca5f1a false-positive: <reason, 20 to 500 characters>

@buluoray

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/security.md and in files beside this PR's files under src/kiro_crew/ and docs/system-specs/modules/ since this head's checks ran (40cfcd6ad440 -> 7325afe29c14, 29 commits); CI re-running.

The patch is unchanged: git patch-id gives the same id before and after. New head 7c947f36dc66.

On the old head 874a46fd9f40 every red check but two failed only with API rate limit exceeded for installation (all review lanes, Dependency License Gate, Analyze (actions), Publish readiness signal). The other two were Backend Tests (3.12, 7) on the known flaky test in #28212 (recorded there) and Coverage Gate, which failed only because that shard did. No review lane published a verdict on that head.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7c947f36dc66f754167b5c79d462645e9bca5f1a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7c947f3

Verdict parsed from the review's SHA-scoped output markers for commit 7c947f36dc66f754167b5c79d462645e9bca5f1a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7c947f36dc66f754167b5c79d462645e9bca5f1a false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 12, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the whole diff at 7c947f3, tests included.

On Windows, a failure of the descendant tracker during a kiro-cli probe used to reach the run as a cancellation, which could fail the gateway boot instead of the one probe. Now the tracker hands its own error to the snapshot the run is waiting on, so the run fails with that error. It ends the leader and every retained child handle, closes the handles, logs the reason once, and returns a failed result with tree_unverified set. A real cancel from outside still ends the same teardown and then propagates.

Checks:

  • Fail-closed is kept. The leader and the retained children are still ended and no handle leaks, in each of the four failure timings (first or later scan, expected or unexpected error), and also when the leader exited first. Nothing is trusted that was not verified.
  • The probe readers treat an unverified run as unknown. _probe_ran is false for it, so a cut-short run with a zero exit and partial help text is not read as "this binary has no KAS". The cached probes keep an unverified run out of their cache for three runs in a row, then cache it as unknown, as a timed-out probe is. Each retry spawns kiro-cli, so the bound matters, and a test pins it for both probes.
  • A timeout is still a timeout, and a tracker that fails under a timeout is still logged.
  • No regex, no new refusal, no change to what a probe on a healthy tree returns.

The GPT review run cancelled at 19:20 has a passing run at 19:22 on this head, so it is not live red. All other checks are green. Approving without auto, since auto-merge is already set.

@buluoray
buluoray merged commit 93b7d36 into main Nov 12, 2026
104 of 105 checks passed
@buluoray
buluoray deleted the crew/slack/windows-probe-tracker-failure branch November 12, 2026 20:27
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_tool_marker_prompt_completes_the_turn and other gateway boot matrix tests die in the kiro-cli version probe with CancelledError on Windows

2 participants