Skip to content

fix: name the cause when pod_up cannot see a scratch worktree - #33254

Merged
bolichen97 merged 1 commit into
mainfrom
crew/slack/devfleet-scratch-worktree
Nov 8, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/slack/devfleet-scratch-worktree

Conversation

@buluoray

@buluoray buluoray commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When an agent asks Dev Fleet for a pod on a worktree it cannot see because the worktree is in a session's scratch directory, the refusal names that cause and tells the agent where to create the worktree instead.

Agents create worktrees under $KIROCREW_SCRATCH, as the system prompt tells them to. pod_up on such a worktree answers worktree not found: <name> although the worktree exists, so other Bots cannot start pods and get no hint why.

Why it matters

Every agent that follows the prompt's worktree advice and then wants a pod hits this, and the message sends them looking for a typo. The only way out today is to know that a pod worktree must live outside scratch.

Not a goal

  • No change to the sandbox's scratch mask. One session must still not read another session's scratch.
  • No pods from a scratch worktree. A scratch directory is private to its session and is deleted when that session's processes are gone; a pod outlives a turn.
  • No change to the system prompt's scratch advice for clones in general.
  • No change to the pod stop timeout (see Related Issues).

What changed (motivation → approach → change)

Symptom: pod_up wt-stop answered worktree not found: wt-stop for a worktree under ~/.kiro/crew/scratch/runtime-8bd91637/.

Cause: Dev Fleet runs git worktree list --porcelain through the agent sandbox (runtime._run_cmd → sandboxed_spawn_argv). The sandbox masks the whole scratch root (sandbox_leaves), so that git reports any worktree in scratch as prunable gitdir file points to non-existent location, and repository._discover_worktrees drops prunable entries. Run through the same sandbox entry point, a worktree in this session's scratch came back prunable and one under <repo>/.worktrees/ did not.

Change:

Part What it does
repository._find_worktree On "not found", looks for a prunable worktree of that name under the scratch root and returns its record and the remedy instead
dev_fleet/scratch_worktrees.py The scratch check and the message (new module; repository.py is already long)
mcp_tools/apps.py The pod_up description says to create the worktree under <repo>/.worktrees/<name>
config/prompt.md The scratch rule names one exception: a worktree you will start a pod from goes in <clone>/.worktrees/<name>
pod-e2e skill Same advice, with the remove-then-add commands
kirocrew-worktree-dev skill One line: a pod worktree never goes under $KIROCREW_SCRATCH
docs/system-specs/modules/dev-fleet.md Discovery documents the scratch case and the remedy

The message, for a worktree under scratch on a branch (paths are shell-quoted). A detached one first gets a branch named after it, git -C <path> switch -c <name> && ..., then is re-added on that branch, so a commit made after the refusal is not orphaned when worktree remove deletes the tree's HEAD:

worktree 'wt-stop' is in a session's private scratch directory (<path>), which Dev Fleet cannot see, so it cannot start a pod from it. Recreate it under <repo>/.worktrees/: commit your changes first, then run git -C <repo> worktree remove <path> && git -C <repo> worktree add <repo>/.worktrees/wt-stop <branch>

Why remove-then-add and not git worktree move: in an agent's sandboxed shell the scratch dir is its own bind mount, so the move's rename fails with Invalid cross-device link. git worktree remove without --force refuses a tree with uncommitted or untracked work, so the command cannot drop work silently; ignored files such as .venv do not block it.

pod_status, pod_restart and pod_down use the same lookup, so they say the same. A name that is truly missing still answers worktree not found.

Why not let pod_up accept the calling session's own scratch: the check here is only a lookup. kirocrew pod up itself also runs in the sandbox and reads the checkout, so every pod command would need a window into the caller's scratch, and the pod would break when that session's scratch is reclaimed.

builder-security checked: no listener, bind, auth or sandbox scope changes. The scratch check runs in the gateway process and only reads paths git already reported.

Backwards compatibility

Compatible: only the error text of a lookup that already failed changes, and only when the missing name is a worktree under the scratch root.

Removes nothing: no route, field, tool or behaviour is taken away.

Tests

Proof: not provable -- other: prove.py reverts every production hunk at once, including the new module the test imports, so the mutated run fails only at collection; reverting repository.py alone turns test_a_worktree_hidden_by_the_scratch_mask_names_its_cause red

  • New test/test_dev_fleet_scratch_worktree.py:
    • a detached prunable worktree under the scratch root gets the scratch message with its path and the exact switch -c <name> + worktree remove + worktree add <name> commands, and never the stale head;
    • one on a branch is recreated on that branch with no switch -c, and the message never offers worktree move;
    • expected paths are built with shlex.quote, as the code does, so the tests hold on Windows paths;
    • a prunable worktree outside scratch still gets worktree not found;
    • a visible worktree is still found.
  • With repository.py reverted to main, both scratch tests fail and the other two pass.
  • Determinism: the earlier revision of this file passed 20/20 at -n0.
  • Builtin-skill, prompt, context and Dev Fleet pod tests (test_builtin_skill_*, test_prompt_*, test_context.py, test_agent_parts.py, test_agent_fork_templates.py, test_dev_fleet_pod_*, test_dev_fleet_app_guards.py, test_dev_fleet_app_prune.py, test_pod_e2e_harness_paths.py, test_live_checkout.py, test_hook_matcher_modes.py): 704 passed, 1 skipped. The earlier revision also passed all 38 test/test_dev_fleet_* files (1,412 passed).
  • flake8, black, isort, mypy on the changed modules, the comment-history and brand-name checks, and scripts/docs-lint.sh pass.

Manual verification

  • Reproduced on main through the live gateway: a worktree under this session's $KIROCREW_SCRATCH → pod_up → worktree not found: scr-podrepro.
  • Through the same sandbox entry point Dev Fleet uses, that worktree read prunable gitdir file points to non-existent location, and <repo>/.worktrees/pod-33124 did not.
  • The remedy path works end to end through the live gateway: <repo>/.worktrees/pod-33124 → pod_up (up on 7902) → pod_status (up health=200) → pod_down (stopped, home reclaimed) → pod_status (down health=0).
  • The remedy commands, run through the same sandbox entry point an agent shell gets (sandboxed_spawn_argv, standard mode, own scratch as a private window): git worktree move <scratch>/x <repo>/.worktrees/x failed with Invalid cross-device link; git worktree remove refused a tree with an edited file, removed a clean one that held an ignored .venv, and git worktree add <repo>/.worktrees/x <branch> then succeeded. The probe worktrees and branches were removed afterwards.
  • The new message itself was not seen through a live gateway: the live gateway runs main. The test drives _find_worktree with the porcelain entries git returns.

Screenshots / video

N/A - no frontend change.

Related Issues

Closes #33252

Not in this PR: the pod_down stop timeout is fixed separately in #33272 (issue #33270).

Sibling sweep

Swept: git grep -n '_discover_worktrees()|_find_worktree(' -- src/kiro_crew/apps/builtins/dev_fleet
Sibling: src/kiro_crew/apps/builtins/dev_fleet/worktree_ops_pods.py -- not affected -- pod_up, pod_restart, pod_status and pod_down all resolve through _find_worktree, so the fix there reaches them
Sibling: src/kiro_crew/apps/builtins/dev_fleet/repository.py -- not affected -- _valid_worktree_names and the fleet list only list worktrees; a scratch worktree there is the same ghost row git already reports as prunable

Pattern harvest

Not generalizable: the cause is one specific mask (scratch) hiding one specific kind of path (a worktree) from one sandboxed reader; the fix names that pair.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 8, 2026 15:26
@buluoray
buluoray requested a review from tim-finnigan November 8, 2026 15:26
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8a57fc7096b776dfe1f9b2541167fde028f81450 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8a57fc7096b776dfe1f9b2541167fde028f81450 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only rewrites an error that was already being returned, and touches no sandbox scope or contract.

Notes

  • config/prompt.md now names Dev Fleet and pod_up in the core system prompt. Every user gets that text, including those without the app enabled. It also ties core to a built-in app that Move every built-in app out to KiroCrewApps and forbid built-in apps #26967 is moving out of core. The pod_up tool description and the pod-e2e skill already carry the advice, so the prompt line is the one to drop.
  • The scratch match is by basename across every session's scratch. This session's agent can therefore receive another session's private scratch path in the error text. The test -d guard stops the remove commands from running from the wrong session, but it does not hide that path.

[DESIGN-REVIEWED] 8a57fc7

Description read

[DESCRIPTION-READ] 0677055d28901ebc0a0a53263fd43dbd73c6fbca75d767e67fbfca26de09631e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 8a57fc7096b776dfe1f9b2541167fde028f81450 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix stops at the error message on purpose, because ## Not a goal keeps the scratch mask and rules out pods from scratch. One helper split has no new caller.

Notes

  • Item 7 — one consumer, generalized: the diff pulls out _usable_worktrees(entries), but its only caller is _discover_worktrees. The new _find_worktree never passes it the entries it already fetched. Instead it calls await _worktree_porcelain_entries() again, which runs git worktree list a second time (item 8). Subtraction: fetch the porcelain entries once in _find_worktree and feed them to both _usable_worktrees and scratch_worktree_entry. Otherwise, inline _usable_worktrees back into _discover_worktrees and drop the split.

What this change ships

Inventory (8 items) — 6 justified

Intent: an agent that cannot start a pod on a worktree in scratch learns the cause and how to fix it. This is a FIX.

  1. pod_up/pod_status/pod_restart/pod_down on a scratch worktree now name the scratch cause and give the recreate commands — justified
  2. The pod_up tool description says to create pod worktrees under <repo>/.worktrees/ — justified
  3. The system prompt's scratch rule gains a pod-worktree exception — justified
  4. The pod-e2e skill gains a section on recreating a scratch worktree — justified
  5. The kirocrew-worktree-dev skill gains a one-line pod-worktree warning — justified
  6. The dev-fleet spec documents scratch discovery and the remedy — justified
  7. _usable_worktrees helper split out of _discover_worktrees — one consumer, generalized, rides along
  8. A not-found lookup now runs git worktree list a second time — undeclared

[FIRST-PRINCIPLES-REVIEWED] 8a57fc7

Description read

[DESCRIPTION-READ] 7e97188efde10f08df086988287dda0fea0b56eb9e08d7ca24ce9c39abee9e71

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8a57fc7096b776dfe1f9b2541167fde028f81450 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8a57fc7

Verdict parsed from the review's SHA-scoped output markers for commit 8a57fc7096b776dfe1f9b2541167fde028f81450.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8a57fc7096b776dfe1f9b2541167fde028f81450 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8a57fc7096b776dfe1f9b2541167fde028f81450 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8a57fc7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8a57fc7096b776dfe1f9b2541167fde028f81450 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 8, 2026
@buluoray
buluoray force-pushed the crew/slack/devfleet-scratch-worktree branch from 1b5ced8 to 589a66e Compare November 8, 2026 16:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 8, 2026
@buluoray
buluoray force-pushed the crew/slack/devfleet-scratch-worktree branch from 589a66e to 4b29f37 Compare November 8, 2026 16:35
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 8, 2026
@buluoray
buluoray force-pushed the crew/slack/devfleet-scratch-worktree branch from 4b29f37 to c169e3d Compare November 8, 2026 16:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 8, 2026
@buluoray
buluoray force-pushed the crew/slack/devfleet-scratch-worktree branch from c169e3d to 26c7e8b Compare November 8, 2026 17:22
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 8, 2026
Dev Fleet runs its git in the agent sandbox, which masks the whole
scratch root so one session cannot read another's scratch. A worktree
an agent created under $KIROCREW_SCRATCH therefore reads as prunable to
that git, discovery drops it, and pod_up, pod_status, pod_restart and
pod_down answered a bare "worktree not found" for a worktree that
exists.

The name lookup now checks, when nothing else matches, for a prunable
worktree of that name under the scratch root, and answers with its path
and copy-ready commands that recreate it under <repo>/.worktrees/:
git worktree remove (which refuses uncommitted work), then git worktree
add on the same branch or commit. git worktree move is not offered: in
a sandboxed shell scratch is its own mount and the rename fails with
EXDEV. The system prompt's scratch rule, the pod_up tool description,
and the pod-e2e and kirocrew-worktree-dev skills say a pod worktree
goes in <repo>/.worktrees/<name>. The sandbox mask is unchanged.
@buluoray
buluoray force-pushed the crew/slack/devfleet-scratch-worktree branch from 26c7e8b to 8a57fc7 Compare November 8, 2026 17:54
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 8, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. When pod_up cannot find a worktree, _find_worktree now checks whether the name matches a prunable worktree recorded under the scratch root. If so, it names that cause instead of a bare "worktree not found". The lookup runs on the executor and only after the normal lookup has failed. Discovery itself is unchanged: _usable_worktrees keeps the same filter.

The suggested recovery is careful:

  • test -d comes first, so a shell that cannot see the path stops before worktree remove drops someone else's record.
  • A detached tree gets a branch first, so a new commit is not orphaned.
  • Remove-then-add is used instead of git worktree move, which fails across the sandbox mount.
  • Every path and branch goes through shlex.quote.

The sandbox mask is not touched, and the prompt, skill and tool docs now say to put pod worktrees under <repo>/.worktrees/. The tests cover the scratch match, a normal miss, a detached tree and a dirty-tree refusal. No regex, nothing gets stricter, and all checks are green.

@bolichen97
bolichen97 merged commit 6d1e817 into main Nov 8, 2026
97 checks passed
@bolichen97
bolichen97 deleted the crew/slack/devfleet-scratch-worktree branch November 8, 2026 19:14
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 8, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Correction to my review above: it wrongly said the tests cover a dirty-tree refusal; they cover the scratch match, the branch case, a real miss and a visible tree, and the approval stands.

KiroCrewCaptain added a commit to KiroCrewCaptain/kirocrew-app-dev-fleet that referenced this pull request Nov 14, 2026
Kiro Crew's built-in Dev Fleet moved on after this tree was cut from
main at 6d07df11. An adopted host runs this tree, so each of those
changes is carried here first, up to main at 46f62dea:

- pod_down waits for the pod's whole stop, and the tool waits 120s
  (kirodotdev/KiroCrew#33272)
- pod_up names a worktree hidden in a session's scratch, and its
  description says where to create one (kirodotdev/KiroCrew#33254)
- the how-to sentence is one message per language
  (kirodotdev/KiroCrew#33327), rendered by i18nNodes, the bundle's
  stand-in for the dashboard's <Trans components>
- Create worktree from latest stable, with Refresh, in a labelled More
  actions menu (kirodotdev/KiroCrew#33391), on the page's own MenuBtn
- pod_up's test-double env override (kirodotdev/KiroCrew#35199)
- provision checks the build input (kirodotdev/KiroCrew#22686), a clean
  closed worktree can prune (kirodotdev/KiroCrew#31124), the secret
  comes from the spawn environment (kirodotdev/KiroCrew#34762), every
  bad-body 400 carries a code (kirodotdev/KiroCrew#32503), and the
  demo skill's Windows tool lookup (kirodotdev/KiroCrew#33556, #33779)

app.json now carries the wheel copy's agentTools block byte for byte,
which an adopted record needs for its tools to stay listed
(kirodotdev/KiroCrew#33141). The suite runs against Kiro Crew main at
46f62dea, the fixtures match that wheel copy, and PORTING.md records
each port.

Co-authored-by: Bolin Chen <bolichen@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pod_up says 'worktree not found' for a worktree under $KIROCREW_SCRATCH

2 participants