Skip to content

fix(worktree): never reuse a worktree whose add never finished - #19012

Closed
javenciu wants to merge 2 commits into
kirodotdev:mainfrom
javenciu:fix/worktree-reuse-refuses-an-unfinished-add
Closed

javenciu wants to merge 2 commits into
kirodotdev:mainfrom
javenciu:fix/worktree-reuse-refuses-an-unfinished-add

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A worktree whose git worktree add never finished, which git leaves locked initializing, is never handed back as a ready, reused worktree.

The "Start in new worktree" endpoint (POST /api/worktree/create, src/kiro_crew/dashboard/handlers/worktree.py) decides a destination is a ready worktree to reuse by reading only its registered path and branch. The root cause: git worktree add registers the worktree, sets its HEAD and writes a locked = "initializing" marker BEFORE it checks files out, and clears the marker only once the checkout finishes. If the add is killed in between — a gateway stop or restart kills its control group — the entry is left registered, partly or not at all checked out, and still locked initializing. The reuse path in _create_worktree_sync (worktree.py) matched that entry and returned 200 {"reused": true}, so a session opened on a tree missing most of its files.

Why it matters

An agent that opens in that worktree sees most of the repository's files as absent. Its first git add -A and commit record them as deleted on the new branch — a data-loss footgun from a routine restart. A real git worktree add of a 20,000-file tree, killed mid-checkout, leaves 214 files present and the entry locked initializing; the retry then reported it ready.

Not a goal

  • Removing the unfinished worktree automatically. Removing a locked worktree needs an unlock plus a forced remove, and a judgment that the lock is ours; that is a larger change. This PR refuses the unfinished entry with a clear error and leaves removal to the user (or a follow-up). Refusing is the smallest change that ends the data loss.
  • Changing how a worktree is created or how git locks it.
  • Matching a translated lock reason. git worktree add writes the lock reason through gettext, and _run_git (dashboard/handlers/worktree.py) does not pin the locale, so under a German locale the reason reads initialisiere. An add interrupted under such a locale is not refused when its branch line is already recorded: it is reused, as on main. Detached, it gets the Directory already exists 409, also as on main. Pinning LC_ALL=C for git, as apps/registry_pipeline/subprocess_env.py does, would close that case; it changes the environment of every git call in worktree.py, so it is a separate change.

What changed (motivation → approach → change)

  • Motivation: never open a session on a worktree whose creation never finished.
  • Approach: read git's lock state, which the endpoint already had in the porcelain output but ignored, and refuse an unfinished entry instead of reusing it.
  • Change:
    • New _worktree_entries(root) parses git worktree list --porcelain -z into path -> (branch, locked_reason), including the locked line git already emits. _worktree_branches becomes a thin projection of it to path -> branch, so its other caller (_cleanup_partial) is unchanged.
    • The reuse path in _create_worktree_sync now reads the entry at the destination:
      • locked initializing → 409 with code: "worktree_unfinished", whatever the branch line says (a killed add may still be detached, with no branch recorded yet);
      • otherwise, on this branch → reused as before (200), whether it is unlocked or locked for another reason. git worktree lock only keeps git from pruning, moving or removing a tree, so a completed tree that someone locked is still usable.

The hot path is unchanged: a first-time create still claims the branch and the directory and runs worktree add; only the reuse branch reads one extra field from a listing it already fetched.

Every path that decides a worktree is "ready"

_create_worktree_sync has exactly one reuse decision — the if os.path.exists(dest): block. After this change it is the only place that returns reused: true, and it returns it only when entries.get(_norm_path(dest)) is present, on this branch, and not locked initializing. The two other _worktree_branches call sites are in _cleanup_partial, which decides what to delete, not what is ready; they are unchanged. No other function returns reused.

Backwards compatibility

Compatible: no persisted format or route change. A completed worktree is reused exactly as before, locked or not. The one new outcome is a 409 with code: "worktree_unfinished" for a destination whose add never finished, which main answered with a wrong 200 {"reused": true}.

Removes nothing: no endpoint, response field or error code is removed.

Tests

test/test_worktree_reuse_initializing.py (new) drives the real _create_worktree_sync against a scratch repository under tmp_path; the OS-sandbox wrapper is replaced by a plain git call so it runs on any host, and the root is realpath'd so the handler's lexical dest keying agrees with git's canonicalized worktree path. The repository's base commit holds 4,000 files, so each real git worktree add in the two race tests checks out long enough to overlap the other requests. The fixture builds that commit with one git fast-import, as a single pack, so it writes no working-tree files and no loose objects; the tree is the same one git add -A of those files gives. Nothing reads the repository's own working tree: the handler reads its refs, config and worktree list. Writing the 4,000 files and running git add -A took longer than pytest's 180 s per-test limit on a Windows runner, with three of these fixtures running at once. Here the fixture's setup takes 0.15-0.25 s per test, down from 0.98-1.28 s.

  • Red on main (test_a_retry_after_an_interrupted_add_is_refused_not_reused): the test leaves the state a killed add leaves, directly and every time. It runs git worktree add --no-checkout on the branch, then git worktree lock --reason initializing, which writes the same marker git's add writes before its checkout. The entry is registered, on its branch, locked initializing, with 0 of 4000 files. On the current main (843abe3) with only the test file added, 1 failed and 4 passed in each of 3 runs. The retry returns 200 reused=True with 0 files, and the test fails with a retry reported a worktree whose add never finished as ready: status 200, reused=True, 0 of 4000 files, git lists it 'locked initializing'. On the fix the retry returns 409 worktree_unfinished, and the entry is left registered and still locked.

  • Controls, green on the fix:

    • a completed add is still reused with all files;
    • a completed worktree locked for another reason (git worktree lock --reason) is still reused (200) and stays locked and complete, as on main;
    • the same-branch race (eight concurrent creates leave exactly one worktree) still passes;
    • the cleanup-prune race (a failing neighbour's cleanup never damages a live worktree) still passes.
    • Both races start their requests through _race (test/test_worktree_reuse_initializing.py:149-184), which bounds every wait: a worker waits at most 60 s at the barrier, a failed Thread.start() aborts the barrier so the workers already waiting stop, and every started worker is joined against one deadline, failing the test by name if it is still alive.
  • Green: the three changed test files give 222 passed in each of 3 serial runs, on this branch rebased onto that main. The 7 test files that use the worktree handler, run with this file first: 409 passed, 8 skipped.

  • Two existing fixtures retargeted. Both mocked the list helper _worktree_branches; each now also mocks the create path's new seam, _worktree_entries, with the same trees, unlocked:

    • the one in test_worktree_create.py (list failure → 503);
    • the sync_env fixture in test_dashboard_worktree_coverage.py.

    Their intent is unchanged.

Manual verification

Not run against a live gateway.

Related Issues

Two open PRs touch worktree.py. #17628 (switch branches from the Git panel) does not conflict with origin/main in this file, but git merge-tree shows this change adds a conflict there; whichever lands second resolves it. #17169 (Set-repo-path on the follow-up card) already conflicts with origin/main in this file, so its conflict is its own staleness, not caused by this change. No open PR already fixes the unfinished-worktree reuse.

A separate change, not yet opened as a PR, also edits this reuse branch of _create_worktree_sync: it refuses to hand back a worktree that another session is using. The two touch the same lines, so whichever lands second keeps both checks, the in-use check inside the same-branch case.

Pattern harvest

Rule candidate: when code decides a resource is ready for reuse from a listing, it must read the listing's own "not ready" marker (here git's locked initializing), not just the resource's identity. A half-created resource that advertises its own incompleteness should never be read as complete.

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 02:17
@javenciu
javenciu requested a review from cruisercohen October 11, 2026 02:17
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision f82e88824845f678967da606ab8bca829145d8c3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of f82e88824845f678967da606ab8bca829145d8c3 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

The reuse rule in docs/architecture/followup-suggestions-trust-model.md (lines 101–105) does not mention the new worktree_unfinished refusal, so the doc no longer matches the code. Update it in this PR.

Suggestions

  • After this 409 the dashboard offers no way out. git worktree prune skips locked entries, so a user who only deletes the directory still can't retry. Name the fix in the error text: git worktree unlock <path> then git worktree remove --force <path>.

[DESIGN-REVIEWED] f82e888

Description read

[DESCRIPTION-READ] 31d13331ca1728a330721fe80643169e4916f47677ab3e54375cf61bc0b9fde5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed f82e88824845f678967da606ab8bca829145d8c3 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f82e888

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of f82e88824845f678967da606ab8bca829145d8c3 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new code: "worktree_unfinished" response field has zero consumers. The dashboard only reads res.error, so the field is unused surface that clients may come to rely on.

Not justified as shipped

  • Item 3 — zero consumers: the hunk "code": "worktree_unfinished", adds a machine-readable error code that nothing reads. Grepping worktree_unfinished and worktree_[a-z_]+ across website/src and src/ finds no caller. The only caller of this endpoint's result, website/src/pages/ChatPage.tsx:2936, reads res?.error. The existing worktree_mkdir_failed code also has 0 non-test consumers, so it doesn't justify adding another.
    Subtraction: delete the "code" key from the 409 body and keep the error string; the tests can assert on status 409 alone.
    Clears when: the key is deleted, or a caller that branches on code == "worktree_unfinished" is counted.

What this change ships

Inventory (5 items) — 4 justified

Intent: stop "Start in new worktree" from opening a session on a worktree whose git worktree add never finished. This is a FIX.

  1. Retrying on a worktree left locked initializing now gets a 409 refusal instead of 200 reused — justified
  2. A finished worktree locked for another reason is still reused — justified
  3. The 409 body carries a new code: "worktree_unfinished" field — zero consumers
  4. Worktree-list parsing now reports lock state, and _worktree_branches becomes a projection of it — justified
  5. New test file with a red-on-main reproduction plus race controls on a 4,000-file repository — justified

[FIRST-PRINCIPLES-REVIEWED] f82e888

Description read

[DESCRIPTION-READ] 60a21823a0a8a429a82d0cd12bbd4107875921453ce5721d71ff13460808d325

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed f82e88824845f678967da606ab8bca829145d8c3 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings that block the merge.

FINDING — src/kiro_crew/dashboard/handlers/worktree.py:106 — git writes the lock reason through gettext (_("initializing"), and that msgid is in the shipped de/LC_MESSAGES/git.mo). _run_git passes on the gateway's locale without changing it; it only sets GIT_TERMINAL_PROMPT. So on a host running a non-English locale such as LANG=de_DE.UTF-8, an interrupted add records a translated reason, entry.locked_reason == _INITIALIZING_LOCK never matches, and a retry still returns the half-checked-out tree as reused: True with status 200 → Fix: set env["LC_ALL"] = "C" (or LANGUAGE=C) next to GIT_TERMINAL_PROMPT in _run_git, so the worktree add this handler spawns records the English reason.

[OPUS-REVIEWED] f82e888

@javenciu

Copy link
Copy Markdown
Contributor Author

Thanks, the finding holds. Fixed in a second commit, 66a3a7c, on top of 85c489c.

  • test_same_branch_race_still_leaves_one_worktree (test/test_worktree_reuse_initializing.py) no longer counts 200 responses. A request that arrives after the winner's add has finished reuses that worktree, which is correct and also answers 200, so the old count depended on timing. The test now asserts exactly one 200 with reused: false, the request that created the worktree, and keeps both other checks: the winner's tree has every file, and git worktree list shows one worktree besides the main checkout.
  • No source change. The description's control line, "eight concurrent creates leave exactly one worktree", is still what the test checks.

Runs on 66a3a7c:

  • The three changed test files: 222 passed in each of 3 serial runs.
  • Red on main 2ada192, this test file copied in: 1 failed, 221 passed in each of 3 runs, the interrupted-add retry as before.
  • The 7 test files that use the worktree handler: 409 passed, 8 skipped, as the description states.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@javenciu
javenciu force-pushed the fix/worktree-reuse-refuses-an-unfinished-add branch from 66a3a7c to ad3a4f5 Compare October 11, 2026 07:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@javenciu
javenciu force-pushed the fix/worktree-reuse-refuses-an-unfinished-add branch from ad3a4f5 to d98490a Compare October 11, 2026 09:08
`git worktree add` registers the worktree and sets its lock to
"initializing" before it checks files out, clearing the lock only once
the checkout finishes. When the add is killed in between (a gateway stop
or restart kills its control group), the entry is left registered,
partly checked out and locked "initializing". The create endpoint's
reuse path read only the path and branch, not the lock, so a retry
returned 200 {"reused": true} for that half-made tree. A session then
opened on a tree missing most of its files, and its first commit would
record them as deletions.

Parse the porcelain `locked` reason (new `_worktree_entries`, which
`_worktree_branches` now projects from). The reuse path refuses a
destination locked "initializing" with 409 `worktree_unfinished`
whatever its branch line says (a killed add may still be detached). A
completed worktree on the requested branch is still reused, whether it
is unlocked or locked for another reason.
A request that finds the winner's finished worktree is a legitimate reuse and also answers 200, so counting 200s depended on timing. The race now asserts exactly one response that created the worktree (reused false), and keeps the file and worktree-count checks.
@javenciu
javenciu force-pushed the fix/worktree-reuse-refuses-an-unfinished-add branch from d98490a to f82e888 Compare October 11, 2026 10:53
@javenciu

Copy link
Copy Markdown
Contributor Author

Thanks, the finding holds. Fixed by amending the first commit, now ebc86c2; the second, f82e888, is unchanged, and both are still on main 843abe3. _race (test/test_worktree_reuse_initializing.py:149-184) now bounds every wait. Each worker passes barrier.wait a timeout of _RACE_TIMEOUT_SECS, 60 s (:163, :146). A failed Thread.start() runs barrier.abort() (:175), so a worker already waiting at the barrier stops instead of blocking forever. Every started worker is joined against one shared deadline (:178-180), and a worker still alive at it fails the test by name (:182), as does a worker's own exception (:183). Checked with a probe that makes the second worker's Thread.start() fail, 3 of 3 runs on each head: on d98490a the first worker was still waiting at the barrier after the helper raised, and on f82e888 the helper re-raises the start failure at once with no worker left in threading.enumerate(). A second probe, with a worker still running at the deadline, fails the test naming both workers within the bound on f82e888, where d98490a waited and named nobody. The two race tests are unchanged and pass 3 of 3, and on main, with only this test file copied in, the file still gives the description's 1 failed, 4 passed.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 11, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

Closing this as already covered by #19013.

Once #19013 landed, a worktree whose git worktree add never finished is refused on reuse: its never-checked-out files read as uncommitted changes, so the in-use check returns 409 worktree_in_use on the retry rather than reusing it. The data-loss path this PR targeted no longer occurs on main.

The one behaviour left is cosmetic: for a half-finished add specifically, a dedicated worktree_unfinished message ("Remove the unfinished worktree") would point at the real remedy instead of a session that no longer exists. Not worth a separate change on its own; folding it into a future worktree-handler PR would be the place for it.

@javenciu javenciu closed this Oct 11, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant