Repository navigation
feat(pod): opt-in uv provisioning of worktree venvs from a shared copy-on-write cache - #10259
Conversation
|
Intent: Make a worktree's Python venv cheap to create — seconds and near-zero unique disk — by letting |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the "Shared Dependency Cache" RFC PR actually lands — dev-fleet.md, the bundled skill, and docstrings all cite a file not yet in the tree. [DESIGN-REVIEWED] e17389e Description read[DESCRIPTION-READ] 8143343e57cfd8aee374bc28673ca11c38cd20554df0d738d47168c04767a166 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Confirm What this change shipsInventory (7 items) — 7 justifiedIntent: let a contributor set one env var to get worktree venvs in seconds that share wheels copy-on-write, leaving the default pip path untouched — ADDITION.
[FIRST-PRINCIPLES-REVIEWED] e17389e Description read[DESCRIPTION-READ] ba6c4a81c956990c8ac7652a5b998375b29190659e5689649772f3675a462b3b That sha256 names the PR description this verdict read. Recompute it with |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNothing blocks; two claims the new docs make about the uv path are contradicted by the code in the same diff. FINDING — src/kiro_crew/pod/provision.py:447 — FINDING — src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-worktree-dev/SKILL.md:34 — "This is the recipe [OPUS-REVIEWED] e17389e |
70067b7 to
9a6ab88
Compare
|
|
|
|
|
|
9a6ab88 to
4f33704
Compare
|
|
|
|
|
|
For a maintainer override, if wanted: |
|
4f33704 to
8c0c822
Compare
|
Design change on this head, Why. The GPT finding (span Clone mode removes the hazard structurally. It is copy-on-write: on a reflink filesystem the venv shares extents with uv's cache, and a write copies the extent, so nothing written in one venv can reach the cache or a sibling. Measured on this host (XFS, Diff vs the previous head: |
d955ce4 to
8e698a4
Compare
…envs Lands the design first: Phase 2 flips what pod provision builds by default (uv out of a shared hardlinked cache instead of pip), and the First Principles lane reads that decision from the base branch. Phase 1 -- the opt-in, the shared uv resolver and the shared-inode write fence -- is open as kirodotdev#10259 and needs no record; Phases 3 (make backend) and 4 (node hardlink dedup) are proposals. Refs kirodotdev#10258
|
|
8e698a4 to
fdef8e8
Compare
|
|
fdef8e8 to
100c63a
Compare
100c63a to
56cfb70
Compare
|
|
|
Read-only review note, holding approval for one edit. Opt-in and off by default (_find_uv returns None unless KIROCREW_PROVISION_USE_UV, pinned for unset/0/false/empty); uv resolved via the wheel first then shutil.which, same trust level as _find_python; clone mode is copy-on-write so one worktree cannot mutate another's deps; pptx_maker keeps the same contract. Please fix the PR title before merge: it still says 'shared hardlinked cache' but the body and code switched to --link-mode clone and reject hardlinks, and a squash merge would write the wrong title into history. Nits: resolve_uv absolutizes a relative PATH entry like '.' rather than skipping it; no test that the pip path runs when the flag is on and uv is absent; uv pip install --group needs a newer uv than the declared uv>=0.5 floor and degrades to pip silently. |
|
Thanks for the read. All four items addressed on
Black baseline (one graduated entry pruned), docs lint and 2,462 focused tests green. Same base |
|
Black baseline, docs lint and the pod/pptx tests green; same base. |
|
|
|
Same tree as |
…y-on-write cache With KIROCREW_PROVISION_USE_UV set, pod provisioning builds the worktree venv with uv out of its shared global cache (--link-mode clone, --seed, --project) and falls back to the unchanged pip path when uv cannot be located or fails; unset, the pip path is byte-for-byte today's. The uv resolution ladder lives once in kiro_crew.env.resolve_uv and pptx-maker delegates to it. Clone mode is copy-on-write: on a reflink filesystem (XFS with reflink, btrfs, APFS, ReFS) the venv shares blocks with the cache and a write copies the block, so an edit in one venv cannot reach the cache or a sibling venv; without reflink uv copies, keeping the speed and losing the disk saving. Hardlink mode was rejected because it shares the inode itself, which is why no write fence is needed. The default stays pip. Flipping it is Phase 2 of the Shared Dependency Cache RFC, which lands on its own PR. Closes kirodotdev#10258
|
|
iamwhatever
left a comment
There was a problem hiding this comment.
Re-reviewed at e17389e. _run_uv catches OSError (PermissionError, ENOEXEC) and returns 1, both uv steps go through it, a nonzero rc falls back to pip, --allow-existing is set and there is no rmtree on the path. Opt-in stays default off via env_flag_enabled; resolve_uv tries the wheel then which and skips relative hits; --link-mode clone on both steps. Title, body and code now agree. The red macOS shard was test_platform_log_append (a timing test this PR does not touch), re-run.
Non-blocking test nits: the rmtree guard at test_pod.py:1909 is weak because .venv does not exist beforehand; the 'could not be started' message is not asserted; only the seed step's spawn failure is exercised.
Problem / Motivation
Goal: Let a contributor who sets one environment variable get a worktree venv in seconds that shares its wheels with every sibling venv, while the default provisioning path stays exactly as it is.
Every worktree venv is a ~400 MB copy of wheels every other worktree already has.
kirocrew pod provisionandpod upbuild it withpython -m venv+ pip, which copies every wheel and takes about a minute. On a host with 27 worktrees that is ~10 GB of identical Python files, and the twentieth worktree costs as much as the first.Why it matters
The repo's own skills recommend one worktree per change and one pod per worktree. Each one pays the full dependency cost, so a busy contributor's disk fills with duplicates, and the minute-long venv build is the reason
pod upnever auto-builds the dist. Cheap venvs make disposable worktrees actually disposable.Not a goal
pod provisionandpod upwithoutKIROCREW_PROVISION_USE_UVrun today'spython -m venv+ pip sequence byte for byte; flipping the default is Phase 2 of the RFC (docs(rfc): record the shared dependency cache decision for worktree venvs #12937) and a separate PR.uv.lock,uv sync, or change to how dependencies are declared.node_modulesinstallation.What changed (motivation → approach → change)
This PR adds an opt-in fast path; the default is unchanged. With
KIROCREW_PROVISION_USE_UV=1(truthy per the repo'senv_flag_enabled, so0/falsedo not opt in),ensure_venvtriesuvfirst:uv venv --seed --python <py> .venv, thenuv pip install --link-mode clone --python .venv/bin/python --project <checkout> --editable <checkout> --group dev. Site-packages become copy-on-write clones (reflinks) of uv's global cache. Measured on this host (XFS with reflink): about 10 s instead of ~60 s, and a 10 MiBdfdelta for 171 MB of apparent site-packages, 0 hardlinked files. Unset,pod provisionandpod uprun byte-for-byte today'spython -m venv+ pip path.Three flags carry the design.
--link-mode cloneis explicit because uv picks clone only on macOS by default, and on Linux its default was observed to copy on a host whose filesystem supports reflinks. Clone means copy-on-write: on XFS with reflink, btrfs, APFS and ReFS the venv shares blocks with the cache until a write, and the write copies the block, so an edit in one venv never reaches the cache or a sibling venv. Where the filesystem cannot reflink (ext4, XFS without reflink, a cache on another filesystem) uv falls back to a plain copy: the install still succeeds and is still faster than pip, it just does not share disk. Hardlink mode was rejected because it shares the inode itself.--project <checkout>makes--group devread the worktree'spyproject.tomlno matter where the caller runs from; the Dev Fleet backend and a login shell provision from different directories, and without it uv errors out looking for apyproject.tomlin the cwd.--seedkeepspipin the venv, which uv omits by default;make backenddrives$(VENV)/bin/pip, so a pod-provisioned worktree must not differ from a pip-built one there.uvis already a declared dependency (setup.cfg:uv>=0.5,<1, shipped as a wheel), and the repo now resolves it in exactly one place:kiro_crew.env.resolve_uv()asks the wheel withuv.find_uv_bin()and falls back toPATH, never raising. Pod provisioning calls it, and pptx-maker'sresolve_uv— which used to carry its own copy of the same ladder — now delegates to it behind its existing process-wide cache. Nothing is lost when uv cannot be found: if the switch is off, the wheel lacks the binary andPATHhas none, or either uv step exits nonzero, the existing pip path runs unchanged over whatever is in.venv, including its own pip-too-old fallback. The fallback deletes nothing:python -m venvtakes over a half-built directory the same way it already does after an interrupted pip run, and provisioning has no lock, so a delete here could remove a venv a concurrent Dev Fleet or CLI provision had just finished. The venv layout is identical, sohas_venv, the pod runtime and the Dev Fleet view are untouched.flowchart LR subgraph Default A1[pod up / provision]:::ctx --> B1[python -m venv + pip]:::ctx --> C1[.venv, ~400 MB copied]:::ctx end subgraph KIROCREW_PROVISION_USE_UV=1 A2[pod up / provision]:::ctx --> D2{uv found?}:::added D2 -- yes --> E2[uv venv --seed + uv pip install --link-mode clone]:::added --> F2[.venv, copy-on-write clone of the cache]:::added D2 -- no / uv failed --> B2[python -m venv + pip]:::ctx --> C2[.venv, ~400 MB copied]:::ctx end classDef added fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:2px classDef changed fill:#FEF3C7,stroke:#D97706,color:#78350F,stroke-width:2px classDef removed fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-dasharray:4 3 classDef ctx fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E linkStyle 2,3,4 stroke:#16A34A,stroke-width:2px🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged
The default path is untouched; opted in, provisioning asks for uv first and falls back to the same pip path when it is missing or fails.
The design record — why uv and copy-on-write clones, the flip of the default as a later phase, why
node_modulesstays onnpm ciand gets ahardlink -tdedup pass instead of pnpm — isdocs/request-for-change/rfc-shared-dependency-cache.md, landing in its own PR so the default flip can be judged against a document already onmain.docs/system-specs/modules/dev-fleet.mddescribes the opt-in venv path; the bundledkirocrew-worktree-devskill keepspython3 -m venv+ pip as the primary manual recipe and adds the uv recipe beneath it with a note on why clone, not hardlink.Backwards compatibility
The declared
uvfloor moves from>=0.5to>=0.6.7, the release that added--grouptouv pip install; an older uv would exit nonzero on the install step and fall back to pip on every provision. Otherwise compatible: withKIROCREW_PROVISION_USE_UVunset,pod provisionandpod uprun the identicalpython -m venv+ pip sequence (pinned bytest_ensure_venv_builds_then_succeedsandtest_find_uv_is_opt_in_and_pip_is_the_default); the venv layout is unchanged sohas_venvand every consumer of it are untouched; pptx-maker'sresolve_uvkeeps its signature, cache andreset_uv_cache.Tests
test/test_pod.py::TestProvisionBuildPaths:test_ensure_venv_prefers_uv_with_shared_cache_flags— opted in with uv found, the venv is built byuv venv --seedthenuv pip install, pip never runs, and the install carries--link-mode clone,--project <checkout>,--editable <checkout>,--group dev.test_ensure_venv_uv_failure_falls_back_to_pip_without_deleting— a nonzerouv pip installhands the existing.venvtopython -m venvuntouched (shutil.rmtreeis patched to fail the test), and provisioning still succeeds.test_find_uv_is_opt_in_and_pip_is_the_default— unset,0,falseand empty resolve no uv;1/true/YESopt in.test_opted_in_without_uv_runs_the_pip_path— with the switch on and uv absent from both the wheel andPATH, the real_find_uvreturnsNoneand the unchangedpython -m venv+ pip sequence runs; no uv command is attempted.test_find_uv_is_the_shared_ladder— pod provisioning resolves throughkiro_crew.env.resolve_uv: the wheel'sfind_uv_bin()wins overPATH, andPATHis the fallback when the wheel raises.test_ensure_venv_builds_then_succeeds— the existing pip-path test, unchanged in behaviour, now runs with the switch unset.test/test_pptx_maker_provision.py::TestResolveUv— unchanged assertions (wheel first,whichfallback, missing file not trusted, never raises, cached) now patch the sharedkiro_crew.envseam, proving pptx-maker consumes the same ladder; two new cases pin that a relativeshutil.whichhit (relativePATHentry) is skipped rather than handed toPopen(cwd=<checkout>), and that the wheel locator's answer is normalised to an absolute path.Manual verification
Built a venv with the opt-in recipe on Linux/XFS (reflink enabled):
uv pip install --link-mode clone --group devproduced 171 MB of apparent site-packages for a 10 MiBdfdelta and 0 hardlinked files, with.venv/bin/pippresent from--seed. Focusedpytest test/test_security_gate_liveness.py test/test_pod.py test/test_pod_provision_steperr.py test/test_env.py test/test_security.py test/test_hooks*.py test/test_pptx_maker_provision.py: 2,459 passed. Black baseline and docs lint green locally. Earlier heads carried a hardlink design plus asite-packageswrite fence; both are removed on this head after the GPT and Opus lanes showed the fence could neither close the hazard for shell writes nor keep its stat bounded.Related Issues
Closes #10258
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)