Skip to content

feat(pod): opt-in uv provisioning of worktree venvs from a shared copy-on-write cache - #10259

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:feat/uv-provision
Sep 30, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:feat/uv-provision

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let a contributor who sets one environment variable get a worktree venv in seconds that shares its wheels with every sibling venv, while the default provisioning path stays exactly as it is.

Every worktree venv is a ~400 MB copy of wheels every other worktree already has. kirocrew pod provision and pod up build it with python -m venv + pip, which copies every wheel and takes about a minute. On a host with 27 worktrees that is ~10 GB of identical Python files, and the twentieth worktree costs as much as the first.

Why it matters

The repo's own skills recommend one worktree per change and one pod per worktree. Each one pays the full dependency cost, so a busy contributor's disk fills with duplicates, and the minute-long venv build is the reason pod up never auto-builds the dist. Cheap venvs make disposable worktrees actually disposable.

Not a goal

  • No change to the default. pod provision and pod up without KIROCREW_PROVISION_USE_UV run today's python -m venv + pip sequence byte for byte; flipping the default is Phase 2 of the RFC (docs(rfc): record the shared dependency cache decision for worktree venvs #12937) and a separate PR.
  • No uv.lock, uv sync, or change to how dependencies are declared.
  • No change to node_modules installation.
  • No shared inodes between venvs. Hardlink mode was considered and rejected; clone mode shares blocks copy-on-write, so nothing an agent writes in one venv can reach the uv cache or a sibling venv, and no write fence is needed.

What changed (motivation → approach → change)

This PR adds an opt-in fast path; the default is unchanged. With KIROCREW_PROVISION_USE_UV=1 (truthy per the repo's env_flag_enabled, so 0/false do not opt in), ensure_venv tries uv first: uv venv --seed --python <py> .venv, then uv pip install --link-mode clone --python .venv/bin/python --project <checkout> --editable <checkout> --group dev. Site-packages become copy-on-write clones (reflinks) of uv's global cache. Measured on this host (XFS with reflink): about 10 s instead of ~60 s, and a 10 MiB df delta for 171 MB of apparent site-packages, 0 hardlinked files. Unset, pod provision and pod up run byte-for-byte today's python -m venv + pip path.

Three flags carry the design. --link-mode clone is explicit because uv picks clone only on macOS by default, and on Linux its default was observed to copy on a host whose filesystem supports reflinks. Clone means copy-on-write: on XFS with reflink, btrfs, APFS and ReFS the venv shares blocks with the cache until a write, and the write copies the block, so an edit in one venv never reaches the cache or a sibling venv. Where the filesystem cannot reflink (ext4, XFS without reflink, a cache on another filesystem) uv falls back to a plain copy: the install still succeeds and is still faster than pip, it just does not share disk. Hardlink mode was rejected because it shares the inode itself. --project <checkout> makes --group dev read the worktree's pyproject.toml no matter where the caller runs from; the Dev Fleet backend and a login shell provision from different directories, and without it uv errors out looking for a pyproject.toml in the cwd. --seed keeps pip in the venv, which uv omits by default; make backend drives $(VENV)/bin/pip, so a pod-provisioned worktree must not differ from a pip-built one there.

uv is already a declared dependency (setup.cfg: uv>=0.5,<1, shipped as a wheel), and the repo now resolves it in exactly one place: kiro_crew.env.resolve_uv() asks the wheel with uv.find_uv_bin() and falls back to PATH, never raising. Pod provisioning calls it, and pptx-maker's resolve_uv — which used to carry its own copy of the same ladder — now delegates to it behind its existing process-wide cache. Nothing is lost when uv cannot be found: if the switch is off, the wheel lacks the binary and PATH has none, or either uv step exits nonzero, the existing pip path runs unchanged over whatever is in .venv, including its own pip-too-old fallback. The fallback deletes nothing: python -m venv takes over a half-built directory the same way it already does after an interrupted pip run, and provisioning has no lock, so a delete here could remove a venv a concurrent Dev Fleet or CLI provision had just finished. The venv layout is identical, so has_venv, the pod runtime and the Dev Fleet view are untouched.

flowchart LR
  subgraph Default
    A1[pod up / provision]:::ctx --> B1[python -m venv + pip]:::ctx --> C1[.venv, ~400 MB copied]:::ctx
  end
  subgraph KIROCREW_PROVISION_USE_UV=1
    A2[pod up / provision]:::ctx --> D2{uv found?}:::added
    D2 -- yes --> E2[uv venv --seed + uv pip install --link-mode clone]:::added --> F2[.venv, copy-on-write clone of the cache]:::added
    D2 -- no / uv failed --> B2[python -m venv + pip]:::ctx --> C2[.venv, ~400 MB copied]:::ctx
  end
  classDef added fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:2px
  classDef changed fill:#FEF3C7,stroke:#D97706,color:#78350F,stroke-width:2px
  classDef removed fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-dasharray:4 3
  classDef ctx fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E
  linkStyle 2,3,4 stroke:#16A34A,stroke-width:2px
Loading

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

The default path is untouched; opted in, provisioning asks for uv first and falls back to the same pip path when it is missing or fails.

The design record — why uv and copy-on-write clones, the flip of the default as a later phase, why node_modules stays on npm ci and gets a hardlink -t dedup pass instead of pnpm — is docs/request-for-change/rfc-shared-dependency-cache.md, landing in its own PR so the default flip can be judged against a document already on main. docs/system-specs/modules/dev-fleet.md describes the opt-in venv path; the bundled kirocrew-worktree-dev skill keeps python3 -m venv + pip as the primary manual recipe and adds the uv recipe beneath it with a note on why clone, not hardlink.

Backwards compatibility

The declared uv floor moves from >=0.5 to >=0.6.7, the release that added --group to uv pip install; an older uv would exit nonzero on the install step and fall back to pip on every provision. Otherwise compatible: with KIROCREW_PROVISION_USE_UV unset, pod provision and pod up run the identical python -m venv + pip sequence (pinned by test_ensure_venv_builds_then_succeeds and test_find_uv_is_opt_in_and_pip_is_the_default); the venv layout is unchanged so has_venv and every consumer of it are untouched; pptx-maker's resolve_uv keeps its signature, cache and reset_uv_cache.

Tests

test/test_pod.py::TestProvisionBuildPaths:

  • test_ensure_venv_prefers_uv_with_shared_cache_flags — opted in with uv found, the venv is built by uv venv --seed then uv pip install, pip never runs, and the install carries --link-mode clone, --project <checkout>, --editable <checkout>, --group dev.
  • test_ensure_venv_uv_failure_falls_back_to_pip_without_deleting — a nonzero uv pip install hands the existing .venv to python -m venv untouched (shutil.rmtree is patched to fail the test), and provisioning still succeeds.
  • test_find_uv_is_opt_in_and_pip_is_the_default — unset, 0, false and empty resolve no uv; 1/true/YES opt in.
  • test_opted_in_without_uv_runs_the_pip_path — with the switch on and uv absent from both the wheel and PATH, the real _find_uv returns None and the unchanged python -m venv + pip sequence runs; no uv command is attempted.
  • test_find_uv_is_the_shared_ladder — pod provisioning resolves through kiro_crew.env.resolve_uv: the wheel's find_uv_bin() wins over PATH, and PATH is the fallback when the wheel raises.
  • test_ensure_venv_builds_then_succeeds — the existing pip-path test, unchanged in behaviour, now runs with the switch unset.

test/test_pptx_maker_provision.py::TestResolveUv — unchanged assertions (wheel first, which fallback, missing file not trusted, never raises, cached) now patch the shared kiro_crew.env seam, proving pptx-maker consumes the same ladder; two new cases pin that a relative shutil.which hit (relative PATH entry) is skipped rather than handed to Popen(cwd=<checkout>), and that the wheel locator's answer is normalised to an absolute path.

Manual verification

Built a venv with the opt-in recipe on Linux/XFS (reflink enabled): uv pip install --link-mode clone --group dev produced 171 MB of apparent site-packages for a 10 MiB df delta and 0 hardlinked files, with .venv/bin/pip present from --seed. Focused pytest test/test_security_gate_liveness.py test/test_pod.py test/test_pod_provision_steperr.py test/test_env.py test/test_security.py test/test_hooks*.py test/test_pptx_maker_provision.py: 2,459 passed. Black baseline and docs lint green locally. Earlier heads carried a hardlink design plus a site-packages write fence; both are removed on this head after the GPT and Opus lanes showed the fence could neither close the hazard for shell writes nor keep its stat bounded.

Related Issues

Closes #10258

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and patrigao September 12, 2026 00:32
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 12, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Intent: Make a worktree's Python venv cheap to create — seconds and near-zero unique disk — by letting pod provision build it with uv from one shared hardlinked cache, with the pip path as the unchanged fallback. Record the wider shared-dependency-cache plan (Makefile, node_modules dedup) as an RFC so those phases are decided on evidence.
Not a goal: Changing what gets installed, adding a Python lockfile, switching the repo's Node package manager or lockfile, adding provisioning locks, or touching make backend in this PR.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 12, 2026
@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of e17389e517119e46dee30f64d57bcc418f8910c2 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the "Shared Dependency Cache" RFC PR actually lands — dev-fleet.md, the bundled skill, and docstrings all cite a file not yet in the tree.

[DESIGN-REVIEWED] e17389e

Description read

[DESCRIPTION-READ] 8143343e57cfd8aee374bc28673ca11c38cd20554df0d738d47168c04767a166

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed e17389e517119e46dee30f64d57bcc418f8910c2 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e17389e

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of e17389e517119e46dee30f64d57bcc418f8910c2 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Confirm uv pip install --group truly shipped in uv 0.6.7; a wrong floor turns the opt-in into a silent pip fallback on every provision.

What this change ships

Inventory (7 items) — 7 justified

Intent: let a contributor set one env var to get worktree venvs in seconds that share wheels copy-on-write, leaving the default pip path untouched — ADDITION.

  1. New opt-in env var KIROCREW_PROVISION_USE_UV: pod provision/pod up build the venv with uv from a shared copy-on-write cache — justified
  2. Missing or failing uv falls back to the unchanged pip path, deleting nothing in .venv — justified
  3. Declared uv floor rises >=0.5 → >=0.6.7 for every install — justified
  4. uv resolution consolidated into one kiro_crew.env.resolve_uv; pptx-maker's private ladder deleted (2 consumers counted: pod/provision.py, pptx-maker; grep find_uv|resolve_uv finds no third copy) — justified
  5. The shared ladder now skips a uv found via a relative PATH entry and absolutizes the wheel's answer, pptx-maker included (base provision.py:77 already pins the absolute-path contract) — justified
  6. dev-fleet spec and worktree-dev skill document the uv recipe, pip staying primary — justified
  7. pod/provision.py and test_pptx_maker_provision.py leave the black baseline with mechanical reformats — justified

[FIRST-PRINCIPLES-REVIEWED] e17389e

Description read

[DESCRIPTION-READ] ba6c4a81c956990c8ac7652a5b998375b29190659e5689649772f3675a462b3b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed e17389e517119e46dee30f64d57bcc418f8910c2 via the fork AI-review pipeline; updated in place on each push.

Review details

Nothing blocks; two claims the new docs make about the uv path are contradicted by the code in the same diff.

FINDING — src/kiro_crew/pod/provision.py:447 — "--project", str(checkout) is credited with making --group resolve "regardless of the caller's cwd", but --project is a uv global option with no effect in the uv pip interface, and the group in fact resolves only because _run(cmd, checkout) spawns with cwd=checkout; the same false guarantee ships to contributors as SKILL.md:42 `--project .` makes `--group` read this worktree's `pyproject.toml` from any cwd — where . is the cwd, so following it from elsewhere resolves a different pyproject.toml or errors — and test_ensure_venv_prefers_uv_with_shared_cache_flags now pins the inert flag (install[install.index("--project") + 1] == str(co)) as a contract → Fix: drop the --project pair from the install argv and its assertion, and state in the docstring, dev-fleet.md and SKILL.md that the group resolves from _run's cwd=<checkout>; use the global --directory <checkout> if cwd-independence is actually wanted.

FINDING — src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-worktree-dev/SKILL.md:34 — "This is the recipe kirocrew pod provision runs when KIROCREW_PROVISION_USE_UV=1 is set" is false against _ensure_venv_uv in the same diff: the documented command installs -e ".[voice]" and requests a version with --python 3.12 (which uv may satisfy with a managed python-build-standalone interpreter), while the code installs "--editable", str(checkout) with no extras and "--python", py from _find_python(), an absolute python3.12 → Fix: drop the equivalence sentence, or say the pod path pins an absolute python3.12 and installs no extras, so this block reads as the contributor variant.

[OPUS-REVIEWED] e17389e

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [SUGGESTIONS] RFC Risks omits the hardlink in-place-edit hazard — fixed in 9a6ab88
    self-added: no

Added as the first entry of the RFC's Risks section: an in-place edit to any worktree's site-packages mutates the shared inode in every sibling venv and in the uv cache until that package version is evicted or re-fetched; pip-built venvs never had this mode. Mitigations recorded: uv cache clean <package> + re-provision repairs every venv at once, the opt-out covers a worktree that must be hand-patched, and the contract is the same one pnpm and Nix impose. A CONTRIBUTING note is scheduled for Phase 2, when the primary checkout adopts it.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [WATCH] Second spelling of uv resolution — fixed in 9a6ab88
    self-added: no

_find_uv now falls through to uv.find_uv_bin() before PATH, which is this Watch item's stated clearing condition. test_find_uv_prefers_the_declared_wheel_over_path pins the order and the wheel-missing fallback so the two ladders cannot diverge on the minimal-PATH case again.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [NOT JUSTIFIED AS SHIPPED] _find_uv duplicates the existing uv resolver — fixed in 9a6ab88
    self-added: no
    mechanism: none added; the resolver ladder now delegates to the declared uv wheel's own locator

uv is a declared dependency (setup.cfg uv>=0.5,<1, wheel located by uv.find_uv_bin()), so _find_uv now uses that locator first and shutil.which second — the same two-rung ladder as apps/builtins/pptx_maker/backend/provision.py::resolve_uv. The ~/.local/bin probe is gone.
Not lifted into one shared helper: pod/ must not import from an app builtin, and moving resolve_uv out of pptx-maker is that app's change; recorded as the RFC's open question so a third consumer triggers the lift.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [WATCH] Root-cause sibling: Makefile:65 still runs python -m venv — code stays; clearing condition is met
    self-added: no

The item's own clearing condition is "Phase 2 lands or the RFC row keeps recording it as deferred". The RFC index row for rfc-shared-dependency-cache.md records Phase 2 (make backend) as a proposal, and the RFC's Design and Rollout sections say why it is a separate PR: make backend also carries --prefer-binary and the macOS re-sign step and touches every contributor's primary checkout, whereas the pod path is worktree-scoped. The lane itself marked this "declared and deferred as RFC Phase 2, accepted".
This ruling covers any finding that asks this PR to also convert the Makefile's venv step.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [SUBTRACTIONS] Shrink _find_uv to the wheel locator — fixed in 9a6ab88
    self-added: no

Subtracted exactly as proposed: the ~/.local/bin probe and the bare-which-only ladder are gone; _find_uv is opt-out check → uv.find_uv_bin() → shutil.which. The KIROCREW_PROVISION_PIP_ONLY check is kept and now reads the flag through env_flag_enabled.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [FINDING] os.environ.get(PIP_ONLY_ENV) treats "0"/"false" as enabled — fixed in 9a6ab88
    self-added: no

The opt-out now reads through the repo's env_flag_enabled (constants.py), which strips, lowercases and compares against {"1","true","yes","on"}; "0", "false" and unset keep uv. test_find_uv_honours_pip_only_opt_out asserts both directions. This ruling covers any presence-versus-truthiness finding on KIROCREW_PROVISION_PIP_ONLY.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 12, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [SUGGESTIONS] uv venv does not seed pip; ad-hoc .venv/bin/pip in a pod-built worktree fails — fixed in 4f33704
    self-added: yes
    mechanism: uv venv --seed

Fixed in code rather than documented: _ensure_venv_uv now runs uv venv --seed, so the venv carries pip like a python -m venv one. The concrete consumer is Makefile:12 (PIP := $(VENV)/bin/pip), which make backend drives — a pod-provisioned worktree would otherwise break make on the primary checkout, not just ad-hoc use. test_ensure_venv_prefers_uv_with_shared_cache_flags pins the flag; the RFC's Phase 1 now lists it as the third design-carrying flag.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [NOT JUSTIFIED AS SHIPPED] kirocrew-worktree-dev/SKILL.md:53 still teaches python3 -m venv + pip — fixed in 4f33704
    self-added: no

The bundled skill's per-worktree recipe is now the uv one — uv venv --seed --python 3.12 .venv then uv pip install --link-mode hardlink --python .venv/bin/python --project . -e ".[voice]" --group dev — with one line each on why the three flags matter, and it names pod provision as running the same recipe. grep -- '-m venv' in the skill now returns nothing. This ruling covers the CONTRIBUTING sibling too; both docs carry the same recipe.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [NOT JUSTIFIED AS SHIPPED] _find_uv duplicates pptx-maker's resolve_uv — fixed in 4f33704
    self-added: yes
    mechanism: kiro_crew/env.py::resolve_uv is the single ladder; pod _find_uv and pptx-maker resolve_uv both call it

The ladder now exists once, in kiro_crew/env.py (already imported by pod/provision.py), and both consumers call it: pod provisioning directly, pptx-maker through its unchanged process-wide cache and reset_uv_cache(). The duplicate body, tuple and carve-out comment are deleted from pptx-maker. grep find_uv_bin outside tests now returns one implementation.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [WATCH] The RFC defers unifying the two ladders on rule-of-three — fixed in 4f33704
    self-added: yes

The clearing condition "the ladder lives once (e.g. kiro_crew/env.py) with both call sites consuming it" is met: kiro_crew/env.py::resolve_uv is the only implementation and both pod/provision.py and apps/builtins/pptx_maker/backend/provision.py consume it. The RFC's open question proposing to wait for a third consumer is deleted; its Phase 1 text now names env.py as the home.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [SUBTRACTIONS] Replace _find_uv's ladder body with one shared helper in kiro_crew/env.py — fixed in 4f33704
    self-added: yes

Done as proposed: _find_uv is now the opt-out check plus a call to kiro_crew.env.resolve_uv(); pptx-maker's resolve_uv is a cached caller of the same function. One spelling instead of two. Net: pod/provision.py and pptx-maker's provision.py each lost their ladder body; env.py gained one.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [WATCH] kirocrew-worktree-dev/SKILL.md:53 still recommends the pip recipe — fixed in 4f33704
    self-added: no

Clearing condition "the skill's recipe is updated" is met: the bundled skill now carries the uv recipe (uv venv --seed + uv pip install --link-mode hardlink --project .) in place of python3 -m venv + pip. The unfixed-sibling count for the CONTRIBUTING change is zero.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [BLOCKING] Hardlinked dependencies let agents poison sibling environments — not a defect; the code keeps --link-mode hardlink
    self-added: no

Not a boundary crossing. Pod separation is documented as operational, not adversarial: src/kiro_crew/pod/README.md states it is "not an adversarial security boundary against arbitrary processes already running as the same Unix UID" and that "such a process can modify user-owned pod storage directly". The agent sandbox (sandbox.py) hides credential paths and seals crew-home leaves; it does not confine writes to the worktree. So an agent in a pod can already open and rewrite a sibling worktree's .venv/…/*.py or ~/.cache/uv/… by path. A hardlink adds no capability that agent lacks; it changes the blast radius of an honest in-place edit, which the RFC's Risks section now records with its repair path (uv cache clean <package> + re-provision). This ruling covers every finding that treats shared inodes between same-UID pod venvs as a security defect.
The proposed fix does not work here. --link-mode clone was measured on this XFS volume: 0 shared extents, 354 MB per venv, no warning — uv silently degrades to a full copy, which erases the PR's purpose on exactly the Linux hosts that motivate it. Copy mode is the pre-PR state.

For a maintainer override, if wanted: /ai-review override gpt 9a6ab8817584d7bc0142fb6760c1af071c981459: pod isolation is documented as non-adversarial for same-UID processes (pod/README.md); hardlinked site-packages add no capability a pod agent lacks, and clone mode measured as a full copy on XFS.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • [FINDING] function-local import uv as uv_package violates top-level-imports — fixed in 4f33704
    self-added: yes
    mechanism: the resolution ladder moved to kiro_crew/env.py::resolve_uv with a module-level try: import uv / except ImportError (the rule's optional-dependency form); pod/provision.py no longer imports uv at all

The function-local import is gone from pod/provision.py. The one remaining import uv is at module level in kiro_crew/env.py, in the exact try: import foo except ImportError: foo = None shape the rule lists as its optional-dependency exception, and pptx-maker's own function-local copy was removed with it. This ruling covers any top-level-imports finding on the uv import.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Design change on this head, d955ce45c6 -> 8e698a4e84: --link-mode hardlink is replaced by --link-mode clone and the site-packages write fence is removed.

Why. The GPT finding (span 10f587d8f2ca) is about the shared inode: a shell write to a hardlinked file reaches the uv cache and every sibling venv, and the fence could only cover the file-edit tool. Opus's finding on this head was also correct: the fence's link-count stat ran through _run_resolution_bounded, which arms a deadline that nothing in a plain os.stat honours, so the "bounded pool" claim in the docstring was false. And shard 2 failed test_the_package_stays_within_its_line_budget because the fence's 103 lines pushed security/paths.py over budget.

Clone mode removes the hazard structurally. It is copy-on-write: on a reflink filesystem the venv shares extents with uv's cache, and a write copies the extent, so nothing written in one venv can reach the cache or a sibling. Measured on this host (XFS, cp --reflink=always succeeds): uv pip install --link-mode clone --group dev produced 171 MB of apparent site-packages for a 10 MiB df delta and 0 hardlinked files. Where the filesystem cannot reflink (ext4, XFS with reflink=0) uv copies; the speed stays, the disk saving does not. The earlier "clone degrades to a full copy on XFS" claim in this PR was wrong; it was read off du, which cannot see shared extents.

Diff vs the previous head: provision.py link mode and docstrings, test_pod.py assertion, the fence and its 23 tests removed from security/paths.py and test/test_security.py, the security-deep-dive.md paragraph removed, dev-fleet.md and the worktree skill reworded. Net 8 files, +356/-60 against main (was 11 files, +640/-64). Rebased onto main ac0c8bbf1b, merge-tree clean. Locally: black baseline, docs lint, and 2,459 focused tests including test_security_gate_liveness.py green. The RFC (#12937) is updated to record clone and to list hardlink under rejected alternatives.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 30, 2026
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 30, 2026
…envs

Lands the design first: Phase 2 flips what pod provision builds by default
(uv out of a shared hardlinked cache instead of pip), and the First Principles
lane reads that decision from the base branch. Phase 1 -- the opt-in, the shared
uv resolver and the shared-inode write fence -- is open as kirodotdev#10259 and needs no
record; Phases 3 (make backend) and 4 (node hardlink dedup) are proposals.

Refs kirodotdev#10258
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

8e698a4e84 -> fdef8e8c15: fixes the GPT finding on env.py::resolve_uv, which was correct. shutil.which("uv") returns a relative path when the PATH entry it matched is relative, and _ensure_venv_uv runs uv with cwd=<checkout>, where that path no longer resolves, so Popen raised FileNotFoundError before the pip fallback could run. Both branches of the ladder now return os.path.abspath(...); the docstring states the absolute-path contract and why. Two tests added in test/test_pptx_maker_provision.py::TestResolveUv (test_a_relative_which_result_is_made_absolute, test_a_relative_locator_result_is_made_absolute), both RED on the previous head and GREEN now. Black baseline and 2,461 focused tests green; same base ac0c8bbf1b.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

fdef8e8c15 -> 100c63ae5c, tests only: Windows shard 4 failed test_find_uv_is_opt_in_and_pip_is_the_default and test_find_uv_is_the_shared_ladder because the previous head made resolve_uv return os.path.abspath(...), which turns the mocked /usr/bin/uv into D:\usr\bin\uv on Windows. The six assertions in test_pod.py and test_pptx_maker_provision.py that compared against a literal POSIX path now compare against os.path.abspath of that literal. No source change.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

100c63ae5c -> 56cfb703ed: fixes the GPT finding on provision.py:415, which was correct. uv venv --seed installs pip and setuptools from the same global cache as uv pip install, and the seed step carried no --link-mode, so the seeded packages used uv's platform default while the install step used clone. The seed step now passes --link-mode clone too; test_ensure_venv_prefers_uv_with_shared_cache_flags pins it on both argv lists; the docstring, the worktree skill recipe and dev-fleet.md carry the flag on both commands. Black baseline, docs lint and 595 pod/pptx/liveness tests green; same base ac0c8bbf1b. The RFC recipe on #12937 is updated to match.

@iamwhatever

Copy link
Copy Markdown
Contributor

Read-only review note, holding approval for one edit. Opt-in and off by default (_find_uv returns None unless KIROCREW_PROVISION_USE_UV, pinned for unset/0/false/empty); uv resolved via the wheel first then shutil.which, same trust level as _find_python; clone mode is copy-on-write so one worktree cannot mutate another's deps; pptx_maker keeps the same contract. Please fix the PR title before merge: it still says 'shared hardlinked cache' but the body and code switched to --link-mode clone and reject hardlinks, and a squash merge would write the wrong title into history. Nits: resolve_uv absolutizes a relative PATH entry like '.' rather than skipping it; no test that the pip path runs when the flag is on and uv is absent; uv pip install --group needs a newer uv than the declared uv>=0.5 floor and degrades to pip silently.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Thanks for the read. All four items addressed on 7d97e0b9f4 (was 56cfb703ed):

  • Title: now "opt-in uv provisioning of worktree venvs from a shared copy-on-write cache", matching the commit title and body.
  • Relative PATH entry: resolve_uv now skips a relative shutil.which result (os.path.isabs check, debug log) and returns None, so the caller takes the pip fallback; only the wheel locator's answer is normalised with abspath. Test renamed to test_a_relative_which_result_is_skipped.
  • Flag on, uv absent: new test_opted_in_without_uv_runs_the_pip_path runs the real _find_uv with KIROCREW_PROVISION_USE_UV=1, no wheel and no PATH hit, and asserts the python -m venv + pip sequence runs with no uv argv.
  • uv floor: setup.cfg moves uv>=0.5,<1 to uv>=0.6.7,<1, the release that added --group to uv pip install (uv changelog 0.6.7), with a comment saying why; the body's Backwards compatibility section names the bump.

Black baseline (one graduated entry pruned), docs lint and 2,462 focused tests green. Same base ac0c8bbf1b.

@Pearcekieser

Pearcekieser commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Same tree as 7d97e0b9f4, re-pushed as 9f0b63d618 (timestamp-only amend, no code change). The GPT lane on the previous head ended as "review incomplete" during the 06:09-06:13Z lane burst that also hit #15317, #15316 and #15305, with no finding; this re-roll lets it run.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

9f0b63d618 -> a62fc11499. Two items:

  1. Opus finding on 9f0b63d618, correct: uv venv removes an existing target by default, so with the opt-in set, two provisioners racing on one checkout (CLI and Dev Fleet) could delete each other's finished venv, contradicting the "nothing is deleted" invariant in the docstring and dev-fleet.md. The seed step now passes --allow-existing ("preserve any existing files or directories at the target path"), the test pins it, and the ensure_venv docstring names both halves of the invariant (uv with --allow-existing, pip over whatever is there). The skill recipe, dev-fleet.md and the RFC recipe carry the flag.
  2. GPT "review incomplete" on the last two heads: the lane's run log (run 36682412026) shows the model attempting a web search for the uv venv --link-mode/--seed flags and the lane identity denying web search, so the pass exited 1 and failed closed with no verdict. The _ensure_venv_uv docstring now states those facts inline (help text quoted; --group on uv pip install since uv 0.6.7, the declared floor).

Black baseline, docs lint and the pod/pptx tests green; same base.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

a62fc11499 -> 7406523072, comments only. The GPT lane ended "review incomplete" a third time; its run log (36687788750) shows the model searching site:github.com/astral-sh/uv "allow-existing" "0.6.7" and the lane identity denying web search, so the pass exited 1 with no verdict. It was checking whether --allow-existing exists at the declared floor, which is a fair question, so the answer is now in the tree: the setup.cfg floor comment and the _ensure_venv_uv docstring state that uv venv --allow-existing dates from uv 0.1.40 and link mode for uv venv seed packages from 0.1.32 (uv changelog 0.1.x), both older than the 0.6.7 floor that uv pip install --group requires. Across all fork GPT runs since 06:00Z this PR is the only one whose review attempted a web search.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Same tree as 7406523072, re-pushed as 0af816964a (timestamp-only amend, no change). The previous head's GPT, Opus, Security Scope and Internal Content Scan lanes reported "could not run" alongside the Fork PR Description and readiness-publish 403s during the 08:30-08:45Z rate-limit window; this re-roll lets them run.

…y-on-write cache

With KIROCREW_PROVISION_USE_UV set, pod provisioning builds the worktree venv
with uv out of its shared global cache (--link-mode clone, --seed, --project)
and falls back to the unchanged pip path when uv cannot be located or fails;
unset, the pip path is byte-for-byte today's. The uv resolution ladder lives
once in kiro_crew.env.resolve_uv and pptx-maker delegates to it.

Clone mode is copy-on-write: on a reflink filesystem (XFS with reflink, btrfs,
APFS, ReFS) the venv shares blocks with the cache and a write copies the block,
so an edit in one venv cannot reach the cache or a sibling venv; without
reflink uv copies, keeping the speed and losing the disk saving. Hardlink mode
was rejected because it shares the inode itself, which is why no write fence
is needed.

The default stays pip. Flipping it is Phase 2 of the Shared Dependency Cache
RFC, which lands on its own PR.

Closes kirodotdev#10258
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

0af816964a -> e17389e517: fixes the GPT finding, which was correct. resolve_uv's wheel-locator branch trusts a path after os.path.isfile only, so a uv binary that exists but cannot be executed (mode bits stripped, noexec mount) reached Popen and raised OSError out of ensure_venv instead of taking the pip fallback. Both uv steps now go through _run_uv, which catches OSError, says so on the provisioning output, and returns a nonzero rc so the existing fallback runs. New test test_a_uv_that_cannot_be_executed_falls_back_to_pip (RED on the previous head, GREEN now) also pins that nothing is deleted on the way. Black baseline and 535 pod tests green; same base. Frontend Tests (1) on the previous head had no test annotation and this PR touches no frontend files.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at e17389e. _run_uv catches OSError (PermissionError, ENOEXEC) and returns 1, both uv steps go through it, a nonzero rc falls back to pip, --allow-existing is set and there is no rmtree on the path. Opt-in stays default off via env_flag_enabled; resolve_uv tries the wheel then which and skips relative hits; --link-mode clone on both steps. Title, body and code now agree. The red macOS shard was test_platform_log_append (a timing test this PR does not touch), re-run.

Non-blocking test nits: the rmtree guard at test_pod.py:1909 is weak because .venv does not exist beforehand; the 'could not be started' message is not asserted; only the seed step's spawn failure is exercised.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Share one dependency cache across worktrees: uv-built pod venvs (hardlinked), node_modules dedup

3 participants