Skip to content

fix(scratch): keep only the work in the scratch quarantine - #22435

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/scratch-delete-clean-trees
Oct 20, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/scratch-delete-clean-trees

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The scratch sweep stops keeping throwaway bytes in .quarantine: pytest basetemps and rebuildable directories are no longer kept for 14 days.

Since #19897 the hourly sweep moves every dead, idle scratch tree that holds a git repository into <scratch root>/.quarantine and keeps it 14 days. On one host .quarantine reached 28 GB over 24 trees in about a day, and the 80 GB disk hit 99%.

Two causes. Pytest puts pytest-of-<user>/ at the top of the scratch dir (it is TMPDIR), and about 112 test files git init there, so nearly every tree that ran tests "holds a repository". And a kept KiroCrew clone (~4.2 GB) keeps node_modules (0.95 GB) and .venv (0.63 GB), which tools rebuild.

Why it matters

Every agent session that runs tests or installs dependencies in its scratch dir parks gigabytes for two weeks. A busy host fills its disk in a day, and then everything on it fails.

Not a goal

What changed (motivation → approach → change)

The quarantine now keeps the work, not what tools rebuild from it.

Directory Repos inside count? Kept in .quarantine?
node_modules, __pycache__, .pytest_cache, .mypy_cache, .ruff_cache, .tox, .nox no no
any dir holding a regular file pyvenv.cfg (a venv) and no repo below it no no
pytest-of-* at the top of a scratch tree no no
any of the above that itself holds .git or is a git dir yes yes
anything inside a .git dir or a bare repo n/a untouched

holds_git_repository in src/kiro_crew/agent_scratch.py skips those directories, so a tree whose only repositories sit in them is deleted at once, as any repo-free tree was. A tree that still holds a repository is moved as before. The quarantine pass of the same sweep then deletes those directories from every not-yet-expired tree in .quarantine, outside the sweep lock, on every hourly sweep. That covers a tree just moved and trees moved before this change, and a second run changes nothing. A link or junction is never removed or followed, whatever its name: it holds no bytes. Each entry fails open. One INFO line per tree names how many directories went.

src/kiro_crew/docs/troubleshooting.md says what the quarantine drops and that a restored tree needs npm ci or a new venv.

Backwards compatibility

Compatible: the sweep's inputs, names, expiry and log line for a move are unchanged; only which bytes stay in .quarantine changes.

Reader: src/kiro_crew/agent_scratch.py:_expire_quarantine -- entry: release (gateway hourly sweep) -- deletes stamped trees after the TTL whatever they hold; test_a_quarantined_tree_is_deleted_only_after_its_ttl
Reader: src/kiro_crew/agent_scratch.py:sweep_dead_scratch -- entry: release (gateway hourly sweep) -- never reads inside .quarantine except through _expire_quarantine; test_entries_the_sweep_did_not_write_are_left_in_the_quarantine
Reader: a person restoring a quarantined tree (src/kiro_crew/docs/troubleshooting.md) -- entry: chat -- the repository, its history and every other file stay byte-identical; installs and venvs are rebuilt, as the doc now says; test_the_quarantine_keeps_the_work_and_drops_what_tools_rebuild

Tests

New in test/test_agent_scratch.py, TestGitTreesAreQuarantined (snapshot style):

  • test_repositories_only_in_rebuildable_dirs_are_not_work: repos only in pytest-of-* and node_modules → tree deleted, nothing quarantined.
  • test_a_basetemp_name_below_the_top_is_searched: pytest-of-* deeper down is not a basetemp.
  • test_the_quarantine_keeps_the_work_and_drops_what_tools_rebuild: a clone loses installs, venvs, caches and the basetemp; every other path is byte-identical, including a .venv with no pyvenv.cfg, a linked pyvenv.cfg, and a clone named node_modules.
  • test_a_venv_holding_a_repository_is_work: a venv with a clone in its src is kept whole.
  • test_nested_venvs_are_searched_once: nested venvs are listed once each, so the search stays linear.
  • test_a_basetemp_name_inside_a_venv_is_work and test_a_quarantined_venv_keeps_a_repository_below_a_basetemp_name: pytest-of-* counts as a basetemp only at the top of the scratch tree.
  • test_nothing_inside_a_git_directory_is_stripped: regenerable names inside .git and a bare repo stay.
  • test_a_linked_install_is_never_followed: node_modules / pytest-of-* links stay as links; in-tree and outside targets stay byte-identical.
  • test_a_tree_already_in_the_quarantine_is_stripped_on_a_later_sweep: strip on a later sweep, idempotent, repo still works.

Proof:

  • prove.py --base origin/main: PROVEN.
  • 20 repeats of the 10 new ids at -n0, all green.
  • 10 repeats at -n 4 --dist loadgroup --max-worker-restart=2 with test/test_agents_janitor.py and test/test_subagent_shared_scratch.py, all green.
  • The whole file in a shuffled order (seed 1361): 80 passed, 1 skipped.

Manual verification

N/A — unit coverage sufficient: the change is a filesystem walk the tests drive against real git repos, links and venvs.

Related Issues

Closes #22431

Pattern harvest

Not generalizable: the defect is this sweep's own definition of "holds a repository", which had no notion of tool-owned directories.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 20, 2026 06:39
@iamwhatever
iamwhatever requested a review from CrysisDeu October 20, 2026 06:39
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 1c47cceabc960ecca0e5b18f524a4fd6fae783d8 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 1c47cceabc960ecca0e5b18f524a4fd6fae783d8 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check one thing before merge: a tree whose only repositories sit in node_modules or a top-level pytest-of-* is now deleted at once, with no quarantine window.

What this change ships

Inventory (5 items) — 5 justified

Intent: stop the hourly scratch sweep from keeping gigabytes of rebuildable bytes in .quarantine for 14 days, which filled a host's disk in a day (#22431). This is a FIX.

  1. A scratch tree whose only git repos are in pytest-of-* at its top, in node_modules/caches, or in a repo-free venv is deleted instead of quarantined — justified
  2. Each quarantined tree loses its node_modules, __pycache__, the tool caches, its venvs and its top-level basetemp on every hourly sweep — justified
  3. Trees quarantined before this change are stripped on later sweeps too — justified
  4. Each stripped tree gets one INFO log line with a count of removed dirs — justified
  5. troubleshooting.md says what the quarantine drops and that a restored tree needs npm ci or a new venv — justified

[FIRST-PRINCIPLES-REVIEWED] 1c47cce

Description read

[DESCRIPTION-READ] 41d924e8d448f6a51986566a122ad8bfc051b69bbeceb4a07d689cb6186ed1c0

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 1c47cceabc960ecca0e5b18f524a4fd6fae783d8 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the strip's rmtree on a venv, which runs outside the lock, cannot race a person moving that tree out of .quarantine in the same hour.

[DESIGN-REVIEWED] 1c47cce

Description read

[DESCRIPTION-READ] c5d61b8b8f04f63e87bc086f9bf02d345a0f8b2d0b456258051dbf3ab552b48c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 1c47cceabc960ecca0e5b18f524a4fd6fae783d8 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1c47cce

Verdict parsed from the review's SHA-scoped output markers for commit 1c47cceabc960ecca0e5b18f524a4fd6fae783d8.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1c47cceabc960ecca0e5b18f524a4fd6fae783d8: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 1c47cceabc960ecca0e5b18f524a4fd6fae783d8 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1c47cce

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1c47cceabc960ecca0e5b18f524a4fd6fae783d8: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the fix/scratch-delete-clean-trees branch from 4049031 to 53cc374 Compare October 20, 2026 06:55
@iamwhatever

iamwhatever commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: removing links with a regenerable name from quarantined trees

  • fixed: link-removal branch deleted (span=973444e4ba3b)

Agreed: a link holds no bytes, so removing it served nothing in the disk-space Goal.
_strip_regenerable now skips every link and junction whatever its name: never removed, never followed.
test_a_linked_install_is_never_followed pins that in-tree and outside targets, and the links themselves, stay byte-identical.
Fixed in 53cc374.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: hourly rescan of unexpired quarantined trees stays cheap

The rescan is a metadata-only lstat/scandir walk; it opens no file and runs no git.
It never enters .git, a bare repo, a link, or a regenerable dir, so a stripped clone costs about its source file count.
After the first strip, .quarantine holds only work, so the walk shrinks with exactly what this PR removes.
The same sweep already walks every live scratch tree in full each hour (_tree_newest_mtime), a larger set.
It runs off the event loop and outside _SWEEP_LOCK, so it delays no spawn.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (3) red is main's own: test/test_chat_follow_default_slot_kinds.py::TestSlotCreatorRegistry (test_no_stale_entry, test_every_creator_is_classified) fails on a clean origin/main 6432cad checkout too, and names slack/interactions*.py, which this diff does not touch. Fix is in #22433; this PR rebases onto it once it lands.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: a resumed session in a restored tree recovers once installs are rebuilt

Restoring a quarantined tree is a manual move-back (troubleshooting.md); no code path resumes a session into .quarantine.
What is removed is only what npm ci / python -m venv / pytest recreate; the repo, its history and all other files are byte-identical (test_the_quarantine_keeps_the_work_and_drops_what_tools_rebuild).
The doc tells the person restoring to run npm ci or recreate the venv, the same step a fresh clone needs.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: real work only under top-level pytest-of- or node_modules*

pytest-of-* is pytest's own basetemp under $TMPDIR; pytest itself deletes all but the newest three, so it is never a safe home for work.
node_modules is rewritten by every npm ci/npm install; work kept only there is lost on the next install anyway.
A directory with either name that is itself a repo (holds .git or is a git dir) is never regenerable, so a clone named node_modules is kept (test_the_quarantine_keeps_the_work_and_drops_what_tools_rebuild).

@iamwhatever
iamwhatever force-pushed the fix/scratch-delete-clean-trees branch from 53cc374 to daae3c4 Compare October 20, 2026 08:35
@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: venv judged by searching its subtree for a repository

  • fixed: nested venvs searched once (span=df0c0cd14661)

Legitimate: the venv check re-searched the same subtree at each nested venv level, doubling per level.
holds_git_repository now walks a venv in place as its own root instead of calling back into itself, so each directory is listed once.
test_nested_venvs_are_searched_once builds 8 nested venvs over a clone and asserts each directory is listed at most once; it is red on the previous head and green on the fix.
Fixed in daae3c4.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@iamwhatever
iamwhatever force-pushed the fix/scratch-delete-clean-trees branch from daae3c4 to eba03af Compare October 20, 2026 08:48
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: pytest-of- judged as a basetemp only at the scratch-tree top (span=033f2bc5b901)*

Legitimate: a venv was walked as a new root, so a pytest-of-* directly inside it was skipped as a basetemp.
The walk is now _search(tree, basetemp_at_top=...): holds_git_repository passes True, the venv check in _regenerable passes False, and venvs are walked in place without resetting the root.
test_a_basetemp_name_inside_a_venv_is_work (sweep) and test_a_quarantined_venv_keeps_a_repository_below_a_basetemp_name (strip) were red on daae3c4 and are green on eba03af.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: unreachable RecursionError catch removed (span=06e9a1eb0896)

Agreed: the walk runs on an explicit stack and _regenerable is never reached for a venv from inside the walk, so nothing recurses.
The except tuple is back to (OSError, ValueError). Fixed in eba03af.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: whole-disk free-space delta removed (span=8f391b2f326d)

Agreed: shutil.disk_usage measured the whole disk, so other writers skewed it.
The strip now logs only the count of removed directories. Fixed in eba03af.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: a real node_modules holding links or junctions (pnpm) loses only the links

The strip removes a regenerable directory with platform_compat.rmtree_force, the same remover the sweep already uses for whole scratch trees on every platform.
shutil.rmtree on the supported Python (>=3.12, pyproject.toml) unlinks a symlink or junction entry instead of descending into it, so link targets outside the tree are not touched.
This PR adds no new removal primitive; a link at the regenerable name itself is skipped entirely (test_a_linked_install_is_never_followed).

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: a repo found only inside node_modules is an accepted loss

npm link makes node_modules/<pkg> a symlink to the real checkout, and a link is never followed or removed, so that checkout is judged where it lives.
A git-dependency checkout npm writes into node_modules is fetched again by every npm ci; a directory named node_modules that is itself a repo is kept.
Treating repos under node_modules as work is what kept most trees whole before this PR; the issue (#22431) names exactly that as the cost to remove.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: stripping trees quarantined before this change

That is the intended behaviour: the 28 GB on the reporting host sits in trees already quarantined, and only a strip of existing trees frees it.
node_modules is rewritten by every npm ci/npm install, so an edit kept only there was never safe; troubleshooting.md now says the quarantine drops it on every sweep.
The repository itself, its history and every file outside the regenerable directories stay byte-identical.

The scratch sweep moves every dead, idle tree that holds a git repository
into .quarantine for 14 days. Two things made that fill the disk:

- pytest basetemps under the scratch dir hold the throwaway repositories
  test suites git init, so nearly every tree counted as holding a repo;
- a quarantined clone kept node_modules, venvs and caches, which are most
  of its size and are rebuilt from the work.

holds_git_repository no longer searches regenerable subtrees
(node_modules, __pycache__, .pytest_cache, .mypy_cache, .ruff_cache, .tox,
.nox, any dir holding a regular pyvenv.cfg, and a top-level pytest-of-*).
A dir that itself holds .git or is a git dir is never regenerable. A tree
that still holds a repo is moved as before, then stripped of those subtrees
outside the sweep lock; trees already in the quarantine are stripped on a
later sweep. Nothing inside a .git dir or a bare repository is touched, and
a link with a regenerable name is removed as a link only.
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (8) on eba03af was this PR's own test: test_nested_venvs_are_searched_once cloned a repo 8 venvs deep and git hit Windows' 260-char path limit ('Filename too long'). The test now marks the repo with a .git file instead of a clone (still red on the exponential version, green on the fix). Pushed as 1c47cce; rebased on main.

@iamwhatever
iamwhatever force-pushed the fix/scratch-delete-clean-trees branch from eba03af to 1c47cce Compare October 20, 2026 09:26
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: strip racing a person moving a tree out of .quarantine

A rename of the tree out of .quarantine is atomic on one filesystem; the strip either finished on the old path or its next scandir/rmtree of a now-missing path fails and is skipped (per-entry fail-open).
rmtree works by path under the quarantined tree, so a tree already moved elsewhere is never reached through the old name.
What a racing strip can remove is only a regenerable directory (installs, venvs, caches), which the doc says a restored tree rebuilds anyway; the repo and all other files are never touched.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: tree whose only repos sit in node_modules or a top-level pytest-of- is deleted at once*

That is the fix for issue #22431: pytest's basetemp repos made nearly every tree that ran tests look like work, which filled the disk.
pytest keeps only its three newest basetemps and npm ci rewrites node_modules, so neither is a place work survives today.
A directory with either name that is itself a repo is kept, and repos anywhere else in the tree still send it to the quarantine.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 20, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every review lane on the head is clean (only runner notices and PR Scope size warnings); held by the triage cron on a category, path or P0-area rule, not a finding. Approved on the watch holder's request.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 20, 2026 13:46
@KiroCrewCaptain
KiroCrewCaptain merged commit 656ab2f into main Oct 20, 2026
104 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/scratch-delete-clean-trees branch October 20, 2026 13:48
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scratch quarantine keeps pytest basetemps and rebuildable dirs, filling the disk

3 participants