Skip to content

fix(snapshot): refuse a bundle restore would refuse, share one size override - #23202

Merged
bolichen97 merged 1 commit into
mainfrom
fix/snapshot-archive-bound-23171
Oct 23, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/snapshot-archive-bound-23171

Conversation

@JLargent42

@JLargent42 JLargent42 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: kirocrew snapshot never writes a bundle that kirocrew restore refuses, and an install larger than 4 GiB has an explicit, bounded way to back up and restore.

On a large install, kirocrew snapshot writes a bundle and kirocrew restore then refuses it: "This archive declares more than 4 GiB of uncompressed content, which no memory bundle produces". The restore bound _MAX_ARCHIVE_BYTES is fixed at 4 GiB, and the writer never stops at that size.

Why it matters

The person hit by this has a backup on disk and no supported way to restore it, and finds out only on the new machine. That is the moment the backup exists for.

Not a goal

  • Removing or raising the default 4 GiB bound. It stays as the decompression guard on every archive reader.
  • Sizing the bound from the archive's own manifest. The manifest sits inside the archive, so it can be read only after extraction, and whoever made the file wrote it.
  • Changing the member-count bound (200,000 entries) or the AWS Control off-host backup path. That path calls snapshot_main with the default bound and now gets the same early refusal.

What changed (motivation → approach → change)

Restore guards against a decompression bomb by walking member headers and refusing past 4 GiB. Snapshot never held itself to that number. Its only check read the finished tarball back after the full compression pass, and it left the unrestorable file under the name --keep and restore look for.

Side Behaviour
kirocrew snapshot _build_snapshot holds the staged tree to the restore bound before the tarball is created. It counts entries and regular-file bytes, the same two quantities the reader counts. Over the bound it exits 1 with no bundle written and nothing pruned. A byte refusal names the largest entry, --components and --max-archive-gib; an entry-count refusal names only --components, since the flag does not move that bound.
both commands --max-archive-gib N (1-1024) sets the byte bound for that run. A bundle written past the default restores with any N at least as large.
kirocrew restore Without the flag, nothing changes. With it, the header walk refuses past N GiB, and also past the free space of the extraction directory, before anything is extracted.

The owners are src/kiro_crew/snapshot_archive.py (_refuse_oversized_tree, archive_byte_bound, and the max_bytes parameter on _refuse_oversized_archive), src/kiro_crew/snapshot_build.py (the pre-tarball check) and the src/kiro_crew/snapshot.py facade (flag validation and the two commands). src/kiro_crew/cli_parser_admin.py adds the flag to the kirocrew CLI. The user guide src/kiro_crew/docs/snapshot-and-restore.md and the CLI spec docs/system-specs/modules/cli.md describe the limit and the flag.

Backwards compatibility

Compatible: without --max-archive-gib, both commands enforce the same 4 GiB and 200,000-entry bounds as main. A bundle over the bound already exited 1 on main. The only difference is that the refusal now lands before the tarball is published rather than after.

Removes nothing: no flag, key or code path is removed. The one taken-away outcome is the unrestorable tarball main left on disk after its exit-1 refusal, and nothing could read that file anyway.

Tests

test/test_snapshot_archive_override.py (new, 14 tests):

  • an entry-count refusal suggests --components and not --max-archive-gib
  • an oversized snapshot never opens a tarball for writing, leaves no bundle or .tmp, keeps the older bundle under --keep 1, and names both remedies
  • a bundle written with --max-archive-gib 1 is refused by restore without the flag and restored with --max-archive-gib 2
  • a normal bundle round-trips with no flag
  • a forged 5 GiB header is refused before extraction, both with and without the flag
  • the raised bound is still held to free space
  • out-of-range values (0, -1, 1025) are refused by both commands
  • the writer's tree bound and the reader's header bound agree at the exact byte and entry boundary

Updated pins: the CLI argument tables in test/test_snapshot_refactor_contracts.py, the owner map in test/test_snapshot_refactor_ownership.py, and the two ordering locators in test/test_snapshot_unrestorable_refusal.py and test/test_snapshot_install_scoped_validation.py, which now match the call with its new keyword argument.

Determinism: 20 runs at -n0, 10 at -n 4 --dist loadgroup --max-worker-restart=2 with test_snapshot_archive_bound.py and test_snapshot_unrestorable_refusal.py as co-tenants, and one shuffled @file run (seed 24117). All green.

Revert-and-retest: prove.py came back INCONCLUSIVE (exit 21) because the mutated run errored at collection. I substituted a manual revert: with the four production files checked out from origin/main, 11 of the 13 new tests fail. The 2 that pass are the no-flag round trip and the no-flag forged-header refusal, which are the unchanged behaviour they exist to lock.

Local gates: the backend scoped run passed 50,679 tests with 1 failure. That failure is test_snapshot_merge_core_files_pinned.py::test_the_rewrite_keeps_the_live_stores_access_control_acl: setfacl is unsupported on this host's filesystem, and the test fails the same way on clean main. black, isort, flake8, mypy, the comment-history, brand, docs-lint and file-line-cap gates all pass, as do the flake-pattern ratchet and semgrep 1.78.0 with --baseline-commit.

Manual verification

N/A — the unit tests drive the real snapshot_main and restore_main against real tarballs on disk, and the bound is lowered by monkeypatch so the over-limit path runs without writing gigabytes.

Related Issues

Closes #23171

Pattern harvest

Rule candidate: review-prompt
Pattern: a reader-side limit (size, count, schema) with no matching writer-side check, so the tool produces output it later refuses to read

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@JLargent42

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/history.md since this head went green (5c32cf3 -> a5ceafb); CI re-running.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a34a48541f714c07141e7c3f22f79228eadc818e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the writer-side check fixes the cause (no writer-side limit to match the reader's), and the flag that moves the limit is off by default.

Notes

  • Item 2: duplicate. The --max-archive-gib help text appears three times: twice as a literal in cli_parser_admin.py and once as _MAX_ARCHIVE_GIB_HELP in snapshot.py. Subtraction: delete the two literals and import _MAX_ARCHIVE_GIB_HELP.
  • Item 1: one unfixed sibling. Of the three _refuse_oversized_archive( call sites (grepped), only snapshot.py:382 (the redaction read-back) still passes no max_bytes. So a bundle made with the raised limit still fails there. The PR has the AWS path use the default limit and refuse early, so this is a known and recorded gap.

What this change ships

Inventory (7 items) — 6 justified

Intent: let a person with a large install make a backup that restore will actually accept. This is a FIX that also adds an opt-in flag.

  1. kirocrew snapshot refuses an over-limit selection before writing a tarball, and prunes nothing — justified
  2. New --max-archive-gib N (1-1024) on snapshot — duplicate of _MAX_ARCHIVE_GIB_HELP text in cli_parser_admin.py
  3. New --max-archive-gib N on restore — justified
  4. With a raised limit, restore also refuses an archive that declares more than the free disk space — justified
  5. An entry-count refusal suggests only --components — justified
  6. The restore refusal message now points at --max-archive-gib — justified
  7. User guide section "Installs larger than 4 GiB" and new CLI spec rows — justified

[FIRST-PRINCIPLES-REVIEWED] a34a485

Description read

[DESCRIPTION-READ] 687f1a2ccf400fddf08c4a7fcaa6cc9e88472156ed909dd7faf340f170629821

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a34a48541f714c07141e7c3f22f79228eadc818e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a34a48541f714c07141e7c3f22f79228eadc818e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: snapshot now refuses before writing a bundle that restore would refuse, and the size override is opt-in, has an upper limit, is checked the same way by both commands, and changes nothing when the flag is absent.

Notes

  • The N a bundle needs is stored nowhere, so whoever restores it must remember it or find it by trial. Printing the N used in snapshot's success line would help without trusting the manifest.

[DESIGN-REVIEWED] a34a485

Description read

[DESCRIPTION-READ] 17b2fd3c61c9bd568acb01e068567460fc78c306bcef4cef5b460cf46d50f283

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a34a48541f714c07141e7c3f22f79228eadc818e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a34a485

Verdict parsed from the review's SHA-scoped output markers for commit a34a48541f714c07141e7c3f22f79228eadc818e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a34a48541f714c07141e7c3f22f79228eadc818e: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a34a48541f714c07141e7c3f22f79228eadc818e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a34a485

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a34a48541f714c07141e7c3f22f79228eadc818e: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 22, 2026
@JLargent42
JLargent42 force-pushed the fix/snapshot-archive-bound-23171 branch from 89eb909 to aa79fdc Compare October 22, 2026 23:01
@JLargent42

JLargent42 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: shared _ArchiveTooLarge handler printed byte-bound advice for the entry-count refusal

  • span=1c0010f3de8c One refusal message gives the operator a fix that cannot work

Outcome: fixed in aa79fdc.
_ArchiveTooLarge carries kind ("bytes" | "entries"); both entry-count raise sites set kind="entries".
The snapshot handler names --max-archive-gib only for a byte refusal; an entry-count refusal names only --components, since no flag moves the 200,000-entry bound.
Regression: test_an_entry_count_refusal_does_not_offer_the_byte_override (red with the handler change reverted, green with it).

@JLargent42

Copy link
Copy Markdown
Contributor Author

GPT 6.1 Review on 89eb909 did not produce a verdict: its pass 2 never completed, so the gate failed closed with no [GPT-REVIEWED] marker. That is a lane infrastructure failure, not a finding. The push to aa79fdc starts a fresh GPT run.

@JLargent42

Copy link
Copy Markdown
Contributor Author

Two reds on aa79fdc that this diff does not cause:

  • Backend Tests (Windows) (7): all 33 cases of test_sibling_list_check.py::test_a_heading_is_found_exactly_when_the_template_check_finds_it fail because bash .github/scripts/pr-description-check.sh exits 1 on runner image 20261021.278.1. This PR touches neither file, and the same shard on main (41708c9) passes the same 21,796 tests. Recorded as test_a_heading_is_found_exactly_when_the_template_check_finds_it fails deterministically on GitHub-hosted windows-latest #23254; the job is re-run once.
  • Sensitive Change Review: the job could not read the PR from the API on three attempts, during the repo-wide installation rate-limit window. Not re-run again; it needs a maintainer re-run or approval.

@JLargent42

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (7) failed again on its single re-run, the same 33 cases. This is deterministic, not a flake. It hits GitHub-hosted windows-latest runners, and same-window shard 7 jobs on the CodeBuild Windows pool pass. Root cause and runner evidence are on #23254 (tier:T1). This PR does not touch the test or the script, so this red belongs to main rather than to this diff.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@JLargent42
JLargent42 force-pushed the fix/snapshot-archive-bound-23171 branch from aa79fdc to a34a485 Compare October 23, 2026 06:56
@JLargent42

Copy link
Copy Markdown
Contributor Author

Rebased once onto main for the #23264 Windows fix.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 23, 2026
@JLargent42

Copy link
Copy Markdown
Contributor Author

Review-ready at a34a485: PR Readiness passed (104 checks, 0 failing). GPT 6.1 and Opus 5.5 posted no blocking findings, and Design and First Principles both passed. Waiting on maintainer approval.

@bolichen97
bolichen97 enabled auto-merge (squash) October 23, 2026 16:13

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review passed — the writer and reader now share the same bounded archive limit, and the optional override remains explicit and capped.

Sensitive change review

  • Head commit: a34a485
  • What rule changed: local operators may raise the snapshot and restore content limit from 4 GiB to at most 1024 GiB.
  • Before -> after: defaults remain 4 GiB; an explicit bounded flag raises both writer and reader limits together.
  • Worst case if wrong: a very large archive could exhaust temporary disk space before extraction.
  • Evidence checked: integer bounds, member-count cap, tar-header preflight, free-space refusal, path traversal checks, link checks, current-head tests and reviewer signals.
  • No regression:
    • Tools still work: no-flag behavior is unchanged.
    • Backward compatible: old bundles and callers use the original default.
    • Existing tests: current-head required checks pass.
  • How to undo: revert this pull request; no data migration is required.

[Security: the override changes a decompression safety limit, but only a local operator can set it, the default remains unchanged, and path, link, member-count, header-size, and free-space guards still apply.]

@bolichen97
bolichen97 merged commit 88786dc into main Oct 23, 2026
106 checks passed
@bolichen97
bolichen97 deleted the fix/snapshot-archive-bound-23171 branch October 23, 2026 16:17
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

snapshot writes a bundle that restore refuses past the 4 GiB archive bound

2 participants