Skip to content

fix(packaging): clean failed bundle setup - #13538

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/13382-packaging-cleanup-refusal
Sep 25, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/13382-packaging-cleanup-refusal

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

build_bundle creates its staging tree and ownership marker before reading the existing report, output directory, and carried plan. A raw filesystem OSError in that setup window escapes before the main cleanup transaction, leaving both staging paths behind so later builds refuse the stale claim. Separately, _open_dir_nofollow_pinned lets Path.resolve() raise RuntimeError for a symlink loop past callers that handle only OSError.

Why it matters

One transient permission, I/O, stale-handle, or long-path failure can become a standing refusal that requires manual deletion. A symlink loop bypasses the builder's fail-closed refusal boundary and surfaces as a traceback. The staging descriptor must also close before early cleanup so repeated failures cannot leak file descriptors.

What changed (motivation → approach → change)

Gap Change
Post-marker setup faults The existing report/output/plan setup is enclosed by one OSError boundary. It keeps the retained staging descriptor open while cleanup captures the path, requires the captured tree's device/inode to match that descriptor, removes only this run's identity-owned staging tree and marker through existing guarded helpers, then closes the descriptor exactly once and raises ExportRefused with the original exception as its cause. The purge reports whether capture/verification/sweep completed: successful cleanup recommends a retry, while safe preservation of a swapped or unpinnable tree names the residue and tells the operator to move it before retrying.
Purge sweep as disposal The private-aside purge leaves the captured, verified tree in the run-private directory for the _rmtree_pinned sweep to delete, so that sweep IS the deletion. A swallowed sweep failure previously let _purge_staging_best_effort return success over a tree still sitting undeleted in the aside. _dispose_via_private_aside now records whether the sweep completed and carries it in its existing bool return: True only when the rename captured the target, verify and settle ran, and the sweep completed; False when the target vanished before capture (removed or moved by another process, so nothing was captured or verified) or when the sweep did not complete. _purge_via_private_aside forwards that result. Only _purge_staging_best_effort reads it; the transaction-path and post-promotion <out>.previous purges ignore it, so a failed sweep never turns a bundle that already landed into a refusal. No flag and no raise: the sweep status travels in the return the one reader already consumed. When the staging purge reports False, the pre-transaction refusal says the tree could not be released safely and that it is still at the staging path or under a .smc-purge-* directory beside it.
Deliberate refusals ExportRefused remains outside the OSError catch, preserving each existing specific message and cleanup path. Promotion and rollback ordering are unchanged.
Resolve loops _open_dir_nofollow_pinned normalizes CPython 3.12 resolve-time RuntimeError and CPython 3.13 component-walk loop errors to OSError(errno.ELOOP, ...) at one boundary. Confirmed symlink components that report ENOTDIR under `O_NOFOLLOW
Contract ratchets Packaging call-site inventories classify the converted probes and resolve boundary. The security mutation anchor follows the refactored setup without weakening its verdict.

The cleanup path never deletes an unowned output or report. A prior bundle and report remain byte-identical after an injected fault, staging and its marker are absent, every retained descriptor is closed, and the same build succeeds when retried.

Backwards compatibility

No successful-build behavior or bundle format changes. Failures that previously escaped as raw OSError or symlink-loop RuntimeError now return the existing packaging refusal type after bounded cleanup. Specific ExportRefused messages continue unchanged.

Tests

  • Current one-commit head a6a0f5013 sits on main 3f16157ba (a clean rebase of 2fc176474, git range-diff identical: 4926e50b3 carries the ci(lint): run hosted black through bounded_black #13698 bounded-black fix for CI: Backend Lint's black step killed by runner shutdown (exit 143) on 4 consecutive fork-PR heads #13386, and 84da367c7 hit the CI flake: test_the_trim_and_count_execute_as_written dies with SIGPIPE (141) under pipefail + head -n1 #13360 SIGPIPE flake in Backend Tests shard 2). 2fc176474 answered the Opus advisory on ec81442d1: when the no-follow component walk in _open_dir_nofollow_pinned meets a link, the ELOOP refusal now names the component and says it is a link where a directory was resolved (a loop, or a change since resolution), instead of calling every such case a "symlink loop" (regression test_a_component_swapped_for_a_link_is_named_not_called_a_loop). ec81442d1 reworded one test docstring from f11a41551 to present tense (the Comment History Gate flagged "used to"). f11a41551 amended 1f403fba7 to answer the two advisory findings on that head: the unconfirmed-release refusal no longer claims the tree is "still there or under a .smc-purge-* directory" (a tree moved by another process before capture is elsewhere), and the _refuse_unless_this_build_wrote_it inventory verdict now records that its sole caller sits inside the OSError boundary.
  • Focused transaction/resolve and hash/promotion suites: 38 passed.
  • test_post_marker_fault_with_staging_moved_away_does_not_claim_where_it_is moves staging away before the purge's capture rename during a post-marker fault and asserts the refusal says deletion was not confirmed and names "wherever another process moved it"; it fails against 1f403fba7's build.py.
  • Crew packaging suite with the worktree source on PYTHONPATH: 539 passed, 1 skipped.
  • test_a_staging_tree_moved_away_before_capture_is_not_reported_as_purged moves staging away before the capture rename and asserts _purge_staging_best_effort returns False, leaves the moved tree untouched and sweeps the empty aside. test_a_failed_purge_sweep_is_not_reported_as_released drives a failed private-aside sweep end to end through the pre-transaction refusal: the message says "could not be released safely", names the .smc-purge-* location, and never says "was released".
  • Sweep-failure regressions in test_hash_and_promotion_authority.py: test_a_failed_private_aside_sweep_reports_residue_not_success forces the sweep to fail after capture+verify and asserts _purge_staging_best_effort returns False with the tree retained; test_MUTATION_swallowing_the_sweep_failure_reports_a_false_success source-mutates return not sweep_failed to return True and shows the purge then reports True over an undeleted tree, so the return value is load-bearing; test_a_failed_sweep_never_raises_so_post_promotion_sites_do_not_refuse asserts _purge_via_private_aside returns False without raising on a failed sweep and leaves residue. Against the prior head's build.py, the changed tests fail (the default purge returned True over the unswept tree; the refusal did not name the aside).
  • Test file applied to untouched base f7d66984485d93a6d8cbd9e2ac1a38e7df8ca6c0: the setup rows escape raw OSError; the symlink-loop rows escape RuntimeError; the non-vacuity controls pass.
  • The swapped-staging regression fails blocked head 3f30b6cac782ccc541fc857ec1c7659f6698de2e because the planted bundle-shaped victim is recursively deleted. The repaired path preserves the victim and displaced original, removes this run's marker, and closes the retained descriptor.
  • The setup-fault matrix covers existing-report is_file, output-directory exists, and carried-plan is_file, and proves staging/marker cleanup, descriptor closure, prior artifact preservation, and successful retry.
  • The repository black gate (scripts/check_black_formatting.py), isort and flake8 on the changed files, and mypy on build.py pass. The diff contains no .github changes.

Manual verification

N/A — deterministic filesystem fault injection and real symlink-loop tests exercise the affected transaction and no-follow boundaries.

Related Issues

Fixes #13382

Pattern harvest

Rule candidate: review-prompt
Pattern: Once a transaction publishes an ownership marker, every setup read before the main mutation must share the transaction's descriptor release, identity-safe cleanup, and typed refusal boundary. A best-effort teardown whose sweep IS the deletion must report whether the sweep completed, not swallow its failure and return success over an undeleted tree.

Checklist

  • One Conventional Commit on current main
  • Existing applicable tests pass and new regressions fail on the old head
  • Self-review completed; code follows project style guidelines
  • No bundle-format or successful-path behavior change
  • No secrets, credentials, internal references, or .github changes in the diff

@jeeshofone
jeeshofone requested review from a team and patrigao September 24, 2026 21:28
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Exact head 64bb1d0bf30f325a84676f6c0c9b9ff68c9bc7e9 is one commit on current main c08b4e2b70d754f4d5caa67170199addb7acd52d. The focused transaction/resolve suite passes 8 tests; the complete packaging suite passes 531 tests with 1 skipped; full mypy and all formatting, history, brand, Agent SDK, async-I/O, subprocess, docs, feature-map, lockdown, diff, and zero-.github gates pass. Exact GPT 5.6 and Opus 5 local reviews report no findings. Please approve the fork workflow runs for this head.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed a6a0f5013b71b66323018ffa78542fe5b9470b9f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a6a0f50

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of a6a0f5013b71b66323018ffa78542fe5b9470b9f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify CI runs CPython 3.13, since the deferred-ELOOP resolve shape is exercised only via a monkeypatched simulation, not a real 3.13 interpreter.

[DESIGN-REVIEWED] a6a0f50

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of a6a0f5013b71b66323018ffa78542fe5b9470b9f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the CPython 3.13 deferred-loop claim on a 3.13 runner; both shapes are normalized either way, so only the test premise rides on it.

What this change ships

Inventory (8 items) — 8 justified

Intent: FIX — a transient filesystem fault or symlink loop during bundle setup no longer strands the staging tree/marker into a standing refusal or escapes as a traceback.

  1. A raw fault after the ownership marker exists now refuses with retry guidance instead of a traceback that blocks every later build — justified
  2. A symlink loop at the pinned opener refuses as OSError(ELOOP) instead of an escaped RuntimeError — justified
  3. A component swapped to a plain link is named, kept ELOOP, ordinary non-directories stay ENOTDIR — justified
  4. Failure-path purge now reports whether deletion happened; refusals distinguish "released" from "residue at .smc-purge-*/moved elsewhere" — justified
  5. Cleanup refuses a staging tree whose device/inode differs from the descriptor this build opened; swapped trees preserved — justified
  6. Mid-transaction and final-cleanup purges get the same identity check — justified
  7. Staging descriptor closed exactly once on every failure path, no fd leak on repeated failures — justified
  8. Call-site inventory and mutation anchor rewritten to record the fixed state — justified

Provenance checked: new tests fail on the untouched base per description and exercise each claim; deleted VERDICTS pins recorded the defects this diff fixes, rewritten in the same commit. Counted: _verify_captured_is_staging_fd has 2 call sites (one passes expected_identity, one legitimately defaults); remaining unconverted .resolve siblings (_dispose_via_private_aside, build_bundle ×2) are pre-staging or recorded as left in the same-commit inventory ratchet.

[FIRST-PRINCIPLES-REVIEWED] a6a0f50

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed a6a0f5013b71b66323018ffa78542fe5b9470b9f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a6a0f50

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 24, 2026
@jeeshofone
jeeshofone force-pushed the fix/13382-packaging-cleanup-refusal branch 3 times, most recently from f8fcd7f to 773d3a7 Compare September 25, 2026 02:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@jeeshofone
jeeshofone force-pushed the fix/13382-packaging-cleanup-refusal branch from 773d3a7 to ad216f0 Compare September 25, 2026 04:29
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Dispositions for every open review finding, on head ad216f0543bdd92e4ab685afd54b4b7b2459f5b5 (one commit on main d6f2ed5a91a92240eda29174b212886480e6f0c1). Line numbers refer to src/kiro_crew/apps/builtins/aws_control/crew/packaging/build.py at this head.

  1. GPT 5.6, build.py:4899 (reviewed b123faf): sweep_failed unbound when os.mkdir raises. Fixed. _dispose_via_private_aside now binds sweep_failed = False before the pinned-parent open, so no error path can reach the read with the name unbound. The adjudicator's point stands too: the mkdir OSError propagates through an except-less try/finally and never reaches that read. The lane flagged this as fenced security-class. If it re-flags on this head, it needs a maintainer override on that rationale.

  2. First Principles, item 4 (reviewed b123faf): the sweep-failure raise reached all four purge sites. Fixed. _purge_via_private_aside takes report_sweep_failure with a default of False. Only _purge_staging_best_effort passes True. The transaction-path and post-promotion <out>.previous purges keep their prior swallow. test_the_default_purge_swallows_a_failed_sweep_so_post_promotion_sites_do_not_raise pins this.

  3. Opus 5, test_transaction_oserror_and_resolve_loop.py:240 (reviewed 64bb1d0): __cause__ is RuntimeError fails on CPython 3.13. Fixed. _open_dir_nofollow_pinned normalizes the 3.12 RuntimeError, a 3.13 OSError(ELOOP) from resolve(), and a component-walk ELOOP or confirmed-link ENOTDIR to OSError(ELOOP) at one boundary. The tests assert only errno == ELOOP. On this head the test docstring now says that the cause's shape depends on the interpreter version.

  4. Opus 5, build.py:5558 (reviewed 64bb1d0): the refusal said "released" when _purge_staging_best_effort swallowed a failed purge. Fixed. _release_pretransaction_staging returns the purge result. The OSError boundary picks "was released ... Re-run the build" only when that result is True. Otherwise it picks "the staging tree at <path> could not be released safely; move it before retrying". test_post_marker_fault_preserves_a_swapped_bundle_shaped_staging_tree and test_a_failed_purge_sweep_is_not_reported_as_released cover both branches.

  5. GPT review: _purge_staging_best_effort could report success when the purge had not completed. Fixed on this head. A failed private-aside sweep already returned False through report_sweep_failure. One path still read as success: when the target vanished before the capture rename (removed or moved by another process), _dispose_via_private_aside took its FileNotFoundError early return, and the helper returned True with nothing captured or verified. _dispose_via_private_aside and _purge_via_private_aside now return whether the capture happened. That early return yields False, and _purge_staging_best_effort returns the value directly, so True means capture, identity verification and sweep all completed. The other purge sites ignore the return value, so their behaviour is unchanged. Regression test_a_staging_tree_moved_away_before_capture_is_not_reported_as_purged fails on prior head f8fcd7fb7 with assert True is False and passes here.

  6. Opus 5 Review "review incomplete" on f8fcd7fb7. That lane never produced a verdict for that head. Superseded: this new head triggers a fresh run.

  7. First Principles Review "review incomplete (advisory)" on f8fcd7fb7. Same cause as item 6, and superseded by this head.

Backend Lint & Type Check (3.12) on f8fcd7fb7 failed with exit code 143 at 00:25:15Z, logged as "The runner has received a shutdown signal". That is runner preemption, not a lint result. Locally on this head the repository black gate (scripts/check_black_formatting.py), isort, flake8 and mypy on build.py all pass. The crew packaging suite passes (537 passed, 1 skipped).

@jeeshofone
jeeshofone force-pushed the fix/13382-packaging-cleanup-refusal branch from 1f403fb to f11a415 Compare September 25, 2026 15:46
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition, GPT 5.6 finding 42eaf796f350 (build.py:5622, reviewed 1f403fba7): fixed on head f11a41551b5d9a53e31b871c1922a2e5adb2dfac.

When _release_pretransaction_staging(remove_marker=True) returns False, the ExportRefused retry guidance in build_bundle (now build.py:5625-5630) no longer asserts the tree is "still there or under a .smc-purge-* directory". It says deletion was not confirmed and lists every place the residue can be: still at the staging path, under a .smc-purge-* directory beside it, or wherever another process moved it. A code comment records why False covers the moved-away and verify-refused-and-restored cases, not only an incomplete sweep.

Regression: packaging/tests/test_transaction_oserror_and_resolve_loop.py::test_post_marker_fault_with_staging_moved_away_does_not_claim_where_it_is injects a post-marker OSError, moves the staging tree away before the purge captures it, and asserts the message contains "deletion was not confirmed" and "wherever another process moved it", does not contain "it is still there", and that the moved tree is left intact.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition, Opus 5 finding 5a5d6a47462a (test_external_input_call_sites.py:373, reviewed 1f403fba7): fixed on head f11a41551b5d9a53e31b871c1922a2e5adb2dfac.

The _refuse_unless_this_build_wrote_it verdict (test_external_input_call_sites.py:373-378) no longer claims a strand. It now reads the same way as the rewritten build_bundle entry: exists() and is_dir()/is_file() collapse absent and invalid entries and raise for unreadable ones, and its sole caller in build_bundle runs it inside the OSError-to-ExportRefused boundary that releases this run's staging tree and marker, so a raised OSError no longer strands them. The residue case when release cannot be confirmed is covered by the GPT 42eaf796f350 fix and its regression test_post_marker_fault_with_staging_moved_away_does_not_claim_where_it_is, which asserts the marker and staging path are gone after the refusal.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@jeeshofone
jeeshofone force-pushed the fix/13382-packaging-cleanup-refusal branch from f11a415 to ec81442 Compare September 25, 2026 16:45
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head ec81442d1821ebb787b6da941a2b598920fd1e85 (one commit on main c6b1ebf24).

On f11a41551 the Comment History Gate failed (job) on test_transaction_oserror_and_resolve_loop.py:254: used to in the docstring of the new regression. Fixed: the docstring now states current behaviour in present tense; range-diff against f11a41551 shows that docstring as the only change. Coverage Gate and Await Fast Gate on f11a41551 were downstream of that Fast Gate failure (backend tests were skipped, frontend-test=skipped -- failing closed).

The GPT 42eaf796f350 and Opus 5a5d6a47462a dispositions above carry over unchanged to this head.

Local on this head: crew packaging suite 538 passed, 1 skipped; isort, flake8, black and check_comment_history.py clean on the changed files.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@jeeshofone
jeeshofone force-pushed the fix/13382-packaging-cleanup-refusal branch from ec81442 to 2fc1764 Compare September 25, 2026 17:28
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition, Opus 5 advisory on build.py:2014 (reviewed ec81442d1): fixed on head 2fc17647437021a5afb83d8a8c97c510e0b1daa6 (one commit on main 37b3210ba).

Confirmed the premise on Linux: os.open(link, O_RDONLY|O_DIRECTORY|O_NOFOLLOW, dir_fd=...) on an ordinary, non-cyclic symlink fails ENOTDIR, so the ENOTDIR + S_ISLNK arm took a post-resolution component swap and reported it as symlink loop resolving <dir>. A no-follow open cannot tell a loop from a swapped link, so _open_dir_nofollow_pinned's walk arm now raises OSError(ELOOP, "component '<part>' of <dir> is a symbolic link where a directory was resolved (a symlink loop, or it changed to a link since resolution): <original error>"). It keeps ELOOP, so every caller's except OSError still fails closed, and it keeps the original errno text and filename. The resolve()-reported cases (3.12 RuntimeError, 3.13 ELOOP) keep symlink loop resolving. Callers that embed {exc} (_write_bytes_nofollow, _dispose_via_private_aside, _publish_report) now carry the component name.

Regression: packaging/tests/test_transaction_oserror_and_resolve_loop.py::test_a_component_swapped_for_a_link_is_named_not_called_a_loop opens swapped/leaf with already_resolved=True where swapped is a plain link to a real directory. It asserts ELOOP, the component name and "changed to a link since resolution" in the message, and no "symlink loop resolving". It fails on ec81442d1's build.py and passes on this head.

Local on this head: crew packaging suite 539 passed, 1 skipped. mypy on build.py, black, isort, flake8 and check_comment_history.py are clean on the changed files. The GPT 42eaf796f350 and Opus 5a5d6a47462a dispositions above carry over unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@jeeshofone
jeeshofone force-pushed the fix/13382-packaging-cleanup-refusal branch from 2fc1764 to 84da367 Compare September 25, 2026 20:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
@jeeshofone
jeeshofone force-pushed the fix/13382-packaging-cleanup-refusal branch from 84da367 to a6a0f50 Compare September 25, 2026 21:48
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 25, 2026 23:11
@iamwhatever
iamwhatever merged commit 2642b6b into kirodotdev:main Sep 25, 2026
97 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

crew packaging: OSError before the cleanup transaction strands the staging marker; resolve() raises RuntimeError past except OSError

2 participants