Skip to content

fix(restore): resolve merge core-file and cron-store names once - #18090

Merged
buluoray merged 2 commits into
mainfrom
fix/snapshot-merge-pinned-copies-8725
Oct 12, 2026
Merged

buluoray merged 2 commits into
mainfrom
fix/snapshot-merge-pinned-copies-8725

Conversation

@buluoray

@buluoray buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A merge restore (CLI kirocrew restore --mode merge and the dashboard Import → Merge) can no longer be redirected by a link or hardlink planted at a core-file name into writing outside the data home or importing a file it was not pointed at.

Merge restore installed memory.db, memory_index.db and crons.json (the dashboard import also hooks.json) with an is_file() name check followed by shutil.copy2, and _merge_crons read the live crons.json by name and rewrote it by name. Each pair is two resolutions of one string:

  • a dangling symlink at the live name answers is_file() == False, and copy2 then follows it and creates the bundle's file wherever it points;
  • a hardlink alias of a credential in the agent-writable staging tree is a regular file every name check accepts, so its bytes were installed as memory.db / crons.json;
  • a symlink swapped in at the live crons.json between the merge's read and its write_text made the rewrite truncate the link target.

Why it matters

The staging tree and the data home are writable by the agent. These are the same validate-then-reopen class the notification merge already closed, left open at the sites next to it: a planted name turns a restore into an arbitrary-file write (the cron rewrite) or a credential read into a live core file.

Not a goal

  • The _merge_memory integrity-check-then-ATTACH DATABASE of the staged database (issue site 6). SQLite accepts only a path, so the posture (copy into a private temp file first, or refuse and document) is an owner decision; it stays open on snapshot restore: six more validate-then-reopen sites in the same path as #8667 #8725.
  • The dashboard import's skills/ tree copy (shutil.copytree / copy2 by name in portability.apply_import_zip). It is a tree, not a core file, and needs the tree primitive rather than this helper.
  • The notifications no-live-file copy2 arm, which open PR fix(snapshot): validate notification records the copy branch installs #8576 owns.
  • Substitution of the rewrite's staged temporary by a same-UID writer in the data home. The replace stages a temporary by name inside the data home, exactly as the cron store's own save (cron.py _save → atomic_write) does on every write; no portable descriptor-based publish exists.

What changed (motivation → approach → change)

Symptom → cause → change:

  • Core-file installs (issue sites 1–3, plus the dashboard twins). New snapshot_merge._install_core_file_if_absent is one call to pinned_fs.copy_file_pinned(..., skip_existing=True) — the config component's existing install path. The source is opened once with O_NOFOLLOW and judged on its descriptor (regular, st_nlink == 1); the destination is created O_CREAT|O_EXCL|O_NOFOLLOW, so "absent" and "this call created it" are one atomic fact and a planted link at either end is refused, never followed. Where the platform has no O_NOFOLLOW (Windows), the source is opened through platform_compat.open_file_no_reparse and handed to copy_file_pinned as src_fd, so a staged reparse point is refused rather than judged by its target. Used for memory.db, memory_index.db (only beside a memory.db this call installed) and crons.json in snapshot._do_merge, and for the same three plus hooks.json in portability.apply_import_zip. A skipped install is printed (CLI) or listed in items + refused_merges (dashboard), and the CLI no longer prints ✅ memory / ✅ crons over it.
  • Cron store rewrite (issue site 5). _merge_crons now opens each store once (_read_cron_store_once): parent pinned with pinned_fs.pin_parent where the platform can, O_NOFOLLOW|O_NONBLOCK, descriptor must be regular and pass pinned_fs.refuse_hardlink_alias, text read through that descriptor. The read descriptor stays open; the rewrite (_write_merged_crons) re-checks that the live name still holds that regular inode (an open inode cannot be freed, so a planted entry cannot reuse its number) and then does an atomic_write replace through the same pinned directory, carrying the store's mode and its access-control xattrs (a named POSIX ACL) from that descriptor; an ACL that cannot be carried refuses the write. A rename replaces the directory entry and never follows it. Any refusal imports nothing and returns False, which both callers already report. Where there is no dir_fd (Windows, reached only with --allow-unpinned-staging or the dashboard import), the final-name open uses platform_compat.open_file_no_reparse and the write is the cron store's own atomic_write.
  • The owning spec (docs/system-specs/modules/cli.md) records both, next to the notification-merge paragraph.

Signatures each guard is meant to cover, with a verdict:

Name at the site Before After
live core file is a dangling symlink followed, file created outside the home refused (O_EXCL), printed
live core file is a directory / FIFO copy2 wrote into the directory, or blocked on the FIFO refused (O_EXCL), printed
staged core file is a symlink followed refused (O_NOFOLLOW)
staged core file is a hardlink alias bytes installed refused (st_nlink)
live crons.json swapped for a link after the read rewrite followed the link inode mismatch, nothing imported
live crons.json is a hardlink alias rewritten in place, alias's other name changed refused (st_nlink)
staged crons.json is a link to another cron file foreign jobs imported refused
live crons.json is a link to a real file (pre-existing) read and rewritten through refused at the open

Not a copy of the defect: _copy_tree_no_overwrite and the config branch already use copy_file_pinned, so they are untouched.

Backwards compatibility

Compatible, with two deliberate changes. Every regular, single-link core file and cron store merges as before (same bytes, same mode), except that a memory_index.db already present beside a freshly installed memory.db is now kept rather than overwritten: the CLI prints memory_index.db: existing index kept, not replaced; the gateway rebuilds it on its next start, and the dashboard import lists memory search index (kept the existing one; rebuilt when the gateway restarts) in items and memory_index in refused_merges. Names that are links, hardlink aliases or non-regular files are refused instead of followed. That includes a live crons.json an operator keeps as a deliberate symlink to a real file: cron merge is skipped for it, and the printed reason names the link (Could not read …/crons.json: [Errno 40] Too many levels of symbolic links: 'crons.json' — skipping cron merge). This is an accepted break.

On macOS the cron merge rewrites the live crons.json in place, through the descriptor it read and validated, instead of replacing it atomically. macOS native ACLs are not reachable through the xattr API that atomic_write carries, so a replace would drop a deny entry the owner set by hand; writing in place keeps the inode and its ACL exactly as the old by-name write did. Residual, macOS only: that rewrite is not atomic, so a crash or a full disk mid-write can leave a torn crons.json, as on base. Linux keeps the atomic replace with the ACL carried; Windows keeps the atomic replace. The macOS branch is tested only by simulating the platform flag on Linux; no real Mac has run it.

Reader: src/kiro_crew/snapshot.py:_do_merge -- entry: chat|cron (kirocrew restore --mode merge) -- prints the refusal and drops the component tick; test_absent_core_files_are_still_installed_with_their_mode
Reader: src/kiro_crew/portability.py:apply_import_zip -- entry: crew page (dashboard Import → Merge) -- refusal listed in items and refused_merges, the shape it already uses for a refused cron merge; test_the_dashboard_import_twins_refuse_a_dangling_link_too
Reader: src/kiro_crew/snapshot_merge.py:_merge_crons -- entry: both of the above -- still returns False on refusal, which both callers already handle; test_an_unswapped_merge_imports_and_keeps_the_store_regular_with_its_mode

Tests

New test/test_snapshot_merge_core_files_pinned.py (real links and hardlinks on a real filesystem, POSIX):

  • dangling link at live memory.db / crons.json / memory_index.db is not followed;
  • hardlink alias and symlink in staging are not installed;
  • live crons.json swapped for a link between the read and the rewrite is not written through (swap performed inside _usable_cron_shape, which runs after both reads and before the write);
  • a live store replaced by a new regular file after the read is not overwritten; a planted link that reuses the read inode's number (seen on CI, simulated here) is still refused;
  • the rewrite keeps the live store's named POSIX ACL (setfacl -m u:nobody:---);
  • hardlinked live cron store is refused; staged cron store that is a link is not imported;
  • a staged link where the open cannot refuse links (Windows shape, simulated on Linux: O_NOFOLLOW removed, supports_pinned_walk false, open_file_no_reparse stood in with Windows' ELOOP semantics) is not installed;
  • happy paths: absent core files still install with their mode and ticks; an unswapped cron merge imports, keeps the store regular, keeps its mode.

Red on base: 14 of the 16 fail on origin/main (the two happy paths pass on both). Mutations, each reddening exactly the named test: drop the inode re-check → the swap test; drop the st_nlink check → the hardlinked-store test; helper back to shutil.copy2 → all seven core-file refusal tests; drop the open_file_no_reparse/src_fd arm → the Windows-shape test; drop the ACL carry → the ACL test; drop the regular-file leg of the re-check → the inode-reuse test.

test/test_snapshot_refactor_ownership.py and test/test_snapshot_refactor_contracts.py list the three new names, as the facade contract requires.

Run locally, by path only: the three files above (50 passed), plus test/test_portability.py, flake8, black, isort, mypy on the three changed modules. The full suite was not run locally; CI covers it.

Manual verification

N/A — the behaviour is filesystem-level and the unit tests drive it with real links on a real filesystem. Windows (no dir_fd, open_file_no_reparse path) is unverified on a real Windows host; it is exercised only by the simulated test above and checked against source.

Related Issues

Refs #8725 (sites 1–3 and 5; site 6 stays open there for the owner's decision)

Pattern harvest

Rule candidate: semgrep
Pattern: shutil.copy2(...) / Path.write_text(...) on a path that was just tested with is_file() / exists() / read_text() inside snapshot*.py or portability.py — a validate-then-reopen by name.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality (new tests added; the full suite was not run locally, CI covers it)
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

Decision for the owner

Issue #8725 asks for the descriptor-once treatment at six validate-then-reopen sites in merge restore and a recorded posture for the SQLite ATTACH of the staged memory.db. This draft pins the five mechanical sites (the memory.db, memory_index.db and crons.json copies and the cron read-then-rewrite) plus their dashboard-import twins, and leaves the ATTACH posture open for you to choose between copying into a private temp file first and refusing with a documented exposure.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ecb7c9d995c07ec120f4c17570ae4392c4349344 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of ecb7c9d995c07ec120f4c17570ae4392c4349344 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The cron rewrite has two write strategies where one is enough. Writing through the validated read descriptor already closes the swap window on every pinning platform.

Not justified as shipped

  • Item 7 — oversized: the Goal is "can no longer be redirected by a link". On macOS, if in_place and dst_fd is not None: ... os.ftruncate(dst_fd, 0) already stops that redirect without the atomic replace. The replace adds pinned_parent_replace_supported, ACL-xattr carry and the stat_at re-check, which buy crash safety that base never had and the Goal doesn't ask for.
    Subtraction: write through the read descriptor on every pinning platform. Delete _cron_rewrite_in_place and the atomic_write(... preserve_access_control_from=..., parent_dir_fd=...) arm on that path.
    Clears when: the second write arm is removed, or the description names a harm in the Goal that only the atomic replace removes.
  • Item 9 — duplicate of snapshot_merge._open_notification_file: _read_cron_store_once repeats its flag set, fstat/S_ISREG and refuse_hardlink_alias judgement. That helper's own docstring warns that "two copies of one security predicate drift independently". I counted 2 implementations of the open-once judgement in snapshot_merge.py.
    Subtraction: replace the judgement block (flags, S_ISREG, hardlink check) with one shared call. Keep only the dir_fd retention that the rewrite needs.
    Clears when: one copy of the descriptor judgement remains.

What this change ships

Inventory (10 items) — 8 justified

Intent: stop a planted link or hardlink at a core-file name from redirecting a merge restore's write or read (FIX).

  1. A merge restore no longer follows a dangling link or non-regular entry at memory.db, memory_index.db or crons.json — justified
  2. A staged core file that is a symlink or hardlink alias is refused instead of installed — justified
  3. Dashboard Import → Merge does the same, plus hooks.json; refusals are listed in items/refused_merges — justified
  4. An existing memory_index.db beside a freshly installed memory.db is kept, and the restore says so — justified
  5. The CLI drops the ✅ memory / ✅ crons tick when an install is skipped — justified
  6. The cron merge reads each store once and refuses links, including a deliberately symlinked crons.json — justified
  7. On Linux and Windows the cron rewrite becomes an atomic replace that carries mode and ACL — oversized
  8. On macOS the cron rewrite writes through the validated descriptor, so a crash mid-write can still leave a torn file — justified
  9. A new open-and-judge helper for the cron store — duplicate of snapshot_merge._open_notification_file
  10. The cli.md spec records both guards — justified

[FIRST-PRINCIPLES-REVIEWED] ecb7c9d

Description read

[DESCRIPTION-READ] 5299bdaac7d1432f65a1b9c2224b00984a6a1b00af53ffd54c6a432632255a5c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ecb7c9d995c07ec120f4c17570ae4392c4349344 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Run the cron-merge tests on a real Mac (macos-on-demand.yml). Every test of the in-place O_RDWR rewrite branch (_cron_rewrite_in_place) fakes the platform flag on Linux.

[DESIGN-REVIEWED] ecb7c9d

Description read

[DESCRIPTION-READ] 2331c73f62a9425938cdc697078203c8b187a38431b5c6b0b8369459f1901fef

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@buluoray
buluoray force-pushed the fix/snapshot-merge-pinned-copies-8725 branch from a068f94 to e75d3ac Compare October 8, 2026 08:46
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ecb7c9d995c07ec120f4c17570ae4392c4349344 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ecb7c9d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ecb7c9d995c07ec120f4c17570ae4392c4349344: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ecb7c9d995c07ec120f4c17570ae4392c4349344 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ecb7c9d

Verdict parsed from the review's SHA-scoped output markers for commit ecb7c9d995c07ec120f4c17570ae4392c4349344.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ecb7c9d995c07ec120f4c17570ae4392c4349344: <one-sentence reason>

@buluoray
buluoray force-pushed the fix/snapshot-merge-pinned-copies-8725 branch from e75d3ac to 9f0201f Compare October 8, 2026 09:51
@buluoray

buluoray commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Dispositions for the review verdicts on e75d3ac525, addressed in 9f0201f227:

GPT 6.1 — BLOCKING (security-class), snapshot_merge.py:150, staged link followed on Windows: fixed. Confirmed against source: copy_file_pinned opens the source with getattr(os, "O_NOFOLLOW", 0), which is 0 on Windows, so a by-name open follows a reparse point and the target passes the regular/one-link checks. _install_core_file_if_absent now opens the source through platform_compat.open_file_no_reparse(..., nonblocking=True) where supports_pinned_walk() is false, hands the descriptor over as src_fd, and reports an ELOOP refusal as a symlink skip (the install returns False, which every caller already reports). New test test_a_staged_link_is_refused_where_open_cannot_refuse_links simulates that platform on Linux (no O_NOFOLLOW, no pinned walk, open_file_no_reparse stood in with Windows' ELOOP behaviour); it fails when the new arm is removed. Not run on a real Windows host.

First Principles — CONCERNS, inline st_nlink predicate in _read_cron_store_once: fixed. The inline link-count branch is replaced by pinned_fs.refuse_hardlink_alias(fd, what=what, name=str(path), refusal=OSError), called outside the close-on-error arm per its contract. What remains specific to this helper is the pinned dir_fd handed back to the rewrite and the Windows open_file_no_reparse arm. test_a_hardlinked_live_store_is_refused fails when the call is removed.

Opus 5.5 — no findings. Design — PASS. UX — skipped (no UI surface).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
Merge restore installed memory.db, memory_index.db and crons.json (and the
dashboard import also hooks.json) with an is_file() name check followed by
shutil.copy2, and _merge_crons read the live crons.json by name and rewrote it
by name. Each is two resolutions of one string: a dangling link at the live
name passed the check and was followed out of the data home, a hardlink alias
in the staging tree was installed as the core file, and a link swapped in
between the cron read and rewrite redirected the rewrite.

Core files now install through copy_file_pinned(skip_existing=True), the
config component's existing path. The cron merge opens each store once, judges
the descriptor (regular, one link), and replaces the live store atomically in
the same pinned directory after re-checking the inode it read.

The _merge_memory ATTACH of the staged database is not covered here.

Refs #8725
@buluoray
buluoray force-pushed the fix/snapshot-merge-pinned-copies-8725 branch from 9f0201f to bf437b8 Compare October 8, 2026 10:26
@buluoray

buluoray commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Dispositions for GPT 6.1 on 9f0201f227, addressed in bf437b8d40:

F2 — BLOCKING (security-class), replacing the store drops its access-control ACL: fixed. Confirmed: atomic_write_at carries permission bits only. The live store's read descriptor is now kept open, and the rewrite goes through atomic_write(..., mode=..., preserve_access_control_from=<that fd>, parent_dir_fd=<pinned dir>), the same carry pinned_io, steering and mcp_cleanup use; an ACL that cannot be carried refuses the write. New test test_the_rewrite_keeps_the_live_stores_access_control_acl sets u:nobody:--- with setfacl and fails when the carry is removed.

CI red on 9f0201f227 (Backend Tests (3.12, 4), test_a_live_store_swapped_for_a_link_between_read_and_rewrite_is_not_written_through): fixed. The runner's filesystem handed the freed inode number to the planted symlink, so the identity re-check alone matched. The re-check now also requires a regular file, and the held read descriptor keeps the inode alive so its number cannot be reused. New test test_a_link_that_reuses_the_read_inode_number_is_still_refused simulates the reuse and fails when the regular-file leg is removed.

F1 — BLOCKING (security-class), the staged temporary can be substituted before the rename: not fixed in this PR, left for the owner. It holds as described: atomic_write stages .crons.json.*.tmp by name in the data home and renames it by name. The adjudicator's reading also holds: a same-UID writer in that directory can already replace crons.json itself, and the cron store's own _save (cron.py → atomic_write) publishes the same way on every write. No portable descriptor-based publish exists (pinned_fs.copy_file_pinned documents why: linkat(AT_EMPTY_PATH) is not exposed). The alternatives are an in-place rewrite through the held descriptor, which gives up crash-atomicity, or a later change to the store's shared publish path. This is recorded under "Not a goal". It is security-class, so it is not overridden here.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@buluoray

buluoray commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Dispositions for First Principles (CONCERNS, advisory) on bf437b8d40. Neither one changes code in this PR.

Item 7, facade re-exports of _read_cron_store_once / _write_merged_crons with zero consumers: kept on purpose. The snapshot family's own contract requires them. test/test_snapshot_refactor_ownership.py::test_each_owner_defines_exactly_its_declared_names requires every top-level name snapshot_merge defines to be declared in OWNERS, and test_the_facade_hands_out_the_owners_objects then requires snapshot.<name> to be that same object for every declared name. With the two re-exports removed, the second test fails (AttributeError: module 'kiro_crew.snapshot' has no attribute '_read_cron_store_once', observed while building this PR). The only way to drop them is to change that contract, which is outside this fix.

Item 5, portability._sanitize_imported_crons reads and rewrites snap / "crons.json" by name: a real sibling, deferred. It is pre-existing code this PR does not touch. Its tree is not the snapshot staging tree described in the PR: apply_import_zip extracts into a fresh tempfile.TemporaryDirectory() that platform_compat.restrict_dir_to_owner locks to the owner, under a random name created by this import. A same-UID writer could still reach it, so this is the same validate-then-reopen class one stage earlier. It belongs with the remaining by-name sites in the dashboard import (the skills/ tree copy is already listed under "Not a goal"), not in this slice. I am leaving the frozen goal sections as they are; the owner decides whether it joins #8725's remainder.

GPT 6.1 on this head: no findings. The temporary-file substitution it raised on 9f0201f227 no longer appears; that residual is described in the earlier disposition comment.

@buluoray
buluoray marked this pull request as ready for review October 8, 2026 15:40
@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners October 8, 2026 15:40
@buluoray
buluoray requested a review from CrysisDeu October 8, 2026 15:40
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

This closes link and hardlink redirection in merge restore. Core files are installed through copy_file_pinned (source opened O_NOFOLLOW and must have one link; destination created O_CREAT|O_EXCL|O_NOFOLLOW). _merge_crons reads each store once by descriptor and replaces it atomically after an inode re-check. The approach fixes the validate-then-reopen root cause at these sites. HOLD because it is security hardening (credential-read and arbitrary-write paths) and needs a maintainer's call. I tried to get past the new checks. A dangling link, directory or FIFO at a live memory.db/crons.json makes is_file() false, but O_EXCL then refuses the install. A staged hardlink is refused by st_nlink. A swap after the cron read fails the regular-file plus held-inode check. I found no bypass at these sites. Open items a maintainer should rule on: (1) the documented residual where a same-UID writer replaces the staged temporary (GPT F1, deliberately not overridden); (2) the unfixed sibling portability._sanitize_imported_crons, which still reads and rewrites the staged crons.json by name; (3) _merge_memory's ATTACH by path, still open on #8725; (4) the Design Watch, still unanswered: operators who keep crons.json as a deliberate symlink now lose cron merge and only see a generic "Could not read" message. That message should name the link as the cause, since this safer default takes something away from them. The Windows open_file_no_reparse arm is only simulated on Linux. — crew-pr-reviewer

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes the root cause at five validate-then-reopen sites in merge restore (CLI and dashboard Import -> Merge): core files are now installed with copy_file_pinned (source opened O_NOFOLLOW and must be regular with one link; destination created O_CREAT|O_EXCL|O_NOFOLLOW), and the crons.json merge reads each store once by descriptor, re-checks the inode, then replaces it atomically while keeping mode and ACL. I read copy_file_pinned on the head and ran probes on head bf437b8. A dangling link at live memory.db is not followed. A live crons.json that links to a real file is refused, the target is left alone, and ELOOP is printed. A hardlink or symlink in staging is refused (author tests, red on base). CI is green on the current head (the cancelled rows are duplicate runs) and GPT/Opus report no findings. New regression I proved with a probe: if memory.db is missing but an old memory_index.db is still there, the old index is now kept without a word (EEXIST returns False with no on_skip report) and the CLI prints "Memory: copied". Base overwrote that index with the bundle's copy. So the FTS index no longer matches the imported memory.db and search misses imported memories. Smaller gaps: the "(see above)" refusal line has nothing above it when the live name is occupied. The Design Watch about a symlinked crons.json was never answered. The First Principles sibling (portability._sanitize_imported_crons rewrites snap/crons.json by name in the import temp dir) was deferred with no deferred-finding issue filed. Requested change: src/kiro_crew/snapshot.py:1377 and src/kiro_crew/portability.py:2021: check what the memory_index.db _install_core_file_if_absent call returns. When it returns False, print or list "memory_index.db: existing index kept, not replaced; rebuild the index" (dashboard: add it to items and refused_merges), and add a test for it. Keep refusing the overwrite. Also open a deferred-finding issue (owner, Due: line) for portability.py:1134-1214 _sanitize_imported_crons, and post a target=design disposition for the symlinked-crons Watch. On the owner decision in the PR body: For the SQLite ATTACH of the staged memory.db (#8725 site 6), copy it into a private O_EXCL temp file first and ATTACH that copy, so merge keeps working and the hole closes. Fold the _sanitize_imported_crons sibling into #8725's remainder. Risk: medium. — crew-pr-reviewer

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 8, 2026
@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • fixed in 1bf3142 span=b5683e4ff0af the kept index was counted as imported

Confirmed against NOT_APPLIED_ITEM in website/src/pages/overview/PortabilityTab.tsx. The dashboard item is now memory search index (kept the existing one; rebuilt when the gateway restarts), whose (kept opener is one the tab already treats as not applied.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • fixed in 1bf3142 span=0d967a0e4187 "(see above)" on the dashboard

The dashboard form (dashboard=True) never carries "(see above)"; the refused-copy case reads memory search index (skipped: the archive's copy was refused; ...). The test asserts no item contains "see above". The CLI keeps it only where the skip hook has already printed the refusal.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • deferred span=64344537195a plain labels for refused store names

The raw refused_merges labels (memory, hooks, crons, config) predate this PR and are rendered the same way for every entry; mapping them is a change to the Portability tab's shared rendering, outside #8725's scope. This PR adds one more label in the existing convention.

@buluoray
buluoray force-pushed the fix/snapshot-merge-pinned-copies-8725 branch from 1bf3142 to 07ee4bc Compare October 8, 2026 20:07
@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: maintainer-requested span=bc835b842bb0 a kept index is listed in refused_merges

iamwhatever's CHANGES_REQUESTED review asked for exactly this: "dashboard: add it to items and refused_merges". It is not harmless: until the gateway rebuilds the index on restart, search misses the imported memories, which is the regression that review proved with a probe. So the refusal stays visible. Whether it should be shown as an error or a warning is the maintainer's call on re-review.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: not this PR's evidence span=14898b214ed4 screenshots show a different PR

This PR attaches no screenshots or recordings. Its description says Manual verification N/A, and its diff touches no file under temp-screenshots/ (git diff --stat origin/main...HEAD lists 7 Python/spec/test files). The sticky-collapse-blank-lane images come from elsewhere in the repo. The change is to summary strings only. The rendered Portability result line has not been seen in a browser; how it is counted rests on the NOT_APPLIED_ITEM regex and a test that asserts the (kept opener.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • declined span=35add0161d02 "bundle" to "backup" in the CLI line

"bundle" is the restore CLI's existing term for the archive. snapshot.py on origin/main uses it 119 times, including in user-facing prints such as "This bundle was REDACTED before it left its host." Renaming one line would make the CLI inconsistent with itself. The dashboard form says "archive", matching that surface.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: the judgement differs span=4b3de33672f4 _read_cron_store_once vs _open_notification_file

The second "Clears when" branch applies. _open_notification_file opens read-write for an append and, where the platform cannot pin (Windows), raises NotificationCopyUnsupported so the caller skips the copy. The cron store must not take that branch: it is a read-only open that goes through platform_compat.open_file_no_reparse, so cron merge still runs on Windows. It also accepts the caller's dir_fd and returns the open descriptor (keep_open), because the atomic rewrite replaces the name in that same pinned directory and carries the store's ACL from that descriptor. Folding both helpers into one would either drop Windows cron merge or give the notification copy a Windows path its own docstring says it cannot settle safely. Both already share pinned_fs.refuse_hardlink_alias for the one-link judgement.

@buluoray
buluoray force-pushed the fix/snapshot-merge-pinned-copies-8725 branch from 07ee4bc to 2577b73 Compare October 8, 2026 20:50
@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • fixed in 2577b73 span=cf735dca7274 ACL test raises AttributeError on macOS instead of skipping

Confirmed: macOS os has no setxattr, and except OSError does not catch AttributeError. The test now skips when os has no setxattr. Mutation: with os.setxattr deleted, the old test fails with AttributeError at line 292 and the new one skips. The production carry path is already guarded: atomic_write checks hasattr for listxattr/getxattr/setxattr, atomic_write.py:105.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: the harm is named span=5a7133a812cd atomic replace instead of writing through the held descriptor

Writing the merged jobs through the held descriptor means truncating and rewriting the live crons.json in place. A crash or a full disk mid-write would leave a torn or empty store, which the scheduler then reads as no jobs. On base, dst_path.write_text has exactly that hazard. An atomic replace removes it: the live store is either the old document or the new one, never a partial one. That is also how the cron store's own save publishes. Replacing the inode is what makes the mode and ACL carry necessary, and both are carried from the held descriptor.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: the divergence is required span=4b3de33672f4 _read_cron_store_once vs _open_notification_file

On a platform that cannot pin (Windows), _open_notification_file raises NotificationCopyUnsupported and the copy is skipped, because a read-write append there has no way to settle the name in the open. The cron store needs a read-only open, which platform_compat.open_file_no_reparse does settle, so cron merge keeps working on Windows. The rewrite also needs the pinned parent's dir_fd and the open descriptor (keep_open) for the same-directory replace and the ACL carry. Both helpers already share pinned_fs.refuse_hardlink_alias for the one-link judgement.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: maintainer-requested span=25070017ef66 the index-kept report branch

The startup rebuild does regenerate the index: MemoryStore.rebuild_index runs DELETE FROM memory_fts and re-inserts every row, memory.py around line 1300. It skips that only when an indexable root is refused, and then it keeps the existing index. Between the merge and that restart, though, search runs against the old index. iamwhatever's CHANGES_REQUESTED review asked for exactly this report ("When it returns False, print or list … and add a test for it. Keep refusing the overwrite."). The line tells the operator that the gap closes on restart. Deleting the branch is the maintainer's call on re-review.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: maintainer decision span=a934a2bdb513 keep the leftover index vs pinned atomic replace

The second "Clears when" branch: the maintainer's review on bf437b8d40 ruled "Keep refusing the overwrite" and asked only for the kept index to be reported, which this head does. The index is derived data: the gateway rebuilds it from scratch on its next start (MemoryStore.rebuild_index). Replacing it with a pinned atomic replace would add a second publish path for a file the restart regenerates anyway. The Backwards compatibility section of the description records the keep.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • acknowledged: unverified, stated span=6b7429e9444e the Windows branches are only simulated

This is correct, and the description already says it. No real Windows host has run the dashboard Import → Merge on this branch. The open_file_no_reparse arms are exercised only by test_a_staged_link_is_refused_where_open_cannot_refuse_links, which simulates them on Linux, and were checked against source. This host cannot produce the Windows run that would clear the item, so it is left open for the maintainer's re-review rather than overridden.

@buluoray

buluoray commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@iamwhatever: GPT 6.1 raised a fenced finding on 2577b7319c, "Cron merge drops native macOS access-control ACLs" (snapshot_merge.py:503). I am not overriding it, and I have held the head with no new push. The fix trades one property against another, so it needs your call.

What I checked in source. The finding holds. On base, the merge rewrote crons.json in place (dst_path.write_text), so the inode and any native macOS ACL survived. On this head the merge publishes the result with an atomic replace. That replace carries the mode, plus the ACL where Linux exposes it as xattrs. macOS has no os.listxattr/getxattr/setxattr (ACCESS_CONTROL_XATTRS_SUPPORTED is False, atomic_write.py:105), so a hand-set macOS deny ACE is dropped. Context: the cron store's own normal save, cron.py:4977 atomic_write(self._path, document), carries no ACL on any platform. So such an ACE is already lost on the first ordinary cron save after any edit; the merge is not the only path that drops it.

Options

  1. On a platform that cannot carry the ACL (macOS), write through the descriptor that was read. The merge truncates and rewrites the already-open, already-verified inode instead of replacing it, so the inode and its ACL survive, as on base. The cost is that a crash mid-write on macOS can leave a torn crons.json, which is what base risked everywhere. This is about 20 lines plus a test that simulates the macOS flag on Linux. I could not run it on a real Mac.
  2. Refuse the cron merge on macOS when an extended ACL is present. The ACL would be detected through acl_get_fd called from libc with ctypes. Nothing is published that loses it. The cost is new native code I cannot run on this Linux host, and cron merge stops working for anyone with such an ACL.
  3. Accept it as residual, since the cron store's own save already drops it, and record it under "Not covered" with a follow-up issue to give the cron store's save path an ACL carry. This needs a maintainer /ai-review override because the lane is fenced; I will not post that override myself.

I recommend 1: it keeps today's macOS behaviour exactly, and it is the only option I can verify, by simulation.

… skipped, keep the macOS cron ACL

A merge that installs memory.db beside an existing memory_index.db keeps that
index (it is never overwritten) but said nothing, so search missed the imported
rows until the next rebuild. The CLI now prints 'memory_index.db: existing index
kept, not replaced; the gateway rebuilds it on its next start'; the dashboard
import lists 'memory search index (kept the existing one; rebuilt when the
gateway restarts)' in items and adds memory_index to refused_merges.

The skip line for an occupied live name pointed at a refusal 'above' that was
never printed; _core_file_not_installed now names the actual cause.

On macOS the atomic cron rewrite could not carry a native ACL onto the fresh
inode. Where the ACL carry is unavailable but pinning works, the merge now
rewrites the validated inode through its read descriptor, as the by-name write
did; a crash mid-write there can tear crons.json.

Refs #8725
@buluoray
buluoray force-pushed the fix/snapshot-merge-pinned-copies-8725 branch from 2577b73 to ecb7c9d Compare October 8, 2026 22:16
@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • fixed in ecb7c9d (owner ruling: option 1) span=b2babda4fafe Cron merge drops native macOS access-control ACLs

Not overridden: macOS now gets base's behaviour back. When the store can be pinned but atomic_write.ACCESS_CONTROL_XATTRS_SUPPORTED is False (macOS), _cron_rewrite_in_place makes _merge_crons open the live store read-write and truncate and rewrite that same validated inode, after the same identity re-check, so a hand-set native ACL is kept. Linux keeps the atomic replace with the ACL carried, and Windows keeps the atomic replace.
Tests (the platform is simulated on Linux, no real Mac): test_where_the_acl_cannot_be_carried_the_store_is_rewritten_in_place asserts the inode is unchanged. Forcing _cron_rewrite_in_place to return False makes it fail with "the store was replaced, so a native ACL on it is dropped". test_the_in_place_rewrite_still_refuses_a_store_swapped_after_the_read covers the swap window on that path, and test_where_the_acl_can_be_carried_the_store_is_still_replaced_atomically pins the Linux path.
Residual, stated in the description and in docs/system-specs/modules/cli.md: on macOS the rewrite is not atomic, so a crash mid-write can tear crons.json, as on base.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: the harm is named span=ff0a09fc3e6f keep the atomic replace beside the macOS in-place arm

Removing the atomic arm would also stop the redirect, but it would make every platform pay macOS's residual: a crash or a full disk mid-write leaves a torn crons.json, and the scheduler then reads it as no jobs. The description and docs/system-specs/modules/cli.md now name that harm as the macOS-only residual. Only the atomic replace removes it, and where the ACL can be carried (Linux) or there is none to carry (Windows) it costs nothing. The owner ruled for this split (option 1), keeping in-place only where an atomic replace would drop a native ACL.

@buluoray

buluoray commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: the divergence is required span=44b3530c54ce _read_cron_store_once vs _open_notification_file

This is the same item as span=4b3de33672f4 on 2577b7319c, re-raised. On Windows _open_notification_file raises NotificationCopyUnsupported, because a read-write append cannot settle the name there. The cron read must not take that branch: it settles the name with open_file_no_reparse, so cron merge keeps working on Windows. It also takes the caller's pinned dir_fd, can open read-write for the macOS in-place rewrite (writable), and returns the held descriptor (keep_open). The one-link judgement is a single shared copy, pinned_fs.refuse_hardlink_alias.

@buluoray
buluoray enabled auto-merge (squash) October 9, 2026 05:27
@buluoray

Copy link
Copy Markdown
Contributor Author

@iamwhatever: status of each request in your CHANGES_REQUESTED review on bf437b8d40, checked again on the current head ecb7c9d995. The earlier reply named a15398b3b0; that work now lives in ecb7c9d995. No new push.

test/test_snapshot_merge_core_files_pinned.py: 22 passed on this head. All check-runs pass (87 success, 14 skipped, 7 cancelled duplicates, 0 failing).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR closes the validate-then-reopen hole in merge restore (CLI and dashboard Import > Merge). memory.db, memory_index.db, crons.json and hooks.json are installed through copy_file_pinned (O_NOFOLLOW source, one link, O_CREAT|O_EXCL|O_NOFOLLOW destination), and the cron merge reads each store once by descriptor and re-checks the inode before writing. I went through our CHANGES_REQUESTED on bf437b8 point by point against head ecb7c9d, and all of it is met. A kept memory_index.db is now reported in the CLI (snapshot.py _core_file_not_installed(..., index=True)) and in the dashboard items plus refused_merges "memory_index", with tests that fail when the report is dropped. The "(see above)" line now names an occupied live name directly. The symlinked-crons Design Watch has a target=design disposition and is recorded as an accepted break; the printed reason says ELOOP. The _sanitize_imported_crons sibling went to deferred-finding #18213, since closed by #18240. I tried to break the new macOS in-place arm (_cron_rewrite_in_place). It opens the live store O_RDWR|O_NOFOLLOW relative to the pinned parent, refuses a non-regular or hardlinked descriptor, and re-checks dev/ino before truncating, so a swapped name is refused and a planted link is never written through. The torn-write residual matches base and is documented. The staged memory.db ATTACH-by-path remains #8725's follow-up, as stated. Checks are green and GPT/Opus have no findings. — crew-pr-reviewer (deep re-check)

@buluoray
buluoray merged commit 4c09bdd into main Oct 12, 2026
148 of 161 checks passed
@buluoray
buluoray deleted the fix/snapshot-merge-pinned-copies-8725 branch October 12, 2026 06:48
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants