Repository navigation
fix(restore): resolve merge core-file and cron-store names once - #18090
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The cron rewrite has two write strategies where one is enough. Writing through the validated read descriptor already closes the swap window on every pinning platform. Not justified as shipped
What this change shipsInventory (10 items) — 8 justifiedIntent: stop a planted link or hardlink at a core-file name from redirecting a merge restore's write or read (FIX).
[FIRST-PRINCIPLES-REVIEWED] ecb7c9d Description read[DESCRIPTION-READ] 5299bdaac7d1432f65a1b9c2224b00984a6a1b00af53ffd54c6a432632255a5c That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Run the cron-merge tests on a real Mac ( [DESIGN-REVIEWED] ecb7c9d Description read[DESCRIPTION-READ] 2331c73f62a9425938cdc697078203c8b187a38431b5c6b0b8369459f1901fef That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
a068f94 to
e75d3ac
Compare
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
e75d3ac to
9f0201f
Compare
|
Dispositions for the review verdicts on GPT 6.1 — BLOCKING (security-class), First Principles — CONCERNS, inline Opus 5.5 — no findings. Design — PASS. UX — skipped (no UI surface). |
Merge restore installed memory.db, memory_index.db and crons.json (and the dashboard import also hooks.json) with an is_file() name check followed by shutil.copy2, and _merge_crons read the live crons.json by name and rewrote it by name. Each is two resolutions of one string: a dangling link at the live name passed the check and was followed out of the data home, a hardlink alias in the staging tree was installed as the core file, and a link swapped in between the cron read and rewrite redirected the rewrite. Core files now install through copy_file_pinned(skip_existing=True), the config component's existing path. The cron merge opens each store once, judges the descriptor (regular, one link), and replaces the live store atomically in the same pinned directory after re-checking the inode it read. The _merge_memory ATTACH of the staged database is not covered here. Refs #8725
9f0201f to
bf437b8
Compare
|
Dispositions for GPT 6.1 on F2 — BLOCKING (security-class), replacing the store drops its access-control ACL: fixed. Confirmed: CI red on F1 — BLOCKING (security-class), the staged temporary can be substituted before the rename: not fixed in this PR, left for the owner. It holds as described: |
|
Dispositions for First Principles (CONCERNS, advisory) on Item 7, facade re-exports of Item 5, GPT 6.1 on this head: no findings. The temporary-file substitution it raised on |
|
This closes link and hardlink redirection in merge restore. Core files are installed through |
iamwhatever
left a comment
There was a problem hiding this comment.
Fixes the root cause at five validate-then-reopen sites in merge restore (CLI and dashboard Import -> Merge): core files are now installed with copy_file_pinned (source opened O_NOFOLLOW and must be regular with one link; destination created O_CREAT|O_EXCL|O_NOFOLLOW), and the crons.json merge reads each store once by descriptor, re-checks the inode, then replaces it atomically while keeping mode and ACL. I read copy_file_pinned on the head and ran probes on head bf437b8. A dangling link at live memory.db is not followed. A live crons.json that links to a real file is refused, the target is left alone, and ELOOP is printed. A hardlink or symlink in staging is refused (author tests, red on base). CI is green on the current head (the cancelled rows are duplicate runs) and GPT/Opus report no findings. New regression I proved with a probe: if memory.db is missing but an old memory_index.db is still there, the old index is now kept without a word (EEXIST returns False with no on_skip report) and the CLI prints "Memory: copied". Base overwrote that index with the bundle's copy. So the FTS index no longer matches the imported memory.db and search misses imported memories. Smaller gaps: the "(see above)" refusal line has nothing above it when the live name is occupied. The Design Watch about a symlinked crons.json was never answered. The First Principles sibling (portability._sanitize_imported_crons rewrites snap/crons.json by name in the import temp dir) was deferred with no deferred-finding issue filed. Requested change: src/kiro_crew/snapshot.py:1377 and src/kiro_crew/portability.py:2021: check what the memory_index.db _install_core_file_if_absent call returns. When it returns False, print or list "memory_index.db: existing index kept, not replaced; rebuild the index" (dashboard: add it to items and refused_merges), and add a test for it. Keep refusing the overwrite. Also open a deferred-finding issue (owner, Due: line) for portability.py:1134-1214 _sanitize_imported_crons, and post a target=design disposition for the symlinked-crons Watch. On the owner decision in the PR body: For the SQLite ATTACH of the staged memory.db (#8725 site 6), copy it into a private O_EXCL temp file first and ATTACH that copy, so merge keeps working and the hole closes. Fold the _sanitize_imported_crons sibling into #8725's remainder. Risk: medium. — crew-pr-reviewer
|
|
|
1bf3142 to
07ee4bc
Compare
|
|
|
|
07ee4bc to
2577b73
Compare
|
|
|
|
|
|
|
@iamwhatever: GPT 6.1 raised a fenced finding on What I checked in source. The finding holds. On base, the merge rewrote Options
I recommend 1: it keeps today's macOS behaviour exactly, and it is the only option I can verify, by simulation. |
… skipped, keep the macOS cron ACL A merge that installs memory.db beside an existing memory_index.db keeps that index (it is never overwritten) but said nothing, so search missed the imported rows until the next rebuild. The CLI now prints 'memory_index.db: existing index kept, not replaced; the gateway rebuilds it on its next start'; the dashboard import lists 'memory search index (kept the existing one; rebuilt when the gateway restarts)' in items and adds memory_index to refused_merges. The skip line for an occupied live name pointed at a refusal 'above' that was never printed; _core_file_not_installed now names the actual cause. On macOS the atomic cron rewrite could not carry a native ACL onto the fresh inode. Where the ACL carry is unavailable but pinning works, the merge now rewrites the validated inode through its read descriptor, as the by-name write did; a crash mid-write there can tear crons.json. Refs #8725
2577b73 to
ecb7c9d
Compare
|
|
|
|
@iamwhatever: status of each request in your CHANGES_REQUESTED review on
|
iamwhatever
left a comment
There was a problem hiding this comment.
This PR closes the validate-then-reopen hole in merge restore (CLI and dashboard Import > Merge). memory.db, memory_index.db, crons.json and hooks.json are installed through copy_file_pinned (O_NOFOLLOW source, one link, O_CREAT|O_EXCL|O_NOFOLLOW destination), and the cron merge reads each store once by descriptor and re-checks the inode before writing. I went through our CHANGES_REQUESTED on bf437b8 point by point against head ecb7c9d, and all of it is met. A kept memory_index.db is now reported in the CLI (snapshot.py _core_file_not_installed(..., index=True)) and in the dashboard items plus refused_merges "memory_index", with tests that fail when the report is dropped. The "(see above)" line now names an occupied live name directly. The symlinked-crons Design Watch has a target=design disposition and is recorded as an accepted break; the printed reason says ELOOP. The _sanitize_imported_crons sibling went to deferred-finding #18213, since closed by #18240. I tried to break the new macOS in-place arm (_cron_rewrite_in_place). It opens the live store O_RDWR|O_NOFOLLOW relative to the pinned parent, refuses a non-regular or hardlinked descriptor, and re-checks dev/ino before truncating, so a swapped name is refused and a planted link is never written through. The torn-write residual matches base and is documented. The staged memory.db ATTACH-by-path remains #8725's follow-up, as stated. Checks are green and GPT/Opus have no findings. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A merge restore (CLI
kirocrew restore --mode mergeand the dashboard Import → Merge) can no longer be redirected by a link or hardlink planted at a core-file name into writing outside the data home or importing a file it was not pointed at.Merge restore installed
memory.db,memory_index.dbandcrons.json(the dashboard import alsohooks.json) with anis_file()name check followed byshutil.copy2, and_merge_cronsread the livecrons.jsonby name and rewrote it by name. Each pair is two resolutions of one string:is_file() == False, andcopy2then follows it and creates the bundle's file wherever it points;memory.db/crons.json;crons.jsonbetween the merge's read and itswrite_textmade the rewrite truncate the link target.Why it matters
The staging tree and the data home are writable by the agent. These are the same validate-then-reopen class the notification merge already closed, left open at the sites next to it: a planted name turns a restore into an arbitrary-file write (the cron rewrite) or a credential read into a live core file.
Not a goal
_merge_memoryintegrity-check-then-ATTACH DATABASEof the staged database (issue site 6). SQLite accepts only a path, so the posture (copy into a private temp file first, or refuse and document) is an owner decision; it stays open on snapshot restore: six more validate-then-reopen sites in the same path as #8667 #8725.skills/tree copy (shutil.copytree/copy2by name inportability.apply_import_zip). It is a tree, not a core file, and needs the tree primitive rather than this helper.copy2arm, which open PR fix(snapshot): validate notification records the copy branch installs #8576 owns.cron.py_save→atomic_write) does on every write; no portable descriptor-based publish exists.What changed (motivation → approach → change)
Symptom → cause → change:
snapshot_merge._install_core_file_if_absentis one call topinned_fs.copy_file_pinned(..., skip_existing=True)— the config component's existing install path. The source is opened once withO_NOFOLLOWand judged on its descriptor (regular,st_nlink == 1); the destination is createdO_CREAT|O_EXCL|O_NOFOLLOW, so "absent" and "this call created it" are one atomic fact and a planted link at either end is refused, never followed. Where the platform has noO_NOFOLLOW(Windows), the source is opened throughplatform_compat.open_file_no_reparseand handed tocopy_file_pinnedassrc_fd, so a staged reparse point is refused rather than judged by its target. Used formemory.db,memory_index.db(only beside amemory.dbthis call installed) andcrons.jsoninsnapshot._do_merge, and for the same three plushooks.jsoninportability.apply_import_zip. A skipped install is printed (CLI) or listed initems+refused_merges(dashboard), and the CLI no longer prints✅ memory/✅ cronsover it._merge_cronsnow opens each store once (_read_cron_store_once): parent pinned withpinned_fs.pin_parentwhere the platform can,O_NOFOLLOW|O_NONBLOCK, descriptor must be regular and passpinned_fs.refuse_hardlink_alias, text read through that descriptor. The read descriptor stays open; the rewrite (_write_merged_crons) re-checks that the live name still holds that regular inode (an open inode cannot be freed, so a planted entry cannot reuse its number) and then does anatomic_writereplace through the same pinned directory, carrying the store's mode and its access-control xattrs (a named POSIX ACL) from that descriptor; an ACL that cannot be carried refuses the write. A rename replaces the directory entry and never follows it. Any refusal imports nothing and returnsFalse, which both callers already report. Where there is nodir_fd(Windows, reached only with--allow-unpinned-stagingor the dashboard import), the final-name open usesplatform_compat.open_file_no_reparseand the write is the cron store's ownatomic_write.docs/system-specs/modules/cli.md) records both, next to the notification-merge paragraph.Signatures each guard is meant to cover, with a verdict:
O_EXCL), printedcopy2wrote into the directory, or blocked on the FIFOO_EXCL), printedO_NOFOLLOW)st_nlink)crons.jsonswapped for a link after the readcrons.jsonis a hardlink aliasst_nlink)crons.jsonis a link to another cron filecrons.jsonis a link to a real file (pre-existing)Not a copy of the defect:
_copy_tree_no_overwriteand the config branch already usecopy_file_pinned, so they are untouched.Backwards compatibility
Compatible, with two deliberate changes. Every regular, single-link core file and cron store merges as before (same bytes, same mode), except that a
memory_index.dbalready present beside a freshly installedmemory.dbis now kept rather than overwritten: the CLI printsmemory_index.db: existing index kept, not replaced; the gateway rebuilds it on its next start, and the dashboard import listsmemory search index (kept the existing one; rebuilt when the gateway restarts)initemsandmemory_indexinrefused_merges. Names that are links, hardlink aliases or non-regular files are refused instead of followed. That includes a livecrons.jsonan operator keeps as a deliberate symlink to a real file: cron merge is skipped for it, and the printed reason names the link (Could not read …/crons.json: [Errno 40] Too many levels of symbolic links: 'crons.json' — skipping cron merge). This is an accepted break.On macOS the cron merge rewrites the live
crons.jsonin place, through the descriptor it read and validated, instead of replacing it atomically. macOS native ACLs are not reachable through the xattr API thatatomic_writecarries, so a replace would drop a deny entry the owner set by hand; writing in place keeps the inode and its ACL exactly as the old by-name write did. Residual, macOS only: that rewrite is not atomic, so a crash or a full disk mid-write can leave a torncrons.json, as on base. Linux keeps the atomic replace with the ACL carried; Windows keeps the atomic replace. The macOS branch is tested only by simulating the platform flag on Linux; no real Mac has run it.Reader: src/kiro_crew/snapshot.py:_do_merge -- entry: chat|cron (
kirocrew restore --mode merge) -- prints the refusal and drops the component tick; test_absent_core_files_are_still_installed_with_their_modeReader: src/kiro_crew/portability.py:apply_import_zip -- entry: crew page (dashboard Import → Merge) -- refusal listed in
itemsandrefused_merges, the shape it already uses for a refused cron merge; test_the_dashboard_import_twins_refuse_a_dangling_link_tooReader: src/kiro_crew/snapshot_merge.py:_merge_crons -- entry: both of the above -- still returns
Falseon refusal, which both callers already handle; test_an_unswapped_merge_imports_and_keeps_the_store_regular_with_its_modeTests
New
test/test_snapshot_merge_core_files_pinned.py(real links and hardlinks on a real filesystem, POSIX):memory.db/crons.json/memory_index.dbis not followed;crons.jsonswapped for a link between the read and the rewrite is not written through (swap performed inside_usable_cron_shape, which runs after both reads and before the write);setfacl -m u:nobody:---);O_NOFOLLOWremoved,supports_pinned_walkfalse,open_file_no_reparsestood in with Windows' ELOOP semantics) is not installed;Red on base: 14 of the 16 fail on
origin/main(the two happy paths pass on both). Mutations, each reddening exactly the named test: drop the inode re-check → the swap test; drop thest_nlinkcheck → the hardlinked-store test; helper back toshutil.copy2→ all seven core-file refusal tests; drop theopen_file_no_reparse/src_fdarm → the Windows-shape test; drop the ACL carry → the ACL test; drop the regular-file leg of the re-check → the inode-reuse test.test/test_snapshot_refactor_ownership.pyandtest/test_snapshot_refactor_contracts.pylist the three new names, as the facade contract requires.Run locally, by path only: the three files above (50 passed), plus
test/test_portability.py,flake8,black,isort,mypyon the three changed modules. The full suite was not run locally; CI covers it.Manual verification
N/A — the behaviour is filesystem-level and the unit tests drive it with real links on a real filesystem. Windows (no
dir_fd,open_file_no_reparsepath) is unverified on a real Windows host; it is exercised only by the simulated test above and checked against source.Related Issues
Refs #8725 (sites 1–3 and 5; site 6 stays open there for the owner's decision)
Pattern harvest
Rule candidate: semgrep
Pattern:
shutil.copy2(...)/Path.write_text(...)on a path that was just tested withis_file()/exists()/read_text()insidesnapshot*.pyorportability.py— a validate-then-reopen by name.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.
Decision for the owner
Issue #8725 asks for the descriptor-once treatment at six validate-then-reopen sites in merge restore and a recorded posture for the SQLite
ATTACHof the stagedmemory.db. This draft pins the five mechanical sites (thememory.db,memory_index.dbandcrons.jsoncopies and the cron read-then-rewrite) plus their dashboard-import twins, and leaves theATTACHposture open for you to choose between copying into a private temp file first and refusing with a documented exposure.