Repository navigation
fix: keep memory.db locks and read office-held files in the viewer - #33918
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. Both defects are fixed at the shared open prefix, and every reader of the removed paths is listed in the description. Notes
[DESIGN-REVIEWED] dba203d Description read[DESCRIPTION-READ] 9c544dade5af6130dc27d93286bea8789f2814b942a24fec8b9663b812546bd4 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Each item traces to one of the two defects in #33880, and every change sits at the code that caused it: the in-process open, and the CRT share mode. What this change shipsInventory (8 items) — 8 justifiedIntent: let a user view or search files in the dashboard without splitting the gateway's memory database, and on Windows open a document that Word or Excel has open. FIX.
[FIRST-PRINCIPLES-REVIEWED] dba203d Description read[DESCRIPTION-READ] 5c03659f489640456c108e66ccdb71eae9f4b1a532fbf6a107c75dde4a6d163d That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsI ran all six candidates through the falsification pass and none survived, so the PR passes. No findings. [OPUS-REVIEWED] dba203d Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
2111bcd to
fa2aaf7
Compare
|
fa2aaf7 to
14b6eaf
Compare
buluoray
left a comment
There was a problem hiding this comment.
Approve on 14b6eaf. Both blockers from the 2111bcd review are fixed. The _EmptyVectorStore stub now has init_for_read, so the 4 export tests pass. init_for_read now runs the owner-only lockdown (make_owner_only_dir and _restrict_memory_files, both by path, no SQLite descriptor) before the mode=ro connect, which answers the GPT ACL finding without reopening the lock-drop issue. A new 8-case test proves it. Every guard is mutation-proven. Decisive lines: test/test_memory_markdown_read.py:68, src/kiro_crew/vector_memory_lifecycle.py:504-507, src/kiro_crew/vector_memory_lifecycle.py:508, src/kiro_crew/dashboard/file_api/pinned_io.py:295
- Dashboard file reads and searches no longer drop the gateway's memory.db SQLite locks, a close that could split the memory database. - The Windows file viewer reads a document Word or Excel holds open (delete-sharing open) instead of answering HTTP 500; busy is 503.
14b6eaf to
dba203d
Compare
|
self-added: yes
|
|
self-added: yes
|
buluoray
left a comment
There was a problem hiding this comment.
Approve on dba203d. Compared with 14b6eaf, this head removes two things. The first is the CLI read-only open (init_for_read, _opening_for_read, both mode=ro connects and their tests). The second is the open_log_file_for_tail alias, whose one caller and its tests now call open_file_for_shared_read directly. Both answer First Principles findings. The CLI verbs go back to main's init(), which already sets keep_wal_on_close (vector_memory_lifecycle.py:497), so dropping the CLI half adds no data-loss path. The dashboard memory.db refusal, which is the actual lock-drop fix, is unchanged and mutation-proven again. Decisive lines: src/kiro_crew/dashboard/file_api/pinned_io.py:295, src/kiro_crew/dashboard/file_api/transfer.py:84, src/kiro_crew/dashboard/file_api/grep.py:485, src/kiro_crew/vector_memory_lifecycle.py:497, src/kiro_crew/cli_server.py:781
…33918) - Dashboard file reads and searches no longer drop the gateway's memory.db SQLite locks, a close that could split the memory database. - The Windows file viewer reads a document Word or Excel holds open (delete-sharing open) instead of answering HTTP 500; busy is 503. (cherry picked from commit 81d3b14) Backport note: learn-cron-dashboard.md conflicted; kept release's text (no 'Editing text-backed viewers' paragraph and release's File read headers wording) and added only the new Windows open-file paragraph. The open_log_file_for_tail row is removed from the platform_compat facade inventory.
Problem / Motivation
Goal: Viewing or searching files in the dashboard never splits the gateway's memory database, and on Windows a document open in Word or Excel opens in the file viewer instead of failing with HTTP 500.
This PR bundles two dashboard file viewer fixes, one per defect in #33880:
memory.dblocks (Linux, macOS). The gateway keeps<data home>/memory.dbopen in WAL mode through its memory store. The dashboard file routes (/api/file-read,/api/file-raw,/api/file-download,/api/file-stream,/api/file-watchand the Python walk of/api/file-grep) opened that file or its-wal/-shmwith a plain descriptor inside the gateway and then closed it. On POSIX that close releases every record lock the process holds on the file, so the gateway kept writing with no lock another process could see. The next process to open and close the database could then delete the-walthe gateway still wrote through, and the database split.read_failedfor a.docxor.xlsxopen in either app, and/api/file-watchstopped refreshing the tab.Why it matters
-wal(Another process's close deletes memory.db-wal while the gateway still writes through it, splitting the memory database #25846, memory.db still splits on Linux x86_64 because pysqlite3 has no setconfig, and #25852's regression test passes with the fix reverted #26331); a viewer read of the live store is the remaining in-gateway path.Not a goal
rmtree_force): fix(platform): remove trees holding read-only directories on POSIX #30455 carries that fix, so it is not bundled here.memory_stores/is already on the sensitive-path floor.file_api/grep.py) still leaves that one document out of its results.What changed (motivation → approach → change)
The file viewer no longer opens the gateway's live memory database, so viewing or searching the data home cannot split it. On Windows the viewer opens a document the same way whether or not Word or Excel has it open, because it asks Windows for the sharing those apps require.
CreateFileWsharing read, write and delete; a holder that shares nothing gets 503file_busyMemory database locks.
_is_live_memory_store_fileindashboard/file_api/pinned_io.pycompares a path by file identity withmemory.dband its-wal,-shmand-journal, so a case variant on a case-insensitive volume matches too._open_checked_file, the shared open step behind file-read, file-raw, file-download, file-stream, file-sheet and the office previews, refuses a match assensitive_path(HTTP 400) before opening it./api/file-watch(file_api/transfer.py) refuses the same files before its poll loop, and the Python engine infile_api/grep.pyskips them without opening them. The memory CLI commands keep their ordinary open: it reads and writes through SQLite, andsqlite_wal.keep_wal_on_closealready stops their close from deleting the-wal(the repro below showslearn listandmemory exportbeside a live gateway commit nothing and leave no split).Office-held files.
_open_rb_nofollowkeeps its Windowslstatlink and reparse-point check, then opens withplatform_compat.open_file_for_shared_read(platform_compat_files.py, forwarded byplatform_compat.py); on POSIX that helper is a plain read-only open. It replaces the log reader'sopen_log_file_for_tail, the same open under a log-only name, andkirocrew logs(cli_server.py) calls it directly. A sharing or lock violation (winerror 32 or 33) logs one warning and answers 503file_busy, "the file is open in another program; close it there and retry" (_FILE_BUSY_ERRORinhandlers/files.py), on file-read (file_api/editor.py), file-raw, file-download, file-stream, file-sheet (file_api/sheet.py), file-office-preview (file_api/office_preview.py) and the office-slides routes (handlers/office_slides.py)./api/file-watchreads each change through the helper.The specs say the same:
security.md(the file API refuses the live store),learn-cron-dashboard.md(the Windows busy answer) andplatform-compat.md(the shared-read helper in place of the log-tail row).Backwards compatibility
Breaking: the dashboard file routes (file-read, file-raw, file-download, file-stream, file-sheet, file-office-preview, the office slides and file-watch) refuse
<data home>/memory.dband its-wal,-shmand-journalwith 400sensitive_path, and the Python file-grep walk leaves them out of its results. Writer sweep on origin/main @ da1c9b0: no code in the repository requests these paths through the file API; a person could reach them only by opening one in the file browser, which now shows the refusal every sensitive path shows. A raw copy of a live WAL database is not a consistent snapshot anyway;kirocrew memory backup,memory exportand the state export read it through SQLite. A Windows sharing violation answered 500read_failedand answers 503file_busy; both are failures to every reader below. On POSIX the new open helper is the same read-only open as before.platform_compat.open_log_file_for_tailis gone; its one caller now callsopen_file_for_shared_read, the same open.Reader: website/src/components/useMarkdownPanelDisk.ts:useMarkdownPanelDisk -- chat -- reads
/api/file-readand/api/file-watchfor a file the user opens; a 400 or a 503 lands in itsfailedstate, as every non-404 refusal does -- test_memory_db_descriptor_locks.py::test_the_file_api_refuses_the_live_memory_db_and_keeps_its_locksReader: website/src/components/FileRenderers.tsx:SheetViewer -- chat -- reads
/api/file-sheetfor a user's spreadsheet, never the data home'smemory.db, and treats the 503 as the failure the 500 was -- test_file_viewer_windows_shared_read.py::test_a_holder_that_shares_nothing_answers_busy_and_names_the_causeReader: website/src/apps/md-notebook/utils.ts:resolveNoteImageSrc -- app -- asks
/api/file-rawonly for images under the notebook's vaultReader: src/kiro_crew/portability_stream.py:_add_sqlite_member -- release -- the state export reads
memory.dbthrough the SQLite backup API, not the file APIReader: src/kiro_crew/cli_commands_memory.py:_memory_verb -- release --
memory exportreadsmemory.dbthrough SQLite, not the file APIReader: src/kiro_crew/cli_server.py:_tail_log_file -- release --
kirocrew logsopens the log withopen_file_for_shared_read, the openopen_log_file_for_tailforwarded to -- test_windows_logs_and_diagnostics.py::test_follow_allows_real_writer_rolloverTests
test/test_memory_db_descriptor_locks.pymemory.db,-waland-shmwhile the store's locks stay held; another database in the data home still reads; the Python grep walk keeps the lockstest/test_file_viewer_windows_shared_read.pyCreateFileWfor read, write and delete sharing; a.docxheld by Word and an.xlsxheld by Excel read as when closed on file-read, file-office-preview and file-download; a holder that shares nothing answers 503file_busyon every route; file-watch streams an Office-held documenttest/test_platform_compat_refactor_facade.py,test/test_windows_logs_and_diagnostics.pyopen_file_for_shared_readand no longeropen_log_file_for_tail;kirocrew logsreads through the shared-read open, including across a real writer rolloverRuns on a Linux DevSpace (Python 3.12) on 2189d9a9e297, on main da1c9b0:
learnandmemoryCLI files and theplatform_compatfiles) at-n 4 --dist loadgroup --max-worker-restart=2: 2106 passed, 19 skipped, 3 failed. The 3 aretest/test_file_grep.py::TestEngines::test_ripgrep_reports_the_same_shape,TestEngines::test_both_engines_answer_one_tree_identicallyandTestHelpers::test_the_sensitive_globs_are_derived_from_the_shared_fence; they fail the same way on a clean main da1c9b0 on that host, so they come from the host, not this diff. The skips are Windows-only tests.check_black_formatting.py,check_subprocess_encoding.py, isort, flake8,mypy --platform linux src/kiro_crew,check_comment_history.py,check_brand_name.py,check_file_line_cap.py,docs-lint.sh, the harness-parity check and the flake-pattern ratchet: each exits 0.Determinism (testing-conventions D12): the three changed test files (
test_memory_db_descriptor_locks.py,test_windows_logs_and_diagnostics.py,test_platform_compat_refactor_facade.py) passed 20/20 runs at-n0, 10/10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest_file_viewer_windows_shared_read.pyandtest_vector_memory_wal_split.py, and in a seeded shuffled order (seed 22326).test_file_viewer_windows_shared_read.pypassed 20/20 at-n0, 10/10 at-n 4and one shuffled run on 987b67f6fa48; this head carries its code unchanged.A repro script for each defect ran on that host against a detached worktree at 2189d9a9e297 and at main da1c9b0, with no process left behind:
sqlite3):GET /api/file-readon the livememory.dbanswered 400, the gateway still held its shared lock onmemory.dband on the-shmafterwards, the-walsurvived asqlite3read, and a fresh reader saw the memory the gateway saved after it. Exit 0 here; on main the lock drops and the database splits (exit 1).learn listandmemory exportrun against the live gateway first and openmemory.dbread-write, as on main; the split check after them stays clean. The script's third check, that those two verbs openO_RDONLY, is dropped because this PR no longer changes them.learn listandmemory exportthrough the ordinaryinit(), beside a store holder that keeps or has already dropped itsmemory.dblock, commit nothing, keep the same-walfile and leave the holder's locks, and the holder's next row reaches a fresh reader. A control withoutkeep_wal_on_closesplits.CreateFileWshare-mode shim on Linux): a.docxheld by Word and an.xlsxheld by Excel answered 200 on file-read, file-office-preview and file-sheet, and a.csvheld with no sharing answered 503file_busy. Exit 0 here, 1 on main. Real Windows was not run.Rebased onto main after review; patch unchanged:
git range-diffshows 2189d9a9e297 = 52d47fe4adbf = dba203d (on main e577c9a, then 04c99b7). On 52d47fe4adbf the 32 related test files ran again (2107 passed, 19 skipped, the same 3 host failures), withdocs-lint.sh;prove.pyran again on dba203d. Main moved in no file this PR changes between them.Semgrep did not run on that host; CI's SAST job scans this head.
Proof: PROVEN test/test_file_viewer_windows_shared_read.py::test_the_windows_open_shares_read_write_and_delete, test/test_file_viewer_windows_shared_read.py::test_a_document_open_in_office_reads_as_when_closed[/api/file-read-report.docx-holder0], test/test_file_viewer_windows_shared_read.py::test_a_document_open_in_office_reads_as_when_closed[/api/file-office-preview-report.docx-holder1], test/test_file_viewer_windows_shared_read.py::test_a_document_open_in_office_reads_as_when_closed[/api/file-download-budget.xlsx-holder2], test/test_file_viewer_windows_shared_read.py::test_a_holder_that_shares_nothing_answers_busy_and_names_the_cause[/api/file-read-notes.csv] (head dba203d)
Manual verification
The two repro scripts above drove a real isolated gateway through its dashboard API and the
kirocrewCLI. The Windows open was checked through a share-mode shim, not on a Windows host.Screenshots / video
N/A: the diff changes no file under
website/; the viewer shows its existing error state for the refused and busy answers.Related Issues
Fixes #33880
Sibling sweep
Swept: git grep -nE 'MEMORY_DB_FILE|"memory.db"' -- src/kiro_crew
Swept: git grep -nE "os.open(|safe_read_prefix(" -- src/kiro_crew/dashboard/file_api src/kiro_crew/dashboard/handlers/office_slides.py
Sibling: src/kiro_crew/dashboard/file_api/pinned_io.py -- changed here
Sibling: src/kiro_crew/dashboard/file_api/transfer.py -- changed here
Sibling: src/kiro_crew/dashboard/file_api/grep.py -- changed here
Sibling: src/kiro_crew/cli_commands_memory.py -- not affected -- opens memory.db through SQLite with
keep_wal_on_close, never a raw descriptor in the gateway's processSibling: src/kiro_crew/portability_stream.py:168 -- not affected -- copies memory.db through the SQLite backup API, which keeps SQLite's own per-file lock bookkeeping
Sibling: src/kiro_crew/memory_backup.py:587 -- not affected -- hashes the staged copy, or the live file only in the startup restore pass before the store opens
Sibling: src/kiro_crew/diagnose_probes.py:583 -- not affected -- reads memory.db through a SQLite connection opened mode=ro
Sibling: src/kiro_crew/dashboard/handlers/office_slides.py:309 -- not affected -- reads the gateway's own rendered-slides cache, which no Office app holds
Pattern harvest
Rule candidate: review-prompt
Pattern: code inside the gateway opens and closes a raw descriptor of a SQLite file the same process holds open, which on POSIX drops every record lock SQLite took on it
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)