Repository navigation
fix(portability): stream each SQLite database into the export - #23244
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
CI note (head 78fead8): every AI-review lane plus Sensitive Change Review failed on infrastructure reads on attempt 1 (23:54Z) and again on attempt 2 (00:02Z). This is not a code finding. Opus could not read the merge-base 95531ce...78fead8 from shallow history and failed closed. Sensitive Change Review could not read the PR from the API (CalledProcessError). GPT, Design, UX and First Principles hit the same PR-read step. Dependency License Gate failed with 'Dependency review is not supported on this repository', a repository setting. Other PRs' claude-review runs failed in the same window (23:59Z-00:03Z). Each lane has been re-run once already, so I am not re-running again; a maintainer re-run is needed once the runners or API recover. Backend and test jobs are still running. |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: each item serves the bounded-memory export goal, and the fix replaces the in-memory backup buffer that caused the problem. What this change shipsInventory (6 items) — 6 justifiedIntent: let someone export a large memory store from Settings > Export without the gateway holding the whole database in RAM. This is a FIX (closes #23234, and the PR adds a bounded-memory test that fails on the buffered form).
[FIRST-PRINCIPLES-REVIEWED] 9176fed Description read[DESCRIPTION-READ] 8d54fb90c811392cd99f041714a9aa23625ac85ee51b7d95fd358b8654d189a3 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change removes the one remaining whole-member buffer, and the archive format, the import path and the callers stay as they were. [DESIGN-REVIEWED] 9176fed Description read[DESCRIPTION-READ] 03316b3acea6776ec3d8a8512b0c098f97ed373e102a758ffe88a93f8d4b1079 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
78fead8 to
bb86bc3
Compare
|
CI note for head bb86bc3 (the prior head was 78fead8):
|
bb86bc3 to
9176fed
Compare
|
Rebased once onto main for the #23264 Windows fix. |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the export backs each SQLite store up to a temp file and streams it into its zip member in fixed-size reads, instead of holding the whole database in memory; a round-trip test covers it.
Problem / Motivation
Goal: Settings > Export holds no SQLite database whole in memory, so a large memory store exports in bounded memory.
The export already streams (#22393): the first byte goes out at once and every file member is copied in fixed-size pieces. Each SQLite database (
memory.db,memory_index.db, every named store's databases) is the one exception._backup_sqlitebacks it up into an in-memory SQLite database, writes that to a temp file, reads the temp file whole, and_build_export_zipcopies it into aBytesIOand callszf.writestr. Measured on a 7.1 MB database: a 27.7 MB Python heap peak, and the whole member handed to the response in one write.Why it matters
A memory store full of embeddings can be hundreds of MB. Exporting it costs several times that in gateway RAM at once, which can push the gateway into memory pressure while it also serves every chat. While that member is being copied the download makes no progress, which a tunnel or proxy can read as a stalled response.
Not a goal
What changed (motivation → approach → change)
Each database now goes straight into its zip member, with no stop in memory.
:memory:database → temp file →read_bytes()→BytesIO→zf.writestr_backup_sqlite(portability_export.py) takes any binary writable, backs up into a private temp file, streams it out withshutil.copyfileobj, returns the byte count, and removes the temp file whether it succeeds or fails. A new_add_sqlite_member(portability_stream.py) opens the member withforce_zip64=True(the size is unknown when the header goes out, as for file members). The member keeps the timestamp and0600modewritestrgave it. The bytes are identical: going through:memory:added nothing._SQLITE_COPY_CHUNK_BYTESjoins the TYPE_CHECKING manifest insrc/kiro_crew/portability.py, and the docstring indashboard/handlers/portability.pysays that every member streams.Backwards compatibility
Compatible: the archive holds the same members with byte-identical database contents, and import reads it unchanged. The only format difference is ZIP64 extra fields on database members, which every other streamed member already carries.
Removes nothing: no member, key, field or caller goes away.
_backup_sqlitekeeps its name and argument order and still accepts aBytesIO.Tests
test/test_portability_memory_round_trip.py::TestDatabaseMembersStream:test_a_large_database_streams_in_bounded_memory: a 7 MB incompressiblememory.dbexports throughstream_export_zipwith a traced heap peak under a third of the database (measured about 0.4 MiB on Linux and 1.4 MiB on Windows, against 27.7 MB buffered), no sink write over 1 MiB, an empty temp directory afterwards, and an archivevalidate_import_zipaccepts, whose restored database holds every row.test_the_streamed_member_matches_a_buffered_backup: the streamed member is byte-identical to a backup into aBytesIO, with mode0600.test_a_failed_copy_removes_the_temp_file: a writer that raises propagates the error and leaves no temp file.Proof: with only
_add_sqlite_member's body reverted to the bufferedBytesIO+writestrform, the bounded-memory test fails (peak 27.7 MB for a 7.1 MB database). Determinism: the class passes 5 consecutive runs, in reversed node order and under-n 4. The data is seeded (random.Random(23234)). The existing streaming tests (test_api_portability.py::test_export_head_arrives_before_the_build_finishes,test_export_streams_every_chunk_the_builder_writes) and the round-trip and import suites pass: 606 passed across the portability, snapshot-store, facade and API files.Manual verification
N/A — the bounded-memory test exercises the real backup and zip writer against a real database; the HTTP streaming path is unchanged and covered by its existing tests.
Related Issues
Closes #23234
Sibling sweep
Swept: git grep -nE 'connect(":memory:")|.backup(' -- src/
Sibling: src/kiro_crew/snapshot_archive.py -- not affected -- backs up disk to disk, no in-memory copy
Sibling: src/kiro_crew/memory_backup.py -- not affected -- backs up into a partial file on disk
Sibling: src/kiro_crew/member_memory_backup.py -- not affected -- backs up into a copied file on disk
Sibling: src/kiro_crew/pod/runtime_home.py -- not affected -- backs up into the target file on disk
Sibling: src/kiro_crew/acp/codex_sqlite.py -- not affected -- backs up into a temporary file on disk
Pattern harvest
Not generalizable: the only whole-buffer site was this one helper, and every other SQLite backup already targets a file on disk.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)