Skip to content

fix(portability): stream each SQLite database into the export - #23244

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/23234-export-stream-sqlite
Oct 23, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/23234-export-stream-sqlite

Conversation

@JLargent42

@JLargent42 JLargent42 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Settings > Export holds no SQLite database whole in memory, so a large memory store exports in bounded memory.

The export already streams (#22393): the first byte goes out at once and every file member is copied in fixed-size pieces. Each SQLite database (memory.db, memory_index.db, every named store's databases) is the one exception. _backup_sqlite backs it up into an in-memory SQLite database, writes that to a temp file, reads the temp file whole, and _build_export_zip copies it into a BytesIO and calls zf.writestr. Measured on a 7.1 MB database: a 27.7 MB Python heap peak, and the whole member handed to the response in one write.

Why it matters

A memory store full of embeddings can be hundreds of MB. Exporting it costs several times that in gateway RAM at once, which can push the gateway into memory pressure while it also serves every chat. While that member is being copied the download makes no progress, which a tunnel or proxy can read as a stalled response.

Not a goal

What changed (motivation → approach → change)

Each database now goes straight into its zip member, with no stop in memory.

Before After
source → :memory: database → temp file → read_bytes() → BytesIO → zf.writestr source → temp file (SQLite backup API) → 1 MiB reads → open zip member
peak about 4x the database peak a few copy chunks, whatever the database size

_backup_sqlite (portability_export.py) takes any binary writable, backs up into a private temp file, streams it out with shutil.copyfileobj, returns the byte count, and removes the temp file whether it succeeds or fails. A new _add_sqlite_member (portability_stream.py) opens the member with force_zip64=True (the size is unknown when the header goes out, as for file members). The member keeps the timestamp and 0600 mode writestr gave it. The bytes are identical: going through :memory: added nothing. _SQLITE_COPY_CHUNK_BYTES joins the TYPE_CHECKING manifest in src/kiro_crew/portability.py, and the docstring in dashboard/handlers/portability.py says that every member streams.

Backwards compatibility

Compatible: the archive holds the same members with byte-identical database contents, and import reads it unchanged. The only format difference is ZIP64 extra fields on database members, which every other streamed member already carries.

Removes nothing: no member, key, field or caller goes away. _backup_sqlite keeps its name and argument order and still accepts a BytesIO.

Tests

test/test_portability_memory_round_trip.py::TestDatabaseMembersStream:

  • test_a_large_database_streams_in_bounded_memory: a 7 MB incompressible memory.db exports through stream_export_zip with a traced heap peak under a third of the database (measured about 0.4 MiB on Linux and 1.4 MiB on Windows, against 27.7 MB buffered), no sink write over 1 MiB, an empty temp directory afterwards, and an archive validate_import_zip accepts, whose restored database holds every row.
  • test_the_streamed_member_matches_a_buffered_backup: the streamed member is byte-identical to a backup into a BytesIO, with mode 0600.
  • test_a_failed_copy_removes_the_temp_file: a writer that raises propagates the error and leaves no temp file.

Proof: with only _add_sqlite_member's body reverted to the buffered BytesIO + writestr form, the bounded-memory test fails (peak 27.7 MB for a 7.1 MB database). Determinism: the class passes 5 consecutive runs, in reversed node order and under -n 4. The data is seeded (random.Random(23234)). The existing streaming tests (test_api_portability.py::test_export_head_arrives_before_the_build_finishes, test_export_streams_every_chunk_the_builder_writes) and the round-trip and import suites pass: 606 passed across the portability, snapshot-store, facade and API files.

Manual verification

N/A — the bounded-memory test exercises the real backup and zip writer against a real database; the HTTP streaming path is unchanged and covered by its existing tests.

Related Issues

Closes #23234

Sibling sweep

Swept: git grep -nE 'connect(":memory:")|.backup(' -- src/
Sibling: src/kiro_crew/snapshot_archive.py -- not affected -- backs up disk to disk, no in-memory copy
Sibling: src/kiro_crew/memory_backup.py -- not affected -- backs up into a partial file on disk
Sibling: src/kiro_crew/member_memory_backup.py -- not affected -- backs up into a copied file on disk
Sibling: src/kiro_crew/pod/runtime_home.py -- not affected -- backs up into the target file on disk
Sibling: src/kiro_crew/acp/codex_sqlite.py -- not affected -- backs up into a temporary file on disk

Pattern harvest

Not generalizable: the only whole-buffer site was this one helper, and every other SQLite backup already targets a file on disk.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@JLargent42
JLargent42 requested review from a team and KiroCrewCaptain as code owners October 22, 2026 23:54
@JLargent42 JLargent42 self-assigned this Oct 22, 2026
@JLargent42
JLargent42 requested a review from buluoray October 22, 2026 23:54
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9176feda041e663cee09c6b62c45305d117d296e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9176feda041e663cee09c6b62c45305d117d296e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9176fed

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9176feda041e663cee09c6b62c45305d117d296e: <one-sentence reason>

@JLargent42

Copy link
Copy Markdown
Contributor Author

CI note (head 78fead8): every AI-review lane plus Sensitive Change Review failed on infrastructure reads on attempt 1 (23:54Z) and again on attempt 2 (00:02Z). This is not a code finding. Opus could not read the merge-base 95531ce...78fead8 from shallow history and failed closed. Sensitive Change Review could not read the PR from the API (CalledProcessError). GPT, Design, UX and First Principles hit the same PR-read step. Dependency License Gate failed with 'Dependency review is not supported on this repository', a repository setting. Other PRs' claude-review runs failed in the same window (23:59Z-00:03Z). Each lane has been re-run once already, so I am not re-running again; a maintainer re-run is needed once the runners or API recover. Backend and test jobs are still running.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 9176feda041e663cee09c6b62c45305d117d296e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: each item serves the bounded-memory export goal, and the fix replaces the in-memory backup buffer that caused the problem.

What this change ships

Inventory (6 items) — 6 justified

Intent: let someone export a large memory store from Settings > Export without the gateway holding the whole database in RAM. This is a FIX (closes #23234, and the PR adds a bounded-memory test that fails on the buffered form).

  1. Each SQLite database goes straight from a temp-file backup into its zip member in 1 MiB reads, with no :memory: copy, BytesIO or writestr — justified
  2. Database members now carry ZIP64 extra fields, as streamed file members already do — justified
  3. _backup_sqlite takes any binary writable, returns the byte count, and removes its temp file on success and on failure — justified
  4. A _SQLITE_COPY_CHUNK_BYTES read size, listed in the facade's TYPE_CHECKING manifest — justified
  5. Database members keep the 0600 mode and write-time timestamp they had before — justified
  6. The export handler's docstring now says every member streams — justified

[FIRST-PRINCIPLES-REVIEWED] 9176fed

Description read

[DESCRIPTION-READ] 8d54fb90c811392cd99f041714a9aa23625ac85ee51b7d95fd358b8654d189a3

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9176feda041e663cee09c6b62c45305d117d296e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change removes the one remaining whole-member buffer, and the archive format, the import path and the callers stay as they were.

[DESIGN-REVIEWED] 9176fed

Description read

[DESCRIPTION-READ] 03316b3acea6776ec3d8a8512b0c098f97ed373e102a758ffe88a93f8d4b1079

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9176feda041e663cee09c6b62c45305d117d296e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9176fed

Verdict parsed from the review's SHA-scoped output markers for commit 9176feda041e663cee09c6b62c45305d117d296e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9176feda041e663cee09c6b62c45305d117d296e: <one-sentence reason>

@JLargent42
JLargent42 force-pushed the fix/23234-export-stream-sqlite branch from 78fead8 to bb86bc3 Compare October 23, 2026 01:13
@JLargent42

Copy link
Copy Markdown
Contributor Author

CI note for head bb86bc3 (the prior head was 78fead8):

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 23, 2026
@JLargent42
JLargent42 force-pushed the fix/23234-export-stream-sqlite branch from bb86bc3 to 9176fed Compare October 23, 2026 06:27
@JLargent42

Copy link
Copy Markdown
Contributor Author

Rebased once onto main for the #23264 Windows fix.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 23, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 23, 2026 09:05

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the export backs each SQLite store up to a temp file and streams it into its zip member in fixed-size reads, instead of holding the whole database in memory; a round-trip test covers it.

@KiroCrewCaptain
KiroCrewCaptain merged commit 9dc6a1c into main Oct 23, 2026
104 of 105 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/23234-export-stream-sqlite branch October 23, 2026 09:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export still holds each SQLite database whole in memory, about 3x its size

2 participants