Repository navigation
fix(memory): build a new store's schema in one transaction - #21034
jeeshofone wants to merge 1 commit into
Conversation
9362195 to
1b43b5c
Compare
|
Disposition for the independent review of 9362195, answered against the new head 1b43b5c (still one commit, same base 5d899b0). MUST-FIX 1 (opening an up-to-date store took the write lock): fixed.
SHOULD-FIX 1 (no rollback test for an existing store mid-upgrade): taken. NIT 1 ( NIT 2 (a migration step could commit mid-transaction): fixed. After every step, NIT 3 (WAL counter ignores frame salts): no code change. The test helper's docstring now says the counter assumes a WAL written since the file was created. Every test using it starts from a fresh file. Also added: Kept and re-measured: a fresh store is still 1 commit (19 for v1 and 23 for crew on main), and reopening a built store is 0. The 200-store probe (32 processes, timing only Gates on 1b43b5c (Linux, Python 3.12). The new file passes 15/15. Every test file that names Matched AUTOSDE rules: First-pass check: R1-R8 OK. One note on R5.1: the new The fork workflows on the previous head were still awaiting maintainer approval, so no CI or AI lane has run on this PR yet. |
1b43b5c to
73f6a1d
Compare
|
Disposition for the independent delta review of 1b43b5c (APPROVE-WITH-FIXES, no must-fix), answered on 73f6a1d. Also rebased onto current main (b9d1104); main touched no memory files.
Gates on this head: the new file 17 passed, 2 xfailed. First-pass check: R1-R8 OK. |
A vector store's first open ran its schema DDL in autocommit, so every CREATE TABLE/INDEX was its own FULL-sync commit: 19 durable commits for a fresh v1 file and 23 for a crew file. Under heavy parallel I/O on Windows CI that setup outran the 180 s per-test timeout and killed the xdist worker during fixture setup. init() now plans the schema work first, with no transaction: missing schema versions, the shared revision metadata tables and indexes, and the crew lineage stamps. An up-to-date file needs nothing, so the schema step takes no write lock. When something is pending, init() takes BEGIN IMMEDIATE, plans again under the lock (another opener may have built it meanwhile) and runs only what is still missing, in one transaction rolled back on any error. A fresh store pays one commit, and a failed build or upgrade leaves the file at its prior schema and version. The open-time metadata pass after the schema step still writes on a store holding accepted revisions; kirodotdev#21056 tracks that, and a strict xfail test pins it. executescript commits an open transaction before it runs, so migration SQL goes through memory_schema.execute_script, which splits statements with sqlite3.complete_statement and runs them inside the caller's transaction. A migration step that commits the setup transaction is refused and not recorded as applied. isolation_level is restored in a finally. journal_mode stays outside (SQLite refuses it inside a transaction), synchronous stays FULL, and the metadata backfill keeps its own transaction so its full-table scan never holds the write lock. Concurrent first opens of a fresh store file do not fail at the WAL switch. That switch upgrades a read lock to a write lock, which SQLite does not cover with busy_timeout, so a second opener raised "database is locked" at once. The store's open sets busy_timeout first, skips the switch when the file is already in WAL, and otherwise retries it within the same 5 s budget. Closes kirodotdev#13732
73f6a1d to
8ba2ed6
Compare
|
Head 8ba2ed6: rebased onto main c55294e, still one commit. No CI lane had run on this PR before, so this head carries what a full first-pass self-check found. Fixed: the new tests opened their own connections with a second SQLite library. The store takes Fixed: unbounded waits in Pinned: the commit check after each setup step ( Removed: two hunks that changed nothing a test can observe. These were The points from the independent review of 9362195 are all still in place on this head:
Repository rules checked against this diff:
Gates on this head:
First-pass check: R1-R8 OK. |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The runtime check that refuses a committing schema step guards against a migration that doesn't exist yet. None of today's steps commit. Not justified as shipped
What this change shipsInventory (8 items) — 6 justifiedIntent: make a fresh memory store's first open build its schema in one durable commit, so Windows test setup stops timing out and a crash can't leave a half-built schema. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 8ba2ed6 Description read[DESCRIPTION-READ] d73d3e90af64b40ade55fbbf48b625b1485e4d952041ccd06c3e2dede76802c5 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm the Windows CI shards stop timing out in fixture setup. The description says Windows was not verified and the fix is inferred from the commit count. Suggestions
[DESIGN-REVIEWED] 8ba2ed6 Description read[DESCRIPTION-READ] baaa0a2edc12d62d10e8a02af7cd12314cc73931c5036dfb68e299e9e97c0c91 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
|
Hold at The change itself reads sound: the schema plan is read without a transaction, rebuilt under The fork review lanes posted after the last author comment, and nothing answers them yet:
This clears when each item has its own — crew-pr-reviewer (r107) |
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Taking this over in #22866 since it has been idle; you are credited as co-author on the commits. Thanks for the groundwork. The fix itself is unchanged. It is ported onto |
A fresh store's first open ran every DDL statement in autocommit, so each CREATE was its own FULL-sync commit: 19 for a fresh v1 file, 23 for a crew file. On a busy Windows runner that is enough for a per-test store's setup to outrun the 180 s pytest-timeout (#13732). _init_database now plans what the file still needs with no transaction open (missing schema_version rows, the revision-metadata objects, the crew stamps). An up-to-date file needs nothing and takes no write lock. Otherwise it takes BEGIN IMMEDIATE, plans again under the lock, runs only what is still missing, and commits once, rolling back on any error. Migration SQL goes through memory_schema.execute_script, because executescript commits a pending transaction first. Ported from #21034 onto the vector_memory_lifecycle split (#20806). The WAL-switch retry and the runtime "step committed" check from #21034 are left out; a committing step already fails the open at the closing COMMIT. Closes #13732 Co-authored-by: Will Laws <167437+jeeshofone@users.noreply.github.com>
|
Thank you for this fix, @jeeshofone. The same change has already landed on main: #22866 "fix(memory): build a new store's schema in one transaction" (merge commit be78b6c) carried this work forward, rebased onto the #20806 split into vector_memory_lifecycle.py. It closed #13732 on 2026-10-22. main now has memory_schema.execute_script and the single-transaction schema build, and the WAL-switch retry followed in #23920. This branch conflicts with main and has nothing left to add, so I am closing it. Please reopen if you see something here that did not make it into #22866. |
Problem / Motivation
Goal: a memory store's first open builds its whole schema in one transaction, so a fresh store pays one durable commit instead of one per DDL statement, and a failure part-way leaves no half-built schema.
On Windows CI, xdist workers are killed by the 180 s pytest-timeout during fixture SETUP, while a fresh per-test store runs its schema DDL (
VectorMemoryStore.init→_init_database→executescript): 15 Windows PR runs in 7 days, per the issue. The cost is commits: DDL runs in autocommit, so everyCREATE TABLE/CREATE INDEXis its own FULL-sync commit. On the parent commit a fresh v1 file takes 19 durable commits and a fresh crew file takes 23.Why it matters
A busy Windows shard that opens a fresh store can lose its whole xdist group. In production a crash mid-setup can leave tables with no
schema_versionrow.Not a goal
journal_modeorsynchronous. Both stay as they are (WAL, FULL).--max-worker-restartor sharing stores across tests. Both only hide or thin out the stall.vector_memory.pycode into owner modules (open refactor: split vector_memory.py into bound owner modules #20806). See "Rebase note".What changed (motivation → approach → change)
_init_databaseasks_schema_steps()what the file still needs, with NO transaction open: missingschema_versionrows, the revision-metadata tables and indexes (record_meta.schema_is_current), and the crew lineage/store-name stamps. It also binds the lineage. An up-to-date file needs nothing, so the schema step issues noBEGINand takes no write lock. The metadata pass after it still writes on a store holding revisions, as on main (Opening a populated memory store takes the write lock even when nothing needs trimming #21056).BEGIN IMMEDIATE, then_schema_steps()runs AGAIN under the lock and only what is still missing runs. Another opener may have built the schema between the two reads. OneCOMMIT,rollback()on any exception, migration log lines after the commit.executescriptCOMMITs an open transaction first, so migration SQL goes throughmemory_schema.py's newexecute_script(db, sql). It splits wheresqlite3.complete_statementdoes (a;in a literal, comment or trigger body does not split) and runs each statement in the caller's transaction.init()checksin_transactionand raises. A migration that committed is not recorded as applied, so the next open runs it again. The rule is documented beside_MIGRATIONSandMIGRATIONS_CREW.isolation_levelis restored in afinally.journal_mode=WALstays outside (SQLite refuses it inside a transaction).reconcileandlimit_v1_accepted_historykeep their own transaction, so the backfill scan never holds the schema write lock.PRAGMA journal_mode=WALupgrades a read lock to a write lock, which SQLite does not cover withbusy_timeout, so concurrent first opens of a fresh file raiseddatabase is locked(on main too)._enable_walskips a file already in WAL and otherwise retries within the same 5 s, withbusy_timeoutset first.memory_record_metadata.py's four additive objects live in one_SCHEMA_OBJECTSlist used byensure_schema(build) andschema_is_current(read), so the check cannot drift from the build.Reader:
_SCHEMA_OBJECTSis read only byensure_schemaandschema_is_current;_schema_stepsonly by_init_database, twice.Why not
BEGIN IMMEDIATEon every open: an empty up-to-date file would wait outbusy_timeoutbehind any writer and fail, where main opens it at once.Why not a
"BEGIN IMMEDIATE;" + sqlscript: v1 runs a Python step between SQL migrations.Other schema paths, checked:
create_member_databaseand member provisioning already use one transaction; the FTS index is one statement; private member opens run no DDL.Backwards compatibility
Compatible: schema, version rows and migration order are unchanged. The schema step of an up-to-date file writes nothing. A file needing v2/v3 gets it in one transaction, or keeps its prior schema, rows and version. Removes nothing public;
VectorMemoryStoregains four private methods (composition-contract pins updated).Tests
test/test_vector_memory_schema_transaction.py(new, 21 tests, v1 and crew where it applies). Its connections use the store's SQLite library (kiro_crew._sqlite_compat, per #18833): a second library keeps its own lock table, so the store would not see their locks.test_a_fresh_store_builds_its_schema_in_one_commit: exactly 1 commit frame in the WAL, the fullschema_versionset, the metadata tables.test_reopening_a_built_store_commits_nothing: 0.test_the_schema_step_of_an_up_to_date_store_takes_no_write_lock: an empty store opens under another connection'sBEGIN IMMEDIATE, with noBEGINtraced.test_a_populated_up_to_date_store_opens_while_another_connection_holds_the_write_lock:xfail(strict=True)for Opening a populated memory store takes the write lock even when nothing needs trimming #21056.test_a_first_open_waits_for_another_openers_lock_on_the_wal_switch: the lock is released at the first retry, no real sleep;test_four_first_opens_of_a_fresh_v1_file_all_succeed: 20 rounds.test_two_openers_racing_on_a_fresh_file_build_its_schema_once: another opener builds everything between one opener's plan and its lock; the re-plan is empty and each migration is logged once.test_a_failed_upgrade_leaves_the_store_at_its_prior_version(v1 and v1+v2 files with rows): schema and rows unchanged.CREATEleaves no tables;execute_scriptedge cases.On main: 19/23 commits, the failed-upgrade tests leave v2/v3 half-applied, both WAL tests raise
database is locked, and a committing step is not refused (prove.py: PROVEN).Run locally (Linux, Python 3.12). Stdlib driver: the new file 19 passed, 2 xfailed; the 139 files naming a touched module (
-n 4): 10312 passed, 65 skipped, 2 xfailed. pysqlite3 (the Linux x86_64 driver):test_vector_memory*,test_memory_*,test_member_memory*: 2752 passed, 21 skipped, 2 xfailed. All*ratchet*files, refactor facade, sqlite-compat, host-isolation: 431 passed. black gate, isort, flake8, mypysrc/kiro_crew/(0 issues), comment-history, file-line-cap, Semgrep 1.78 (0 findings).Manual verification
200 fresh
VectorMemoryStore.init()calls across 32 processes on a Linux host, timing onlyinit(), two runs each:Not verified: Windows. The effect on the stall is inferred from the commit count; the Windows shards are the check.
Rebase note (#20806)
#20806 moves
_init_databaseintovector_memory_lifecycle.py. Whoever lands second moves this block and its helpers with it.vector_memory.pygrows here (a reported-only line-cap row) and the split absorbs it.Related Issues
Closes #13732
Pattern harvest
Rule candidate: semgrep
Pattern:
executescript(...)or a run of DDL in autocommit on a store's open path. Each statement is a separate FULL-sync commit, andexecutescriptsilently commits any transaction it runs inside. Also: a write lock taken to find nothing to write, andjournal_mode=WALwith no retry.Checklist