Repository navigation
fix(memory): build a new store's schema in one transaction - #22866
Conversation
A fresh store's first open ran every DDL statement in autocommit, so each CREATE was its own FULL-sync commit: 19 for a fresh v1 file, 23 for a crew file. On a busy Windows runner that is enough for a per-test store's setup to outrun the 180 s pytest-timeout (#13732). _init_database now plans what the file still needs with no transaction open (missing schema_version rows, the revision-metadata objects, the crew stamps). An up-to-date file needs nothing and takes no write lock. Otherwise it takes BEGIN IMMEDIATE, plans again under the lock, runs only what is still missing, and commits once, rolling back on any error. Migration SQL goes through memory_schema.execute_script, because executescript commits a pending transaction first. Ported from #21034 onto the vector_memory_lifecycle split (#20806). The WAL-switch retry and the runtime "step committed" check from #21034 are left out; a committing step already fails the open at the closing COMMIT. Closes #13732 Co-authored-by: Will Laws <167437+jeeshofone@users.noreply.github.com>
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: it fixes the root cause (one durable commit per DDL statement in autocommit), and the plan-and-recheck under Notes
[DESIGN-REVIEWED] 5242ab2 Description read[DESCRIPTION-READ] d5235e86829a540b7a91e581cb2af696f30a19c129baa7b44719730e3b8c54e6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The fix works at the mechanism level (the store's DDL ran in autocommit), the defect has provenance (#13732 and tests that fail on base), and every helper the change adds has a consumer that uses it. Notes
What this change shipsInventory (8 items) — 7 justifiedIntent: a memory store's first open no longer stalls Windows CI workers or leaves a half-built schema. FIX.
[FIRST-PRINCIPLES-REVIEWED] 5242ab2 Description read[DESCRIPTION-READ] 74ec95409679b09b4584065f54d95c993a4dfa91d260f08a4eead7eb0e3222e0 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: disposition for First Principles PASS note item 8 on 5242ab2.
|
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Disposition for the cancelled
macOS Tests runs on demand, so main has no baseline. Here is the same shard on other PRs today:
Slowest tests: the shard prints only the top 5 (
Our wall time is +0.4 min (1%) over #21867, which ran at the same time. That PR passed with only 16s to spare. So this shard sits right at the cap for every PR. It is not a slowdown from the one-transaction DDL. The single allowed re-run is spent, and no further re-run will be issued. |
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: disposition for Design PASS note 2 of 2 on 5242ab2 (split out of comment 6325077369, one finding per record).
|
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: disposition for Design PASS note 1 of 2 on 5242ab2.
|
iamwhatever
left a comment
There was a problem hiding this comment.
Approve. Reviewed head 5242ab2.
The fix targets the real cost: schema DDL ran in autocommit, so each statement was its own FULL-sync commit. Setup now plans with no transaction open, takes BEGIN IMMEDIATE only when something is pending, plans again under the lock and commits once, rolling back on any error (vector_memory_lifecycle.py#L304). Migration SQL goes through execute_script (memory_schema.py#L812), so no step commits early. I checked the pieces it leans on: the store lock is re-entrant, the crew stamp writes do not commit, and _migrate_v2's duplicate-column fallback does not abort the transaction. The new tests cover one-commit builds, rollback, failed upgrades, racing openers and a step that commits. All lane notes are answered, the deferred items are tracked in #22865 and #21056, and every current check is green, including all eight Windows shards. I read CI and the test source; I did not run the tests.
— crew-pr-reviewer (r119)
Opening a V1-algorithm store that holds any accepted revision ran one DELETE per record in limit_v1_accepted_history. On a steady-state store each matched no row, but under implicit transactions the first DML still began a write, so the open waited out busy_timeout behind any other writer and then raised "database is locked". Both lineages were affected, because algorithm_version (not the lineage) selects the bound. The open-time pass now selects only the records over V1_ACCEPTED_REVISION_LIMIT (GROUP BY ... HAVING count(*) > ?) and deletes for those alone, so an open with nothing to trim issues no DML. The per-record path a normal write calls is unchanged. reconcile already skips unchanged rows and issues no DML on a steady-state open. #22866 pinned this case with a strict xfail; it is now a normal test that also traces the reopen and asserts no BEGIN or DML statement ran. Closes #21056
…4602) Opening a V1-algorithm store that holds any accepted revision ran one DELETE per record in limit_v1_accepted_history. On a steady-state store each matched no row, but under implicit transactions the first DML still began a write, so the open waited out busy_timeout behind any other writer and then raised "database is locked". Both lineages were affected, because algorithm_version (not the lineage) selects the bound. The open-time pass now selects only the records over V1_ACCEPTED_REVISION_LIMIT (GROUP BY ... HAVING count(*) > ?) and deletes for those alone, so an open with nothing to trim issues no DML. The per-record path a normal write calls is unchanged. reconcile already skips unchanged rows and issues no DML on a steady-state open. #22866 pinned this case with a strict xfail; it is now a normal test that also traces the reopen and asserts no BEGIN or DML statement ran. Closes #21056
Problem / Motivation
Goal: a memory store's first open builds its whole schema in one transaction, so a fresh store pays one durable commit instead of one per DDL statement, and a failure part-way leaves no half-built schema.
On Windows CI, xdist workers are killed by the 180 s pytest-timeout during fixture SETUP while a fresh per-test store builds its schema (
VectorMemoryStore.init→_init_database→executescript). The issue thread has three such runs in different files and shards. The DDL runs in autocommit, so everyCREATE TABLE/CREATE INDEXis its own FULL-sync commit: 19 for a fresh v1 file, 23 for a fresh crew file.Why it matters
A busy Windows shard that opens fresh stores can lose its whole xdist group, which reds unrelated PRs. In production, a crash part-way through setup can leave tables with no
schema_versionrow.Not a goal
journal_modeorsynchronous. Both stay WAL and FULL.--max-worker-restartor sharing stores across tests. Both only hide or thin out the stall.What changed (motivation → approach → change)
Supersedes #21034 by @jeeshofone (co-author). Review verdict on #21034: sound. The plan-then-build-in-one-transaction approach targets the real cost (one fsync per DDL statement), its tests are meaningful, and its CI was green on all eight Windows shards at 8ba2ed6. It no longer applied because #20806 moved
_init_databaseintovector_memory_lifecycle.py, so this PR ports it there and answers the open lane items._init_databaseasks_schema_steps()what the file still needs, with no transaction open: missingschema_versionrows, the revision-metadata objects (record_meta.schema_is_current), and the crew lineage/store-name stamps. It also binds the lineage. An up-to-date file needs nothing, so its schema step takes no write lock.BEGIN IMMEDIATE, then_schema_steps()runs again under the lock and only what is still missing runs. OneCOMMIT,rollback()on any error, migration log lines after the commit.executescriptcommits an open transaction first, so migration SQL goes through the newmemory_schema.execute_script(db, sql). It splits wheresqlite3.complete_statementdoes, so a;in a literal, comment or trigger body does not split a statement.memory_record_metadata.py's four additive objects live in one_SCHEMA_OBJECTSlist, read by bothensure_schema(build) andschema_is_current(check), so the two cannot drift.reconcileandlimit_v1_accepted_historykeep their own transaction, so the backfill scan never holds the schema write lock.Dropped from #21034, per its First Principles lane:
_require_setup_transactionand its two call sites. No shipped step commits. A step that does still fails the open, at the closingCOMMIT("no transaction is active");test_a_migration_step_that_commits_fails_the_openpins that._enable_wal) and its two tests. That race is on main too (2 of 40 rounds of four concurrent first opens fail on main and on this branch, always atPRAGMA journal_mode=WAL), so it moves to Concurrent first opens of a fresh memory file fail at PRAGMA journal_mode=WAL #22865 with its own acceptance.Backwards compatibility
Compatible: the schema, the version rows and the migration order are unchanged. An up-to-date file's schema step writes nothing. A file that needs v2/v3 gets it in one transaction or keeps its prior schema, rows and version.
Removes nothing: no public name, key or row is removed.
VectorMemoryStoregains four private methods bound fromvector_memory_lifecycle(composition and owner pins updated).Tests
test/test_vector_memory_schema_transaction.py(new; v1 and crew where it applies). Its connections use the store's own SQLite library (kiro_crew._sqlite_compat).test_a_fresh_store_builds_its_schema_in_one_commit: exactly 1 commit frame in the WAL, the fullschema_versionset, the metadata tables.test_reopening_a_built_store_commits_nothing: 0 commit frames.test_a_failure_after_the_first_create_leaves_no_partial_schema: no tables after an injected failure; the next open builds everything.test_a_failed_upgrade_leaves_the_store_at_its_prior_version(v1 and v1+v2 files with rows): schema and rows unchanged.test_the_schema_step_of_an_up_to_date_store_takes_no_write_lock: opens under another connection'sBEGIN IMMEDIATE, with noBEGINtraced.test_two_openers_racing_on_a_fresh_file_build_its_schema_once: the re-plan under the lock is empty, each migration logged once.test_a_migration_step_that_commits_fails_the_open.test_a_populated_up_to_date_store_opens_while_another_connection_holds_the_write_lock:xfail(strict=True)for Opening a populated memory store takes the write lock even when nothing needs trimming #21056.execute_scriptedge cases, and a crew file that lost its lineage stamp gets it back.Revert check: with
_init_database's body put back to main's (helpers kept), 9 tests in the new file go red, among themtest_a_fresh_store_builds_its_schema_in_one_commit[v1]and[crew], bothtest_a_failed_upgrade_leaves_the_store_at_its_prior_versioncases andtest_two_openers_racing_on_a_fresh_file_build_its_schema_once.Run locally (Linux, Python 3.12, stdlib driver): the new file plus the composition-contract, owners and lock-discipline files: 155 passed, 3 skipped, 2 xfailed. Every test file naming
vector_memory,memory_schema,memory_record_metadataorVectorMemoryStore(165 files,-n 8): 8578 passed, 34 skipped, 2 xfailed. black, isort, flake8 and mypy clean on the touched modules.Manual verification
Not verified: Windows. This host is Linux. The effect on the Windows stall is inferred from the commit count (19/23 durable commits per fresh store down to 1); the Windows backend shards (
Backend Tests (Windows) (1)through(8)) on this PR are the real check. #21034 measured fresh-init latency on Linux at 186–241 ms median before and 25–28 ms after; that figure is theirs and was not re-measured here.Related Issues
Closes #13732
Supersedes #21034
Follow-up: #22865 (WAL-switch race), #21056 (open-time metadata write)
Pattern harvest
Rule candidate: semgrep
Pattern:
executescript(...)or a run of DDL in autocommit on a store's open path. Each statement is a separate FULL-sync commit, andexecutescriptsilently commits any transaction it runs inside.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.