Skip to content

fix(memory): build a new store's schema in one transaction - #22866

Merged
iamwhatever merged 1 commit into
mainfrom
fix/memory-schema-one-transaction-13732
Oct 22, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/memory-schema-one-transaction-13732

Conversation

@buluoray

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a memory store's first open builds its whole schema in one transaction, so a fresh store pays one durable commit instead of one per DDL statement, and a failure part-way leaves no half-built schema.

On Windows CI, xdist workers are killed by the 180 s pytest-timeout during fixture SETUP while a fresh per-test store builds its schema (VectorMemoryStore.init → _init_database → executescript). The issue thread has three such runs in different files and shards. The DDL runs in autocommit, so every CREATE TABLE / CREATE INDEX is its own FULL-sync commit: 19 for a fresh v1 file, 23 for a fresh crew file.

Why it matters

A busy Windows shard that opens fresh stores can lose its whole xdist group, which reds unrelated PRs. In production, a crash part-way through setup can leave tables with no schema_version row.

Not a goal

What changed (motivation → approach → change)

Supersedes #21034 by @jeeshofone (co-author). Review verdict on #21034: sound. The plan-then-build-in-one-transaction approach targets the real cost (one fsync per DDL statement), its tests are meaningful, and its CI was green on all eight Windows shards at 8ba2ed6. It no longer applied because #20806 moved _init_database into vector_memory_lifecycle.py, so this PR ports it there and answers the open lane items.

  • Plan, then build only if needed. _init_database asks _schema_steps() what the file still needs, with no transaction open: missing schema_version rows, the revision-metadata objects (record_meta.schema_is_current), and the crew lineage/store-name stamps. It also binds the lineage. An up-to-date file needs nothing, so its schema step takes no write lock.
  • When something is pending: BEGIN IMMEDIATE, then _schema_steps() runs again under the lock and only what is still missing runs. One COMMIT, rollback() on any error, migration log lines after the commit.
  • executescript commits an open transaction first, so migration SQL goes through the new memory_schema.execute_script(db, sql). It splits where sqlite3.complete_statement does, so a ; in a literal, comment or trigger body does not split a statement.
  • memory_record_metadata.py's four additive objects live in one _SCHEMA_OBJECTS list, read by both ensure_schema (build) and schema_is_current (check), so the two cannot drift.
  • reconcile and limit_v1_accepted_history keep their own transaction, so the backfill scan never holds the schema write lock.
  • The no-commit migration contract is written in the memory spec ("Schema setup is one transaction"), beside the lineage table.

Dropped from #21034, per its First Principles lane:

  • _require_setup_transaction and its two call sites. No shipped step commits. A step that does still fails the open, at the closing COMMIT ("no transaction is active"); test_a_migration_step_that_commits_fails_the_open pins that.
  • The WAL-switch retry (_enable_wal) and its two tests. That race is on main too (2 of 40 rounds of four concurrent first opens fail on main and on this branch, always at PRAGMA journal_mode=WAL), so it moves to Concurrent first opens of a fresh memory file fail at PRAGMA journal_mode=WAL #22865 with its own acceptance.

Backwards compatibility

Compatible: the schema, the version rows and the migration order are unchanged. An up-to-date file's schema step writes nothing. A file that needs v2/v3 gets it in one transaction or keeps its prior schema, rows and version.

Removes nothing: no public name, key or row is removed. VectorMemoryStore gains four private methods bound from vector_memory_lifecycle (composition and owner pins updated).

Tests

test/test_vector_memory_schema_transaction.py (new; v1 and crew where it applies). Its connections use the store's own SQLite library (kiro_crew._sqlite_compat).

  • test_a_fresh_store_builds_its_schema_in_one_commit: exactly 1 commit frame in the WAL, the full schema_version set, the metadata tables.
  • test_reopening_a_built_store_commits_nothing: 0 commit frames.
  • test_a_failure_after_the_first_create_leaves_no_partial_schema: no tables after an injected failure; the next open builds everything.
  • test_a_failed_upgrade_leaves_the_store_at_its_prior_version (v1 and v1+v2 files with rows): schema and rows unchanged.
  • test_the_schema_step_of_an_up_to_date_store_takes_no_write_lock: opens under another connection's BEGIN IMMEDIATE, with no BEGIN traced.
  • test_two_openers_racing_on_a_fresh_file_build_its_schema_once: the re-plan under the lock is empty, each migration logged once.
  • test_a_migration_step_that_commits_fails_the_open.
  • test_a_populated_up_to_date_store_opens_while_another_connection_holds_the_write_lock: xfail(strict=True) for Opening a populated memory store takes the write lock even when nothing needs trimming #21056.
  • Also: execute_script edge cases, and a crew file that lost its lineage stamp gets it back.

Revert check: with _init_database's body put back to main's (helpers kept), 9 tests in the new file go red, among them test_a_fresh_store_builds_its_schema_in_one_commit[v1] and [crew], both test_a_failed_upgrade_leaves_the_store_at_its_prior_version cases and test_two_openers_racing_on_a_fresh_file_build_its_schema_once.

Run locally (Linux, Python 3.12, stdlib driver): the new file plus the composition-contract, owners and lock-discipline files: 155 passed, 3 skipped, 2 xfailed. Every test file naming vector_memory, memory_schema, memory_record_metadata or VectorMemoryStore (165 files, -n 8): 8578 passed, 34 skipped, 2 xfailed. black, isort, flake8 and mypy clean on the touched modules.

Manual verification

Not verified: Windows. This host is Linux. The effect on the Windows stall is inferred from the commit count (19/23 durable commits per fresh store down to 1); the Windows backend shards (Backend Tests (Windows) (1) through (8)) on this PR are the real check. #21034 measured fresh-init latency on Linux at 186–241 ms median before and 25–28 ms after; that figure is theirs and was not re-measured here.

Related Issues

Closes #13732
Supersedes #21034
Follow-up: #22865 (WAL-switch race), #21056 (open-time metadata write)

Pattern harvest

Rule candidate: semgrep
Pattern: executescript(...) or a run of DDL in autocommit on a store's open path. Each statement is a separate FULL-sync commit, and executescript silently commits any transaction it runs inside.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): the memory spec's "Two schema lineages" section
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

A fresh store's first open ran every DDL statement in autocommit, so each
CREATE was its own FULL-sync commit: 19 for a fresh v1 file, 23 for a crew
file. On a busy Windows runner that is enough for a per-test store's setup
to outrun the 180 s pytest-timeout (#13732).

_init_database now plans what the file still needs with no transaction
open (missing schema_version rows, the revision-metadata objects, the crew
stamps). An up-to-date file needs nothing and takes no write lock.
Otherwise it takes BEGIN IMMEDIATE, plans again under the lock, runs only
what is still missing, and commits once, rolling back on any error.
Migration SQL goes through memory_schema.execute_script, because
executescript commits a pending transaction first.

Ported from #21034 onto the vector_memory_lifecycle split (#20806). The
WAL-switch retry and the runtime "step committed" check from #21034 are
left out; a committing step already fails the open at the closing COMMIT.

Closes #13732

Co-authored-by: Will Laws <167437+jeeshofone@users.noreply.github.com>
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 5242ab2427207b231fe9cd6334fadc73425503e3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 5242ab2427207b231fe9cd6334fadc73425503e3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: it fixes the root cause (one durable commit per DDL statement in autocommit), and the plan-and-recheck under BEGIN IMMEDIATE fits the existing lifecycle owner.

Notes

  • The PR still only infers the Windows stall fix from the commit count. The Windows backend shards on this PR are the evidence, so check that all eight are green before merging.
  • execute_script relies on authors not calling executescript in a step, and nothing enforces that. The semgrep rule the PR proposes under "Pattern harvest" would turn that runtime failure into a lint error.

[DESIGN-REVIEWED] 5242ab2

Description read

[DESCRIPTION-READ] d5235e86829a540b7a91e581cb2af696f30a19c129baa7b44719730e3b8c54e6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 5242ab2427207b231fe9cd6334fadc73425503e3 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix works at the mechanism level (the store's DDL ran in autocommit), the defect has provenance (#13732 and tests that fail on base), and every helper the change adds has a consumer that uses it.

Notes

  • Item 8 — rides along, undeclared: +_BUSY_TIMEOUT_MS = 5000 has 1 consumer (grepped _BUSY_TIMEOUT_MS|busy_timeout=5000 in src/kiro_crew). The member-file path in the same file, at vector_memory_lifecycle.py:238, still spells "PRAGMA busy_timeout=5000", so one value now has two spellings. Subtraction: delete _BUSY_TIMEOUT_MS and put back the literal at line 288.

What this change ships

Inventory (8 items) — 7 justified

Intent: a memory store's first open no longer stalls Windows CI workers or leaves a half-built schema. FIX.

  1. A fresh store builds its whole schema in one durable commit, not 19 or 23 — justified
  2. A failed first build or v2/v3 upgrade leaves the file at its prior schema and rows — justified
  3. Opening an up-to-date store reads first and takes no write lock — justified
  4. Writing the crew lineage and store-name stamps moves into the setup transaction — justified
  5. New memory_schema.execute_script runs migration SQL without committing — justified
  6. record_meta.schema_is_current reads the shared _SCHEMA_OBJECTS list to decide whether metadata DDL is still needed — justified
  7. The spec and the migration-list comments now say a migration step must not commit — justified
  8. Busy-timeout literal pulled into _BUSY_TIMEOUT_MS — rides along, undeclared; sibling literal at line 238 left as is

[FIRST-PRINCIPLES-REVIEWED] 5242ab2

Description read

[DESCRIPTION-READ] 74ec95409679b09b4584065f54d95c993a4dfa91d260f08a4eead7eb0e3222e0

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5242ab2427207b231fe9cd6334fadc73425503e3 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5242ab2

Verdict parsed from the review's SHA-scoped output markers for commit 5242ab2427207b231fe9cd6334fadc73425503e3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5242ab2427207b231fe9cd6334fadc73425503e3: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 5242ab2427207b231fe9cd6334fadc73425503e3 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5242ab2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5242ab2427207b231fe9cd6334fadc73425503e3: <one-sentence reason>

@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: disposition for First Principles PASS note item 8 on 5242ab2.

_BUSY_TIMEOUT_MS is kept. It has a second user: test_a_populated_up_to_date_store_opens_while_another_connection_holds_the_write_lock patches it to 200 ms. That keeps the strict xfail for #21056 to a short, bounded wait. With a literal, each of its two cases would wait the full 5 s on every shard. The member-file path at vector_memory_lifecycle.py:238 still says busy_timeout=5000 because this PR does not touch that path. Using the constant there too is a fair cleanup, but it is not needed for this fix.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Disposition for the cancelled macOS Tests / Backend Tests (macOS) (2) on head 5242ab2. Both attempts hit the job's 40-minute limit. Neither failed a test, and the diff does not slow the shard.

  • Attempt 1, job 119418497192: cut at 19:45Z after 40:59, at [99%] of 58,589 items, 0 F/E.
  • Attempt 2, job 119460327638: pytest finished 57387 passed, 1200 skipped, 2 xfailed in 2331.04s (0:38:51), 0 F/E. The job was then cut during the artifact upload at exactly 40:00.

macOS Tests runs on demand, so main has no baseline. Here is the same shard on other PRs today:

Shard 2 job Items pytest wall 50% at 99% at Job
#22866 attempt 2 58,589 38:51 16.3 min 36.1 min 40:00 (cut)
#22866 attempt 1 58,589 (cut) 18.3 min 37.2 min 40:59 (cut)
#21867 119461026776 (ran alongside ours) 58,604 38:29 18.0 min 35.8 min 39:44 (passed)
#22854 119455627581 58,572 36:44 16.7 min 34.0 min 37:53 (passed)

Slowest tests: the shard prints only the top 5 (--durations=5), not 20. Those five are the same tests in all three completed logs, at similar times. None of them are memory or SQLite tests:

Test #22866 #21867 #22854
facade_mirror_guards every_name..._reaches_its_readers 51.05s 49.72s 51.18s
facade_mirror_guards no_test_writes_a_mirrored_facade_namespace 49.32s 49.37s 54.89s
facade_mirror_guards ..._create_true 37.95s 46.35s 41.27s
stt_engine (window test) 39.45s 38.78s <35s
dashboard_peer_auth loopback_urlopen_stale_socket... 35.63s 35.61s 35.61s

Our wall time is +0.4 min (1%) over #21867, which ran at the same time. That PR passed with only 16s to spare. So this shard sits right at the cap for every PR. It is not a slowdown from the one-transaction DDL. The single allowed re-run is spent, and no further re-run will be issued.

@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: disposition for Design PASS note 2 of 2 on 5242ab2 (split out of comment 6325077369, one finding per record).

  • Nothing enforces "no executescript in a step": agreed, and nothing is lost meanwhile. A step that commits fails the open loudly at the closing COMMIT, and test_a_migration_step_that_commits_fails_the_open pins that. The semgrep rule is recorded under "Pattern harvest", which is where this repo collects rule candidates. Writing it is not part of this fix.

@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: disposition for Design PASS note 1 of 2 on 5242ab2.

  • Windows shards are the evidence: checked. On this head Backend Tests (Windows) (1) through (8) all passed, and so did Windows Fail-Closed Sandbox Contract, Pod Boot Canary (Windows) and E2E Gateway Boot (windows-latest). One green run shows the change does not break Windows. It cannot prove the intermittent stall is gone, since the stall was seen in only three runs. The PR body says so.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@buluoray
buluoray marked this pull request as ready for review October 21, 2026 22:36
@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners October 21, 2026 22:36
@buluoray
buluoray requested a review from mbriones98 October 21, 2026 22:36
@buluoray
buluoray enabled auto-merge (squash) October 21, 2026 22:36
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 21, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Reviewed head 5242ab2.

The fix targets the real cost: schema DDL ran in autocommit, so each statement was its own FULL-sync commit. Setup now plans with no transaction open, takes BEGIN IMMEDIATE only when something is pending, plans again under the lock and commits once, rolling back on any error (vector_memory_lifecycle.py#L304). Migration SQL goes through execute_script (memory_schema.py#L812), so no step commits early. I checked the pieces it leans on: the store lock is re-entrant, the crew stamp writes do not commit, and _migrate_v2's duplicate-column fallback does not abort the transaction. The new tests cover one-commit builds, rollback, failed upgrades, racing openers and a step that commits. All lane notes are answered, the deferred items are tracked in #22865 and #21056, and every current check is green, including all eight Windows shards. I read CI and the test source; I did not run the tests.

— crew-pr-reviewer (r119)

@iamwhatever
iamwhatever merged commit be78b6c into main Oct 22, 2026
117 of 119 checks passed
@iamwhatever
iamwhatever deleted the fix/memory-schema-one-transaction-13732 branch October 22, 2026 06:45
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 22, 2026
cruisercohen added a commit that referenced this pull request Oct 27, 2026
Opening a V1-algorithm store that holds any accepted revision ran one
DELETE per record in limit_v1_accepted_history. On a steady-state store
each matched no row, but under implicit transactions the first DML still
began a write, so the open waited out busy_timeout behind any other
writer and then raised "database is locked". Both lineages were
affected, because algorithm_version (not the lineage) selects the bound.

The open-time pass now selects only the records over
V1_ACCEPTED_REVISION_LIMIT (GROUP BY ... HAVING count(*) > ?) and
deletes for those alone, so an open with nothing to trim issues no DML.
The per-record path a normal write calls is unchanged. reconcile already
skips unchanged rows and issues no DML on a steady-state open.

#22866 pinned this case with a strict xfail; it is now a normal test
that also traces the reopen and asserts no BEGIN or DML statement ran.

Closes #21056
bolichen97 pushed a commit that referenced this pull request Oct 27, 2026
…4602)

Opening a V1-algorithm store that holds any accepted revision ran one
DELETE per record in limit_v1_accepted_history. On a steady-state store
each matched no row, but under implicit transactions the first DML still
began a write, so the open waited out busy_timeout behind any other
writer and then raised "database is locked". Both lineages were
affected, because algorithm_version (not the lineage) selects the bound.

The open-time pass now selects only the records over
V1_ACCEPTED_REVISION_LIMIT (GROUP BY ... HAVING count(*) > ?) and
deletes for those alone, so an open with nothing to trim issues no DML.
The per-record path a normal write calls is unchanged. reconcile already
skips unchanged rows and issues no DML on a steady-state open.

#22866 pinned this case with a strict xfail; it is now a normal test
that also traces the reopen and asserts no BEGIN or DML statement ran.

Closes #21056
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests: memory_cli_store_scope worker killed by 180s timeout inside SQLite schema DDL during setup (Windows)

2 participants