Skip to content

fix(memory): open a populated store without taking the write lock - #24602

Merged
bolichen97 merged 1 commit into
mainfrom
fix/21056-write-free-open-pass
Oct 27, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/21056-write-free-open-pass

Conversation

@cruisercohen

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: opening a memory store that already holds facts writes nothing when there is nothing to trim, so the open never waits on, or fails behind, another writer.

Opening a V1-algorithm store that holds any accepted revision takes the SQLite write lock on every open. When another connection is writing to the same file at that moment, VectorMemoryStore.init() waits out its 5 s busy_timeout and then raises sqlite3.OperationalError: database is locked. Reproduced on main a3ee04f57: the strict-xfail test #22866 added for this case fails with --runxfail for both lineages, raised from limit_v1_accepted_history.

Why it matters

Every open of a populated store (the default store, and every named crew store) contends for the write lock with whatever else is writing to that file. A busy writer turns a read-only open into a 5 s stall and then a hard failure.

Not a goal

What changed (motivation → approach → change)

  • Symptom: a reopen of a store with one fact traces BEGIN; DELETE FROM memory_revisions ...; COMMIT, even though the DELETE matches no row.
  • Root cause: the open-time pass in _init_database calls limit_v1_accepted_history(db) with no record id, which ran one DELETE ... NOT IN (... LIMIT 20) per record holding an accepted revision. Under the connection's implicit-transaction mode the first DML statement issues BEGIN and takes the RESERVED lock whether or not it deletes anything. algorithm_version (not the lineage) selects this bound, so v1 and crew files are both hit.
  • Change: with no record id, limit_v1_accepted_history now selects only the records over V1_ACCEPTED_REVISION_LIMIT (GROUP BY record_id HAVING count(*) > ?) and runs the unchanged DELETE for those alone. A steady-state open issues no DML. The per-record path a normal write calls is unchanged.
  • reconcile audit (asked for in the issue): it skips every row whose content hash is unchanged, and get_record_metadata only reads, so a steady-state open issues no DML there. The new test traces the whole reopen and asserts no BEGIN, INSERT, UPDATE, DELETE or REPLACE ran.
  • The memory spec gains one paragraph after "Schema setup is one transaction": the open-time metadata pass writes only when it has work.

Backwards compatibility

Compatible: the retention bound is the same 20 accepted revisions per record. A record over the bound is trimmed on open exactly as before; a record within it was never changed by the DELETE.

Removes nothing: the change only skips DELETE statements that matched no row. No row, key, column or public name is removed, and over-limit trimming still runs on open (test_existing_journal_catchup_keeps_latest_evidence_before_reconcile and test_failed_init_pruning_rolls_back_reconciliation_while_v2_still_opens cover it).

Tests

Proof: PROVEN test/test_memory_revision_retention.py::test_populated_pre_metadata_v1_open_avoids_newer_sql[v1] (head c819b87)

  • test_a_populated_up_to_date_store_opens_while_another_connection_holds_the_write_lock[v1] and [crew] (in the schema-transaction test file): was xfail(strict=True) for this issue, now a normal test. The store holds one accepted revision; it reopens while another connection holds BEGIN IMMEDIATE, and a statement trace of the reopen shows no BEGIN and no DML. With the production hunk reverted both fail with database is locked (an OperationalError, which is why prove.py names only the assertion failure).
  • test_populated_pre_metadata_v1_open_avoids_newer_sql[v1]: the open-time pass now runs the over-limit selection and, with one revision, no DELETE. Its legacy-syntax connection still refuses ROW_NUMBER/OVER/ON CONFLICT, so the new selection is pinned to older SQLite syntax too.
  • Run locally (Linux, Python 3.12): every test file naming vector_memory, memory_record_metadata, VectorMemoryStore or memory_schema (171 files, -n 16): 7154 passed, 33 skipped, 8 xfailed. black, isort, flake8 and mypy clean on the touched modules; the comment-history, file-line-cap, sync-io-in-async and memory-store-seam gates pass.

Manual verification

N/A — unit coverage sufficient: the regression is a lock-contention condition the test reproduces directly with a second connection holding BEGIN IMMEDIATE, plus a statement trace of the open.

Related Issues

Closes #21056
Follow-up to #22866, which pinned this case with a strict xfail.

Sibling sweep

Swept: git grep -n -E "limit_v1_accepted_history\(|record_meta\.reconcile\(" -- src/
Siblings: none -- the store open is the only caller of the bulk trim; reconcile, beside it on that path, skips unchanged rows and issues no DML, and the per-record trim runs inside a write its caller already began

Pattern harvest

Rule candidate: review-prompt
Pattern: unconditional DML (DELETE/UPDATE that usually matches nothing) on a read-mostly open path; under implicit transactions it still begins a write and takes the write lock.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): the memory spec's schema-lineage section
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to ship on the code. A maintainer check remains before approval.

Head reviewed: c819b87.

What I verified:

  • Equivalence. The old open-time pass ran one DELETE per record with an accepted revision. That DELETE removes rows only when a record holds more than V1_ACCEPTED_REVISION_LIMIT accepted revisions. The new GROUP BY ... HAVING count(*) > ? selection picks exactly those records, so the retention result is unchanged and a steady-state open issues no DML.
  • The per-record path that sync_record calls is unchanged.
  • The open-time block in vector_memory_lifecycle.py is a plain connection context manager. With no DML it opens no transaction, so the trace asserting no BEGIN holds.
  • I ran test/test_vector_memory_schema_transaction.py and test/test_memory_revision_retention.py at this head on Linux: 39 passed.
  • Mutation check: with the production hunk reverted, the two populated-store lock tests fail with "database is locked" and test_populated_pre_metadata_v1_open_avoids_newer_sql fails. The tests go red when the behaviour breaks.
  • The spec paragraph matches the code. The linked issue is a T1 bug. No security path is touched.

What remains: the five AI review lanes on this head ended with the review step skipped (they failed closed), so there is no readiness signal for this head yet. I did not approve for that reason. Once the lanes run and readiness passes, this can merge.

No findings.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8ed14708db6ea2185fe8834efe744abbb3d299e5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 8ed14708db6ea2185fe8834efe744abbb3d299e5 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: this is a mechanism-level fix to the one bulk caller (vector_memory_lifecycle.py:537), and the xfail and issue #21056 record the defect.

What this change ships

Inventory (4 items) — 4 justified

Intent: let a populated memory store open while another connection is writing, instead of stalling for 5 s and then failing — a FIX.

  1. Opening a populated V1 or crew store no longer takes the write lock or fails with "database is locked" — justified
  2. Over-limit trimming at open now runs only for records over 20 accepted revisions — justified
  3. Former strict xfail (fix(memory): build a new store's schema in one transaction #22866) is now a passing test that asserts the reopen issues no DML — justified
  4. Memory spec gains a paragraph saying the open-time pass writes only when it has work — justified

[FIRST-PRINCIPLES-REVIEWED] 8ed1470

Description read

[DESCRIPTION-READ] 5a827f573c62c3f81167989903519ea790350874b5b90e82935a9320f0b0ebeb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8ed14708db6ea2185fe8834efe744abbb3d299e5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

The open-time trim now runs only on records over the limit. That removes the empty DML that took the write lock, and the per-record write path is unchanged.

Notes

  • The flipped DELETE assertion in test_populated_pre_metadata_v1_open_avoids_newer_sql checked that older SQLite syntax still works; it was not a decision to write on every open. That test no longer runs the DELETE path. An over-limit fixture opened on the same legacy-syntax connection would keep that check.

[DESIGN-REVIEWED] 8ed1470

Description read

[DESCRIPTION-READ] 422bdd72b898a2ab27331687fdc80aa666dd77e5779b24f8db4f83bfc5f0ce08

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8ed14708db6ea2185fe8834efe744abbb3d299e5 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8ed1470

Verdict parsed from the review's SHA-scoped output markers for commit 8ed14708db6ea2185fe8834efe744abbb3d299e5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8ed14708db6ea2185fe8834efe744abbb3d299e5 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8ed14708db6ea2185fe8834efe744abbb3d299e5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8ed1470

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8ed14708db6ea2185fe8834efe744abbb3d299e5 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
Opening a V1-algorithm store that holds any accepted revision ran one
DELETE per record in limit_v1_accepted_history. On a steady-state store
each matched no row, but under implicit transactions the first DML still
began a write, so the open waited out busy_timeout behind any other
writer and then raised "database is locked". Both lineages were
affected, because algorithm_version (not the lineage) selects the bound.

The open-time pass now selects only the records over
V1_ACCEPTED_REVISION_LIMIT (GROUP BY ... HAVING count(*) > ?) and
deletes for those alone, so an open with nothing to trim issues no DML.
The per-record path a normal write calls is unchanged. reconcile already
skips unchanged rows and issues no DML on a steady-state open.

#22866 pinned this case with a strict xfail; it is now a normal test
that also traces the reopen and asserts no BEGIN or DML statement ran.

Closes #21056
@cruisercohen
cruisercohen force-pushed the fix/21056-write-free-open-pass branch from c819b87 to 8ed1470 Compare October 27, 2026 01:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at head 8ed1470.

This head is a rebase of c819b87 onto current main. A range diff shows the PR commit's patch is identical, so the findings from my earlier review still hold: the GROUP BY ... HAVING count(*) > limit selection picks exactly the records the old per-record DELETE could trim, the per-record path that sync_record uses is unchanged, and a steady-state open now issues no DML and so takes no write lock.

Gates checked on this head:

  • readiness: passed
  • All required checks green (91 success, 16 skipped, 0 failing), including Dependency License Gate
  • Design, First Principles, Opus and GPT lanes PASS with no blocking findings; UX skipped (no user-facing surface)
  • Linked issue is tier T1, 4 files changed, no security-sensitive path touched, base is main, no merge conflict

No findings.

@bolichen97
bolichen97 merged commit e086a08 into main Oct 27, 2026
108 checks passed
@bolichen97
bolichen97 deleted the fix/21056-write-free-open-pass branch October 27, 2026 02:22
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Opening a populated memory store takes the write lock even when nothing needs trimming

2 participants