Repository navigation
Description
Existing issues
- I searched open issues and this is not already reported.
Related: #25846 describes the mechanism, and #25852 fixed it where Connection.setconfig exists. #26091 (open) covers a stdlib sqlite3 connection inside the gateway corrupting or crashing the store's own reads, and the audit of which modules open memory.db that way. This report is about the cross-process close that #25852 guards: on Linux x86_64 that guard does nothing, and the regression test cannot detect it.
KiroCrew version
Field case: a 0.11.0 prerelease build that predates #25852 (its diagnose_probes.py still imports the stdlib sqlite3). Reproduction: main at fb4da54. No tag contains #25852 yet.
Release channel
Not sure (a prerelease build).
How is it installed?
pip / pipx (a managed venv). The reproduction runs from source.
Platform
Linux x86_64
What happened
Summary: on Linux x86_64, setup.cfg installs pysqlite3-binary, so _sqlite_compat binds the memory store to pysqlite3. A pysqlite3 connection has no setconfig, so sqlite_wal.keep_wal_on_close returns False and the store still closes with SQLite's default checkpoint-and-delete. #25852 states this gap in its body: "On Linux x86_64 the store uses pysqlite3, which has no setconfig, so its close keeps SQLite's default and those hosts stay exposed." I found no issue that tracks closing it. This report adds a field occurrence with lost rows on such a host, a reproduction against main's VectorMemoryStore, and a gap in the regression test.
Field case (one host, times UTC):
- 03:18:17:
diagnose_settingsran. The audit log shows the gateway servingGET /api/cards/agent/diagnose, which runsprobe_embedding_coveragein a gateway thread. In that build the probe opensmemory.dbwith the stdlibsqlite3(mode=ro, because a-walexisted). - 05:01:33:
kirocrew memory listran from a shell. It opens the store read-write in its own process.memory.dbwas last modified at 05:01:41, when that command finished, and nothing wrote it after that. - From then until a restart at 16:39, the gateway held
memory.db-wal (deleted)andmemory.db-shm (deleted)open (lsof +L1) and kept committing into them. Every other process readmemory.dbas it was at 05:01:41. - 16:00: the pre-update snapshot failed with
memory backup of <config dir>/memory.db failed: disk I/O error, andPOST /api/updatereturned 500memory_snapshot_failed. Akirocrew memory backuprun from a shell three minutes later reported success, and its copy held only the 05:01:41 state. - Rows only in the deleted log: 3 semantic, 8 episodic, 23 events, 11
memory_record_meta, 16 revisions, and newer versions of 131 existing rows. They were recovered from/proc/<gateway pid>/fdbefore the restart. The file a plain restart would have reopened, kept aside asmemory.db.superseded.*, still held only the 05:01:41 state, so a restart without that copy would have lost those rows.
That the 03:18 close removed the gateway's locks, and that the 05:01 close deleted the log, is inferred from this timeline and the reproduction below; the host recorded no lock state at those times.
The diagnosis-probe open in this chain is gone on main, because #25852 moved diagnose_probes.py to the store's driver. What stays open on pysqlite3 hosts is the close: once the gateway loses its locks by any route, the next read-write close in another process deletes the log.
Measured (scratch databases):
- One stdlib
mode=roopen and close inside the writer's process removes the writer's POSIX locks on bothmemory.dbandmemory.db-shm(/proc/locks: 1 and 1 before, 0 and 0 after). - Which lock loss lets another process's read-write close delete the
-wal. Same result on SQLite 3.51.1 (pysqlite3-binary0.5.4.post2), 3.53.1 and 3.53.2:
| writer lost its lock on | -wal deleted by the other process's close |
|---|---|
| nothing (control) | no |
memory.db only |
no |
memory.db and memory.db-shm |
yes; a fresh reader then sees 1 of 6 rows |
- With fix(memory): keep the shared -wal when another process closes the memory store #25852's two calls applied to the other process's close (
setconfig(SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE, True), andPRAGMA wal_checkpoint(TRUNCATE)before close), the-walsurvived the both-locks case and a fresh reader saw all 6 rows. So the fix works wheresetconfigexists. - With
main's own store on the default Linux x86_64 driver, the chain still splits. After the writer loses both locks, by a stdlibmode=roconnection or by a plain open and close ofmemory.dbandmemory.db-shm, a second process'sVectorMemoryStoreinit and close deletes the-wal, and a fresh reader sees only the first key. Test below.
The regression test cannot catch this: test/test_vector_memory_wal_split.py::test_another_process_close_keeps_the_wal_a_lockless_writer_uses
- With
pysqlite3-binaryinstalled it is skipped:SKIPPED ... driver has no Connection.setconfig(measured in a venv with 0.5.4.post2). The Linux x86_64 CI lanes install the same package, so I expect the same skip there (not checked in CI logs). - Where it runs,
_drop_this_process_locksdrops only thememory.dblock, which the table above shows is not enough. With the stdlib driver forced (SQLite 3.53.1):
| run | result |
|---|---|
| test as merged | passed |
keep_wal_on_close reverted to return False |
passed |
reverted, and the helper also opens and closes memory.db-shm |
failed: the second process deleted the -wal the gateway writes through |
| as merged, with that helper | passed |
To force the stdlib driver I put a package pysqlite3/__init__.py with no connect first on PYTHONPATH, which resolve_sqlite3 sends down the stdlib path.
Steps to reproduce
Scratch files only. With a Python that has pysqlite3 installed: python split_repro.py <empty dir>.
split_repro.py
"""Scratch files only. Needs a Python with pysqlite3 installed: python split_repro.py <empty dir>"""
import os
import sqlite3 as stdlib
import subprocess
import sys
from pathlib import Path
import pysqlite3.dbapi2 as store
d = Path(sys.argv[1])
d.mkdir(parents=True, exist_ok=True)
db, wal, shm = d / "memory.db", d / "memory.db-wal", d / "memory.db-shm"
for p in (db, wal, shm, d / "copy.db"):
p.unlink(missing_ok=True)
def locks(p):
ino = p.stat().st_ino
return sum(f":{ino} " in r and f" {os.getpid()} " in r for r in open("/proc/locks"))
def other_process(code, *args):
cmd = [sys.executable, "-c", code, *map(str, args)]
return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout.strip()
print("sqlite", store.sqlite_version)
# The gateway: a store-driver connection in WAL mode that stays open.
gw = store.connect(db, isolation_level=None, check_same_thread=False)
gw.execute("PRAGMA journal_mode=WAL")
gw.execute("CREATE TABLE t(x)")
gw.execute("INSERT INTO t VALUES (1)")
print("gateway locks: db", locks(db), "shm", locks(shm))
# Inside the gateway, a stdlib connection opens, reads and closes (the diagnosis probe).
probe = stdlib.connect(db.resolve().as_uri() + "?mode=ro", uri=True)
probe.execute("SELECT count(*) FROM t").fetchone()
probe.close()
print("after the stdlib open/close: db", locks(db), "shm", locks(shm))
# A CLI command: another process opens read-write with the store driver, reads, closes.
other_process(
"import sys, pysqlite3.dbapi2 as s\n"
"c = s.connect(sys.argv[1]); c.execute('SELECT count(*) FROM t').fetchone(); c.close()",
db,
)
print("after the CLI close: -wal exists", wal.exists(), "-shm exists", shm.exists())
for i in range(2, 7):
gw.execute("INSERT INTO t VALUES (?)", (i,))
print("gateway sees", gw.execute("SELECT count(*) FROM t").fetchone()[0], "rows")
seen = other_process(
"import sys, pysqlite3.dbapi2 as s\n"
"c = s.connect('file:' + sys.argv[1] + '?mode=ro', uri=True)\n"
"print(c.execute('SELECT count(*) FROM t').fetchone()[0])",
db,
)
print("another process sees", seen, "rows")
# The pre-update snapshot: a new store-driver connection inside the gateway process.
src = store.connect(db.resolve().as_uri() + "?mode=ro", uri=True)
dst = store.connect(d / "copy.db")
try:
src.backup(dst)
print("in-gateway backup copied", dst.execute("SELECT count(*) FROM t").fetchone()[0], "rows")
except store.OperationalError as exc:
print("in-gateway backup failed:", exc)Output with pysqlite3 on SQLite 3.53.2 (3.51.1 prints the same lines):
sqlite 3.53.2
gateway locks: db 1 shm 1
after the stdlib open/close: db 0 shm 0
after the CLI close: -wal exists False -shm exists False
gateway sees 6 rows
another process sees 1 rows
in-gateway backup failed: disk I/O error
The last line is the error the pre-update snapshot hit in the field case.
Against main's store: save this as test/test_pysqlite_split_repro.py (it uses the repo's opened fixture) and run pytest test/test_pysqlite_split_repro.py -n0 on Linux x86_64.
test_pysqlite_split_repro.py
"""Repro: the #25846 split with main's VectorMemoryStore on the default driver."""
from __future__ import annotations
import os
import sqlite3 as stdlib_sqlite3
import subprocess
import sys
from pathlib import Path
import pytest
from kiro_crew import _sqlite_compat
from kiro_crew.vector_memory import VectorMemoryStore
_SECOND_PROCESS = """
import sys
from pathlib import Path
from kiro_crew.vector_memory import VectorMemoryStore
store = VectorMemoryStore(db_path=Path(sys.argv[1]))
store.init()
store.get_all_semantic()
store.close()
"""
def _lose_locks(db: Path, how: str) -> None:
if how == "stdlib-ro-connection":
conn = stdlib_sqlite3.connect(db.resolve().as_uri() + "?mode=ro", uri=True)
conn.execute("SELECT count(*) FROM semantic_memory").fetchone()
conn.close()
else:
for suffix in ("", "-shm"):
with open(f"{db}{suffix}", "rb") as handle:
handle.read(16)
@pytest.mark.parametrize("how", ["stdlib-ro-connection", "plain-open-close-db-and-shm"])
def test_split_on_default_driver(tmp_path: Path, opened, how: str) -> None:
driver = _sqlite_compat.sqlite3
print(f"driver={driver.__name__} sqlite={driver.sqlite_version}")
db = tmp_path / "memory.db"
gateway = opened(VectorMemoryStore(db_path=db))
gateway.init()
gateway.set_semantic("pref.before", "1", 0.9, "user_explicit")
wal = Path(f"{db}-wal")
wal_inode = wal.stat().st_ino
_lose_locks(db, how)
subprocess.run(
[sys.executable, "-c", _SECOND_PROCESS, str(db)],
cwd=tmp_path,
check=True,
timeout=120,
env=os.environ.copy(),
)
gateway.set_semantic("pref.after", "2", 0.9, "user_explicit")
kept = wal.exists() and wal.stat().st_ino == wal_inode
reader = stdlib_sqlite3.connect(db.resolve().as_uri() + "?mode=ro", uri=True)
try:
keys = sorted(row[0] for row in reader.execute("SELECT key FROM semantic_memory"))
finally:
reader.close()
assert kept, f"the second process deleted the -wal; a fresh reader sees only {keys}"Result with pysqlite3-binary 0.5.4.post2: both cases fail with AssertionError: the second process deleted the -wal; a fresh reader sees only ['pref.before'].
Expected: on every supported driver, closing the store in one process never deletes a -wal that another process still writes through, and a test covers that on Linux x86_64 and fails when the guard is reverted.
Relevant log output
The field case's pre-update snapshot, install prefix shortened and lower frames omitted:
WARNING kiro_crew.memory_backup: memory backup failed for <config dir>/memory.db
File "<install>/kiro_crew/memory_backup.py", line 187, in backup_store
src.backup(dst)
pysqlite3.dbapi2.OperationalError: disk I/O error
kiro_crew.memory_backup.MemoryBackupFailed: memory backup of <config dir>/memory.db failed: disk I/O error
WARNING kiro_crew.dashboard.handlers.updates: Update refused: pre-update memory snapshot failed: 1 store(s) not copiedAnything else
Questions for the maintainers; the design call is yours:
- What should guard the close where the driver has no
setconfig?pysqlite3'ssrc/connection.conmasterhas nosetconfig, and the newestpysqlite3-binaryis 0.5.4.post2 (2025-12-03). - Should
_drop_this_process_locksalso drop the-shmlock, and should some form of the test run wherepysqlite3is the driver? - Should the gateway notice this state? It can see that its own
-waldescriptor is unlinked. Here the first visible symptom was the update refusal 11 hours later, and its message ends "repair it: kirocrew memory backups", a command that lists backups.
Recovery that worked here: before any restart, copy the gateway's memory.db and memory.db-wal (deleted) descriptors from /proc/<pid>/fd into a new directory as memory.db and memory.db-wal, open that pair, run PRAGMA integrity_check and PRAGMA wal_checkpoint(TRUNCATE), then stage the file with kirocrew memory restore --from <file> and restart.
Activity
KiroCrewCaptain commentedon Oct 30, 2026
Thanks for the report. This issue has been triaged by Crew Captain and will be assigned to a maintainer, who will follow up here.