Skip to content

memory.db still splits on Linux x86_64 because pysqlite3 has no setconfig, and #25852's regression test passes with the fix reverted #26331

Closed

Description

@swchoi0102

Existing issues

  • I searched open issues and this is not already reported.

Related: #25846 describes the mechanism, and #25852 fixed it where Connection.setconfig exists. #26091 (open) covers a stdlib sqlite3 connection inside the gateway corrupting or crashing the store's own reads, and the audit of which modules open memory.db that way. This report is about the cross-process close that #25852 guards: on Linux x86_64 that guard does nothing, and the regression test cannot detect it.

KiroCrew version

Field case: a 0.11.0 prerelease build that predates #25852 (its diagnose_probes.py still imports the stdlib sqlite3). Reproduction: main at fb4da54. No tag contains #25852 yet.

Release channel

Not sure (a prerelease build).

How is it installed?

pip / pipx (a managed venv). The reproduction runs from source.

Platform

Linux x86_64

What happened

Summary: on Linux x86_64, setup.cfg installs pysqlite3-binary, so _sqlite_compat binds the memory store to pysqlite3. A pysqlite3 connection has no setconfig, so sqlite_wal.keep_wal_on_close returns False and the store still closes with SQLite's default checkpoint-and-delete. #25852 states this gap in its body: "On Linux x86_64 the store uses pysqlite3, which has no setconfig, so its close keeps SQLite's default and those hosts stay exposed." I found no issue that tracks closing it. This report adds a field occurrence with lost rows on such a host, a reproduction against main's VectorMemoryStore, and a gap in the regression test.

Field case (one host, times UTC):

  • 03:18:17: diagnose_settings ran. The audit log shows the gateway serving GET /api/cards/agent/diagnose, which runs probe_embedding_coverage in a gateway thread. In that build the probe opens memory.db with the stdlib sqlite3 (mode=ro, because a -wal existed).
  • 05:01:33: kirocrew memory list ran from a shell. It opens the store read-write in its own process. memory.db was last modified at 05:01:41, when that command finished, and nothing wrote it after that.
  • From then until a restart at 16:39, the gateway held memory.db-wal (deleted) and memory.db-shm (deleted) open (lsof +L1) and kept committing into them. Every other process read memory.db as it was at 05:01:41.
  • 16:00: the pre-update snapshot failed with memory backup of <config dir>/memory.db failed: disk I/O error, and POST /api/update returned 500 memory_snapshot_failed. A kirocrew memory backup run from a shell three minutes later reported success, and its copy held only the 05:01:41 state.
  • Rows only in the deleted log: 3 semantic, 8 episodic, 23 events, 11 memory_record_meta, 16 revisions, and newer versions of 131 existing rows. They were recovered from /proc/<gateway pid>/fd before the restart. The file a plain restart would have reopened, kept aside as memory.db.superseded.*, still held only the 05:01:41 state, so a restart without that copy would have lost those rows.

That the 03:18 close removed the gateway's locks, and that the 05:01 close deleted the log, is inferred from this timeline and the reproduction below; the host recorded no lock state at those times.

The diagnosis-probe open in this chain is gone on main, because #25852 moved diagnose_probes.py to the store's driver. What stays open on pysqlite3 hosts is the close: once the gateway loses its locks by any route, the next read-write close in another process deletes the log.

Measured (scratch databases):

  • One stdlib mode=ro open and close inside the writer's process removes the writer's POSIX locks on both memory.db and memory.db-shm (/proc/locks: 1 and 1 before, 0 and 0 after).
  • Which lock loss lets another process's read-write close delete the -wal. Same result on SQLite 3.51.1 (pysqlite3-binary 0.5.4.post2), 3.53.1 and 3.53.2:
writer lost its lock on -wal deleted by the other process's close
nothing (control) no
memory.db only no
memory.db and memory.db-shm yes; a fresh reader then sees 1 of 6 rows
  • With fix(memory): keep the shared -wal when another process closes the memory store #25852's two calls applied to the other process's close (setconfig(SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE, True), and PRAGMA wal_checkpoint(TRUNCATE) before close), the -wal survived the both-locks case and a fresh reader saw all 6 rows. So the fix works where setconfig exists.
  • With main's own store on the default Linux x86_64 driver, the chain still splits. After the writer loses both locks, by a stdlib mode=ro connection or by a plain open and close of memory.db and memory.db-shm, a second process's VectorMemoryStore init and close deletes the -wal, and a fresh reader sees only the first key. Test below.

The regression test cannot catch this: test/test_vector_memory_wal_split.py::test_another_process_close_keeps_the_wal_a_lockless_writer_uses

  • With pysqlite3-binary installed it is skipped: SKIPPED ... driver has no Connection.setconfig (measured in a venv with 0.5.4.post2). The Linux x86_64 CI lanes install the same package, so I expect the same skip there (not checked in CI logs).
  • Where it runs, _drop_this_process_locks drops only the memory.db lock, which the table above shows is not enough. With the stdlib driver forced (SQLite 3.53.1):
run result
test as merged passed
keep_wal_on_close reverted to return False passed
reverted, and the helper also opens and closes memory.db-shm failed: the second process deleted the -wal the gateway writes through
as merged, with that helper passed

To force the stdlib driver I put a package pysqlite3/__init__.py with no connect first on PYTHONPATH, which resolve_sqlite3 sends down the stdlib path.

Steps to reproduce

Scratch files only. With a Python that has pysqlite3 installed: python split_repro.py <empty dir>.

split_repro.py
"""Scratch files only. Needs a Python with pysqlite3 installed: python split_repro.py <empty dir>"""

import os
import sqlite3 as stdlib
import subprocess
import sys
from pathlib import Path

import pysqlite3.dbapi2 as store

d = Path(sys.argv[1])
d.mkdir(parents=True, exist_ok=True)
db, wal, shm = d / "memory.db", d / "memory.db-wal", d / "memory.db-shm"
for p in (db, wal, shm, d / "copy.db"):
    p.unlink(missing_ok=True)


def locks(p):
    ino = p.stat().st_ino
    return sum(f":{ino} " in r and f" {os.getpid()} " in r for r in open("/proc/locks"))


def other_process(code, *args):
    cmd = [sys.executable, "-c", code, *map(str, args)]
    return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout.strip()


print("sqlite", store.sqlite_version)

# The gateway: a store-driver connection in WAL mode that stays open.
gw = store.connect(db, isolation_level=None, check_same_thread=False)
gw.execute("PRAGMA journal_mode=WAL")
gw.execute("CREATE TABLE t(x)")
gw.execute("INSERT INTO t VALUES (1)")
print("gateway locks: db", locks(db), "shm", locks(shm))

# Inside the gateway, a stdlib connection opens, reads and closes (the diagnosis probe).
probe = stdlib.connect(db.resolve().as_uri() + "?mode=ro", uri=True)
probe.execute("SELECT count(*) FROM t").fetchone()
probe.close()
print("after the stdlib open/close: db", locks(db), "shm", locks(shm))

# A CLI command: another process opens read-write with the store driver, reads, closes.
other_process(
    "import sys, pysqlite3.dbapi2 as s\n"
    "c = s.connect(sys.argv[1]); c.execute('SELECT count(*) FROM t').fetchone(); c.close()",
    db,
)
print("after the CLI close: -wal exists", wal.exists(), "-shm exists", shm.exists())

for i in range(2, 7):
    gw.execute("INSERT INTO t VALUES (?)", (i,))
print("gateway sees", gw.execute("SELECT count(*) FROM t").fetchone()[0], "rows")
seen = other_process(
    "import sys, pysqlite3.dbapi2 as s\n"
    "c = s.connect('file:' + sys.argv[1] + '?mode=ro', uri=True)\n"
    "print(c.execute('SELECT count(*) FROM t').fetchone()[0])",
    db,
)
print("another process sees", seen, "rows")

# The pre-update snapshot: a new store-driver connection inside the gateway process.
src = store.connect(db.resolve().as_uri() + "?mode=ro", uri=True)
dst = store.connect(d / "copy.db")
try:
    src.backup(dst)
    print("in-gateway backup copied", dst.execute("SELECT count(*) FROM t").fetchone()[0], "rows")
except store.OperationalError as exc:
    print("in-gateway backup failed:", exc)

Output with pysqlite3 on SQLite 3.53.2 (3.51.1 prints the same lines):

sqlite 3.53.2
gateway locks: db 1 shm 1
after the stdlib open/close: db 0 shm 0
after the CLI close: -wal exists False -shm exists False
gateway sees 6 rows
another process sees 1 rows
in-gateway backup failed: disk I/O error

The last line is the error the pre-update snapshot hit in the field case.

Against main's store: save this as test/test_pysqlite_split_repro.py (it uses the repo's opened fixture) and run pytest test/test_pysqlite_split_repro.py -n0 on Linux x86_64.

test_pysqlite_split_repro.py
"""Repro: the #25846 split with main's VectorMemoryStore on the default driver."""

from __future__ import annotations

import os
import sqlite3 as stdlib_sqlite3
import subprocess
import sys
from pathlib import Path

import pytest

from kiro_crew import _sqlite_compat
from kiro_crew.vector_memory import VectorMemoryStore

_SECOND_PROCESS = """
import sys
from pathlib import Path
from kiro_crew.vector_memory import VectorMemoryStore

store = VectorMemoryStore(db_path=Path(sys.argv[1]))
store.init()
store.get_all_semantic()
store.close()
"""


def _lose_locks(db: Path, how: str) -> None:
    if how == "stdlib-ro-connection":
        conn = stdlib_sqlite3.connect(db.resolve().as_uri() + "?mode=ro", uri=True)
        conn.execute("SELECT count(*) FROM semantic_memory").fetchone()
        conn.close()
    else:
        for suffix in ("", "-shm"):
            with open(f"{db}{suffix}", "rb") as handle:
                handle.read(16)


@pytest.mark.parametrize("how", ["stdlib-ro-connection", "plain-open-close-db-and-shm"])
def test_split_on_default_driver(tmp_path: Path, opened, how: str) -> None:
    driver = _sqlite_compat.sqlite3
    print(f"driver={driver.__name__} sqlite={driver.sqlite_version}")
    db = tmp_path / "memory.db"
    gateway = opened(VectorMemoryStore(db_path=db))
    gateway.init()
    gateway.set_semantic("pref.before", "1", 0.9, "user_explicit")
    wal = Path(f"{db}-wal")
    wal_inode = wal.stat().st_ino
    _lose_locks(db, how)

    subprocess.run(
        [sys.executable, "-c", _SECOND_PROCESS, str(db)],
        cwd=tmp_path,
        check=True,
        timeout=120,
        env=os.environ.copy(),
    )

    gateway.set_semantic("pref.after", "2", 0.9, "user_explicit")
    kept = wal.exists() and wal.stat().st_ino == wal_inode
    reader = stdlib_sqlite3.connect(db.resolve().as_uri() + "?mode=ro", uri=True)
    try:
        keys = sorted(row[0] for row in reader.execute("SELECT key FROM semantic_memory"))
    finally:
        reader.close()
    assert kept, f"the second process deleted the -wal; a fresh reader sees only {keys}"

Result with pysqlite3-binary 0.5.4.post2: both cases fail with AssertionError: the second process deleted the -wal; a fresh reader sees only ['pref.before'].

Expected: on every supported driver, closing the store in one process never deletes a -wal that another process still writes through, and a test covers that on Linux x86_64 and fails when the guard is reverted.

Relevant log output

The field case's pre-update snapshot, install prefix shortened and lower frames omitted:

WARNING kiro_crew.memory_backup: memory backup failed for <config dir>/memory.db
  File "<install>/kiro_crew/memory_backup.py", line 187, in backup_store
    src.backup(dst)
pysqlite3.dbapi2.OperationalError: disk I/O error
kiro_crew.memory_backup.MemoryBackupFailed: memory backup of <config dir>/memory.db failed: disk I/O error
WARNING kiro_crew.dashboard.handlers.updates: Update refused: pre-update memory snapshot failed: 1 store(s) not copied

Anything else

Questions for the maintainers; the design call is yours:

  1. What should guard the close where the driver has no setconfig? pysqlite3's src/connection.c on master has no setconfig, and the newest pysqlite3-binary is 0.5.4.post2 (2025-12-03).
  2. Should _drop_this_process_locks also drop the -shm lock, and should some form of the test run where pysqlite3 is the driver?
  3. Should the gateway notice this state? It can see that its own -wal descriptor is unlinked. Here the first visible symptom was the update refusal 11 hours later, and its message ends "repair it: kirocrew memory backups", a command that lists backups.

Recovery that worked here: before any restart, copy the gateway's memory.db and memory.db-wal (deleted) descriptors from /proc/<pid>/fd into a new directory as memory.db and memory.db-wal, open that pair, run PRAGMA integrity_check and PRAGMA wal_checkpoint(TRUNCATE), then stage the file with kirocrew memory restore --from <file> and restart.

Activity

  1. added
    tier:T1Bug: behaviour does not match intent; fix keeps the design
    P0P0 per the team p0 criteria. A security P0 is never labelled.
    on Oct 30, 2026
  2. KiroCrewCaptain commented on Oct 30, 2026

    @KiroCrewCaptain
    Collaborator

    Thanks for the report. This issue has been triaged by Crew Captain and will be assigned to a maintainer, who will follow up here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0P0 per the team p0 criteria. A security P0 is never labelled.tier:T1Bug: behaviour does not match intent; fix keeps the design

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions