v0.2.0: one index run at a time, fail-closed vault switch, durable cleanup debt

CHANGELOG.md

@@ -14,6 +14,9 @@ A write path, through an inbox you approve.
 - An unchanged note whose mtime or size moved (an identical save, a sync tool) now has them updated without being embedded again, so `puremind_recent` orders it correctly.
 - `puremind_propose` holds a lock across the duplicate check, the 100-proposal cap and the file creation, so two agents proposing at once can no longer pass the cap or save the same proposal twice.
 - The MCP server's instructions and `puremind connect` now say that the server cannot write to the vault but accepts proposals. The test that listed exactly three tools now lists four.
+- Index runs hold a process lock beside the database until they finish. A `puremind index` that overlaps another run is refused with a clear message, and a `puremind watch` run waits for the other run to finish and then checks the model again, so a watch on an old model can no longer write its vectors into an index rebuilt for another model. A read-only index is still reported as read-only, and the lock file is never opened through a symlink.
+- Rebuilding for another vault, or for a vault that moved, first has the model embed a test passage, so a model that cannot run leaves the index as it was. It then clears the old rows and records the new root in one transaction before indexing the notes. If embedding fails after that, searches no longer return the old vault's text labelled as the new vault; results stay empty until new notes are indexed.
+- Replacing or deleting a note records the cleanup owed in the same transaction. A run that crashes leaves that record for the next run, and compaction keeps it until the WAL checkpoint succeeds, including when a reader holds the log open.
 
 ## 0.1.6 (2026-10-22)
 

README.md

@@ -93,7 +93,7 @@ Limits: a title of at most 200 characters, content of at most 20,000, and 100 pr
 - **Store.** One SQLite file, `~/.puremind/index.sqlite`, readable only by you: files and hashes, chunks, an FTS5 keyword index (used by `puremind search --lexical-only`), and float16 vectors. Vectors are loaded into a numpy matrix for dense search.
 - **Retrieval.** BM25 over English-stemmed words (top 50) and Qwen3-Embedding-0.6B cosine search (top 50, 1024 dimensions, with the query instruction prefix) are fused with reciprocal rank fusion (k=60). The bge-reranker-v2-m3 cross-encoder then re-scores the 50 fused candidates, reading the first 800 characters of each. `--no-rerank` skips that last step.
 - **Languages.** The embedder is multilingual, so search by meaning works in most languages. Keyword matching uses an English stemmer. In hybrid search, Chinese, Japanese and Korean text is matched by single characters and overlapping pairs of characters; `search --lexical-only` does not do this, and treats an unspaced run of those characters as one word. Thai, Lao, Khmer and Burmese are not split into words for keyword matching. In scripts that need more than one token per character, such as Telugu, Burmese, Khmer and Ethiopic, a full-length chunk is longer than the 2,048 tokens the embedder reads, so search by meaning misses its end (keyword search does not); raise `embed_max_seq` to about 3,200 if most of your notes are in such a script, then run `puremind index --rebuild`.
-- **Watcher.** `puremind watch` indexes once, then again each time the vault has been quiet for `--debounce` seconds (default 2) after a change to a visible note or folder. Reads, hidden folders and other file types never trigger a run, and runs never overlap. It follows a vault directory that is replaced or made again (for example by a remount or a sync tool) or that cannot be read for a while, and if nothing has happened for 10 minutes it scans anyway, in case a change was missed. A run that fails is reported on one line and watching goes on, except when no later run could succeed (a vault symlink that now names another directory than the index's, or a system limit such as the inotify watch limit): then `puremind watch` stops with an error.
+- **Watcher.** `puremind watch` indexes once, then again each time the vault has been quiet for `--debounce` seconds (default 2) after a change to a visible note or folder. Reads, hidden folders and other file types never trigger a run, and runs never overlap, also not with other processes: a `puremind index` started during a run on the same index stops with an error and changes nothing, and a `puremind watch` run waits for a `puremind index` that is still running, then checks the model again. It follows a vault directory that is replaced or made again (for example by a remount or a sync tool) or that cannot be read for a while, and if nothing has happened for 10 minutes it scans anyway, in case a change was missed. A run that fails is reported on one line and watching goes on, except when no later run could succeed (a vault symlink that now names another directory than the index's, or a system limit such as the inotify watch limit): then `puremind watch` stops with an error.
 
 ## Memory and speed
 
@@ -130,7 +130,7 @@ The reranker costs about 0.2 seconds per candidate on a CPU, so with the default
 
 - **Model revisions.** The default models are pinned to `Qwen/Qwen3-Embedding-0.6B` at `97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3` and `BAAI/bge-reranker-v2-m3` at `953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e`. Another model uses its repository's default branch as it was at the first download, and is not refreshed afterwards unless you set a revision or clear the cache.
 - **Model names.** A name is a Hugging Face model id unless it starts with `/`, `./`, `../` or `~`, which mark a local directory. A directory in your working directory is never picked up by accident. Models load with `trust_remote_code` off.
-- **Changing the embedding model**, `embedder_revision` or `embed_max_seq` takes `puremind index --rebuild`, since vectors from different models are not comparable, and a new window changes the vectors of long chunks. Until then `puremind index` and `puremind watch` stop with an error and change nothing, and searches are refused. A `puremind watch` keeps the model and window it started with, so restart it after the change; one left running stops instead of wiping the rebuilt index. A running `puremind serve` keeps them too, and refuses searches until it is restarted. With `--rebuild`, the new model must first load and embed a test passage, so a mistyped name leaves the existing index untouched. The index is then emptied, and searches find nothing until the run finishes. `puremind index --rebuild` with the same model, by contrast, replaces one note at a time, so searches keep working.
+- **Changing the embedding model**, `embedder_revision` or `embed_max_seq` takes `puremind index --rebuild`, since vectors from different models are not comparable, and a new window changes the vectors of long chunks. Until then `puremind index` and `puremind watch` stop with an error and change nothing, and searches are refused. A `puremind watch` keeps the model and window it started with, so restart it after the change; one left running stops instead of wiping the rebuilt index. A running `puremind serve` keeps them too, and refuses searches until it is restarted. With `--rebuild`, the new model must first load and embed a test passage, so a mistyped name leaves the existing index untouched. The index is then emptied, and searches find nothing until the run finishes. `puremind index --rebuild` with the same model, by contrast, replaces one note at a time, so searches keep working. The exception is a vault at another path (one that moved, or another vault), which the index cannot tell apart: its rebuild also checks the model first, then empties the index.
 - **The reranker window.** 800 characters is the setting the benchmark below was measured with. Raising `rerank_chars` lets the reranker see more of each passage, at the cost of slower searches.
 - **Indexes from 0.1.1 or earlier** are brought up to date by the next `puremind index` or `puremind watch` run, which cuts and embeds every note again. It takes about as long as the first index did, and searches keep working meanwhile.
 

src/puremind_mac/cli.py

@@ -139,7 +139,8 @@ def run() -> None:  # always called on this thread, one run at a time
         purge = first[0] or (owed_since[0] is not None and now - owed_since[0] >= PURGE_AT_LEAST_EVERY)
         first[0] = False
         try:
-            stats = index_vault(vault, svc.store, svc.embedder, purge=purge)
+            # A `puremind index` still running is waited for; this run then checks the model again.
+            stats = index_vault(vault, svc.store, svc.embedder, purge=purge, wait=True)
         except (VaultMismatch, ModelMismatch) as exc:
             # A symlinked vault that now names another directory, or an index rebuilt for another
             # model since this watch read its config: every later run would fail the same way.

src/puremind_mac/indexer.py

@@ -3,13 +3,15 @@
 from __future__ import annotations
 
 import errno
+import fcntl
 import hashlib
 import logging
 import os
 import re
 import sqlite3
 import stat
 from collections.abc import Callable, Sequence
+from contextlib import contextmanager
 from dataclasses import dataclass, field
 from pathlib import Path
 
@@ -46,7 +48,7 @@
 
 
 class EmbedderError(RuntimeError):
-    """The embedding model could not be loaded or failed. The index is left as it was."""
+    """The embedding model could not be loaded or failed."""
 
 
 class VaultMismatch(ValueError):
@@ -454,6 +456,43 @@ def _same_directory(known: str, vault: Path) -> bool:
         return False
 
 
+def _flock(fd: int, path: str, *, block: bool) -> bool:
+    """Take the exclusive lock on ``fd``; False if another run holds it and ``block`` is off."""
+    try:
+        fcntl.flock(fd, fcntl.LOCK_EX if block else fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except BlockingIOError:
+        return False
+    except OSError as exc:  # e.g. a file system without locks: running unlocked could interleave two runs
+        raise StoreError(f"cannot lock {path}: {_one_line(exc)}") from exc
+    return True
+
+
+@contextmanager
+def _index_lock(store: Store, *, wait: bool = False):
+    """Hold ``<db>.lock`` for one run. Another run is refused, or with ``wait`` waited for.
+
+    The kernel drops the lock when its holder exits, however that happens, so none is left stale.
+    """
+    if store.path == ":memory:":
+        yield
+        return
+    store._check_writable()  # a read-only index says so, instead of failing on its lock file
+    path = str(Path(store.path).resolve()) + ".lock"
+    try:
+        fd = os.open(path, os.O_CREAT | os.O_RDWR | getattr(os, "O_NOFOLLOW", 0), 0o600)
+    except OSError as exc:
+        raise StoreError(f"cannot open the index lock {path}: {_one_line(exc)}") from exc
+    try:
+        if not _flock(fd, path, block=False):
+            if not wait:
+                raise StoreError(f"another index run holds {path}; try again after it finishes")
+            log.warning("another index run holds %s; waiting for it to finish", path)
+            _flock(fd, path, block=True)
+        yield
+    finally:
+        os.close(fd)  # keep the sidecar: unlinking it would let writers lock different inodes
+
+
 def index_vault(
     vault: Path,
     store: Store,
@@ -462,29 +501,49 @@ def index_vault(
     *,
     should_stop: Callable[[], bool] | None = None,
     purge: bool = True,
+    wait: bool = False,
 ) -> IndexStats:
     """Bring the index in line with the vault.
 
     ``rebuild`` embeds every note again, replacing each one's rows in turn, so searches keep
     finding the old version meanwhile. A new embedding model, revision or configuration is refused
     with ``ModelMismatch`` unless ``rebuild`` is set, and then discards the index first, since
-    vectors from two models cannot be searched together. Notes cut by an earlier chunker are cut
+    vectors from two models cannot be searched together. A vault at another path (another vault,
+    or the same one moved) is refused with ``VaultMismatch`` unless ``rebuild`` is set, and then
+    also discards the old rows, recording the new root in the same transaction. Either way the
+    model must first embed a test passage, so one that cannot run leaves the index as it was.
+    One run at a time works on a database: another run is refused with ``StoreError``, or with
+    ``wait`` waited for. Notes cut by an earlier chunker are cut
     and embedded again. ``should_stop`` is polled between notes; once it returns True the run
     ends early without removing anything.
 
     Text a run removes or replaces leaves the tables at once. Its words leave the keyword index
     when the index is compacted (``Store.purge``), which takes about half a second at 50,000
     chunks. With ``purge`` the run compacts at its end, also paying a compaction an earlier run
-    put off; without it, the run only records that one is owed (``stats.purge_deferred``) and the
+    put off; without it, the debt recorded with each write stays (``stats.purge_deferred``) and the
     caller pays it later, as ``puremind watch`` does.
     """
+    with _index_lock(store, wait=wait):
+        return _index_vault(vault, store, embedder, rebuild, should_stop=should_stop, purge=purge)
+
+
+def _index_vault(
+    vault: Path,
+    store: Store,
+    embedder: Embedder,
+    rebuild: bool,
+    *,
+    should_stop: Callable[[], bool] | None,
+    purge: bool,
+) -> IndexStats:
     vault = vault.expanduser().resolve()
     if not vault.is_dir():
         raise NotADirectoryError(f"vault directory not found: {vault}")
     stats = IndexStats()
 
     known_root = store.get_meta("vault_root")
-    if not rebuild and known_root and not _same_directory(known_root, vault) and store.stats()["files"]:
+    root_changed = known_root is not None and not _same_directory(known_root, vault)
+    if not rebuild and root_changed and store.stats()["files"]:
         raise VaultMismatch(
             f"this index belongs to {known_root}, not {vault}; use a different PUREMIND_DB "
             "(or db = in config.toml) for this vault, or run `puremind index --rebuild` if the vault moved"
@@ -499,8 +558,8 @@ def index_vault(
     )
     config_changed = known_model is not None and known_config is not None and known_config != config
     changed = model_changed or config_changed
-    if changed and store.stats()["files"]:
-        if not rebuild:  # else a `watch` left on the old config and a fresh `index` wipe each other in turn
+    if (changed or root_changed) and store.stats()["files"]:  # the rows are about to be discarded
+        if changed and not rebuild:  # else a `watch` left on the old config and a fresh `index` wipe each other in turn
             if model_changed:  # the larger change; a new window alongside it needs the same rebuild
                 built = model_label(known_model) + (f" at revision {known_rev}" if known_rev else "")
                 wanted = model_label(embedder.name) + (f" at revision {revision}" if revision else "")
@@ -516,7 +575,8 @@ def index_vault(
             "embedding model or configuration changed (%s%s -> %s%s); rebuilding the index",
             known_model, f" at {known_rev}" if known_rev else "", embedder.name, f" at {revision}" if revision else "",
         )
-        store.clear()
+    if changed or root_changed:
+        store.clear(vault_root=str(vault))
     store.set_meta("embedder", model_key(embedder.name))
     store.set_meta("embedder_revision", revision)
     store.set_meta("embedding_config", config)
@@ -548,14 +608,12 @@ def index_vault(
             store.delete_file(rel)
             stats.removed += 1
     emptied = True
-    if stats.removed or stats.updated:  # clear() purges by itself
+    if store.purge_pending():
         if purge:
             emptied = store.purge()
         else:
             emptied = store.defer_purge()
             stats.purge_deferred = True
-    elif purge and store.purge_pending():  # owed by a `puremind watch` that stopped before paying it
-        emptied = store.purge()
     if not emptied:
         log.warning(
             "another process was reading the index, so text deleted or replaced in this run stays in "