diff --git a/CHANGELOG.md b/CHANGELOG.md index 2370635..b857532 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,10 @@ A write path, through an inbox you approve. - `puremind serve --read-only`, or `[inbox] enabled = false` in `config.toml`, leaves out `puremind_propose`, so the server has only its three read-only tools. `[inbox] folder` sets where approved notes go. `puremind init --force` keeps both settings. - A Homebrew formula in `packaging/homebrew/puremind.rb`, for use from a tap of your own (no public tap yet). - README: results by question type for the benchmark (Recall@5 and judged QA for each of the six LongMemEval types). +- `puremind_search` refuses a query over 4,096 characters, in the MCP schema and in the handler, instead of tokenizing it and echoing it back. +- The index records its embedding window (`embed_max_seq`). Changing the window is refused like a model change, until `puremind index --rebuild`, so long notes no longer keep vectors cut at the old window. The error names the recorded and the configured window, and a model change made at the same time is reported as a model change. An index that never recorded the window adopts the current one without a rebuild. The README and the `config.toml` template say that changing `embed_max_seq` takes a rebuild. +- An unchanged note whose mtime or size moved (an identical save, a sync tool) now has them updated without being embedded again, so `puremind_recent` orders it correctly. +- `puremind_propose` holds a lock across the duplicate check, the 100-proposal cap and the file creation, so two agents proposing at once can no longer pass the cap or save the same proposal twice. - The MCP server's instructions and `puremind connect` now say that the server cannot write to the vault but accepts proposals. The test that listed exactly three tools now lists four. ## 0.1.6 (2026-10-22) diff --git a/README.md b/README.md index 415e4b8..8e1a6c5 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ The MCP server offers four tools. Three only read your vault and are marked read | Tool | Purpose | |---|---| -| `puremind_search(query, limit=10)` | Hybrid search. Returns up to 50 matching passages, best first, each with its note path, heading path and score. | +| `puremind_search(query, limit=10)` | Hybrid search. Returns up to 50 matching passages, best first, each with its note path, heading path and score. A query is at most 4,096 characters. | | `puremind_get(path)` | Reads one note by the path search returned. It serves only the notes the indexer reads (see Vault below) and cuts text beyond 200,000 characters, saying so. | | `puremind_recent(days=7, limit=20)` | Lists the notes modified in the last `days` days, newest first, as of the last index run. | | `puremind_propose(title, content)` | Proposes a memory. It is saved in an inbox outside the vault and does nothing else until you approve it. | @@ -92,7 +92,7 @@ Limits: a title of at most 200 characters, content of at most 20,000, and 100 pr - **Chunker.** Heading-aware: splits on `#` to `####` and keeps a breadcrumb such as `Project > Setup`. Headings inside fenced code blocks and YAML frontmatter do not count. A heading with no text of its own joins the subsection under it, and most sections under 100 characters merge with the one before. A section longer than 2,048 characters is cut into near-even windows with about 20% overlap, preferably at blank lines. A code block or frontmatter that fits in a window is usually kept whole, but one that is longer, or that starts late in a long section, can be cut; every part of a cut block keeps its own opening and closing fence lines. Frontmatter of 100 characters or more is a chunk of its own. - **Store.** One SQLite file, `~/.puremind/index.sqlite`, readable only by you: files and hashes, chunks, an FTS5 keyword index (used by `puremind search --lexical-only`), and float16 vectors. Vectors are loaded into a numpy matrix for dense search. - **Retrieval.** BM25 over English-stemmed words (top 50) and Qwen3-Embedding-0.6B cosine search (top 50, 1024 dimensions, with the query instruction prefix) are fused with reciprocal rank fusion (k=60). The bge-reranker-v2-m3 cross-encoder then re-scores the 50 fused candidates, reading the first 800 characters of each. `--no-rerank` skips that last step. -- **Languages.** The embedder is multilingual, so search by meaning works in most languages. Keyword matching uses an English stemmer. In hybrid search, Chinese, Japanese and Korean text is matched by single characters and overlapping pairs of characters; `search --lexical-only` does not do this, and treats an unspaced run of those characters as one word. Thai, Lao, Khmer and Burmese are not split into words for keyword matching. In scripts that need more than one token per character, such as Telugu, Burmese, Khmer and Ethiopic, a full-length chunk is longer than the 2,048 tokens the embedder reads, so search by meaning misses its end (keyword search does not); raise `embed_max_seq` to about 3,200 if most of your notes are in such a script. +- **Languages.** The embedder is multilingual, so search by meaning works in most languages. Keyword matching uses an English stemmer. In hybrid search, Chinese, Japanese and Korean text is matched by single characters and overlapping pairs of characters; `search --lexical-only` does not do this, and treats an unspaced run of those characters as one word. Thai, Lao, Khmer and Burmese are not split into words for keyword matching. In scripts that need more than one token per character, such as Telugu, Burmese, Khmer and Ethiopic, a full-length chunk is longer than the 2,048 tokens the embedder reads, so search by meaning misses its end (keyword search does not); raise `embed_max_seq` to about 3,200 if most of your notes are in such a script, then run `puremind index --rebuild`. - **Watcher.** `puremind watch` indexes once, then again each time the vault has been quiet for `--debounce` seconds (default 2) after a change to a visible note or folder. Reads, hidden folders and other file types never trigger a run, and runs never overlap. It follows a vault directory that is replaced or made again (for example by a remount or a sync tool) or that cannot be read for a while, and if nothing has happened for 10 minutes it scans anyway, in case a change was missed. A run that fails is reported on one line and watching goes on, except when no later run could succeed (a vault symlink that now names another directory than the index's, or a system limit such as the inotify watch limit): then `puremind watch` stops with an error. ## Memory and speed @@ -130,7 +130,7 @@ The reranker costs about 0.2 seconds per candidate on a CPU, so with the default - **Model revisions.** The default models are pinned to `Qwen/Qwen3-Embedding-0.6B` at `97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3` and `BAAI/bge-reranker-v2-m3` at `953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e`. Another model uses its repository's default branch as it was at the first download, and is not refreshed afterwards unless you set a revision or clear the cache. - **Model names.** A name is a Hugging Face model id unless it starts with `/`, `./`, `../` or `~`, which mark a local directory. A directory in your working directory is never picked up by accident. Models load with `trust_remote_code` off. -- **Changing the embedding model**, or `embedder_revision`, takes `puremind index --rebuild`, since vectors from different models are not comparable. Until then `puremind index` and `puremind watch` stop with an error and change nothing, and searches are refused. A `puremind watch` keeps the model it started with, so restart it after the change; one left running stops instead of wiping the rebuilt index. With `--rebuild`, the new model must first load and embed a test passage, so a mistyped name leaves the existing index untouched. The index is then emptied, and searches find nothing until the run finishes. `puremind index --rebuild` with the same model, by contrast, replaces one note at a time, so searches keep working. +- **Changing the embedding model**, `embedder_revision` or `embed_max_seq` takes `puremind index --rebuild`, since vectors from different models are not comparable, and a new window changes the vectors of long chunks. Until then `puremind index` and `puremind watch` stop with an error and change nothing, and searches are refused. A `puremind watch` keeps the model and window it started with, so restart it after the change; one left running stops instead of wiping the rebuilt index. A running `puremind serve` keeps them too, and refuses searches until it is restarted. With `--rebuild`, the new model must first load and embed a test passage, so a mistyped name leaves the existing index untouched. The index is then emptied, and searches find nothing until the run finishes. `puremind index --rebuild` with the same model, by contrast, replaces one note at a time, so searches keep working. - **The reranker window.** 800 characters is the setting the benchmark below was measured with. Raising `rerank_chars` lets the reranker see more of each passage, at the cost of slower searches. - **Indexes from 0.1.1 or earlier** are brought up to date by the next `puremind index` or `puremind watch` run, which cuts and embeds every note again. It takes about as long as the first index did, and searches keep working meanwhile. @@ -147,7 +147,7 @@ The index is a second copy of your notes' text. It is created readable only by y | Command | What it does | |---|---| | `puremind init [--force]` | Writes `config.toml`. | -| `puremind index [vault_dir] [--rebuild]` | Indexes the vault incrementally; `--rebuild` embeds every note again, and is needed after changing the embedding model. | +| `puremind index [vault_dir] [--rebuild]` | Indexes the vault incrementally; `--rebuild` embeds every note again, and is needed after changing the embedding model or `embed_max_seq`. | | `puremind watch [vault_dir] [--debounce SECONDS]` | Indexes, then re-indexes when notes change. | | `puremind search [--limit N] [--no-rerank] [--lexical-only]` | Searches from the terminal; `--limit` is 1 to 50. | | `puremind serve [--no-rerank] [--read-only]` | Runs the MCP server on stdio. Your agent starts this for you. `--read-only` leaves out `puremind_propose`. | diff --git a/src/puremind_mac/config.py b/src/puremind_mac/config.py index 17c276a..8113f70 100644 --- a/src/puremind_mac/config.py +++ b/src/puremind_mac/config.py @@ -277,6 +277,7 @@ def render_config(vault: Path, cfg: Config) -> str: '# "auto" picks mps on Apple Silicon, cuda if present, else cpu.', f"device = {toml_string(cfg.device)}", "# Tokens per chunk the embedder reads; a chunk is at most 2,048 characters.", + "# Changing it takes `puremind index --rebuild`.", f"embed_max_seq = {cfg.embed_max_seq}", "", "[retrieval]", diff --git a/src/puremind_mac/inbox.py b/src/puremind_mac/inbox.py index 717ace1..2aba4f2 100644 --- a/src/puremind_mac/inbox.py +++ b/src/puremind_mac/inbox.py @@ -9,6 +9,7 @@ from __future__ import annotations import errno +import fcntl import json import os import re @@ -128,35 +129,38 @@ def propose(home: Path, title: str, content: str) -> tuple[Proposal, bool]: root = inbox_dir(home) _private_dir(home, create=True) _private_dir(root, create=True) - pending, unreadable = list_proposals(home) - for p in pending: - if p.title == title and p.content == content: - return p, False - if len(pending) + len(unreadable) >= MAX_PENDING: - raise InboxError( - f"the inbox already holds {MAX_PENDING} proposals; ask the user to review them with `puremind inbox`" - ) - now = datetime.now(UTC) - for _ in range(8): - pid = f"{now:%Y%m%d}-{secrets.token_hex(3)}" - proposal = Proposal(pid, now.strftime("%Y-%m-%dT%H:%M:%SZ"), title, content) - try: - fd = os.open(root / f"{pid}.json", os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_NOFOLLOW", 0), 0o600) - except FileExistsError: - continue - # Written in place: a reader that meets the half-written file counts it as unreadable, and - # the writer is a short-lived server process that holds the whole text already. - try: - with os.fdopen(fd, "w", encoding="utf-8") as fh: - json.dump(proposal.__dict__, fh) - except BaseException: + flags = os.O_RDWR | os.O_CREAT | getattr(os, "O_NOFOLLOW", 0) + with os.fdopen(os.open(root / ".propose.lock", flags, 0o600), "a") as lock: + fcntl.flock(lock.fileno(), fcntl.LOCK_EX) + pending, unreadable = list_proposals(home) + for p in pending: + if p.title == title and p.content == content: + return p, False + if len(pending) + len(unreadable) >= MAX_PENDING: + raise InboxError( + f"the inbox already holds {MAX_PENDING} proposals; ask the user to review them with `puremind inbox`" + ) + now = datetime.now(UTC) + for _ in range(8): + pid = f"{now:%Y%m%d}-{secrets.token_hex(3)}" + proposal = Proposal(pid, now.strftime("%Y-%m-%dT%H:%M:%SZ"), title, content) try: - os.unlink(root / f"{pid}.json") - except OSError: - pass - raise - return proposal, True - raise InboxError("could not save the proposal; try again") + fd = os.open(root / f"{pid}.json", os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_NOFOLLOW", 0), 0o600) + except FileExistsError: + continue + # Written in place: a reader that meets the half-written file counts it as unreadable, and + # the writer is a short-lived server process that holds the whole text already. + try: + with os.fdopen(fd, "w", encoding="utf-8") as fh: + json.dump(proposal.__dict__, fh) + except BaseException: + try: + os.unlink(root / f"{pid}.json") + except OSError: + pass + raise + return proposal, True + raise InboxError("could not save the proposal; try again") def find(home: Path, ref: str) -> Proposal: diff --git a/src/puremind_mac/indexer.py b/src/puremind_mac/indexer.py index 873911f..2448e02 100644 --- a/src/puremind_mac/indexer.py +++ b/src/puremind_mac/indexer.py @@ -22,7 +22,15 @@ frontmatter_end, ) from .chunker import VERSION as CHUNKER_VERSION -from .models import Embedder, ModelLoadError, embedder_revision, model_key, model_label +from .models import ( + Embedder, + ModelLoadError, + embedder_revision, + embedding_config, + embedding_config_label, + model_key, + model_label, +) from .store import Store, StoreError log = logging.getLogger("puremind.index") @@ -46,7 +54,7 @@ class VaultMismatch(ValueError): class ModelMismatch(ValueError): - """The index was built with another embedding model or revision, and no rebuild was asked for. + """The index was built with another embedding model, revision or configuration, and no rebuild was asked for. Nothing was changed.""" @@ -285,10 +293,16 @@ def __init__(self, vault: Path, store: Store, embedder: Embedder, stats: IndexSt # embedded again, and none cut by an earlier chunker. current = set() if every else store.chunked_with(CHUNKER_VERSION) self.keep = {rel: digest for rel, digest in self.old.items() if rel in current} + self.stat = {row.path: (row.mtime, row.size) for row in store.list_files()} self.seen: set[str] = set() self.queue: list[_Job] = [] self.queued = 0 + def refresh_stat(self, rel: str, digest: str, st: os.stat_result) -> None: + """Keep an unchanged note's mtime and size current for ``puremind_recent``, writing only when they moved.""" + if self.stat.get(rel) != (st.st_mtime, st.st_size): + self.store.update_file_metadata(rel, digest, st.st_mtime, st.st_size) + def fail(self, rel: str, why: str) -> None: self.stats.skipped.append(f"{rel} ({why})") @@ -332,6 +346,7 @@ def visit(self, path: Path) -> None: self.seen.add(rel) digest = hashlib.sha256(raw).hexdigest() if self.keep.get(rel) == digest: + self.refresh_stat(rel, digest, st) self.stats.unchanged += 1 return try: @@ -398,6 +413,7 @@ def commit(self, job: _Job, vectors) -> None: if digest == job.digest: break if self.keep.get(job.rel) == digest: + self.refresh_stat(job.rel, digest, st) self.stats.unchanged += 1 # edited back to the indexed text return if job.attempt >= RECHECK_ATTEMPTS: @@ -450,7 +466,7 @@ def index_vault( """Bring the index in line with the vault. ``rebuild`` embeds every note again, replacing each one's rows in turn, so searches keep - finding the old version meanwhile. A new embedding model, or a new revision of it, is refused + finding the old version meanwhile. A new embedding model, revision or configuration is refused with ``ModelMismatch`` unless ``rebuild`` is set, and then discards the index first, since vectors from two models cannot be searched together. Notes cut by an earlier chunker are cut and embedded again. ``should_stop`` is polled between notes; once it returns True the run @@ -476,23 +492,34 @@ def index_vault( known_model = store.get_meta("embedder") known_rev = store.get_meta("embedder_revision") # None in an index from 0.1.1, which pinned none revision = embedder_revision(embedder) - changed = known_model is not None and ( + config = embedding_config(embedder) + known_config = store.get_meta("embedding_config") # None in an index built before the window was recorded + model_changed = known_model is not None and ( model_key(known_model) != model_key(embedder.name) or (known_rev is not None and known_rev != revision) ) + config_changed = known_model is not None and known_config is not None and known_config != config + changed = model_changed or config_changed if changed and store.stats()["files"]: if not rebuild: # else a `watch` left on the old config and a fresh `index` wipe each other in turn - built = model_label(known_model) + (f" at revision {known_rev}" if known_rev else "") - wanted = model_label(embedder.name) + (f" at revision {revision}" if revision else "") - raise ModelMismatch(f"the index was built with {built} but {wanted} is configured; nothing was changed") + if model_changed: # the larger change; a new window alongside it needs the same rebuild + built = model_label(known_model) + (f" at revision {known_rev}" if known_rev else "") + wanted = model_label(embedder.name) + (f" at revision {revision}" if revision else "") + raise ModelMismatch(f"the index was built with {built} but {wanted} is configured; nothing was changed") + raise ModelMismatch( + f"the index was built with another embedding configuration ({embedding_config_label(known_config)}, " + f"but {embedding_config_label(config)} is configured); run `puremind index --rebuild`; " + "nothing was changed" + ) _embed(embedder, [PROBE_TEXT]) # a model that cannot run must not cost the user an index if changed: log.warning( - "embedder changed (%s%s -> %s%s); rebuilding the index", + "embedding model or configuration changed (%s%s -> %s%s); rebuilding the index", known_model, f" at {known_rev}" if known_rev else "", embedder.name, f" at {revision}" if revision else "", ) store.clear() store.set_meta("embedder", model_key(embedder.name)) store.set_meta("embedder_revision", revision) + store.set_meta("embedding_config", config) store.set_meta("vault_root", str(vault)) scan = _scan(vault) diff --git a/src/puremind_mac/models.py b/src/puremind_mac/models.py index e9f615e..711dc64 100644 --- a/src/puremind_mac/models.py +++ b/src/puremind_mac/models.py @@ -128,6 +128,22 @@ def embedder_revision(embedder: object) -> str: return getattr(embedder, "revision", None) or "" +def embedding_config(embedder: object) -> str: + """Fingerprint of vector-affecting settings, alongside the model and revision metadata.""" + return json.dumps({"embed_max_seq": getattr(embedder, "max_seq", None)}, sort_keys=True) + + +def embedding_config_label(config: str | None) -> str: + """The settings an ``embedding_config`` fingerprint records, for messages: ``embed_max_seq 2048``.""" + try: + values = json.loads(config) if config else None + except ValueError: + values = None + if not isinstance(values, dict) or not values: + return "an unrecorded configuration" + return ", ".join(f"{key} {'(unset)' if value is None else value}" for key, value in sorted(values.items())) + + def _offline_switch() -> str | None: """The Hugging Face offline variable that is set, if any.""" for name in ("HF_HUB_OFFLINE", "TRANSFORMERS_OFFLINE"): @@ -351,6 +367,10 @@ def __init__( # are not safe for concurrent forward passes on one model. self._busy = threading.Lock() + @property + def max_seq(self) -> int: + return self._max_seq + def _open(self): path = resolve_model(self.name, self._revision, namespace="sentence-transformers") try: diff --git a/src/puremind_mac/server.py b/src/puremind_mac/server.py index 85dcc41..d1aecbd 100644 --- a/src/puremind_mac/server.py +++ b/src/puremind_mac/server.py @@ -27,7 +27,9 @@ # puremind_propose writes a file, but only into pureMind's own inbox, never into the user's notes. PROPOSE = ToolAnnotations(readOnlyHint=False, destructiveHint=False, idempotentHint=False, openWorldHint=False) -Query = Annotated[str, Field(description="Keywords or a natural-language question.")] +Query = Annotated[ + str, Field(max_length=tools.MAX_QUERY_CHARS, description="Keywords or a natural-language question.") +] Limit = Annotated[ int, Field(ge=1, le=tools.MAX_LIMIT, description=f"Most passages to return, 1 to {tools.MAX_LIMIT}.") ] diff --git a/src/puremind_mac/service.py b/src/puremind_mac/service.py index d9568bc..195508f 100644 --- a/src/puremind_mac/service.py +++ b/src/puremind_mac/service.py @@ -12,6 +12,8 @@ Reranker, SentenceTransformerEmbedder, embedder_revision, + embedding_config, + embedding_config_label, model_key, model_label, ) @@ -95,6 +97,13 @@ def search(self, query: str, limit: int = 10, rerank: bool = True) -> list[Hit]: f"the index was built with {model_label(built_with)} at revision {built_rev or '(none)'} but " f"revision {revision or '(none)'} is configured; run `puremind index --rebuild`" ) + built_config = self.store.get_meta("embedding_config") # None in an index built before the window was recorded + config = embedding_config(self.embedder) + if built_with and built_config is not None and built_config != config: + raise IndexMismatch( + f"the index was built with another embedding configuration ({embedding_config_label(built_config)}, " + f"but {embedding_config_label(config)} is configured); run `puremind index --rebuild`" + ) snap = self._loaded() if not len(snap.index): return [] diff --git a/src/puremind_mac/store.py b/src/puremind_mac/store.py index b77b1ea..7e182d8 100644 --- a/src/puremind_mac/store.py +++ b/src/puremind_mac/store.py @@ -470,6 +470,15 @@ def replace_file( ) self._bump() + def update_file_metadata(self, path: str, sha256: str, mtime: float, size: int) -> None: + """Refresh an unchanged note's stat only while its indexed content still matches.""" + self._check_writable() + with self.db: + self.db.execute( + "UPDATE files SET mtime = ?, size = ? WHERE path = ? AND sha256 = ?", + (mtime, size, path, sha256), + ) + def delete_file(self, path: str) -> None: self._check_writable() with self.db: diff --git a/src/puremind_mac/tools.py b/src/puremind_mac/tools.py index e5c3b42..be0b362 100644 --- a/src/puremind_mac/tools.py +++ b/src/puremind_mac/tools.py @@ -30,6 +30,7 @@ log = logging.getLogger("puremind.tools") MAX_LIMIT = 50 +MAX_QUERY_CHARS = 4096 MAX_GET_CHARS = 200_000 MAX_RECENT_LIMIT = 200 MAX_RECENT_DAYS = 3650 @@ -86,6 +87,8 @@ def vault_mismatch(service: Service) -> tuple[Path, Path] | None: def puremind_search(service: Service, query: str, limit: int = 10) -> str: + if query and len(query) > MAX_QUERY_CHARS: + raise ToolRefused(f"the query is longer than {MAX_QUERY_CHARS} characters") query = (query or "").strip() if not query: raise ToolRefused("query is empty") diff --git a/tests/test_inbox.py b/tests/test_inbox.py index 5e1837b..727363a 100644 --- a/tests/test_inbox.py +++ b/tests/test_inbox.py @@ -1,10 +1,14 @@ """The inbox: proposals from an agent, and the approval that turns one into a note.""" import dataclasses +import fcntl import io import json import os +import select import stat +import subprocess +import sys import pytest @@ -42,6 +46,104 @@ def test_the_same_proposal_twice_is_kept_once(home): assert len(inbox.list_proposals(home)[0]) == 1 +@pytest.mark.parametrize("same", [False, True]) +def test_concurrent_processes_deduplicate_and_obey_the_cap(home, same): + home.mkdir() + (home / "inbox").mkdir() + script = """ +import sys, time +from pathlib import Path +from puremind_mac import inbox +inbox.MAX_PENDING = 1 +original = inbox.list_proposals +def slow_list(home): + result = original(home) + time.sleep(0.1) + return result +inbox.list_proposals = slow_list +print('ready', flush=True) +sys.stdin.readline() +try: + proposal, new = inbox.propose(Path(sys.argv[1]), sys.argv[2], 'content') + print(proposal.id, new, flush=True) +except inbox.InboxError: + print('full', flush=True) +""" + children = [] + try: + for n in range(4): + children.append(subprocess.Popen( + [sys.executable, "-c", script, str(home), "same" if same else f"note {n}"], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + )) + for child in children: + assert select.select([child.stdout], [], [], 10)[0] + assert child.stdout.readline().strip() == "ready" + for child in children: + child.stdin.write("go\n") + child.stdin.flush() + results = [] + for child in children: + out, err = child.communicate(timeout=10) + assert child.returncode == 0, err + results.append(out.strip()) + pending, unreadable = inbox.list_proposals(home) + assert len(pending) == 1 and not unreadable + assert results.count(f"{pending[0].id} True") == 1 + assert results.count(f"{pending[0].id} False" if same else "full") == 3 + finally: + for child in children: + if child.poll() is None: + child.kill() + child.communicate(timeout=10) + + +def test_proposing_waits_for_the_sidecar_lock_before_checking_the_inbox(home): + made(home) + with (home / "inbox" / ".propose.lock").open("a") as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + script = """ +import sys +from pathlib import Path +from puremind_mac import inbox +original = inbox.list_proposals +def checked(home): + print('checked', flush=True) + return original(home) +inbox.list_proposals = checked +print('ready', flush=True) +inbox.propose(Path(sys.argv[1]), 'other', 'content') +""" + child = subprocess.Popen( + [sys.executable, "-c", script, str(home)], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + ) + try: + assert select.select([child.stdout], [], [], 10)[0] + assert child.stdout.readline().strip() == "ready" + assert not select.select([child.stdout], [], [], 0.2)[0] + fcntl.flock(lock, fcntl.LOCK_UN) + out, err = child.communicate(timeout=10) + assert child.returncode == 0 and out.strip() == "checked", err + finally: + if child.poll() is None: + child.kill() + child.communicate(timeout=10) + + +def test_failed_creation_releases_the_lock(home, monkeypatch): + def broken(*args): + raise OSError("write failed") + + with monkeypatch.context() as patch: + patch.setattr(inbox.json, "dump", broken) + with pytest.raises(OSError, match="write failed"): + made(home) + with (home / "inbox" / ".propose.lock").open("a") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + assert inbox.list_proposals(home) == ([], []) + assert made(home) + + @pytest.mark.parametrize( ("title", "content", "message"), [ diff --git a/tests/test_indexer.py b/tests/test_indexer.py index b621e2f..69e8ef4 100644 --- a/tests/test_indexer.py +++ b/tests/test_indexer.py @@ -51,11 +51,21 @@ def test_second_index_is_a_no_op_and_embeds_nothing(vault, store, embedder): assert embedder.texts_embedded == embedded -def test_touching_without_changing_content_does_not_reindex(vault, store, embedder): +def test_touching_without_changing_content_refreshes_recent_without_reembedding(vault, store, embedder, service): index_vault(vault, store, embedder) + before = embedder.texts_embedded + chunks, vectors = store.load_index() p = vault / "projects" / "garden.md" - p.write_text(p.read_text()) + touched = time.time() + 60 + os.utime(p, (touched, touched)) assert index_vault(vault, store, embedder).unchanged == 3 + row = store.recent(0, 1)[0] + assert (row.path, row.mtime, row.size) == ("projects/garden.md", p.stat().st_mtime, p.stat().st_size) + assert embedder.texts_embedded == before and store.load_index()[0] == chunks + np.testing.assert_array_equal(store.load_index()[1], vectors) + from puremind_mac.tools import puremind_recent + + assert puremind_recent(service).splitlines()[1].startswith("- projects/garden.md") def test_only_changed_file_is_reembedded(vault, store, embedder): @@ -546,12 +556,87 @@ def test_an_index_without_a_recorded_revision_is_kept(vault, store, embedder): assert index_vault(vault, store, embedder).unchanged == 3 +def test_a_changed_embedding_window_requires_rebuild(vault, store, embedder): + embedder.max_seq = 2048 + index_vault(vault, store, embedder) + fingerprint = store.get_meta("embedding_config") + assert '"embed_max_seq": 2048' in fingerprint + before = embedder.texts_embedded + embedder.max_seq = 1024 + with pytest.raises( + ModelMismatch, match=r"embedding configuration \(embed_max_seq 2048, but embed_max_seq 1024 is configured\).*rebuild" + ): + index_vault(vault, store, embedder) + assert store.get_meta("embedding_config") == fingerprint + assert store.stats()["files"] == 3 and embedder.texts_embedded == before + stats = index_vault(vault, store, embedder, rebuild=True) + assert (stats.added, stats.unchanged) == (3, 0) + assert embedder.texts_embedded == before + stats.chunks + 1 # probe before clearing + assert index_vault(vault, store, embedder).unchanged == 3 + + +def test_a_new_model_with_a_new_window_is_reported_as_a_model_change(vault, store, embedder): + embedder.max_seq = 2048 + index_vault(vault, store, embedder) + embedder.name, embedder.max_seq = "another-model", 1024 + with pytest.raises(ModelMismatch, match="built with fake-embedder but another-model is configured; nothing was"): + index_vault(vault, store, embedder) + assert store.get_meta("embedder") == "fake-embedder" and store.stats()["files"] == 3 + + +def test_an_index_that_never_recorded_the_window_adopts_it(vault, store, embedder): + embedder.max_seq = 2048 + index_vault(vault, store, embedder) + fingerprint = store.get_meta("embedding_config") + with store.db: + store.db.execute("DELETE FROM meta WHERE key = 'embedding_config'") + before = embedder.texts_embedded + assert index_vault(vault, store, embedder).unchanged == 3 # upgrading must not force a rebuild + assert embedder.texts_embedded == before + assert store.get_meta("embedding_config") == fingerprint + + +def test_unchanged_notes_write_their_stat_only_when_it_moved(vault, store, embedder, monkeypatch): + index_vault(vault, store, embedder) + calls = [] + real = store.update_file_metadata + monkeypatch.setattr(store, "update_file_metadata", lambda *a: calls.append(a[0]) or real(*a)) + assert index_vault(vault, store, embedder).unchanged == 3 + assert calls == [] + rel = min(store.file_hashes()) + note = vault / rel + os.utime(note, (note.stat().st_atime, note.stat().st_mtime + 60)) + assert index_vault(vault, store, embedder).unchanged == 3 + assert calls == [rel] + + def test_a_local_model_path_spelled_with_a_tilde_matches_its_absolute_form(vault, store, embedder): index_vault(vault, store, _Wrapped(embedder, name="~/models/embedder")) stats = index_vault(vault, store, _Wrapped(embedder, name=os.path.expanduser("~/models/embedder"))) assert stats.unchanged == stats.scanned == 3 # the same model: nothing is cleared or embedded again +def test_a_failed_window_change_rebuild_preserves_the_index(vault, store, embedder, monkeypatch): + embedder.max_seq = 2048 + index_vault(vault, store, embedder) + fingerprint = store.get_meta("embedding_config") + chunks, vectors = store.load_index() + embedder.max_seq = 1024 + monkeypatch.setattr(embedder, "embed_documents", _boom) + with pytest.raises(EmbedderError, match="could not load the model"): + index_vault(vault, store, embedder, rebuild=True) + assert store.get_meta("embedding_config") == fingerprint + assert store.load_index()[0] == chunks + np.testing.assert_array_equal(store.load_index()[1], vectors) + + +def test_an_empty_index_can_adopt_the_embedding_config(vault, store, embedder): + store.set_meta("embedder", embedder.name) + embedder.max_seq = 1024 + assert index_vault(vault, store, embedder).added == 3 + assert store.get_meta("embedding_config") == '{"embed_max_seq": 1024}' + + @pytest.mark.skipif(not hasattr(os, "geteuid") or os.geteuid() == 0, reason="root reads everything") def test_a_real_unreadable_directory_keeps_its_notes(vault, store, embedder): index_vault(vault, store, embedder) @@ -691,9 +776,13 @@ def test_a_note_edited_back_to_the_indexed_text_needs_no_update(vault, store, em def restore(texts): note.write_text(original) + touched = time.time() + 60 + os.utime(note, (touched, touched)) stats = index_vault(vault, store, _Wrapped(embedder, restore)) assert (stats.updated, stats.unchanged) == (0, 3) + row = store.recent(0, 1)[0] + assert (row.path, row.mtime, row.size) == ("projects/garden.md", note.stat().st_mtime, note.stat().st_size) # ---- bounded work per note diff --git a/tests/test_models.py b/tests/test_models.py index a7a189c..a3969b6 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -494,6 +494,17 @@ def test_the_embedder_reports_the_revision_it_loads(tmp_path): assert models.embedder_revision(SentenceTransformerEmbedder("org/other")) == "" +def test_embedding_config_records_the_window_without_loading_a_model(): + a = SentenceTransformerEmbedder("org/other", max_seq=2048) + b = SentenceTransformerEmbedder("org/other", max_seq=1024) + assert a.max_seq == 2048 and b.max_seq == 1024 + assert models.embedding_config(a) == '{"embed_max_seq": 2048}' + assert models.embedding_config(a) != models.embedding_config(b) + assert models.embedding_config(a) == models.embedding_config( + SentenceTransformerEmbedder("org/other", device="cpu", max_seq=2048, batch_size=1) + ) + + def test_offline_switch_set_to_zero_does_not_block_the_download(stubs, monkeypatch): monkeypatch.setenv("HF_HUB_OFFLINE", "0") models.resolve_model(DEFAULT_EMBEDDER) @@ -669,3 +680,11 @@ def call(): t.join(20) assert len(calls) == 1 and len(errors) == 8 assert all(isinstance(e.__cause__, OSError) for e in errors) + + +def test_embedding_config_label_names_the_recorded_settings(): + from puremind_mac.models import embedding_config_label + + assert embedding_config_label('{"embed_max_seq": 2048}') == "embed_max_seq 2048" + assert embedding_config_label('{"embed_max_seq": null}') == "embed_max_seq (unset)" + assert embedding_config_label(None) == embedding_config_label("not json") == "an unrecorded configuration" diff --git a/tests/test_server_stdio.py b/tests/test_server_stdio.py index 58142f1..850b87a 100644 --- a/tests/test_server_stdio.py +++ b/tests/test_server_stdio.py @@ -164,6 +164,7 @@ def test_the_wire_carries_only_json_rpc_and_declares_a_read_only_server(server): } assert not tool.get("outputSchema") search = listed["puremind_search"]["inputSchema"]["properties"] + assert search["query"]["maxLength"] == 4096 assert (search["limit"]["minimum"], search["limit"]["maximum"], search["limit"]["default"]) == (1, 50, 10) recent = listed["puremind_recent"]["inputSchema"]["properties"] assert (recent["days"]["minimum"], recent["days"]["maximum"]) == (1, 3650) @@ -216,6 +217,7 @@ def test_the_switch_that_turns_reranking_off_reaches_the_tool_text(server): REFUSED = [ ("puremind_search", {"query": " "}, "query is empty"), + ("puremind_search", {"query": "x" * 4097}, "4096"), ("puremind_get", {"path": "../outside.md"}, "hidden"), ("puremind_get", {"path": ".private/diary.md"}, "hidden"), ("puremind_get", {"path": "/etc/passwd"}, "relative to the vault"), @@ -236,6 +238,7 @@ def test_refusals_are_protocol_errors_without_absolute_paths(server, tmp_path): assert result["isError"] is True, (name, arguments, result) assert expected in message, (name, arguments, message) assert str(tmp_path) not in message and "Traceback" not in message, message + assert "x" * 4097 not in message and len(message) < 2000 assert client.close() == 0 diff --git a/tests/test_service.py b/tests/test_service.py index c942fd6..f4a870b 100644 --- a/tests/test_service.py +++ b/tests/test_service.py @@ -96,6 +96,30 @@ def test_the_server_searches_an_index_built_with_a_tilde_model_path(cfg, store, assert svc.search("espresso grinder", rerank=False)[0].path == "projects/espresso.md" +def test_search_refuses_a_changed_embedding_window(service, vault, monkeypatch): + from puremind_mac.service import IndexMismatch + + service.embedder.max_seq = 2048 + index_vault(vault, service.store, service.embedder) + service.embedder.max_seq = 1024 + + def unexpected(query): + pytest.fail("an incompatible query reached the embedder") + + monkeypatch.setattr(service.embedder, "embed_query", unexpected) + with pytest.raises( + IndexMismatch, match=r"embedding configuration \(embed_max_seq 2048, but embed_max_seq 1024 is configured\).*rebuild" + ): + service.search("espresso", rerank=False) + + +def test_search_accepts_an_index_that_never_recorded_the_window(service, vault): + index_vault(vault, service.store, service.embedder) + with service.store.db: + service.store.db.execute("DELETE FROM meta WHERE key = 'embedding_config'") + assert service.search("espresso", rerank=False) + + def test_an_index_built_at_another_embedder_revision_is_refused(cfg, store, vault, embedder, reranker): from puremind_mac.service import IndexMismatch diff --git a/tests/test_store.py b/tests/test_store.py index 480ee92..e833a5a 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -79,6 +79,24 @@ def test_delete_file_cascades(store): assert store.fts_search("one") == [] +def test_metadata_refresh_changes_only_the_matching_file_stat(store): + path = "it's a note.md" + put(store, path=path) + put(store, path="other.md") + chunks, vectors = store.load_index() + generation = store.generation() + store.update_file_metadata(path, "h1", 200.0, 42) + rows = {r.path: r for r in store.list_files()} + assert (rows[path].mtime, rows[path].size, rows[path].sha256) == (200.0, 42, "h1") + assert (rows["other.md"].mtime, rows["other.md"].size) == (100.0, 10) + assert store.load_index()[0] == chunks + np.testing.assert_array_equal(store.load_index()[1], vectors) + assert store.generation() == generation + store.update_file_metadata(path, "stale hash", 300.0, 99) + store.update_file_metadata("missing.md", "h1", 300.0, 99) + assert {r.path: r for r in store.list_files()} == rows + + def test_file_with_no_chunks(store): put(store, chunks=(), vectors=None) assert store.stats() == {"files": 1, "chunks": 0, "vectors": 0} @@ -393,6 +411,7 @@ def writes_refused(s): for call in ( lambda: s.set_meta("k", "v"), lambda: put(s, "c.md"), + lambda: s.update_file_metadata("a.md", "h1", 200.0, 42), lambda: s.delete_file("a.md"), lambda: s.clear(), ): diff --git a/tests/test_tools.py b/tests/test_tools.py index e61aedb..e61670c 100644 --- a/tests/test_tools.py +++ b/tests/test_tools.py @@ -99,6 +99,7 @@ def test_tool_schemas_defaults_and_bounds(service): tools_ = _listed(service) s = _schema(tools_["puremind_search"]) assert s["required"] == ["query"] and s["properties"]["limit"]["default"] == 10 + assert s["properties"]["query"]["maxLength"] == tools.MAX_QUERY_CHARS == 4096 assert (s["properties"]["limit"]["minimum"], s["properties"]["limit"]["maximum"]) == (1, tools.MAX_LIMIT) r = _schema(tools_["puremind_recent"])["properties"] assert r["days"]["default"] == 7 and r["limit"]["default"] == 20 @@ -149,6 +150,32 @@ def test_server_turns_a_refusal_into_a_tool_error(service, vault): # ---- search +@pytest.mark.parametrize("query", ["x" * 4097, " " * 4096 + "x", "é" * 4097]) +def test_long_queries_are_refused_before_search_without_echoing_them(service, monkeypatch, query): + def unexpected(*args, **kwargs): + pytest.fail("an oversized query reached search") + + monkeypatch.setattr(service, "search", unexpected) + assert refusal(tools.puremind_search, service, query) == "the query is longer than 4096 characters" + with pytest.raises(ToolError) as excinfo: + asyncio.run(build_server(service).call_tool("puremind_search", {"query": query})) + assert query not in str(excinfo.value) and len(str(excinfo.value)) < 1000 + + +def test_a_query_at_the_character_limit_is_accepted(service, monkeypatch): + seen = [] + + def search(query, **kwargs): + seen.append(query) + return [] + + monkeypatch.setattr(service, "search", search) + query = "é" * tools.MAX_QUERY_CHARS + assert "No results" in tools.puremind_search(service, query) + asyncio.run(build_server(service).call_tool("puremind_search", {"query": query})) + assert seen == [query, query] + + def test_search_returns_path_heading_and_score(service, vault): indexed(service, vault) out = tools.puremind_search(service, "espresso grinder ethiopian", 3)