diff --git a/docs/mcp-tools.md b/docs/mcp-tools.md index 9903ee8f..fcbb013e 100644 --- a/docs/mcp-tools.md +++ b/docs/mcp-tools.md @@ -93,6 +93,31 @@ No parameters. **Returns:** version, tier, message count, health status, and capabilities. +`health.clustering` reports dependency availability, the last maintenance outcome, +source freshness, coverage and cluster count. Zero clusters can be a successful +empty or all-noise result. A missing dependency is reported separately. + +All health entries retain `status: "ok" | "degraded"`. Clustering also has a +`state`: `unknown`, `pending`, `deferred`, `degraded` or `ready`. A disconnected +database is unknown; busy foreground/model ownership is deferred. Pending +caller writes and unverified vector-generation coverage cannot report ready. +`last_error`, `dependency_error` and `process_error` contain categories only. +The last process failure clears after committed successful clustering work. +Success pending a caller's outer commit preserves that evidence; a later +committed maintenance pass can clear it. + +Clustering remains optional. Installation guidance is included only when the +HDBSCAN module is confirmed missing; a failed native import alone does not +produce a missing-package recommendation. + +## truememory_status + +Inspect rebuild progress and subsystem health. `status_id` is optional and +defaults to `0`, the most recent tier-switch rebuild. Clustering appears under +`degradation.clustering`, with the same fields as stats. Its inspection uses an +existing connection when available and never constructs a memory instance, +opens a database, loads a native package/model, or starts maintenance. + --- ## truememory_configure diff --git a/docs/resource-budgets.md b/docs/resource-budgets.md index d38cd82d..813c54aa 100644 --- a/docs/resource-budgets.md +++ b/docs/resource-budgets.md @@ -1082,3 +1082,207 @@ continue. This checkpoint adds no engine routing, automatic rebuild, model load, corpus scan on open, or user-facing maintenance status. Synthetic arithmetic and transaction checks do not establish retrieval quality or native performance acceptance. + + +### Foreground model admission before writer ownership + +Owned foreground add and incremental vector publication now acquire the model +identity fence before opening the SQLite write transaction. Add first owns its +engine write lock, so another mutation on that same engine cannot hold the +engine lock while waiting for a model fence held by the queued add. The model +fence validates the captured generation and remains held through commit or +rollback. There is one model acquisition, with no nested acquisition of the +nonreentrant model lock. + +This permits an owned foreground call to wait for a concurrent cluster source +snapshot without holding SQLite's writer or failing a valid vector publication. +Model loading and inference finish before owned writer admission. A model +generation change during loading, encoding or admission still rejects stale +publication; add and update propagate those typed errors instead of silently +falling back to a source-only write. Update retains its existing distinction: +capture failure precedes its source write, while a later vector-publication +failure can occur after the source update committed. + +Caller transactions use nonblocking model capture and publication admission. +Add and update classify the connection during model capture under their engine +write lock, so another same-engine mutation's transaction cannot be mistaken +for borrowed caller work. They release that lock before encoding. +Capture accepts only a cached model observed together with its identity under +the model lock. Busy ownership or an unloaded model produces a typed retry +error before any additional source write; the caller's transaction remains +intact. Cached-model inference can still run inside a caller-held transaction. +The publication fence protects the nested savepoint release, not a later outer +commit controlled by that caller. Streamed rebuilds retain nonblocking model +fences after writer admission and use the same cached-only borrowed capture. + +SQLite and event regressions cover actual clustering snapshot contention, +same-engine mutation ordering, stale generations, borrowed sentinels, atomic +add publication, and model release after admission/publication/commit failures. +This checkpoint changes admission ordering rather than bounding wait duration: +cluster snapshots may still hold the model lock while scanning the corpus. +Native integration and latency remain separate acceptance checks. + +### Engine maintenance routing + +Engine startup, committed mutations and later safe API boundaries now use the +same durable eight-layer planner. The default append trigger remains +`insert_count - attempted_insert_count >= 25`; 24 committed appends do not +qualify and the 25th does, including across engine instances and reopen. +`TRUEMEMORY_AUTO_CONSOLIDATE_EVERY` retains its positive-integer override. +Initial historical or empty input attempts once. Corrections, source epochs and +dependency changes follow their separate eligibility rules. A successful empty +cluster result no longer starts every layer again on each open. Source edits +committed through another handle become visible at the next safe API boundary; +there is no idle polling timer. Uncommitted caller changes do not schedule work. + +Foreground scheduling reads eight checkpoint records, source counters and small +dependency metadata. It does not scan source/vector rows, import native +frameworks, wait for the model lock or perform consolidation in `add()`. The +foreground write lock is acquired nonblocking for this probe. Package versions +are cached in three records and refreshed on connection setup and worker/manual +setup. One extension-failure record is bound to the probed model/table identity +and version snapshot. Actual worker extension loading supplies that evidence; +metadata alone is not proof of availability. Epoch checks prevent a late worker +from overwriting newer shared capability evidence, while that worker still uses +its own actual failure for its current attempt. A transient model-busy probe +cannot erase an existing extension-failure baseline. + +Each canonical file has at most one active owner and one pending notification +generation. A notification stores only a generation and threshold, not an +engine, caller connection or source content. Multiple foreground notifications +replace that single pending slot. A same-process owner's release services an +existing pending notification; it creates no notification of its own. A pending +coordinator behind a local owner is retained only until that exact owner's +release or cancellation, so closing the last engine does not lose the wake. +Cross-process contention reports busy/pending and needs a later API boundary; +this checkpoint has no automatic cross-process release notification. Deferred +model ownership and failed attempts do not schedule their own retry loops. + +Workers create and close their own file connection. `engine.close()` detaches +and closes only the foreground connection; it does not cancel or join a shared +worker. Cancellation clears queued notifications but retains ownership until +active computation reaches a cooperative boundary and its connection closes. +Thread-start failure closes an unclaimed owner descriptor; interruption after +the target started leaves teardown with that exact worker and cannot cancel a +later successor. Child processes discard inherited owner descriptors, pending +references and registry state. Daemon threads do not guarantee completion when +the process exits; durable abandoned-attempt recovery remains necessary. + +Explicit `consolidate()` forces one pass with the same eight result keys plus +the existing nonpersisted preference extraction result. A busy owner produces +nine explicit `BUSY` entries and does not queue a hidden forced pass. A file +call without an existing transaction uses a dedicated connection. An existing +caller transaction uses the original connection's protected savepoints and +reports `pending caller commit`; its writes and maintenance output remain +rollbackable together. `:memory:` automatic work reports pending without opening +an empty replacement database or creating a worker. Explicit manual maintenance +uses the original in-memory handle. Caller-held writer and later outer-commit +limitations from the borrowed-runner contract still apply. + +Successful manual summary/cluster publication restores the calling engine's +corresponding capability flags, including successful empty output. Failed or +unavailable layers never enable those flags or clear a previously usable +capability. The existing automatic-consolidation opt-out remains until explicit +manual repair. Cluster retrieval still requires live vector availability. +`get_stats()` exposes bounded coordinator and per-layer outcome, freshness, +coverage and categorical error state; it does not expose source text, paths or +exception messages. Unverified vector-generation and legacy-contact coverage +remain explicit limitations, rather than complete success claims. + +This routing checkpoint preserves builder algorithms and the A17 foreground +vector-publication guards. Ingest database replacement, profile/style getter +freshness and custom-schema source tracking remain separate work. Synthetic +SQLite and event tests establish scheduling and ownership behavior; native +latency, throughput and memory acceptance still require measurement. + +### Clustering health visibility (#720) + +Clustering health reuses the scheduler's dependency identity and persisted +attempt/provenance row. It reports availability, outcome, freshness, coverage, +output count and categorical errors separately. A successful empty or all-noise +result remains `success_empty` with zero output; a missing dependency cannot +become an empty success. Existing unverified vector-generation coverage remains +visible and does not certify complete health. + +The engine accessor does not connect or initialize a coordinator. It attempts +the foreground lock without waiting, then reads the existing connection under +one read snapshot. Model ownership is also tried without waiting. Foreground +and model contention report deferred observation; a disconnected engine reports +unknown readiness. An existing caller transaction is preserved and reported as +pending caller commit. Source mutations, pending work and failed attempts retain +their distinct outcomes and prior published counts. + +Package inspection uses at most six top-level spec lookups and three distribution +records per observation. It imports no native module and loads no model. +Distribution metadata +failure can report dependency unavailability, but installation guidance requires +a confirmed absent HDBSCAN module. A cached worker extension failure is reused +only while its dependency identity matches. Health reads do not change durable +attempts, capability epochs or scheduling, and do not create a worker or database. + +Actual maintenance and bulk-ingest failures emit a categorical warning once per +failure transition in the shared process coordinator. Repeated status reads and +foreground notifications do not log warnings. Committed successful clustering, +including empty output, clears that process failure. A borrowed savepoint release +does not prove the caller's outer commit and preserves prior process failures +through rollback. There is no outer-commit callback; a subsequent committed +successful maintenance pass clears the retained evidence. A process failure is not a durable +cross-process diagnostic; persisted scheduler outcomes remain independently +visible. The optional clustering extra, eight persisted layers, nine manual +result keys, builder algorithms and preference stage are unchanged. + +### Private style maintenance preparation (#747) + +Explicit style maintenance can enroll a private `style_vectors` checkpoint. +Ordinary database initialization and spec construction do not enroll it or +scan profiles. The eight public adapters and nine consolidation result keys +remain unchanged; explicit enrollment adds a ninth physical checkpoint row. +Fixed output INSERT/UPDATE/DELETE triggers invalidate only that private row +and clear its attempt baseline. Trigger repair invalidates provenance using +metadata without scanning messages or profiles. Caller rollback also rolls +back enrollment, trigger installation and invalidation. + +Tracked rebuilds retain the existing 256-dimensional character n-gram +arithmetic, sender ordering, case merging and ordinary-message filter. They +capture source before computation, acquire writer ownership before checking +source and future accumulator versions, then publish profiles, raw sums, +hash version 2 and complete provenance in the runner's final transaction. +Unsupported future formats are retained. Valid legacy getter output remains +readable; getters never rebuild. Low-level standalone builders retain their +existing SQLite-fenced contract and do not gain global coordination or the +tracked future-format guard. + +Style work checks cooperative cancellation during capture, computation and +publication. SQLite progress-handler installation additionally requires the +explicit `connection_owned=True` opt-in and an owned runner transaction; a +clean transaction alone does not establish ownership of the connection. That +handler is disabled before rollback or commit. Other connections keep their +caller's unknown handler, and borrowed transactions retain final commit +ownership. Verified stale source, cancellation and SQLite writer +contention defer without establishing a failure baseline after confirmed +rollback; unexpected SQL failures remain failures. Failed publication retains +the previous output and successful provenance. + +Initial enrollment and trigger repair can also defer on lock contention: +the preparation boundary must establish that no writes started or confirm +its owned rollback or caller savepoint rollback. Caller work stays pending; +unknown errors and an unconfirmed transaction end do not become deferral. +Read-only baseline probes before a running diagnostic, including the repeated +layer probe and cancelled-result probe, can defer only for a known lock error +with unchanged transaction ownership. Unavailable observations report an +unknown count and unverified coverage; no attempt baseline is advanced. +Lock contention before the running diagnostic leaves its prior checkpoint +unchanged. If a writer instead blocks restoration after a deferred build, the +running diagnostic remains persisted; a later owner can recover it after the +writer releases. The deferred result does not claim that restoration succeeded. +An owned SQLite interruption can attest automatic rollback only when this +operation's installed callback returned cancellation and SQLite reported the +interrupt code, or the exact legacy interrupt message on Python 3.10. The +callback is cleared before cleanup. Merely observing an ended transaction +does not establish rollback. Error categories also support Python 3.10's +missing SQLite code attributes without hiding unexpected SQL errors. + +This checkpoint does not activate engine routing, automatic migration, +incremental coverage, style status or coordinator style-only jobs. Those +remaining migration steps are required before release. Synthetic transaction +proofs do not establish native performance or retrieval quality. diff --git a/tests/test-clustering-health-720.py b/tests/test-clustering-health-720.py new file mode 100644 index 00000000..9cd063ca --- /dev/null +++ b/tests/test-clustering-health-720.py @@ -0,0 +1,480 @@ +"""Clustering health with SQLite in memory and synthetic native boundaries.""" + +import ast +import builtins +import json +import logging +import os +import re +import sqlite3 +import sys +import threading +import time +import types +import unittest +from pathlib import Path +from unittest.mock import Mock, patch + + +ROOT = Path(__file__).resolve().parents[1] + + +def load_functions(path: str, names: set[str], namespace: dict, *, method: bool = False) -> dict: + tree = ast.parse((ROOT / path).read_text(encoding="utf-8")) + body = next(node.body for node in tree.body if isinstance(node, ast.ClassDef) + and node.name == "TrueMemoryEngine") if method else tree.body + nodes = [ast.ImportFrom(module="__future__", names=[ast.alias(name="annotations")], level=0)] + for node in body: + if isinstance(node, ast.FunctionDef) and node.name in names: + node.decorator_list = [] + nodes.append(node) + exec(compile(ast.fix_missing_locations(ast.Module(body=nodes, type_ignores=[])), path, "exec"), namespace) + return namespace + + +def isolated_builtins(modules: dict) -> dict: + def safe_import(name: str, globals: dict | None = None, locals: dict | None = None, + fromlist: tuple = (), level: int = 0) -> object: + if name in modules: + return modules[name] + if name.startswith("truememory") or name.split(".")[0] in {"torch", "numpy", "hdbscan", "sqlite_vec"}: + raise AssertionError("Unexpected production or native import: " + name) + return builtins.__import__(name, globals, locals, fromlist, level) + return dict(vars(builtins), __import__=safe_import) + + +def load_primitives() -> tuple[types.ModuleType, types.ModuleType]: + modules = {} + for name in ("storage", "_platform", "maintenance"): + module = types.ModuleType("synthetic_clustering_health_" + name) + module.__dict__["__builtins__"] = isolated_builtins(modules) + path = ROOT / "truememory" / (name + ".py") + exec(compile(path.read_text(encoding="utf-8"), str(path), "exec"), module.__dict__) + modules["truememory." + name] = module + return modules["truememory.storage"], modules["truememory.maintenance"] + + +STORAGE, MAINTENANCE = load_primitives() +INSTALLED_DEPENDENCY = MAINTENANCE._installed_dependency + + +class HealthFixture(unittest.TestCase): + def setUp(self) -> None: + self.conn = STORAGE.create_db(":memory:") + self.addCleanup(self.conn.close) + self.conn.execute("CREATE TABLE synthetic_vectors(embedding BLOB)") + self.conn.execute( + "INSERT INTO vector_cache_registry VALUES " + "('edge','synthetic_vectors','synthetic_separation',0,0,'synthetic-model',2,0,0)" + ) + self.conn.commit() + self.vector = types.SimpleNamespace( + _lock=threading.Lock(), EMBEDDING_MODEL="synthetic-model", _embedding_dim=2, + _active_tier_group=lambda: "edge", _active_vec_table=lambda conn: "synthetic_vectors", + ) + self.versions = {"numpy": "synthetic", "hdbscan": "synthetic", "sqlite_vec": "synthetic"} + self.absent: set[str] = set() + self.probes = [] + + def find_spec(name: str) -> object: + self.probes.append(name) + return None if name in self.absent else object() + + self.enter_context(patch.dict(sys.modules, {"truememory.vector_search": self.vector})) + self.enter_context(patch.object(MAINTENANCE, "_installed_dependency", lambda name, dist: self.versions[name])) + self.enter_context(patch.object(MAINTENANCE, "importlib", types.SimpleNamespace( + util=types.SimpleNamespace(find_spec=find_spec), + import_module=Mock(side_effect=AssertionError("Health imported a module")), + ))) + self.coordinator = MAINTENANCE.MaintenanceCoordinator(None) + self.coordinator.refresh_capabilities() + self.namespace = load_functions("truememory/engine.py", {"get_clustering_health"}, { + "__builtins__": isolated_builtins({"truememory.maintenance": MAINTENANCE}), + }, method=True) + self.engine = types.SimpleNamespace( + conn=self.conn, _write_lock=threading.Lock(), _maintenance_coordinator=self.coordinator, + _maintenance_handle=self.conn, + _ensure_connection=Mock(side_effect=AssertionError("Health opened database")), + _get_maintenance_coordinator=Mock(side_effect=AssertionError("Health initialized scheduler")), + ) + self.engine.get_clustering_health = types.MethodType(self.namespace["get_clustering_health"], self.engine) + + def enter_context(self, context: object) -> object: + result = context.__enter__() + self.addCleanup(context.__exit__, None, None, None) + return result + + def health(self) -> dict: + return self.engine.get_clustering_health() + + def publish(self, count: int = 0, coverage: str = "complete") -> None: + dependency = MAINTENANCE._cluster_schedule_dependency(self.conn, versions=self.versions) + source = MAINTENANCE.read_source_revision(self.conn) + self.conn.execute("BEGIN") + MAINTENANCE.record_layer_success_in_transaction( + self.conn, layer="clusters", dependency=dependency, source=source, + output_count=count, run_generation="synthetic", coverage=coverage, + ) + self.conn.commit() + + +class TestClusteringHealth(HealthFixture): + def test_package_inspection_is_bounded_and_import_free(self) -> None: + versions = Mock(return_value="synthetic") + with patch.object(MAINTENANCE, "_installed_dependency", INSTALLED_DEPENDENCY), \ + patch.object(MAINTENANCE.importlib, "metadata", types.SimpleNamespace(version=versions), create=True): + self.probes.clear() + health = MAINTENANCE.clustering_health() + self.assertEqual(health["state"], "unknown") + self.assertEqual(len(self.probes), 6) + self.assertEqual(versions.call_count, 3) + MAINTENANCE.importlib.import_module.assert_not_called() + + def test_missing_hdbscan_before_any_attempt_is_visible(self) -> None: + self.absent.add("hdbscan") + self.versions["hdbscan"] = None + health = self.health() + self.assertEqual((health["status"], health["availability"], health["outcome"]), + ("degraded", "unavailable", "pending")) + self.assertEqual(health["missing_dependencies"], ["hdbscan"]) + self.assertIn("truememory[clustering]", health["guidance"]) + self.assertEqual(health["dependency_error"], "DependencyMissing") + self.assertFalse(self.coordinator.snapshot()["active"]) + + def test_metadata_failure_does_not_claim_package_is_missing(self) -> None: + self.versions["hdbscan"] = None + health = self.health() + self.assertEqual(health["availability"], "unavailable") + self.assertEqual(health["missing_dependencies"], []) + self.assertIsNone(health["guidance"]) + + def test_disconnected_engine_is_unknown_without_opening_or_scheduling(self) -> None: + self.engine.conn = None + health = self.health() + self.assertEqual((health["status"], health["state"], health["availability"]), + ("degraded", "unknown", "unknown")) + self.engine._ensure_connection.assert_not_called() + self.engine._get_maintenance_coordinator.assert_not_called() + self.assertFalse(self.coordinator.snapshot()["active"]) + + def test_foreground_and_model_busy_remain_distinct_and_do_not_wait(self) -> None: + self.engine._write_lock.acquire() + try: + health = self.health() + finally: + self.engine._write_lock.release() + self.assertEqual((health["state"], health["pending_reason"]), ("deferred", "foreground_busy")) + self.vector._lock.acquire() + try: + health = self.health() + finally: + self.vector._lock.release() + self.assertEqual((health["state"], health["availability"], health["pending_reason"]), + ("deferred", "deferred", "model_busy")) + self.assertFalse(self.conn.in_transaction) + self.assertEqual(self.conn.execute("SELECT outcome FROM maintenance_layers WHERE layer='clusters'").fetchone()[0], + "pending") + + def test_pending_and_successful_empty_are_not_missing_dependency(self) -> None: + pending = self.health() + self.assertEqual((pending["state"], pending["availability"], pending["freshness"]), + ("pending", "available", "untrusted")) + self.publish() + empty = self.health() + self.assertEqual((empty["status"], empty["state"], empty["outcome"], empty["output_count"]), + ("ok", "ready", "success_empty", 0)) + self.assertEqual(empty["freshness"], "current") + + def test_unverified_vector_coverage_cannot_claim_full_health(self) -> None: + self.publish(2, "vector_generation_unverified") + health = self.health() + self.assertEqual((health["status"], health["outcome"], health["output_count"]), ("degraded", "success", 2)) + self.assertEqual((health["coverage"], health["pending_reason"]), + ("vector_generation_unverified", "coverage_unverified")) + self.assertIsNone(health["last_error"]) + + def test_append_and_correction_preserve_prior_count_but_report_pending(self) -> None: + self.publish(2) + self.conn.execute("INSERT INTO messages(id,content) VALUES(1,'synthetic')") + self.conn.commit() + health = self.health() + self.assertEqual((health["state"], health["freshness"], health["output_count"]), + ("pending", "append_pending", 2)) + self.conn.execute("UPDATE messages SET content='synthetic changed' WHERE id=1") + self.conn.commit() + self.assertEqual(self.health()["freshness"], "correction_pending") + + def test_borrowed_transaction_is_never_committed_or_rolled_back(self) -> None: + self.publish() + self.conn.execute("INSERT INTO messages(id,content) VALUES(1,'synthetic pending')") + queries = [] + self.conn.set_trace_callback(queries.append) + health = self.health() + self.conn.set_trace_callback(None) + self.assertTrue(health["pending_caller_commit"]) + self.assertEqual(health["pending_reason"], "pending_caller_commit") + self.assertTrue(self.conn.in_transaction) + self.assertFalse(any(q.strip().upper() in {"COMMIT", "ROLLBACK", "BEGIN"} for q in queries)) + self.conn.rollback() + self.assertEqual(self.conn.execute("SELECT count(*) FROM messages").fetchone()[0], 0) + + def test_runtime_import_failure_remains_degraded_with_installed_packages(self) -> None: + self.publish(3) + self.conn.execute("UPDATE maintenance_layers SET outcome='unavailable',error_category='LayerUnavailableError' " + "WHERE layer='clusters'") + self.conn.commit() + health = self.health() + self.assertEqual((health["status"], health["availability"], health["last_error"], health["output_count"]), + ("degraded", "available", "LayerUnavailableError", 3)) + self.assertIsNone(health["guidance"]) + self.publish(3) + self.assertEqual(self.health()["status"], "ok") + + def test_worker_extension_failure_and_dependency_recovery_are_observed(self) -> None: + dependency = MAINTENANCE._cluster_schedule_dependency(self.conn, versions=self.versions) + failure = MAINTENANCE.make_layer_dependency(1, MAINTENANCE._dependency_parameters(dependency), + available=False, error_category="VectorExtensionUnavailable") + epoch = self.coordinator.capability_snapshot()[0] + self.coordinator._publish_extension_evidence(epoch, failure) + self.assertEqual(self.health()["dependency_error"], "VectorExtensionUnavailable") + self.coordinator.refresh_capabilities() + self.assertEqual(self.health()["availability"], "available") + + def test_errors_are_categorical_and_inspection_is_read_only(self) -> None: + self.publish(3) + self.conn.execute("UPDATE maintenance_layers SET outcome='failed',error_category=? WHERE layer='clusters'", + ("synthetic private text\n/path",)) + self.conn.commit() + queries = [] + self.conn.set_trace_callback(queries.append) + changes = self.conn.total_changes + health = self.health() + self.conn.set_trace_callback(None) + self.assertEqual(health["last_error"], "UnknownError") + self.assertNotIn("synthetic private", json.dumps(health)) + self.assertEqual(self.conn.total_changes, changes) + self.assertFalse(any(q.lstrip().upper().startswith(("INSERT", "UPDATE", "DELETE", "CREATE", "PRAGMA")) + for q in queries)) + self.assertFalse(self.conn.in_transaction) + self.assertFalse(self.coordinator.snapshot()["active"]) + + def test_unavailable_tracking_reports_unknown_and_releases_locks(self) -> None: + with patch.object(MAINTENANCE, "read_layer_states", side_effect=sqlite3.OperationalError("synthetic private")): + health = self.health() + self.assertEqual((health["state"], health["last_error"]), ("unknown", "OperationalError")) + self.assertNotIn("synthetic private", json.dumps(health)) + self.assertFalse(self.conn.in_transaction) + self.assertTrue(self.engine._write_lock.acquire(blocking=False)) + self.engine._write_lock.release() + + def test_rollback_failure_is_categorical_and_releases_engine_lock(self) -> None: + class ReadBoundary: + def __getattr__(boundary, name: str) -> object: + return getattr(self.conn, name) + + def rollback(boundary) -> None: + raise sqlite3.OperationalError("synthetic private rollback") + + self.engine.conn = ReadBoundary() + health = self.health() + self.assertEqual((health["state"], health["last_error"], health["pending_reason"]), + ("unknown", "OperationalError", "inspection_rollback_failed")) + self.assertNotIn("synthetic private", json.dumps(health)) + self.assertTrue(self.engine._write_lock.acquire(blocking=False)) + self.engine._write_lock.release() + self.assertTrue(self.conn.in_transaction) + self.conn.rollback() + + def test_missing_package_repair_changes_live_evidence_without_starting_work(self) -> None: + self.absent.add("hdbscan") + self.versions["hdbscan"] = None + self.assertEqual(self.health()["availability"], "unavailable") + self.absent.clear() + self.versions["hdbscan"] = "synthetic-repaired" + self.assertEqual((self.health()["availability"], self.health()["state"]), ("available", "pending")) + self.assertFalse(self.coordinator.snapshot()["active"]) + + +class TestClusteringWarnings(HealthFixture): + def test_borrowed_success_and_rollback_preserve_prior_process_failure(self) -> None: + self.conn.execute("CREATE TABLE synthetic_cluster_output(id INTEGER)") + self.conn.execute("INSERT INTO synthetic_cluster_output VALUES(1)") + self.conn.commit() + self.publish(1) + personality = types.SimpleNamespace(extract_preferences=lambda conn: {}) + for count in (0, 2): + with self.subTest(count=count): + with patch.object(MAINTENANCE.logger, "warning"): + self.coordinator.observe_clustering_outcome("failed", "SyntheticBulkFailure") + dependency = MAINTENANCE._cluster_schedule_dependency(self.conn, versions=self.versions) + + def build(conn: sqlite3.Connection) -> None: + conn.execute("DELETE FROM synthetic_cluster_output") + conn.executemany("INSERT INTO synthetic_cluster_output VALUES(?)", [(i,) for i in range(count)]) + + spec = MAINTENANCE.LayerSpec( + "clusters", "cluster_messages", lambda: dependency, build, + lambda conn: conn.execute("SELECT count(*) FROM synthetic_cluster_output").fetchone()[0], + ) + self.conn.execute("BEGIN") + try: + with patch.object(MAINTENANCE, "engine_layer_specs", return_value=(spec,)), \ + patch.object(MAINTENANCE.importlib, "import_module", return_value=personality, side_effect=None): + report = MAINTENANCE.run_engine_maintenance( + self.conn, self.coordinator, force=True, prepare_extensions=False, allow_caller_transaction=True) + self.assertEqual(report.results[0].output_count, count) + self.assertTrue(report.results[0].pending_caller_commit) + self.assertEqual(self.coordinator.clustering_failure(), ("failed", "SyntheticBulkFailure")) + finally: + self.conn.rollback() + self.assertEqual(self.health()["process_error"], "SyntheticBulkFailure") + self.assertEqual(self.conn.execute("SELECT count(*) FROM synthetic_cluster_output").fetchone()[0], 1) + self.assertEqual(self.health()["output_count"], 1) + with patch.object(MAINTENANCE, "engine_layer_specs", return_value=(spec,)), \ + patch.object(MAINTENANCE.importlib, "import_module", return_value=personality, side_effect=None): + report = MAINTENANCE.run_engine_maintenance( + self.conn, self.coordinator, force=True, prepare_extensions=False) + self.assertFalse(report.results[0].pending_caller_commit) + self.assertIsNone(self.coordinator.clustering_failure()) + + def test_bulk_ingest_failure_is_visible_and_recovery_clears_process_error(self) -> None: + namespace = { + "__builtins__": isolated_builtins({}), "Path": Path, "time": time, "os": os, + "logger": logging.getLogger("synthetic_ingest_health"), + "create_db": lambda path: self.conn, "load_messages_from_file": lambda conn, path: 0, + "init_vec_table": lambda conn: None, "build_vectors": lambda conn: 0, + "build_separation_vectors": lambda conn: 0, + "cluster_messages": Mock(side_effect=ImportError("synthetic private sentinel")), + } + for flag in ("VECTOR", "HYBRID", "CLUSTERING", "PERSONALITY", "STYLE_VEC", "CONSOLIDATION", + "PREDICTIVE", "TEMPORAL"): + namespace["_HAS_" + flag] = flag in {"VECTOR", "CLUSTERING"} + load_functions("truememory/engine.py", {"ingest"}, namespace, method=True) + self.engine.db_path = Path(":memory:") + self.engine.stats = {} + for flag in ("hybrid", "temporal", "salience", "personality", "style_vec", "consolidation", + "predictive", "reranker", "hyde"): + setattr(self.engine, "_has_" + flag, False) + self.engine._get_maintenance_coordinator = lambda: self.coordinator + with self.assertLogs(MAINTENANCE.logger, level=logging.WARNING) as captured: + for _ in range(2): + result = namespace["ingest"](self.engine, "synthetic-unused-input") + self.assertEqual(len(captured.records), 1) + self.assertNotIn("synthetic private sentinel", " ".join(captured.output)) + self.assertIn("ERROR", result["build_clusters"]) + self.assertEqual(self.health()["process_error"], "ImportError") + self.assertEqual(self.health()["status"], "degraded") + namespace["cluster_messages"] = lambda conn: 0 + namespace["ingest"](self.engine, "synthetic-unused-input") + self.assertIsNone(self.health()["process_error"]) + + def test_warnings_are_sanitized_deduplicated_and_clear_after_success(self) -> None: + self.absent.add("hdbscan") + with self.assertLogs(MAINTENANCE.logger, level=logging.WARNING) as captured: + for _ in range(3): + self.coordinator.observe_clustering_outcome("unavailable", "synthetic private\n/path") + self.coordinator.observe_clustering_outcome("success_empty") + self.coordinator.observe_clustering_outcome("unavailable", "DependencyMissing") + self.assertEqual(len(captured.records), 2) + self.assertIn("UnknownError", captured.output[0]) + self.assertNotIn("synthetic private", " ".join(captured.output)) + self.assertTrue(all("truememory[clustering]" in line for line in captured.output)) + + def test_runtime_failure_has_no_false_install_guidance_and_reads_never_warn(self) -> None: + with self.assertLogs(MAINTENANCE.logger, level=logging.WARNING) as captured: + self.coordinator.observe_clustering_outcome("failed", "RuntimeError") + self.assertNotIn("pip install", captured.output[0]) + with patch.object(MAINTENANCE.logger, "warning") as warning: + for _ in range(3): + self.health() + self.coordinator.observe_clustering_outcome("deferred", "ModelBusy") + warning.assert_not_called() + + def test_runner_reports_only_actual_cluster_attempts(self) -> None: + result = MAINTENANCE.LayerResult("clusters", "cluster_messages", "unavailable", None, 0, + "LayerUnavailableError", True) + personality = types.SimpleNamespace(extract_preferences=lambda conn: {}) + with patch.object(MAINTENANCE, "engine_layer_specs", return_value=()), \ + patch.object(MAINTENANCE, "run_layers", return_value=(result,)), \ + patch.object(MAINTENANCE.importlib, "import_module", return_value=personality, side_effect=None), \ + patch.object(self.coordinator, "observe_clustering_outcome") as observed: + MAINTENANCE.run_engine_maintenance(self.conn, self.coordinator, prepare_extensions=False) + observed.assert_called_once_with("unavailable", "LayerUnavailableError") + observed.reset_mock() + with patch.object(MAINTENANCE, "run_layers", return_value=(result._replace(attempted=False),)): + MAINTENANCE.run_engine_maintenance(self.conn, self.coordinator, prepare_extensions=False) + observed.assert_not_called() + + +class TestMCPClusteringHealth(HealthFixture): + def server(self, memory: object | None) -> dict: + modules = { + "truememory.maintenance": MAINTENANCE, + "truememory.engine": types.SimpleNamespace(get_vectors_load_error=lambda: "SyntheticVectorError"), + "truememory.tier_switch.manager": types.SimpleNamespace(RebuildManager=types.SimpleNamespace( + get_instance=lambda: types.SimpleNamespace(get_status=lambda status_id: {"status_id": status_id}))), + } + return load_functions("truememory/mcp_server.py", {"_build_health_payload", "truememory_status", "truememory_stats"}, { + "__builtins__": isolated_builtins(modules), "json": json, "re": re, "_memory": memory, + "_get_memory": Mock(side_effect=AssertionError("Health constructed Memory")), + "_reranker_error_lock": threading.Lock(), "_reranker_last_error": "SyntheticRerankerError", + "_llm_error_lock": threading.Lock(), "_llm_last_error": {"synthetic": "SyntheticLLMError"}, + "_current_llm_provider_name": "synthetic", "_encoding_gate_error_lock": threading.Lock(), + "_encoding_gate_last_error": "SyntheticGateError", "_encoding_gate_degradation_count": 2, + "_build_model_server_health": lambda: {"status": "ok", "running": True}, + "_load_config": lambda: {"tier": "edge"}, "__version__": "synthetic", + "_get_rss_mb": lambda: 0, "_MAX_RSS_MB": 0, + }) + + def test_existing_health_contract_and_values_are_preserved(self) -> None: + server = self.server(types.SimpleNamespace(_engine=self.engine)) + health = server["_build_health_payload"]() + self.assertEqual(health["reranker"], {"status": "degraded", "last_error": "SyntheticRerankerError"}) + self.assertEqual(health["hyde_llm"], {"status": "degraded", "active_provider": "synthetic", + "last_error_by_provider": {"synthetic": "SyntheticLLMError"}}) + self.assertEqual(health["vectors"], {"status": "degraded", "last_error": "SyntheticVectorError"}) + self.assertEqual(health["encoding_gate"], {"status": "degraded", "last_error": "SyntheticGateError", + "degradation_count": 2}) + self.assertEqual(health["model_server"], {"status": "ok", "running": True}) + self.assertTrue(all(item["status"] in {"ok", "degraded"} for item in health.values())) + self.assertIn("clustering", health) + self.assertEqual(health["clustering"]["state"], "pending") + + def test_status_before_memory_initialization_reports_missing_dependency_without_initialization(self) -> None: + self.absent.add("hdbscan") + self.versions["hdbscan"] = None + server = self.server(None) + result = json.loads(server["truememory_status"](17)) + self.assertEqual(result["rebuild"], {"status_id": 17}) + self.assertIn("clustering", result["degradation"]) + self.assertEqual(result["degradation"]["clustering"]["availability"], "unavailable") + self.assertEqual(result["degradation"]["clustering"]["missing_dependencies"], ["hdbscan"]) + server["_get_memory"].assert_not_called() + + def test_stats_tool_includes_the_same_clustering_health(self) -> None: + self.engine._ensure_connection = Mock() + memory = types.SimpleNamespace(_engine=self.engine, stats=lambda: {"message_count": 0}) + server = self.server(memory) + server["_get_memory"] = lambda: memory + result = json.loads(server["truememory_stats"]()) + self.assertEqual(result["health"]["clustering"], self.health()) + self.assertEqual(result["message_count"], 0) + self.engine._ensure_connection.assert_called_once_with() + + def test_clustering_inspection_failure_cannot_hide_existing_subsystems(self) -> None: + server = self.server(types.SimpleNamespace(_engine=self.engine)) + expected = server["_build_health_payload"]() + expected.pop("clustering") + self.engine.get_clustering_health = Mock(side_effect=RuntimeError("synthetic private inspection")) + result = server["_build_health_payload"]() + clustering = result.pop("clustering") + self.assertEqual(result, expected) + self.assertEqual((clustering["status"], clustering["state"], clustering["last_error"]), + ("degraded", "unknown", "RuntimeError")) + self.assertNotIn("synthetic private", json.dumps(clustering)) + result = json.loads(server["truememory_status"]()) + self.assertEqual(result["degradation"]["clustering"], clustering) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test-l0-style-maintenance-747.py b/tests/test-l0-style-maintenance-747.py new file mode 100644 index 00000000..f5a84483 --- /dev/null +++ b/tests/test-l0-style-maintenance-747.py @@ -0,0 +1,929 @@ +"""Private style maintenance with stdlib source and synthetic in-memory SQLite.""" + +import builtins +import json +import runpy +import sqlite3 +import sys +import threading +import types +import unittest +import uuid +from pathlib import Path +from unittest.mock import patch + + +ROOT = Path(__file__).resolve().parents[1] +ACCUMULATOR = runpy.run_path(str(Path(__file__).with_name("test-l0-style-accumulator-747.py"))) +STYLE, STORAGE = ACCUMULATOR["STYLE"], ACCUMULATOR["STORAGE"] + + +def load_maintenance() -> types.ModuleType: + modules = {"truememory.storage": STORAGE, + "truememory._platform": ACCUMULATOR["load_stdlib_source"]("_platform")} + + def safe_import(name: str, globals: dict | None = None, locals: dict | None = None, + fromlist: tuple = (), level: int = 0) -> object: + if name in modules: + return modules[name] + if name.split(".", 1)[0] not in sys.stdlib_module_names: + raise AssertionError("Non-stdlib import forbidden: " + name) + return builtins.__import__(name, globals, locals, fromlist, level) + + module = types.ModuleType("synthetic_style_maintenance") + module.__dict__["__builtins__"] = dict(vars(builtins), __import__=safe_import) + source = ROOT / "truememory/maintenance.py" + exec(compile(source.read_text(encoding="utf-8"), str(source), "exec"), module.__dict__) + + def import_module(name: str) -> types.ModuleType: + if name != "truememory.personality_style_vec": + raise AssertionError("Unexpected style dependency: " + name) + return STYLE + + module.importlib = types.SimpleNamespace(import_module=import_module) + return module + + +MAINTENANCE = load_maintenance() + + +class PublicationFailure: + """Invalidate cached statements before denying one real COMMIT or RELEASE.""" + + def __init__(self, conn: sqlite3.Connection, *, borrowed: bool) -> None: + self.conn, self.borrowed = conn, borrowed + self.matches = 0 + self.denied = 0 + + def __getattr__(self, name: str) -> object: + return getattr(self.conn, name) + + def execute(self, sql: str, *args: object) -> sqlite3.Cursor: + match = sql.startswith("RELEASE truememory_layer_") if self.borrowed else sql == "COMMIT" + if match: + self.matches += 1 + if self.matches == (1 if self.borrowed else 2): + self.denied += 1 + denied = ((sqlite3.SQLITE_SAVEPOINT, "RELEASE", sql.split()[1]) if self.borrowed + else (sqlite3.SQLITE_TRANSACTION, "COMMIT", None)) + + def authorize(action: int, one: str | None, two: str | None, *_rest: object) -> int: + return sqlite3.SQLITE_DENY if (action, one, two) == denied else sqlite3.SQLITE_OK + + self.conn.set_authorizer(authorize) + try: + return self.conn.execute(sql, *args) + finally: + self.conn.set_authorizer(ACCUMULATOR["allow_authorized_action"]) + return self.conn.execute(sql, *args) + + +class LegacySqliteModule: + """Python 3.10 lacks these primary result-code exports.""" + + def __getattr__(self, name: str) -> object: + if name in {"SQLITE_BUSY", "SQLITE_LOCKED", "SQLITE_INTERRUPT"}: + raise AttributeError(name) + return getattr(sqlite3, name) + + +class StyleMaintenanceTests(unittest.TestCase): + def connection(self, cache: int = 100, *, uri: str | None = None) -> sqlite3.Connection: + conn = sqlite3.connect(uri or ":memory:", uri=uri is not None, timeout=0, cached_statements=cache) + self.addCleanup(conn.close) + conn.executescript(STORAGE._SCHEMA_SQL) + STORAGE._initialize_maintenance_tracking(conn) + conn.execute("CREATE TABLE IF NOT EXISTS sentinel(value TEXT)") + conn.commit() + return conn + + def add(self, conn: sqlite3.Connection, text: str = "synthetic source", *, sender: str = "Synthetic", + directive: int = 0) -> None: + conn.execute("INSERT INTO messages(sender,content,timestamp,directive) VALUES (?,?,'2026-01-01',?)", + (sender, text, directive)) + conn.commit() + + def state(self, conn: sqlite3.Connection) -> object: + spec = MAINTENANCE.style_layer_spec(conn) + parameters = json.loads(spec.resolve_dependency().key)["parameters"] + self.assertEqual(parameters, {"accumulator_version": STYLE._STYLE_ACCUMULATOR_VERSION, + "hash_version": 2, "dimension": STYLE.DIM, "ngrams": list(STYLE.NGRAM_SIZES), + "sum_format": STYLE._STYLE_SUM.format}) + return MAINTENANCE.read_layer_states(conn, (spec,))["style_vectors"] + + def output(self, conn: sqlite3.Connection) -> list[tuple]: + return conn.execute("SELECT * FROM entity_style_vectors ORDER BY entity").fetchall() + + def checkpoint(self, conn: sqlite3.Connection) -> tuple | None: + return MAINTENANCE._read_layer_row(conn, "style_vectors") + + def test_spec_and_ordinary_schema_do_not_activate_or_scan(self) -> None: + conn = self.connection() + source = MAINTENANCE.read_source_revision(conn) + before = conn.total_changes + + def authorize(action: int, table: str | None, *_rest: object) -> int: + if action == sqlite3.SQLITE_READ and table in {"messages", "entity_style_vectors"}: + return sqlite3.SQLITE_DENY + return sqlite3.SQLITE_OK + + conn.set_authorizer(authorize) + spec = MAINTENANCE.style_layer_spec(conn) + state = MAINTENANCE.read_layer_states(conn, (spec,))[spec.layer] + self.assertEqual(state.successful_coverage, "unverified") + self.assertEqual(conn.total_changes, before) + self.assertEqual(len(STORAGE._MAINTENANCE_LAYERS), 8) + self.assertEqual(len(MAINTENANCE.all_layer_specs(conn)), 8) + self.assertEqual(len(MAINTENANCE.maintenance_busy_result()), 9) + self.assertEqual(conn.execute("SELECT count(*) FROM maintenance_layers").fetchone()[0], 8) + MAINTENANCE.prepare_style_maintenance(conn) + self.assertEqual(MAINTENANCE.read_source_revision(conn), source) + self.assertEqual(conn.execute("SELECT count(*) FROM maintenance_layers").fetchone()[0], 9) + self.assertIsNone(self.state(conn).successful_revision) + before = conn.total_changes + MAINTENANCE.prepare_style_maintenance(conn) + self.assertEqual(conn.total_changes, before) + conn.set_authorizer(ACCUMULATOR["allow_authorized_action"]) + + def test_preparation_is_owned_or_part_of_caller_transaction(self) -> None: + for borrowed in (False, True): + conn = self.connection() + if borrowed: + conn.execute("INSERT INTO sentinel VALUES ('synthetic pending')") + MAINTENANCE.prepare_style_maintenance(conn) + self.assertEqual(conn.in_transaction, borrowed) + if borrowed: + conn.rollback() + self.assertIsNone(self.checkpoint(conn)) + self.assertEqual(conn.execute("SELECT count(*) FROM sqlite_master WHERE name LIKE 'truememory_style_output_%'").fetchone()[0], 0) + else: + self.assertIsNotNone(self.checkpoint(conn)) + + def test_cached_preparation_commit_and_release_faults_do_not_enroll(self) -> None: + for cache in (0, 5, 100): + for borrowed in (False, True): + with self.subTest(cache=cache, borrowed=borrowed): + conn = self.connection(cache) + if borrowed: + conn.execute("INSERT INTO sentinel VALUES ('synthetic pending')") + sql = "RELEASE SAVEPOINT truememory_style_tracking" if borrowed else "COMMIT" + action = ((sqlite3.SQLITE_SAVEPOINT, "RELEASE", "truememory_style_tracking") if borrowed + else (sqlite3.SQLITE_TRANSACTION, "COMMIT", None)) + probe = ACCUMULATOR["StatementFailure"](conn, sql, action) + with self.assertRaises(sqlite3.DatabaseError): + MAINTENANCE.prepare_style_maintenance(probe) + self.assertEqual(probe.armed, [sql]) + self.assertIsNone(self.checkpoint(conn)) + self.assertEqual(conn.execute("SELECT count(*) FROM sqlite_master WHERE name LIKE 'truememory_style_output_%'").fetchone()[0], 0) + self.assertEqual(conn.in_transaction, borrowed) + self.assertEqual(conn.execute("SELECT count(*) FROM sentinel").fetchone()[0], int(borrowed)) + conn.rollback() + + def test_batch_arithmetic_case_zero_directive_and_hash_are_unchanged(self) -> None: + conn = self.connection() + for sender, text, directive in (("Synthetic", "abc", 0), ("synthetic", "a", 0), + ("Synthetic", "", 0), ("Other", "ignored", 1), ("", "ignored", 0)): + self.add(conn, text, sender=sender, directive=directive) + expected = STYLE.build_entity_style_vectors(conn) + baseline = conn.execute("SELECT entity,vector,message_count,vector_sum,accumulator_version FROM entity_style_vectors").fetchall() + token = MAINTENANCE.read_source_revision(conn) + public_rows = conn.execute("SELECT * FROM maintenance_layers ORDER BY layer").fetchall() + result = MAINTENANCE.run_style_maintenance(conn) + self.assertEqual((result.outcome, result.output_count, result.coverage), ("success", 1, "complete")) + self.assertEqual(conn.execute("SELECT entity,vector,message_count,vector_sum,accumulator_version FROM entity_style_vectors").fetchall(), baseline) + self.assertEqual(STYLE.get_entity_style_vector(conn, "SYNTHETIC"), expected["synthetic"]) + self.assertEqual(conn.execute("SELECT value FROM metadata WHERE key='style_vec_hash_version'").fetchone(), ("2",)) + self.assertEqual(MAINTENANCE.read_source_revision(conn), token) + self.assertEqual(conn.execute("SELECT * FROM maintenance_layers WHERE layer!='style_vectors' ORDER BY layer").fetchall(), public_rows) + self.assertFalse(MAINTENANCE.run_style_maintenance(conn).attempted) + self.assertEqual(self.state(conn).attempted_insert_count, 5) + + def test_empty_success_retirement_and_source_corrections(self) -> None: + for statement in ("UPDATE messages SET content='synthetic changed'", + "UPDATE messages SET sender='Other'", "UPDATE messages SET directive=1", "DELETE FROM messages"): + with self.subTest(statement=statement): + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + conn.execute(statement) + conn.commit() + self.assertEqual(len(MAINTENANCE.plan_layers(conn, (MAINTENANCE.style_layer_spec(conn),))), 1) + result = MAINTENANCE.run_style_maintenance(conn) + self.assertIn(result.outcome, {"success", "success_empty"}) + self.assertEqual(self.state(conn).successful_revision, MAINTENANCE.read_source_revision(conn).revision) + empty = self.connection() + result = MAINTENANCE.run_style_maintenance(empty) + self.assertEqual((result.outcome, result.output_count), ("success_empty", 0)) + self.assertEqual(self.state(empty).attempted_insert_count, 0) + + def test_output_changes_invalidate_failed_baseline_and_rollback_atomically(self) -> None: + for statement in ("UPDATE entity_style_vectors SET message_count=999", + "DELETE FROM entity_style_vectors", + "INSERT INTO entity_style_vectors(entity,vector) VALUES ('other','[]')"): + with self.subTest(statement=statement): + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + with patch.object(STYLE, "compute_style_vector", side_effect=ValueError("synthetic")): + self.assertEqual(MAINTENANCE.run_style_maintenance(conn, force=True).outcome, "failed") + self.assertEqual(MAINTENANCE.plan_layers(conn, (MAINTENANCE.style_layer_spec(conn),)), ()) + old, token = self.checkpoint(conn), MAINTENANCE.read_source_revision(conn) + conn.execute(statement) + self.assertIsNone(self.state(conn).attempted_revision) + self.assertTrue(self.state(conn).full_rebuild_required) + self.assertEqual(MAINTENANCE.read_source_revision(conn), token) + conn.rollback() + self.assertEqual(self.checkpoint(conn), old) + conn.execute(statement) + conn.commit() + self.assertEqual(len(MAINTENANCE.plan_layers(conn, (MAINTENANCE.style_layer_spec(conn),))), 1) + self.assertEqual(MAINTENANCE.run_style_maintenance(conn).outcome, "success") + + def test_trigger_repair_is_metadata_only_and_does_not_certify_v1_rows(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + conn.execute("DROP TRIGGER truememory_style_output_update") + conn.execute("CREATE TRIGGER truememory_style_output_update AFTER UPDATE ON entity_style_vectors BEGIN SELECT 1; END") + conn.commit() + output, source = self.output(conn), MAINTENANCE.read_source_revision(conn) + + def authorize(action: int, table: str | None, *_rest: object) -> int: + return sqlite3.SQLITE_DENY if action == sqlite3.SQLITE_READ and table in {"messages", "entity_style_vectors"} else sqlite3.SQLITE_OK + + conn.set_authorizer(authorize) + MAINTENANCE.prepare_style_maintenance(conn) + self.assertIsNone(self.state(conn).attempted_revision) + self.assertTrue(self.state(conn).full_rebuild_required) + conn.set_authorizer(ACCUMULATOR["allow_authorized_action"]) + self.assertEqual(self.output(conn), output) + self.assertEqual(MAINTENANCE.read_source_revision(conn), source) + + def test_future_custom_format_is_retained_and_low_level_still_has_its_old_contract(self) -> None: + conn = self.connection() + self.add(conn) + conn.execute("INSERT INTO entity_style_vectors VALUES ('synthetic','[1]',9,'synthetic old',X'1234',9)") + conn.execute("INSERT INTO metadata(key,value) VALUES ('style_vec_hash_version','1')") + conn.commit() + before = self.output(conn) + result = MAINTENANCE.run_style_maintenance(conn) + self.assertEqual((result.outcome, result.error_category), ("unavailable", "StyleAccumulatorUnsupported")) + self.assertEqual(self.output(conn), before) + self.assertEqual(conn.execute("SELECT value FROM metadata WHERE key='style_vec_hash_version'").fetchone(), ("1",)) + self.assertIsNone(self.state(conn).successful_revision) + self.assertFalse(MAINTENANCE.run_style_maintenance(conn).attempted) + STYLE.build_entity_style_vectors(conn) + self.assertEqual(conn.execute("SELECT accumulator_version FROM entity_style_vectors").fetchone(), (1,)) + self.assertIsNone(self.state(conn).attempted_revision) + + def test_custom_future_default_is_guarded_even_without_rows(self) -> None: + for with_source in (False, True): + with self.subTest(with_source=with_source): + conn = self.connection() + conn.execute("DROP TABLE entity_style_vectors") + conn.execute("CREATE TABLE entity_style_vectors(entity TEXT PRIMARY KEY,vector TEXT,message_count INTEGER," + "updated_at TEXT,vector_sum BLOB,accumulator_version INTEGER NOT NULL DEFAULT 9)") + conn.commit() + if with_source: + self.add(conn) + schema = conn.execute("PRAGMA table_info(entity_style_vectors)").fetchall() + result = MAINTENANCE.run_style_maintenance(conn) + self.assertEqual((result.outcome, result.error_category), ("unavailable", "StyleAccumulatorUnsupported")) + self.assertEqual(conn.execute("PRAGMA table_info(entity_style_vectors)").fetchall(), schema) + self.assertEqual(self.output(conn), []) + self.assertIsNone(conn.execute("SELECT value FROM metadata WHERE key='style_vec_hash_version'").fetchone()) + + def test_missing_output_trigger_cannot_certify_through_an_unprepared_spec(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + before = self.output(conn) + conn.execute("DROP TRIGGER truememory_style_output_update") + conn.commit() + spec = MAINTENANCE.style_layer_spec(conn) + self.assertFalse(spec.resolve_dependency().available) + result = MAINTENANCE.run_layers(conn, (spec,), force=True)[0] + self.assertEqual((result.outcome, result.error_category), ("unavailable", "StyleTrackingUnprepared")) + self.assertEqual(self.output(conn), before) + self.assertEqual(MAINTENANCE.run_style_maintenance(conn).outcome, "success") + + def test_borrowed_success_remains_pending_and_outer_rollback_restores_everything(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + before, checkpoint = self.output(conn), self.checkpoint(conn) + conn.execute("INSERT INTO sentinel VALUES ('synthetic caller')") + conn.execute("UPDATE messages SET content='synthetic changed'") + result = MAINTENANCE.run_style_maintenance(conn, allow_caller_transaction=True) + self.assertTrue(result.pending_caller_commit) + self.assertEqual(result.outcome, "success") + self.assertTrue(conn.in_transaction) + conn.rollback() + self.assertEqual(self.output(conn), before) + self.assertEqual(self.checkpoint(conn), checkpoint) + self.assertEqual(conn.execute("SELECT count(*) FROM sentinel").fetchone()[0], 0) + + def test_legacy_and_invalid_sums_rebuild_from_source_without_double_counting(self) -> None: + for version, total in ((0, None), (1, b"synthetic invalid sum")): + with self.subTest(version=version): + conn = self.connection() + self.add(conn, "synthetic first") + self.add(conn, "synthetic second") + conn.execute("INSERT INTO entity_style_vectors VALUES ('synthetic','[1]',99,'synthetic old',?,?)", + (total, version)) + conn.commit() + self.assertEqual(STYLE.get_entity_style_vector(conn, "Synthetic"), [1]) + self.assertEqual(MAINTENANCE.run_style_maintenance(conn).outcome, "success") + self.assertEqual(conn.execute("SELECT message_count,accumulator_version,length(vector_sum) FROM entity_style_vectors").fetchone(), (2, 1, 2048)) + self.assertEqual(self.state(conn).attempted_insert_count, 2) + + def test_cached_commit_and_release_faults_retain_prior_output(self) -> None: + for cache in (0, 5, 100): + for borrowed in (False, True): + with self.subTest(cache=cache, borrowed=borrowed): + conn = self.connection(cache) + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + before, prior = self.output(conn), self.state(conn) + conn.execute("UPDATE metadata SET value='1' WHERE key='style_vec_hash_version'") + conn.commit() + if borrowed: + conn.execute("INSERT INTO sentinel VALUES ('synthetic caller')") + probe = PublicationFailure(conn, borrowed=borrowed) + result = MAINTENANCE.run_style_maintenance(probe, force=True, allow_caller_transaction=borrowed) + self.assertEqual((result.outcome, probe.denied), ("failed", 1)) + self.assertEqual(self.output(conn), before) + self.assertEqual(self.state(conn).successful_revision, prior.successful_revision) + self.assertEqual(conn.execute("SELECT value FROM metadata WHERE key='style_vec_hash_version'").fetchone(), ("1",)) + self.assertEqual(conn.in_transaction, borrowed) + if borrowed: + self.assertEqual(conn.execute("SELECT count(*) FROM sentinel").fetchone()[0], 1) + conn.rollback() + + def test_verified_source_change_defers_without_consuming_attempt(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + before, checkpoint = self.output(conn), self.checkpoint(conn) + compute = STYLE.compute_style_vector + + def changed(text: str) -> list[float]: + conn.execute("UPDATE messages SET recipient='synthetic changed'") + return compute(text) + + with patch.object(STYLE, "compute_style_vector", changed): + result = MAINTENANCE.run_style_maintenance(conn, force=True) + self.assertEqual((result.outcome, result.error_category), ("deferred", "StyleSourceChanged")) + self.assertEqual(self.output(conn), before) + self.assertEqual(self.checkpoint(conn), checkpoint) + self.assertFalse(conn.in_transaction) + + def test_cancellation_during_compute_and_after_writer_retains_generation(self) -> None: + for phase in ("before", "compute", "writer"): + with self.subTest(phase=phase): + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + before, checkpoint = self.output(conn), self.checkpoint(conn) + cancel = threading.Event() + compute = STYLE.compute_style_vector + + def cancelled(text: str) -> list[float]: + if phase == "compute": + cancel.set() + return compute(text) + + def trace(sql: str) -> None: + if phase == "writer" and sql == "DELETE FROM entity_style_vectors": + cancel.set() + + if phase == "before": + cancel.set() + conn.set_trace_callback(trace) + with patch.object(STYLE, "compute_style_vector", cancelled): + result = MAINTENANCE.run_style_maintenance(conn, force=True, cancel=cancel, connection_owned=True) + conn.set_trace_callback(None) + self.assertEqual((result.outcome, result.error_category), ("deferred", "StyleCancelled")) + self.assertEqual(self.output(conn), before) + self.assertEqual(self.checkpoint(conn), checkpoint) + self.assertFalse(conn.in_transaction) + + def test_sql_capture_cancellation_disables_handler_before_rollback(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + before, checkpoint = self.output(conn), self.checkpoint(conn) + cancel = threading.Event() + + class ProgressBoundary: + def __getattr__(boundary, name: str) -> object: + return getattr(conn, name) + + def set_progress_handler(boundary, handler: object, steps: int) -> None: + conn.set_progress_handler(handler, 1 if handler is not None else 0) + + def execute(boundary, sql: str, *args: object) -> sqlite3.Cursor: + if sql.startswith("SELECT sender, content, timestamp"): + cancel.set() + return conn.execute(sql, *args) + + result = MAINTENANCE.run_style_maintenance(ProgressBoundary(), force=True, cancel=cancel, connection_owned=True) + self.assertEqual((result.outcome, result.error_category), ("deferred", "StyleCancelled")) + self.assertFalse(conn.in_transaction) + self.assertEqual(self.output(conn), before) + self.assertEqual(self.checkpoint(conn), checkpoint) + + def test_borrowed_progress_handler_is_never_replaced(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.prepare_style_maintenance(conn) + conn.execute("INSERT INTO sentinel VALUES ('synthetic caller')") + calls = [] + conn.set_progress_handler(lambda: calls.append(1) or 0, 1) + + class BorrowedBoundary: + def __getattr__(boundary, name: str) -> object: + return getattr(conn, name) + + def set_progress_handler(boundary, *_args: object) -> None: + raise AssertionError("Borrowed handler must remain untouched") + + result = MAINTENANCE.run_style_maintenance(BorrowedBoundary(), cancel=threading.Event(), + allow_caller_transaction=True, connection_owned=True) + self.assertEqual(result.outcome, "success") + self.assertTrue(calls) + before = len(calls) + conn.execute("SELECT count(*) FROM messages").fetchone() + self.assertGreater(len(calls), before) + conn.set_progress_handler(None, 0) + conn.rollback() + + def test_clean_caller_connection_keeps_its_unknown_progress_handler_by_default(self) -> None: + conn = self.connection() + self.add(conn) + calls = [] + conn.set_progress_handler(lambda: calls.append(1) or 0, 1) + + class CallerBoundary: + def __getattr__(boundary, name: str) -> object: + return getattr(conn, name) + + def set_progress_handler(boundary, *_args: object) -> None: + raise AssertionError("Clean transaction does not prove connection ownership") + + result = MAINTENANCE.run_style_maintenance(CallerBoundary(), cancel=threading.Event()) + self.assertEqual(result.outcome, "success") + before = len(calls) + conn.execute("SELECT count(*) FROM messages").fetchone() + self.assertGreater(len(calls), before) + conn.set_progress_handler(None, 0) + + def test_refused_rollback_never_restores_a_deferred_checkpoint(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + cancel = threading.Event() + compute = STYLE.compute_style_vector + + def cancelled(text: str) -> list[float]: + cancel.set() + return compute(text) + + class RefusedRollback: + def __getattr__(boundary, name: str) -> object: + return getattr(conn, name) + + def rollback(boundary) -> None: + raise sqlite3.OperationalError("synthetic rollback refusal") + + with patch.object(STYLE, "compute_style_vector", cancelled): + with self.assertRaises(MAINTENANCE._LayerRollbackFailed): + MAINTENANCE.run_style_maintenance(RefusedRollback(), force=True, cancel=cancel) + self.assertTrue(conn.in_transaction) + self.assertEqual(self.state(conn).outcome, "running") + conn.rollback() + self.assertEqual(self.state(conn).outcome, "running") + + def test_stale_deferred_restore_cannot_overwrite_new_invalidation(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + with MAINTENANCE.maintenance_owner(None) as owner: + previous = self.checkpoint(conn) + conn.execute("UPDATE maintenance_layers SET outcome='running',run_generation=? WHERE layer='style_vectors'", (owner.generation,)) + conn.commit() + running = self.checkpoint(conn) + conn.execute("UPDATE entity_style_vectors SET message_count=111") + conn.commit() + invalidated = self.checkpoint(conn) + with self.assertRaises(MAINTENANCE.MaintenanceBusyError): + MAINTENANCE._restore_deferred_attempt(conn, "style_vectors", previous, running, owner) + self.assertEqual(self.checkpoint(conn), invalidated) + self.assertIsNone(self.state(conn).attempted_revision) + + def test_two_connections_observe_only_committed_style_output(self) -> None: + uri = "file:synthetic-style-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + first = self.connection(uri=uri) + second = self.connection(uri=uri) + self.add(first) + MAINTENANCE.run_style_maintenance(first) + self.assertEqual(self.output(first), self.output(second)) + self.assertEqual(self.checkpoint(first), self.checkpoint(second)) + reopened = load_maintenance() + self.assertFalse(reopened.run_style_maintenance(second).attempted) + + def test_writer_contention_defers_only_after_confirmed_output_rollback(self) -> None: + uri = "file:synthetic-style-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + first = self.connection(uri=uri) + second = self.connection(uri=uri) + self.add(first) + MAINTENANCE.run_style_maintenance(first) + before, checkpoint = self.output(first), self.checkpoint(first) + compute = STYLE.compute_style_vector + rollbacks = [] + + def contend(text: str) -> list[float]: + second.execute("BEGIN IMMEDIATE") + second.execute("INSERT INTO sentinel VALUES ('synthetic competing writer')") + return compute(text) + + class WriterBoundary: + def __getattr__(boundary, name: str) -> object: + return getattr(first, name) + + def rollback(boundary) -> None: + first.rollback() + rollbacks.append(not first.in_transaction) + second.rollback() + + with patch.object(STYLE, "compute_style_vector", contend): + result = MAINTENANCE.run_style_maintenance(WriterBoundary(), force=True) + self.assertEqual((result.outcome, result.error_category), ("deferred", "StyleWriterBusy")) + self.assertEqual(rollbacks, [True]) + self.assertEqual(self.output(first), before) + self.assertEqual(self.checkpoint(first), checkpoint) + + def test_unexpected_sql_failure_stays_failed_and_modelbusy_default_stays_unchanged(self) -> None: + conn = self.connection() + self.add(conn) + with patch.object(STYLE, "compute_style_vector", side_effect=sqlite3.OperationalError("synthetic unexpected")): + result = MAINTENANCE.run_style_maintenance(conn) + self.assertEqual((result.outcome, result.error_category), ("failed", "OperationalError")) + self.assertEqual(MAINTENANCE.LayerDeferredError("synthetic").category, "ModelBusy") + with self.assertRaises(ValueError): + MAINTENANCE.LayerDeferredError("synthetic", category="Unreviewed") + + def test_existing_adapter_modelbusy_result_and_checkpoint_are_preserved(self) -> None: + conn = self.connection() + dependency = MAINTENANCE.make_layer_dependency(1) + + def busy(_conn: sqlite3.Connection) -> None: + raise MAINTENANCE.LayerDeferredError("synthetic model owner") + + spec = MAINTENANCE.LayerSpec("summaries", "build_summaries", lambda: dependency, busy, lambda _conn: 0) + before = MAINTENANCE._read_layer_row(conn, "summaries") + restore = MAINTENANCE._restore_deferred_attempt + + def positional_restore(*args: object) -> None: + self.assertEqual(len(args), 5) + restore(*args) + + with patch.object(MAINTENANCE, "_restore_deferred_attempt", positional_restore): + result = MAINTENANCE.run_layers(conn, (spec,))[0] + self.assertEqual((result.outcome, result.error_category), ("deferred", "ModelBusy")) + self.assertEqual(MAINTENANCE._read_layer_row(conn, "summaries"), before) + + def test_python310_errors_and_extended_codes_retain_specific_categories(self) -> None: + cases = ( + ("synthetic unexpected", None, False, "failed", "OperationalError"), + ("database is locked", None, False, "deferred", "StyleWriterBusy"), + ("database table is locked: messages", None, False, "deferred", "StyleWriterBusy"), + ("database table is locked: maintenance_layers", None, False, "deferred", "StyleWriterBusy"), + ("interrupted", None, True, "deferred", "StyleCancelled"), + ("synthetic busy snapshot", 517, False, "deferred", "StyleWriterBusy"), + ("synthetic shared cache", 262, False, "deferred", "StyleWriterBusy"), + ("database is locked", 10, False, "failed", "OperationalError"), + ) + for message, code, cancelled, outcome, category in cases: + with self.subTest(code=code, category=category, message=message): + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + before = self.output(conn) + cancel = threading.Event() + error = sqlite3.OperationalError(message) + self.assertFalse(hasattr(error, "sqlite_errorcode")) + if code is not None: + error.sqlite_errorcode = code + + def fail(_text: str) -> list[float]: + if cancelled: + cancel.set() + raise error + + with patch.object(MAINTENANCE, "sqlite3", LegacySqliteModule()), \ + patch.object(STYLE, "compute_style_vector", fail): + result = MAINTENANCE.run_style_maintenance(conn, force=True, cancel=cancel) + self.assertEqual((result.outcome, result.error_category), (outcome, category)) + self.assertEqual(self.output(conn), before) + self.assertFalse(conn.in_transaction) + + def test_running_diagnostic_contention_defers_without_recording_an_attempt(self) -> None: + uri = "file:synthetic-style-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + first = self.connection(uri=uri) + second = self.connection(uri=uri) + self.add(first) + MAINTENANCE.run_style_maintenance(first) + before, checkpoint = self.output(first), self.checkpoint(first) + second.execute("BEGIN IMMEDIATE") + second.execute("INSERT INTO sentinel VALUES ('synthetic writer stays active')") + try: + result = MAINTENANCE.run_style_maintenance(first, force=True) + self.assertEqual((result.outcome, result.error_category, result.attempted), + ("deferred", "StyleWriterBusy", False)) + self.assertTrue(second.in_transaction) + self.assertFalse(first.in_transaction) + self.assertEqual(self.output(first), before) + self.assertEqual(self.checkpoint(first), checkpoint) + finally: + second.rollback() + self.assertEqual(MAINTENANCE.run_style_maintenance(first, force=True).outcome, "success") + + def test_restoration_contention_leaves_running_diagnostic_for_later_recovery(self) -> None: + uri = "file:synthetic-style-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + first = self.connection(uri=uri) + second = self.connection(uri=uri) + self.add(first) + MAINTENANCE.run_style_maintenance(first) + before, checkpoint = self.output(first), self.checkpoint(first) + compute = STYLE.compute_style_vector + + def contend(text: str) -> list[float]: + second.execute("BEGIN IMMEDIATE") + second.execute("INSERT INTO sentinel VALUES ('synthetic writer stays active')") + return compute(text) + + try: + with patch.object(STYLE, "compute_style_vector", contend): + result = MAINTENANCE.run_style_maintenance(first, force=True) + self.assertEqual((result.outcome, result.error_category), ("deferred", "StyleWriterBusy")) + self.assertTrue(second.in_transaction) + self.assertFalse(first.in_transaction) + self.assertEqual(self.output(first), before) + self.assertEqual(self.state(first).outcome, "running") + self.assertNotEqual(self.checkpoint(first), checkpoint) + finally: + second.rollback() + self.assertEqual(MAINTENANCE.run_style_maintenance(first).outcome, "success") + self.assertEqual(self.state(first).outcome, "success") + + def test_actual_delete_interrupt_confirms_owned_automatic_rollback(self) -> None: + for legacy in (False, True): + with self.subTest(legacy=legacy): + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + before, checkpoint = self.output(conn), self.checkpoint(conn) + cancel = threading.Event() + observations = [] + + class InterruptBoundary: + handler_active = False + + def __getattr__(boundary, name: str) -> object: + return getattr(conn, name) + + def set_progress_handler(boundary, handler: object, steps: int) -> None: + boundary.handler_active = handler is not None + conn.set_progress_handler(handler, 1 if handler is not None else 0) + + def execute(boundary, sql: str, *args: object) -> sqlite3.Cursor: + if sql == "DELETE FROM entity_style_vectors": + cancel.set() + elif cancel.is_set(): + self.assertFalse(boundary.handler_active) + try: + return conn.execute(sql, *args) + except sqlite3.OperationalError as error: + if sql == "DELETE FROM entity_style_vectors": + observations.append((str(error), conn.in_transaction)) + if legacy: + old_error = sqlite3.OperationalError(str(error)) + self.assertFalse(hasattr(old_error, "sqlite_errorcode")) + raise old_error from error + raise + + with patch.object(MAINTENANCE, "sqlite3", LegacySqliteModule() if legacy else sqlite3): + result = MAINTENANCE.run_style_maintenance(InterruptBoundary(), force=True, + cancel=cancel, connection_owned=True) + self.assertEqual(observations, [("interrupted", False)]) + self.assertEqual((result.outcome, result.error_category), ("deferred", "StyleCancelled")) + self.assertFalse(conn.in_transaction) + self.assertEqual(self.output(conn), before) + self.assertEqual(self.checkpoint(conn), checkpoint) + + def test_unwitnessed_interrupt_cannot_claim_an_ended_transaction_rolled_back(self) -> None: + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + cancel = threading.Event() + + def ended(_text: str) -> list[float]: + conn.commit() + cancel.set() + error = sqlite3.OperationalError("interrupted") + error.sqlite_errorcode = 9 + raise error + + with patch.object(STYLE, "compute_style_vector", ended): + with self.assertRaises(MAINTENANCE._LayerRollbackFailed): + MAINTENANCE.run_style_maintenance(conn, force=True, cancel=cancel, connection_owned=True) + self.assertEqual(self.state(conn).outcome, "running") + + def test_first_enrollment_and_trigger_repair_contention_preserve_caller(self) -> None: + for repair in (False, True): + for borrowed in (False, True): + with self.subTest(repair=repair, borrowed=borrowed): + uri = "file:synthetic-style-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + first = self.connection(uri=uri) + second = self.connection(uri=uri) + self.add(first) + if repair: + MAINTENANCE.run_style_maintenance(first) + first.execute("DROP TRIGGER truememory_style_output_update") + first.commit() + before, checkpoint = self.output(first), self.checkpoint(first) + first.execute("CREATE TEMP TABLE caller_pending(value TEXT)") + if borrowed: + first.execute("BEGIN") + first.execute("INSERT INTO caller_pending VALUES ('synthetic pending')") + second.execute("BEGIN IMMEDIATE") + second.execute("INSERT INTO sentinel VALUES ('synthetic persistent writer')") + try: + result = MAINTENANCE.run_style_maintenance(first, allow_caller_transaction=borrowed) + self.assertEqual((result.outcome, result.error_category, result.attempted), + ("deferred", "StyleWriterBusy", False)) + self.assertEqual(result.pending_caller_commit, borrowed) + self.assertTrue(second.in_transaction) + self.assertEqual(first.in_transaction, borrowed) + self.assertEqual(first.execute("SELECT count(*) FROM caller_pending").fetchone()[0], int(borrowed)) + self.assertEqual(self.output(first), before) + self.assertEqual(self.checkpoint(first), checkpoint) + self.assertFalse(STORAGE._style_output_tracking_ready(first)) + finally: + first.rollback() + second.rollback() + self.assertEqual(MAINTENANCE.run_style_maintenance(first).outcome, "success") + + def test_preparation_unknown_errors_or_unconfirmed_transaction_end_do_not_defer(self) -> None: + for ended in (False, True): + with self.subTest(ended=ended): + conn = self.connection() + self.add(conn) + + class UnsafePreparation: + def __getattr__(boundary, name: str) -> object: + return getattr(conn, name) + + def execute(boundary, sql: str, *args: object) -> sqlite3.Cursor: + if (ended and sql == "COMMIT") or (not ended and sql == "BEGIN IMMEDIATE"): + if ended: + conn.commit() + raise sqlite3.OperationalError("database is locked" if ended else "synthetic unknown") + return conn.execute(sql, *args) + + with self.assertRaises(sqlite3.OperationalError): + MAINTENANCE.run_style_maintenance(UnsafePreparation()) + self.assertFalse(conn.in_transaction) + self.assertEqual(self.checkpoint(conn) is not None, ended) + + def test_source_metadata_read_contention_before_enrollment_defers(self) -> None: + uri = "file:synthetic-style-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + first = self.connection(uri=uri) + second = self.connection(uri=uri) + self.add(first) + second.execute("BEGIN IMMEDIATE") + second.execute("INSERT INTO messages(sender,content) VALUES ('Synthetic','synthetic pending writer')") + try: + result = MAINTENANCE.run_style_maintenance(first) + self.assertEqual((result.outcome, result.error_category, result.attempted), + ("deferred", "StyleWriterBusy", False)) + self.assertTrue(second.in_transaction) + self.assertFalse(first.in_transaction) + self.assertIsNone(self.checkpoint(first)) + finally: + second.rollback() + self.assertEqual(MAINTENANCE.run_style_maintenance(first).outcome, "success") + + def test_failed_preparation_savepoint_rollback_is_not_deferred(self) -> None: + uri = "file:synthetic-style-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + first = self.connection(uri=uri) + second = self.connection(uri=uri) + self.add(first) + first.execute("BEGIN") + second.execute("BEGIN IMMEDIATE") + second.execute("INSERT INTO sentinel VALUES ('synthetic writer')") + + class RefusedSavepointRollback: + def __getattr__(boundary, name: str) -> object: + return getattr(first, name) + + def execute(boundary, sql: str, *args: object) -> sqlite3.Cursor: + if sql == "ROLLBACK TO SAVEPOINT truememory_style_tracking": + raise sqlite3.OperationalError("synthetic rollback refused") + return first.execute(sql, *args) + + try: + with self.assertRaisesRegex(sqlite3.OperationalError, "synthetic rollback refused"): + MAINTENANCE.run_style_maintenance(RefusedSavepointRollback(), allow_caller_transaction=True) + self.assertTrue(first.in_transaction) + self.assertTrue(second.in_transaction) + self.assertIsNone(self.checkpoint(first)) + finally: + first.rollback() + second.rollback() + + def test_read_baseline_races_after_preparation_initial_read_and_cancellation_defer(self) -> None: + for phase in ("initial", "layer", "cancelled"): + with self.subTest(phase=phase): + uri = "file:synthetic-style-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + first = self.connection(uri=uri) + second = self.connection(uri=uri) + self.add(first) + MAINTENANCE.run_style_maintenance(first) + before, checkpoint = self.output(first), self.checkpoint(first) + prepare = MAINTENANCE.prepare_style_maintenance + read = MAINTENANCE.read_layer_states + reads = [] + cancel = threading.Event() + if phase == "cancelled": + cancel.set() + + def contend() -> None: + second.execute("BEGIN IMMEDIATE") + second.execute("INSERT INTO messages(sender,content) VALUES ('Synthetic','synthetic pending writer')") + + def prepared(conn: sqlite3.Connection, **kwargs: object) -> None: + prepare(conn, **kwargs) + if phase == "initial": + contend() + + def baseline(conn: sqlite3.Connection, specs: tuple) -> dict: + reads.append(1) + result = read(conn, specs) + if len(reads) == 1 and phase != "initial": + contend() + return result + + try: + with patch.object(MAINTENANCE, "prepare_style_maintenance", prepared), \ + patch.object(MAINTENANCE, "read_layer_states", baseline): + result = MAINTENANCE.run_style_maintenance(first, force=True, cancel=cancel) + self.assertEqual((result.outcome, result.error_category, result.attempted), + ("deferred", "StyleWriterBusy", False)) + self.assertEqual(len(reads), 1 if phase == "initial" else 2) + self.assertTrue(second.in_transaction) + self.assertFalse(first.in_transaction) + self.assertEqual(self.output(first), before) + self.assertEqual(self.checkpoint(first), checkpoint) + finally: + second.rollback() + self.assertEqual(MAINTENANCE.run_style_maintenance(first, force=True).outcome, "success") + + def test_read_deferral_requires_known_lock_and_unchanged_caller_boundary(self) -> None: + for borrowed in (False, True): + for changed_boundary in (False, True): + for locked in (False, True): + with self.subTest(borrowed=borrowed, changed_boundary=changed_boundary, locked=locked): + conn = self.connection() + self.add(conn) + MAINTENANCE.run_style_maintenance(conn) + conn.execute("CREATE TEMP TABLE caller_pending(value TEXT)") + if borrowed: + conn.execute("BEGIN") + conn.execute("INSERT INTO caller_pending VALUES ('synthetic pending')") + + def failed_read(_conn: sqlite3.Connection, _specs: tuple) -> dict: + if changed_boundary: + if borrowed: + conn.commit() + else: + conn.execute("BEGIN") + raise sqlite3.OperationalError("database table is locked: maintenance_source_state" + if locked else "synthetic unexpected read") + + with patch.object(MAINTENANCE, "read_layer_states", failed_read): + if locked and not changed_boundary: + result = MAINTENANCE.run_style_maintenance(conn, force=True, allow_caller_transaction=borrowed) + self.assertEqual((result.outcome, result.error_category, result.pending_caller_commit), + ("deferred", "StyleWriterBusy", borrowed)) + self.assertEqual(conn.in_transaction, borrowed) + self.assertEqual(conn.execute("SELECT count(*) FROM caller_pending").fetchone()[0], int(borrowed)) + else: + with self.assertRaises(sqlite3.OperationalError): + MAINTENANCE.run_style_maintenance(conn, force=True, allow_caller_transaction=borrowed) + conn.rollback() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test-maintenance-adapters-753.py b/tests/test-maintenance-adapters-753.py index a0454f79..ebc0601c 100644 --- a/tests/test-maintenance-adapters-753.py +++ b/tests/test-maintenance-adapters-753.py @@ -57,7 +57,11 @@ def setUp(self) -> None: ("vec_source_v1:vec_messages_edge", '{"version":1,"generation":"synthetic-a","cursor":0,"complete":true}'), ]) self.conn.commit() - self.enterContext(patch.dict(sys.modules, {"truememory.vector_search": self.vector})) + # Detached production workers must not discover this fixture's model + # or hold its lock through the process-wide module registry. + self.enterContext(patch.object(MAINTENANCE, "sys", types.SimpleNamespace( + modules={"truememory.vector_search": self.vector}, version_info=sys.version_info, + ))) self.enterContext(patch.object(MAINTENANCE, "importlib", types.SimpleNamespace( import_module=lambda name: self.modules[name], metadata=importlib.metadata, util=importlib.util, ))) diff --git a/tests/test-maintenance-engine-routing-753.py b/tests/test-maintenance-engine-routing-753.py new file mode 100644 index 00000000..5c8c02ff --- /dev/null +++ b/tests/test-maintenance-engine-routing-753.py @@ -0,0 +1,783 @@ +"""Engine routing with real SQLite, event-controlled owners and synthetic models.""" + +import ast +import builtins +import gc +import logging +import os +import select +import signal +import runpy +import sqlite3 +import sys +import threading +import time +import types +import unittest +import warnings +import weakref +from contextlib import contextmanager +from pathlib import Path +from unittest.mock import patch + + +ROOT = Path(__file__).resolve().parents[1] +ADAPTERS = runpy.run_path(str(Path(__file__).with_name("test-maintenance-adapters-753.py"))) +STORAGE, MAINTENANCE = ADAPTERS["STORAGE"], ADAPTERS["MAINTENANCE"] + + +def stdlib_module(name: str, modules: dict) -> types.ModuleType: + module = types.ModuleType("synthetic_routing_" + name) + + def safe_import(name, globals=None, locals=None, fromlist=(), level=0): + if name in modules: + return modules[name] + if name.startswith("truememory"): + raise ImportError("Synthetic boundary excludes native modules") + return builtins.__import__(name, globals, locals, fromlist, level) + + module.__dict__["__builtins__"] = dict(vars(builtins), __import__=safe_import) + path = ROOT / "truememory" / (name + ".py") + with patch.dict(sys.modules, {module.__name__: module}): + exec(compile(path.read_text(encoding="utf-8"), str(path), "exec"), module.__dict__) + return module + + +def load_engine(modules: dict) -> types.ModuleType: + module = types.ModuleType("synthetic_routing_engine") + + def safe_import(name, globals=None, locals=None, fromlist=(), level=0): + if name in modules: + return modules[name] + if name.startswith("truememory"): + raise ImportError("Synthetic boundary excludes native modules") + return builtins.__import__(name, globals, locals, fromlist, level) + + module.__dict__.update({"__builtins__": dict(vars(builtins), __import__=safe_import), + "__file__": str(ROOT / "truememory/engine.py"), "Path": Path, "sqlite3": sqlite3, + "threading": threading, "time": time, "os": os, "warnings": warnings, + "logger": logging.getLogger("synthetic-routing"), "MAX_CONTENT_LENGTH": 50_000, + "_env_int": lambda name, default, **kwargs: default}) + source = ast.parse((ROOT / "truememory/engine.py").read_text(encoding="utf-8")) + nodes = [ast.ImportFrom(module="__future__", names=[ast.alias(name="annotations")], level=0)] + for node in source.body: + if isinstance(node, (ast.FunctionDef, ast.ClassDef)): + nodes.append(node) + elif isinstance(node, ast.ImportFrom) and node.module == "truememory.storage": + module.__dict__.update({item.asname or item.name: getattr(STORAGE, item.name) for item in node.names}) + elif isinstance(node, ast.Assign): + for target in node.targets: + if isinstance(target, ast.Name): + if target.id.startswith("_HAS_"): + module.__dict__[target.id] = False + elif target.id in {"_ALLOWED_TABLES", "_ALLOWED_COLUMNS", "_ALL_VEC_TABLES", "_SQLITE_IN_CHUNK"}: + nodes.append(node) + module.__dict__["_HAS_CONSOLIDATION"] = True + exec(compile(ast.fix_missing_locations(ast.Module(body=nodes, type_ignores=[])), + str(ROOT / "truememory/engine.py"), "exec"), module.__dict__) + return module + + +class RoutingFixture(ADAPTERS["AdapterFixture"]): + def setUp(self) -> None: + style = stdlib_module("personality_style_vec", {}) + self.enterContext(patch.dict(sys.modules, {"truememory.personality_style_vec": style})) + super().setUp() + self.modules.update({"truememory.storage": STORAGE, "truememory.maintenance": MAINTENANCE, + "truememory.vector_search": self.vector, + "sqlite_vec": types.SimpleNamespace(load=lambda conn: None)}) + self.modules["truememory.rebuild_source"] = stdlib_module("rebuild_source", self.modules) + self.engine_module = load_engine(self.modules) + create = STORAGE.create_db + + class ExtensionBoundary: + def __init__(boundary, conn): + boundary.conn = conn + + def __getattr__(boundary, name): + return getattr(boundary.conn, name) + + def enable_load_extension(boundary, enabled): + # The fixture uses ordinary BLOB tables and a fake sqlite-vec + # loader, including hosts compiled without extension support. + self.assertIs(type(enabled), bool) + + self.enterContext(patch.object(MAINTENANCE, "create_db", lambda path: ExtensionBoundary(create(path)))) + self.enterContext(patch.object(self.engine_module, "create_db", lambda path: ExtensionBoundary(create(path)))) + self.engine = self.new_engine(self.conn) + self.coordinator = self.engine._get_maintenance_coordinator() + self.addCleanup(self.coordinator.wait, 3) + self.addCleanup(self.coordinator.cancel) + + def new_engine(self, conn: sqlite3.Connection | None = None): + engine = self.engine_module.TrueMemoryEngine(self.path) + engine.conn = conn or self.open_db() + engine.ready = True + engine._has_consolidation = False + self.addCleanup(engine.close) + return engine + + def completed(self) -> None: + self.assertTrue(self.coordinator.wait(3), "Synthetic maintenance did not terminate") + self.assertFalse(self.coordinator.snapshot()["active"]) + + def fake_report(self): + return MAINTENANCE.MaintenanceReport((), "SKIPPED (synthetic)") + + +class TestCoordinatorHandoff(RoutingFixture): + def test_local_owner_retains_one_pending_wake_after_last_engine_close(self) -> None: + path = self.path.with_name("synthetic-detached.sqlite") + engine = self.engine_module.TrueMemoryEngine(path) + engine.conn, engine.ready = STORAGE.create_db(path), True + coordinator = engine._get_maintenance_coordinator() + reference, engine_reference = weakref.ref(coordinator), weakref.ref(engine) + entered, release = threading.Event(), threading.Event() + observed = [] + + def work(conn, owner, **kwargs): + observed.append(kwargs["threshold"]) + entered.set() + self.assertTrue(release.wait(3)) + return self.fake_report() + + with patch.object(MAINTENANCE, "run_engine_maintenance", work): + try: + with MAINTENANCE.maintenance_owner(path): + coordinator.request_layers(threshold=31) + coordinator.request_layers(threshold=37) + self.assertEqual(len(MAINTENANCE._owner_waiters), 1) + engine.close() + del engine, coordinator + gc.collect() + self.assertIsNone(engine_reference()) + self.assertIsNotNone(reference()) + self.assertTrue(entered.wait(2)) + self.assertEqual(MAINTENANCE._owner_waiters, {}) + coordinator = reference() + self.assertIsNotNone(coordinator) + finally: + release.set() + if reference() is not None: + self.assertTrue(reference().wait(3)) + self.assertEqual(observed, [37]) + + def test_cancel_releases_local_waiter_and_admission_cannot_restore_it(self) -> None: + original = MAINTENANCE._acquire_owner + + def cancelled_admission(path, on_busy=None): + self.coordinator.cancel() + return original(path, on_busy=on_busy) + + with patch.object(MAINTENANCE, "run_engine_maintenance") as work: + with MAINTENANCE.maintenance_owner(self.path): + self.coordinator.request_layers() + self.assertEqual(len(MAINTENANCE._owner_waiters), 1) + self.coordinator.cancel() + self.assertEqual(MAINTENANCE._owner_waiters, {}) + with patch.object(MAINTENANCE, "_acquire_owner", cancelled_admission): + self.assertFalse(self.coordinator.request_layers()) + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 0) + self.assertEqual(MAINTENANCE._owner_waiters, {}) + work.assert_not_called() + self.assertEqual(self.coordinator.status, ("cancelled", None)) + + def test_delayed_start_exception_cannot_cancel_successor_or_its_pending_wake(self) -> None: + start = threading.Thread.start + entered, release = threading.Event(), threading.Event() + attempts, observed, cancellation = [], [], [] + + def work(conn, coordinator, **kwargs): + observed.append(kwargs["threshold"]) + if kwargs["threshold"] == 31: + entered.set() + self.assertTrue(release.wait(3)) + cancellation.append(kwargs["cancel"].is_set()) + return self.fake_report() + + def late_failure(worker): + attempts.append(worker) + start(worker) + if len(attempts) == 1: + self.completed() + self.assertTrue(self.coordinator.request_layers(threshold=31)) + self.assertTrue(entered.wait(2)) + self.assertFalse(self.coordinator.request_layers(threshold=37)) + raise RuntimeError("synthetic delayed start failure") + + with patch.object(MAINTENANCE.threading.Thread, "start", late_failure), \ + patch.object(MAINTENANCE, "run_engine_maintenance", work): + try: + with self.assertRaisesRegex(RuntimeError, "synthetic delayed start failure"): + self.coordinator.request_layers() + self.assertFalse(self.coordinator._cancel.is_set()) + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 1) + finally: + release.set() + self.completed() + self.assertEqual(observed, [25, 31, 37]) + self.assertEqual(cancellation, [False]) + + @unittest.skipUnless(hasattr(os, "fork"), "POSIX fork required") + def test_fork_discards_inherited_pending_worker_and_rejects_stale_coordinator(self) -> None: + entered, release = threading.Event(), threading.Event() + + def active(conn, cancel): + entered.set() + self.assertTrue(release.wait(3)) + + self.coordinator.request(active) + self.assertTrue(entered.wait(2)) + self.coordinator.request_layers() + waiting_path = self.path.with_name("synthetic-fork-waiter.sqlite") + self.enterContext(MAINTENANCE.maintenance_owner(waiting_path)) + waiting = MAINTENANCE.get_coordinator(waiting_path) + self.addCleanup(waiting.cancel) + waiting.request_layers() + self.assertEqual(len(MAINTENANCE._owner_waiters), 1) + read_fd, write_fd = os.pipe() + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + child = os.fork() + if child == 0: + os.close(read_fd) + try: + rejected = False + try: + self.coordinator.request_layers() + except RuntimeError: + rejected = True + fresh = MAINTENANCE.get_coordinator(self.path) + clean = (not fresh.snapshot()["active"] and fresh.snapshot()["pending_generations"] == 0 + and not MAINTENANCE._owner_waiters) + os.write(write_fd, b"PASS" if rejected and clean else b"FAIL") + finally: + os.close(write_fd) + os._exit(0) + os.close(write_fd) + try: + self.assertTrue(select.select([read_fd], [], [], 2)[0], "Synthetic child did not answer") + self.assertEqual(os.read(read_fd, 4), b"PASS") + finally: + os.close(read_fd) + if os.waitpid(child, os.WNOHANG)[0] == 0: + os.kill(child, signal.SIGKILL) + os.waitpid(child, 0) + self.coordinator.cancel() + release.set() + self.completed() + + def test_start_failure_services_only_a_real_later_notification(self) -> None: + start = threading.Thread.start + attempts, observed = [], [] + + def fail_once(worker): + attempts.append(True) + if len(attempts) == 1: + self.assertFalse(self.coordinator.request_layers(threshold=31)) + raise RuntimeError("synthetic start failure") + return start(worker) + + with patch.object(MAINTENANCE.threading.Thread, "start", fail_once), \ + patch.object(MAINTENANCE, "run_engine_maintenance", lambda *args, **kwargs: observed.append(kwargs["threshold"]) or self.fake_report()): + with self.assertRaisesRegex(RuntimeError, "synthetic start failure"): + self.coordinator.request_layers() + self.completed() + self.assertEqual(observed, [31]) + self.assertEqual(len(attempts), 2) + + def test_start_failure_without_notification_has_no_retry(self) -> None: + with patch.object(MAINTENANCE.threading.Thread, "start", side_effect=RuntimeError("synthetic")) as start: + with self.assertRaises(RuntimeError): + self.coordinator.request_layers() + self.completed() + start.assert_called_once() + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 0) + self.assertEqual(self.coordinator.status, ("failed", "worker_start")) + + def test_close_detaches_without_cancelling_or_retaining_engine(self) -> None: + entered, release = threading.Event(), threading.Event() + + def active(conn, cancel): + entered.set() + self.assertTrue(release.wait(2)) + self.assertFalse(cancel.is_set()) + self.assertEqual(conn.execute("SELECT 1").fetchone()[0], 1) + + with patch.object(MAINTENANCE, "run_engine_maintenance", lambda *args, **kwargs: self.fake_report()): + self.coordinator.request(active) + self.assertTrue(entered.wait(2)) + engine = self.engine_module.TrueMemoryEngine(self.path) + engine.conn, engine.ready = self.open_db(), True + reference = weakref.ref(engine) + engine._maybe_auto_consolidate() + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 1) + engine.close() + del engine + gc.collect() + self.assertIsNone(reference()) + self.assertFalse(self.coordinator.wait(0)) + release.set() + self.completed() + self.assertEqual(self.coordinator.status, ("success", None)) + + def test_busy_admission_and_concurrent_owner_release_cannot_lose_wake(self) -> None: + held, release, leaving, ran = (threading.Event() for _ in range(4)) + original_acquire, original_release = MAINTENANCE._acquire_owner, MAINTENANCE._release_owner + + def owner(): + with MAINTENANCE.maintenance_owner(self.path): + held.set() + self.assertTrue(release.wait(2)) + + def release_owner(fd): + leaving.set() + original_release(fd) + + def acquire(path, on_busy=None): + def busy(): + on_busy() + release.set() + self.assertTrue(leaving.wait(2)) + return original_acquire(path, on_busy=busy if on_busy is not None else None) + + owner_thread = threading.Thread(target=owner) + with patch.object(MAINTENANCE, "_release_owner", release_owner), \ + patch.object(MAINTENANCE, "run_engine_maintenance", lambda *args, **kwargs: ran.set() or self.fake_report()): + owner_thread.start() + try: + self.assertTrue(held.wait(2)) + with patch.object(MAINTENANCE, "_acquire_owner", acquire): + self.assertFalse(self.coordinator.request_layers()) + self.assertTrue(ran.wait(2)) + self.completed() + finally: + release.set() + owner_thread.join(3) + self.assertFalse(owner_thread.is_alive()) + + def test_many_active_notifications_coalesce_to_one_static_successor(self) -> None: + entered, release = threading.Event(), threading.Event() + observed = [] + + def active(conn, cancel): + entered.set() + self.assertTrue(release.wait(2)) + + def successor(conn, coordinator, **kwargs): + observed.append((conn is self.conn, kwargs["threshold"])) + return self.fake_report() + + with patch.object(MAINTENANCE, "run_engine_maintenance", successor): + self.assertTrue(self.coordinator.request(active)) + self.assertTrue(entered.wait(2)) + for threshold in range(1, 101): + self.assertFalse(self.coordinator.request_layers(threshold=threshold)) + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 1) + self.assertFalse(self.coordinator.wait(0)) + release.set() + self.completed() + self.assertEqual(observed, [(False, 100)]) + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 0) + + def test_local_synchronous_owner_release_services_pending_wake(self) -> None: + observed = [] + with patch.object(MAINTENANCE, "run_engine_maintenance", lambda *args, **kwargs: observed.append(True) or self.fake_report()): + with MAINTENANCE.maintenance_owner(self.path): + self.assertFalse(self.coordinator.request_layers()) + self.assertEqual(self.coordinator.status, ("busy", None)) + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 1) + self.assertEqual(observed, []) + self.completed() + self.assertEqual(observed, [True]) + + def test_external_busy_requires_new_boundary_and_does_not_spin(self) -> None: + observed = [] + with patch.object(MAINTENANCE, "try_file_lock", return_value=None), \ + patch.object(MAINTENANCE.threading, "Thread") as thread: + self.assertFalse(self.coordinator.request_layers()) + thread.assert_not_called() + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 1) + self.assertEqual(MAINTENANCE._owner_waiters, {}) + with patch.object(MAINTENANCE, "run_engine_maintenance", lambda *args, **kwargs: observed.append(True) or self.fake_report()): + self.assertTrue(self.coordinator.request_layers()) + self.completed() + self.assertEqual(observed, [True]) + + def test_wake_between_descriptor_release_and_idle_transition_is_not_lost(self) -> None: + released, finish = threading.Event(), threading.Event() + original_release = MAINTENANCE._release_owner + observed = [] + first = [] + + def pause(fd): + original_release(fd) + if not first: + first.append(True) + released.set() + self.assertTrue(finish.wait(2)) + + with patch.object(MAINTENANCE, "_release_owner", pause), \ + patch.object(MAINTENANCE, "run_engine_maintenance", lambda *args, **kwargs: observed.append(True) or self.fake_report()): + self.assertTrue(self.coordinator.request(lambda conn, cancel: None)) + self.assertTrue(released.wait(2)) + self.assertFalse(self.coordinator.request_layers()) + self.assertFalse(self.coordinator.wait(0)) + finish.set() + self.completed() + self.assertEqual(observed, [True]) + + def test_model_deferral_has_no_self_generated_successor(self) -> None: + observed = [] + report = MAINTENANCE.MaintenanceReport((MAINTENANCE.LayerResult( + "clusters", "cluster_messages", "deferred", None, 0., "ModelBusy", False),), "SKIPPED") + with patch.object(MAINTENANCE, "run_engine_maintenance", lambda *args, **kwargs: observed.append(True) or report): + self.assertTrue(self.coordinator.request_layers()) + self.completed() + self.assertEqual(observed, [True]) + self.assertEqual(self.coordinator.status, ("deferred", "ModelBusy")) + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 0) + + def test_cancellation_drops_pending_but_keeps_active_ownership(self) -> None: + entered, release = threading.Event(), threading.Event() + + def active(conn, cancel): + entered.set() + self.assertTrue(release.wait(2)) + + with patch.object(MAINTENANCE, "run_engine_maintenance") as successor: + self.coordinator.request(active) + self.assertTrue(entered.wait(2)) + self.coordinator.request_layers() + self.coordinator.cancel() + self.assertFalse(self.coordinator.wait(0)) + self.assertFalse(self.coordinator.request_layers()) + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 0) + release.set() + self.completed() + successor.assert_not_called() + self.assertEqual(self.coordinator.status, ("cancelled", None)) + + +class TestEngineRouting(RoutingFixture): + def test_status_reports_categorical_layer_failure_and_caller_transaction(self) -> None: + self.add(1) + with patch.object(self.cluster, "cluster_messages", side_effect=RuntimeError("synthetic private sentinel")): + self.engine.consolidate() + state = self.engine.get_stats()["maintenance"] + self.assertEqual(len(state["layers"]), 8) + self.assertEqual(state["layers"]["clusters"]["outcome"], "failed") + self.assertEqual(state["layers"]["clusters"]["error_category"], "RuntimeError") + self.assertEqual(state["layers"]["clusters"]["freshness"], "untrusted") + self.assertNotIn("synthetic private sentinel", str(state)) + self.assertNotIn(str(self.path), str(state)) + self.conn.execute("UPDATE messages SET content='synthetic uncommitted status'") + self.assertTrue(self.engine.get_stats()["maintenance"]["pending_caller_commit"]) + self.assertTrue(self.conn.in_transaction) + self.conn.rollback() + + def test_manual_repair_restores_actual_cluster_search_and_automatic_eligibility(self) -> None: + self.add(5) + self.native.fit = lambda: setattr(self.native, "labels", [0, 0, 1, 1, 2]) + self.engine._has_clustering = False + self.engine._has_vectors = True + self.modules["truememory.agentic_search"] = types.SimpleNamespace(normalize_scores=lambda rows: rows) + with patch.object(self.coordinator, "request_layers") as request, \ + patch.object(self.engine, "search", return_value=[{"id": 1, "score": .5}]), \ + patch.object(self.engine, "_entity_focused_search", return_value=[]), \ + patch.object(self.engine, "_check_sufficiency", return_value=True), \ + patch.object(self.engine, "_apply_surprise_boost", side_effect=lambda rows: rows), \ + patch.object(self.engine, "_clean_results", side_effect=lambda rows, *_args, **_kwargs: rows), \ + patch.object(self.engine_module, "search_clustered", return_value=[{"id": 2, "score": .5}], create=True) as search: + self.engine._maybe_auto_consolidate() + request.assert_not_called() + self.engine.search_agentic("synthetic query", use_hyde=False, use_reranker=False) + search.assert_not_called() + result = self.engine.consolidate() + self.assertIn("3 clusters", result["cluster_messages"]) + self.assertTrue(self.engine._has_clustering) + self.assertTrue(self.engine._has_consolidation) + rows = self.engine.search_agentic("synthetic query", use_hyde=False, use_reranker=False) + search.assert_called_once() + self.assertEqual([row["id"] for row in rows], [1, 2]) + request.assert_not_called() + self.conn.execute("UPDATE messages SET content='synthetic repaired correction' WHERE id=1") + self.conn.commit() + self.engine._maybe_auto_consolidate() + request.assert_called_once_with(threshold=25) + + def test_failed_manual_repair_does_not_enable_capabilities_or_remove_prior_success(self) -> None: + self.engine._has_clustering = False + with patch.object(self.cluster, "cluster_messages", side_effect=RuntimeError("synthetic")), \ + patch.object(self.modules["truememory.consolidation"], "build_summaries", side_effect=RuntimeError("synthetic")): + result = self.engine.consolidate() + self.assertIn("ERROR", result["cluster_messages"]) + self.assertIn("ERROR", result["build_summaries"]) + self.assertFalse(self.engine._has_clustering) + self.assertFalse(self.engine._has_consolidation) + self.engine._has_clustering = self.engine._has_consolidation = True + self.engine.consolidate() + self.assertTrue(self.engine._has_clustering) + self.assertTrue(self.engine._has_consolidation) + + def test_unavailable_manual_cluster_repair_preserves_only_prior_capability(self) -> None: + self.engine._has_clustering = False + with patch.object(self.modules["sqlite_vec"], "load", side_effect=RuntimeError("synthetic")): + self.assertIn("UNAVAILABLE", self.engine.consolidate()["cluster_messages"]) + self.assertFalse(self.engine._has_clustering) + self.engine._has_clustering = True + self.assertIn("UNAVAILABLE", self.engine.consolidate()["cluster_messages"]) + self.assertTrue(self.engine._has_clustering) + + def test_first_open_manual_does_not_start_its_own_competitor(self) -> None: + self.add(1) + engine = self.engine_module.TrueMemoryEngine(self.path) + self.addCleanup(engine.close) + with patch.object(self.coordinator, "request_layers") as request: + result = engine.consolidate() + request.assert_not_called() + self.assertTrue(engine.ready) + self.assertEqual(set(result), set(MAINTENANCE._MAINTENANCE_RESULT_KEYS)) + + def test_current_empty_startup_and_rolled_back_correction_do_not_schedule(self) -> None: + self.engine.consolidate() + self.engine._has_consolidation = True + with patch.object(self.coordinator, "request_layers") as request: + self.engine._maybe_startup_consolidate() + self.engine._ensure_connection() + request.assert_not_called() + self.add(1) + self.engine.consolidate() + with patch.object(self.coordinator, "request_layers") as request: + self.conn.execute("UPDATE messages SET content='synthetic uncommitted'") + self.engine._ensure_connection() + self.assertEqual(self.engine._maintenance_pending_reason, "pending_caller_commit") + self.conn.rollback() + self.engine._ensure_connection() + request.assert_not_called() + self.conn.execute("UPDATE messages SET content='synthetic committed'") + self.conn.commit() + self.engine._ensure_connection() + request.assert_called_once_with(threshold=25) + + def test_add_probe_uses_no_corpus_scan_native_import_or_blocking_model_lock(self) -> None: + self.engine.consolidate() + self.engine._has_consolidation = True + operations = [] + underlying = self.vector._lock + + class ProbeLock: + def acquire(lock, blocking=True): + self.assertFalse(blocking) + return underlying.acquire(blocking=False) + + def release(lock): + underlying.release() + + def authorize(action, table, column, *_args): + if action == sqlite3.SQLITE_READ: + operations.append((table, column)) + return sqlite3.SQLITE_OK + + self.conn.set_authorizer(authorize) + try: + with patch.object(self.vector, "_lock", ProbeLock()), \ + patch.object(MAINTENANCE, "_installed_dependency", side_effect=AssertionError("per-add package scan")), \ + patch.object(self.coordinator, "request_layers", side_effect=AssertionError("unexpected work")): + self.engine._maybe_auto_consolidate() + self.assertEqual(self.engine._maintenance_pending_reason, "awaiting_eligibility") + finally: + self.conn.set_authorizer(lambda *_args: sqlite3.SQLITE_OK) + self.assertFalse({table for table, _ in operations}.intersection({"messages", "vec_messages_edge", "summaries", "cluster_centroids"})) + + def test_source_commit_followed_by_a17_vector_error_still_notifies(self) -> None: + self.add(1) + self.engine.consolidate() + self.engine._has_vectors = self.engine._has_consolidation = True + changed = type("VectorPublicationChanged", (RuntimeError,), {}) + + @contextmanager + def publication(*args): + raise changed("synthetic model replacement") + yield + + def capture(conn: sqlite3.Connection) -> tuple[object, str]: + self.assertIs(conn, self.engine.conn) + return object(), "synthetic-identity" + + fields = {"_capture_rebuild_model": capture, + "_encode_with_mps_fallback": lambda *args: [[1., 2.]], + "_build_sep_text": lambda *args: "synthetic separated text", + "_foreground_vector_publication": publication, + "serialize_f32": lambda values: b"synthetic-unused-vector", + "_write_embedder_metadata_no_commit": lambda conn: None, + "VectorPublicationChanged": changed} + with patch.dict(self.vector.__dict__, fields), patch.object(self.coordinator, "request_layers") as request: + with self.assertRaisesRegex(changed, "Source update committed"): + self.engine.update(1, content="synthetic corrected source") + request.assert_called_once_with(threshold=25) + self.assertFalse(self.conn.in_transaction) + self.assertFalse(self.engine._write_lock.locked()) + self.assertEqual(self.conn.execute("SELECT content FROM messages WHERE id=1").fetchone()[0], "synthetic corrected source") + + def test_manual_busy_is_truthful_and_does_not_queue_forced_work(self) -> None: + entered, release = threading.Event(), threading.Event() + + def active(conn, cancel): + entered.set() + self.assertTrue(release.wait(2)) + + self.coordinator.request(active) + self.assertTrue(entered.wait(2)) + try: + result = self.engine.consolidate() + self.assertEqual(set(result), set(MAINTENANCE._MAINTENANCE_RESULT_KEYS)) + self.assertTrue(all(value.startswith("BUSY") for value in result.values())) + self.assertEqual(self.coordinator.snapshot()["pending_generations"], 0) + finally: + release.set() + self.completed() + + def test_file_manual_uses_dedicated_connection_all_eight_and_preferences(self) -> None: + self.add(5) + seen = [] + original = MAINTENANCE.run_engine_maintenance + + def run(conn, coordinator, **kwargs): + seen.append(conn) + self.assertIsNot(conn, self.conn) + self.assertTrue(self.engine._write_lock.acquire(blocking=False)) + self.engine._write_lock.release() + return original(conn, coordinator, **kwargs) + + with patch.object(MAINTENANCE, "run_engine_maintenance", run), \ + patch.object(self.coordinator, "request_layers") as automatic: + result = self.engine.consolidate() + automatic.assert_not_called() + self.assertEqual(set(result), set(MAINTENANCE._MAINTENANCE_RESULT_KEYS)) + self.assertTrue(all("ERROR" not in value and "UNAVAILABLE" not in value for value in result.values()), result) + self.assertIn("vector_generation_unverified", result["cluster_messages"]) + with self.assertRaises(sqlite3.ProgrammingError): + seen[0].execute("SELECT 1") + + def test_24_then_25_committed_adds_across_engines_use_durable_counts(self) -> None: + self.engine.consolidate() + second = self.new_engine() + self.engine._has_consolidation = second._has_consolidation = True + with patch.object(self.coordinator, "request_layers", wraps=self.coordinator.request_layers) as request: + for index in range(24): + engine = self.engine if index % 2 else second + engine.add("synthetic append", sender="synthetic-sender") + request.assert_not_called() + second.add("synthetic twenty fifth", sender="synthetic-sender") + self.completed() + self.assertGreaterEqual(request.call_count, 1) + states = MAINTENANCE.read_layer_states(self.conn, self.specs()) + self.assertTrue(all(state.attempted_insert_count == 25 for state in states.values()), states) + + def test_borrowed_manual_preserves_caller_writes_and_rollback(self) -> None: + self.add(1, commit=False) + with patch.object(self.engine_module, "create_db", side_effect=AssertionError("new handle")): + result = self.engine.consolidate() + self.assertTrue(all("pending caller commit" in value for value in result.values()), result) + self.assertTrue(self.conn.in_transaction) + self.conn.rollback() + self.assertEqual(self.conn.execute("SELECT count(*) FROM messages").fetchone()[0], 0) + self.assertEqual(self.output(), ([], [])) + + def test_in_memory_auto_stays_pending_and_manual_uses_original_rows(self) -> None: + conn = STORAGE.create_db(":memory:") + self.addCleanup(conn.close) + engine = self.engine_module.TrueMemoryEngine(":memory:") + engine.conn, engine.ready = conn, True + self.addCleanup(engine.close) + conn.execute("INSERT INTO messages(content,timestamp) VALUES ('synthetic in memory','2026-01-01')") + conn.commit() + with patch.object(MAINTENANCE, "create_db", side_effect=AssertionError("new handle")), \ + patch.object(MAINTENANCE.threading, "Thread", side_effect=AssertionError("memory worker")): + engine._maybe_auto_consolidate() + self.assertEqual(engine._maintenance_coordinator.status, ("pending_in_memory", None)) + result = engine.consolidate() + self.assertIn("1 episodes", result["detect_episodes"]) + self.assertEqual(conn.execute("SELECT count(*) FROM messages").fetchone()[0], 1) + + +class TestCapabilityEvidence(RoutingFixture): + def test_transient_model_busy_cannot_clear_prior_extension_failure_baseline(self) -> None: + self.add(5) + with patch.object(self.modules["sqlite_vec"], "load", side_effect=RuntimeError("synthetic")): + self.engine.consolidate() + prior = self.coordinator.capability_snapshot()[2] + self.add(1) + worker = MAINTENANCE.create_db(self.path) + try: + with self.vector._lock: + MAINTENANCE._prepare_worker_extensions(worker, self.coordinator) + finally: + worker.close() + self.assertEqual(self.coordinator.capability_snapshot()[2], prior) + self.assertEqual(MAINTENANCE.plan_layers(self.conn, MAINTENANCE.engine_layer_specs(self.conn, self.coordinator)), ()) + + def test_reopen_during_worker_failure_preserves_current_run_local_evidence(self) -> None: + self.add(1) + + def fail_and_refresh(conn): + self.coordinator.refresh_capabilities() + raise RuntimeError("synthetic load failure after reopen") + + with patch.object(self.modules["sqlite_vec"], "load", fail_and_refresh): + result = self.engine.consolidate() + self.assertIn("VectorExtensionUnavailable", result["cluster_messages"]) + self.assertIsNone(self.coordinator.capability_snapshot()[2]) + + def test_worker_extension_failure_keeps_same_key_and_25_insert_baseline(self) -> None: + self.add(5) + loader = self.modules["sqlite_vec"] + with patch.object(loader, "load", side_effect=RuntimeError("synthetic private sentinel")): + result = self.engine.consolidate() + self.assertIn("VectorExtensionUnavailable", result["cluster_messages"]) + self.engine._has_consolidation = True + with patch.object(self.coordinator, "request_layers") as request: + self.engine.add("synthetic sixth") + self.engine._ensure_connection() + request.assert_not_called() + state = MAINTENANCE.read_layer_states(self.conn, MAINTENANCE.engine_layer_specs(self.conn, self.coordinator))["clusters"] + self.assertEqual(state.attempted_insert_count, 5) + self.assertEqual(state.dependency.key, state.attempted_dependency) + repaired = self.engine.consolidate() + self.assertNotIn("UNAVAILABLE", repaired["cluster_messages"]) + self.assertIsNone(self.coordinator.capability_snapshot()[2]) + + def test_reopen_refreshes_static_evidence_and_new_model_invalidates_failure_key(self) -> None: + self.add(1) + with patch.object(self.modules["sqlite_vec"], "load", side_effect=RuntimeError("synthetic")): + self.engine.consolidate() + epoch, versions, failure = self.coordinator.capability_snapshot() + self.assertEqual(len(versions), 3) + self.assertIsNotNone(failure) + self.vector.EMBEDDING_MODEL = "synthetic-model-c" + dependency = MAINTENANCE.engine_layer_specs(self.conn, self.coordinator)[0].resolve_dependency() + self.assertTrue(dependency.available) + engine = self.new_engine() + self.assertIs(engine._get_maintenance_coordinator(), self.coordinator) + self.assertGreater(self.coordinator.capability_snapshot()[0], epoch) + self.assertIsNone(self.coordinator.capability_snapshot()[2]) + + def test_old_worker_cannot_publish_capability_error_after_refresh(self) -> None: + epoch, _, _ = self.coordinator.capability_snapshot() + failure = MAINTENANCE.make_layer_dependency(1, available=False, error_category="VectorExtensionUnavailable") + self.coordinator.refresh_capabilities() + self.coordinator._publish_extension_evidence(epoch, failure) + self.assertIsNone(self.coordinator.capability_snapshot()[2]) + + def test_actual_connection_unavailability_is_not_overridden_by_cached_success(self) -> None: + dependency = self.spec().resolve_dependency() + unavailable = MAINTENANCE.make_layer_dependency(1, MAINTENANCE._dependency_parameters(dependency), + available=False, error_category="VectorExtensionUnavailable") + with patch.object(MAINTENANCE, "_cluster_schedule_dependency", return_value=unavailable): + observed = MAINTENANCE.engine_layer_specs(self.conn, self.coordinator)[0].resolve_dependency() + self.assertEqual(observed, unavailable) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test-maintenance-fixture-isolation-753.py b/tests/test-maintenance-fixture-isolation-753.py new file mode 100644 index 00000000..f52397c5 --- /dev/null +++ b/tests/test-maintenance-fixture-isolation-753.py @@ -0,0 +1,220 @@ +"""Cross-fixture worker isolation using actual fixtures and SQLite in memory.""" + +import ast +import builtins +import sqlite3 +import sys +import threading +import types +import unittest +import uuid +from collections.abc import Callable +from pathlib import Path +from unittest.mock import patch + + +ROOT = Path(__file__).resolve().parents[1] + + +def fixture_loader(process_sys: object) -> Callable[[str], dict]: + """Keep fixture imports and module registries inside one synthetic process.""" + loaded = {} + + def safe_import(name: str, globals: dict | None = None, locals: dict | None = None, + fromlist: tuple = (), level: int = 0) -> object: + if name == "sys": + return process_sys + if name == "runpy": + return types.SimpleNamespace(run_path=load) + if name == "truememory.personality_style_vec": + return process_sys.modules[name] + if name.startswith("truememory") or name.split(".")[0] in {"torch", "numpy", "hdbscan", "sqlite_vec"}: + raise AssertionError("Unexpected application/native import: " + name) + return builtins.__import__(name, globals, locals, fromlist, level) + + def safe_exec(code: object, namespace: dict) -> None: + namespace.setdefault("__builtins__", fixture_builtins) + exec(code, namespace) + + fixture_builtins = dict(vars(builtins), __import__=safe_import, exec=safe_exec) + + def load(path: str) -> dict: + path = Path(path) + if path.name in loaded: + return loaded[path.name] + tree = ast.parse(path.read_text(encoding="utf-8")) + # Existing fixture loaders also read production source. Pin their + # decoding in this harness on Python 3.10/Windows without editing it. + for node in ast.walk(tree): + if (isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) + and node.func.attr == "read_text" and not node.args + and not any(keyword.arg == "encoding" for keyword in node.keywords)): + node.keywords.append(ast.keyword(arg="encoding", value=ast.Constant(value="utf-8"))) + namespace = {"__file__": str(path), "__name__": "synthetic_fixture_" + path.stem, + "__builtins__": fixture_builtins} + exec(compile(ast.fix_missing_locations(tree), str(path), "exec"), namespace) + loaded[path.name] = namespace + return namespace + + return load + + +class TestFixtureIsolation(unittest.TestCase): + def setUp(self) -> None: + values = vars(sys).copy() + values["modules"] = {} + self.process_sys = types.SimpleNamespace(**values) + load = fixture_loader(self.process_sys) + self.routing = load(str(ROOT / "tests/test-maintenance-engine-routing-753.py")) + self.adapters = self.routing["ADAPTERS"] + self.storage = self.routing["STORAGE"] + self.maintenance = self.routing["MAINTENANCE"] + _, self.foreign_vector, _ = self.adapters["CLUSTERS"]["load_clustering"]() + self.process_sys.modules["truememory.vector_search"] = self.foreign_vector + load_stdlib = self.routing["stdlib_module"] + style = load_stdlib("personality_style_vec", {}) + self.process_sys.modules["truememory.personality_style_vec"] = style + self.enter_context(patch.dict(self.routing, {"stdlib_module": lambda name, modules: + style if name == "personality_style_vec" else load_stdlib(name, modules)})) + self.registry_before = self.process_sys.modules.copy() + # Primitive loaders have their own import guard. Bind their sys view + # explicitly so both modules start in the same synthetic process. + self.enter_context(patch.object(self.maintenance, "sys", self.process_sys)) + self.foreign = types.ModuleType("synthetic_foreign_maintenance") + self.foreign.__dict__.update(self.maintenance.__dict__) + # Functions need their own globals to represent an unrelated worker. + source = (ROOT / "truememory/maintenance.py").read_text(encoding="utf-8") + exec(compile(source, "synthetic-foreign-maintenance", "exec"), self.foreign.__dict__) + self.foreign.sys = self.process_sys + + self.uri = "file:synthetic-fixture-" + uuid.uuid4().hex + "?mode=memory&cache=shared" + self.connect = sqlite3.connect + self.keeper = self.connect(self.uri, uri=True, check_same_thread=False) + self.addCleanup(self.keeper.close) + self.create = self.storage.create_db + self.coordinator = self.maintenance.MaintenanceCoordinator(None) + fixture_test = self + + class MemoryDirectory: + name = ":memory:" + + def cleanup(self) -> None: + pass + + class MemoryFixture(self.routing["RoutingFixture"]): + def open_db(self) -> sqlite3.Connection: + conn = fixture_test.open_memory() + self.addCleanup(conn.close) + return conn + + self.enter_context(patch.dict(self.adapters, {"tempfile": types.SimpleNamespace( + TemporaryDirectory=lambda **kwargs: MemoryDirectory())})) + self.enter_context(patch.object(self.storage, "create_db", self.open_memory)) + self.enter_context(patch.object(self.maintenance, "get_coordinator", lambda path: self.coordinator)) + self.fixture = MemoryFixture() + self.addCleanup(self.fixture.doCleanups) + self.fixture.setUp() + + def enter_context(self, context: object) -> object: + result = context.__enter__() + self.addCleanup(context.__exit__, None, None, None) + return result + + def open_memory(self, path: object = None) -> sqlite3.Connection: + sqlite_view = types.SimpleNamespace(**dict(vars(sqlite3), connect=lambda *args, **kwargs: + self.connect(self.uri, uri=True, check_same_thread=False))) + with patch.object(self.storage, "sqlite3", sqlite_view): + return self.create(":memory:") + + def states(self, conn: sqlite3.Connection | None = None) -> dict: + conn = conn or self.fixture.engine.conn + specs = self.maintenance.engine_layer_specs(conn, self.coordinator) + return self.maintenance.read_layer_states(conn, specs) + + def reopen(self) -> object: + self.fixture.engine.close() + reopened = self.fixture.new_engine() + reopened._has_consolidation = True + return reopened + + def test_fixture_does_not_publish_its_vector_into_process_registry(self) -> None: + self.assertIs(self.process_sys.modules["truememory.vector_search"], self.foreign_vector) + self.assertIs(self.maintenance.sys.modules["truememory.vector_search"], self.fixture.vector) + self.assertIsNot(self.maintenance.sys.modules, self.process_sys.modules) + self.assertEqual(self.process_sys.modules, self.registry_before) + + def test_interloper_cannot_defer_bootstrap_or_reset_reopen_threshold(self) -> None: + foreign_conn = self.connect(":memory:", check_same_thread=False) + self.fixture.conn.backup(foreign_conn) + entered, release = threading.Event(), threading.Event() + observed = [] + + class BlockedSnapshot: + def __getattr__(boundary, name: str) -> object: + return getattr(foreign_conn, name) + + def execute(boundary, sql: str, *args: object) -> sqlite3.Cursor: + if sql.startswith("SELECT type,name,rootpage,sql"): + entered.set() + if not release.wait(3): + raise AssertionError("Synthetic worker was not released") + return foreign_conn.execute(sql, *args) + + worker_coordinator = self.foreign.MaintenanceCoordinator(None) + + def work(conn: sqlite3.Connection, cancel: threading.Event) -> object: + observed.append(self.foreign.sys.modules["truememory.vector_search"]) + self.foreign._cluster_schedule_dependency(conn, versions=dict(self.fixture.versions)) + return self.foreign.MaintenanceReport((), "SKIPPED (synthetic)") + + with patch.object(self.foreign, "create_db", lambda path: BlockedSnapshot()), \ + patch.object(self.foreign, "_release_owner", lambda fd: None): + worker = threading.Thread(target=worker_coordinator._run, args=(-1, work)) + worker.start() + try: + self.assertTrue(entered.wait(3)) + result = self.fixture.engine.consolidate() + states = self.states() + finally: + release.set() + worker.join(3) + foreign_conn.close() + self.assertFalse(worker.is_alive()) + self.assertEqual(worker_coordinator.status, ("success", None)) + self.assertNotIn("DEFERRED", result["cluster_messages"]) + self.assertEqual(observed, [self.foreign_vector]) + self.assertEqual(len(states), 8) + self.assertTrue(all(state.attempted_insert_count == 0 for state in states.values()), states) + self.assertTrue(all(state.outcome == "success_empty" for state in states.values()), states) + self.fixture.add(24) + before = self.maintenance.read_source_revision(self.fixture.conn) + versions = self.coordinator.capability_snapshot()[1] + reopened = self.reopen() + with patch.object(self.coordinator, "request_layers") as request: + reopened._maybe_auto_consolidate() + request.assert_not_called() + self.assertEqual(self.maintenance.read_source_revision(reopened.conn), before) + self.assertEqual(self.coordinator.capability_snapshot()[1], versions) + reopened.add("synthetic twenty fifth") + request.assert_called_once_with(threshold=25) + + def test_intentionally_deferred_bootstrap_still_recovers_at_twenty_four(self) -> None: + with self.fixture.vector._lock: + result = self.fixture.engine.consolidate() + self.assertEqual(result["cluster_messages"], "DEFERRED (ModelBusy)") + self.assertIsNone(self.states()["clusters"].attempted_insert_count) + self.fixture.add(24) + reopened = self.reopen() + with patch.object(self.coordinator, "request_layers") as request: + reopened._maybe_auto_consolidate() + request.assert_called_once_with(threshold=25) + report = self.maintenance.run_engine_maintenance( + reopened.conn, self.coordinator, threshold=25, prepare_extensions=False) + self.assertEqual([item.layer for item in report.results if item.attempted], ["clusters"]) + states = self.states(reopened.conn) + self.assertEqual(states["clusters"].attempted_insert_count, 24) + self.assertTrue(all(state.attempted_insert_count == 0 for name, state in states.items() if name != "clusters")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test-maintenance-foreground-admission-753.py b/tests/test-maintenance-foreground-admission-753.py new file mode 100644 index 00000000..d295387e --- /dev/null +++ b/tests/test-maintenance-foreground-admission-753.py @@ -0,0 +1,491 @@ +"""Foreground publication admission with real SQLite and synthetic models.""" + +import runpy +import sqlite3 +import threading +import types +import unittest +from pathlib import Path +from unittest.mock import patch + + +FOREGROUND = runpy.run_path(str(Path(__file__).with_name("test-rebuild-foreground-publication-756.py"))) +CLUSTERS = runpy.run_path(str(Path(__file__).with_name("test-cluster-publication-749.py"))) + + +class AdmissionFixture(unittest.TestCase): + setUp = FOREGROUND["TestForegroundPublication"].setUp + seed = FOREGROUND["TestForegroundPublication"].seed + ids = FOREGROUND["TestForegroundPublication"].ids + writer = FOREGROUND["TestForegroundPublication"].writer + engine = FOREGROUND["TestForegroundPublication"].engine + + def start(self, action, name="synthetic-worker"): + results, done = [], threading.Event() + + def run(): + try: + results.append(action()) + except BaseException as error: + results.append(error) + finally: + done.set() + + thread = threading.Thread(target=run, name=name, daemon=True) + thread.start() + self.addCleanup(thread.join, 3) + return thread, results, done + + def released(self): + self.assertTrue(self.vector._lock.acquire(blocking=False)) + self.vector._lock.release() + + def sentinel(self): + self.conn.execute("INSERT INTO metadata(key,value) VALUES ('synthetic-caller','pending')") + + def preserved(self): + self.assertTrue(self.conn.in_transaction) + self.assertEqual(self.conn.execute("SELECT value FROM metadata WHERE key='synthetic-caller'").fetchone()[0], "pending") + + +class TestForegroundAdmission(AdmissionFixture): + def test_style_computation_runs_outside_capture_and_publication_locks(self) -> None: + engine = self.engine() + engine._has_style_vec = True + events = [] + style = [0.25, 0.75] + + def compute(content: str) -> list[float]: + self.assertFalse(engine._write_lock.locked()) + self.assertFalse(self.vector._lock.locked()) + self.assertFalse(self.conn.in_transaction) + events.append("compute") + return style + + def publish(conn: sqlite3.Connection, sender: str, content: str, + _pre_computed_vec: list[float]) -> None: + self.assertTrue(engine._write_lock.locked()) + self.assertTrue(self.vector._lock.locked()) + self.assertTrue(conn.in_transaction) + self.assertIs(_pre_computed_vec, style) + events.append("publish") + + module = types.ModuleType("synthetic_style") + module.compute_style_vector = compute + with patch.dict(self.modules, personality_style_vec=module), \ + patch.dict(engine.add.__globals__, _update_style_vec=publish): + engine.add("synthetic styled content", sender="synthetic sender") + self.assertEqual(events, ["compute", "publish"]) + self.assertEqual(self.ids(), [1]) + self.assertEqual(self.ids("vec_messages_sep"), [1]) + + def test_actual_cluster_snapshot_delays_add_before_writer_then_add_commits_both_vectors(self): + engine = self.engine() + cluster, runtime, _native = CLUSTERS["load_clustering"]() + encoding, snapshot, admission, release = (threading.Event() for _ in range(4)) + underlying = self.vector._lock + fixture = self + + class ObservedLock: + def acquire(self, blocking=True): + if threading.current_thread().name == "synthetic-add" and snapshot.is_set(): + fixture.assertTrue(blocking) + fixture.assertFalse(engine.conn.in_transaction) + fixture.assertTrue(engine._write_lock.locked()) + admission.set() + return underlying.acquire(blocking=blocking) + + def release(self): + underlying.release() + + def __enter__(self): + self.acquire() + return self + + def __exit__(self, *_args): + self.release() + + self.vector._lock = runtime._lock = ObservedLock() + runtime.EMBEDDING_MODEL = self.vector.EMBEDDING_MODEL + cluster_conn, independent = self.writer(), self.writer() + original = cluster._get_all_embeddings + + def read(conn): + snapshot.set() + self.assertTrue(release.wait(3)) + return original(conn) + + def encode(texts): + if len(self.calls) == 1: + encoding.set() + self.assertTrue(snapshot.wait(3)) + return [[1., 0.] for _ in texts] + + def consolidate(): + self.assertTrue(encoding.wait(3)) + return cluster.cluster_messages(cluster_conn) + + self.on_encode = encode + with patch.object(cluster, "_get_all_embeddings", read): + cluster_thread, cluster_result, cluster_done = self.start(consolidate) + add_thread, result, done = self.start(lambda: engine.add("synthetic first add"), "synthetic-add") + try: + self.assertTrue(admission.wait(2)) + self.assertFalse(done.is_set()) + independent.execute("INSERT INTO metadata(key,value) VALUES ('synthetic-independent','committed')") + independent.commit() + self.assertFalse(self.conn.in_transaction) + finally: + release.set() + add_thread.join(3) + cluster_thread.join(3) + self.assertTrue(done.is_set()) + self.assertTrue(cluster_done.is_set()) + self.assertIsInstance(cluster_result[0], (int, RuntimeError), cluster_result) + self.assertIsInstance(result[0], dict, result) + self.assertEqual(self.ids(), [1]) + self.assertEqual(self.ids("vec_messages_sep"), [1]) + self.assertEqual(self.conn.execute("SELECT content FROM messages").fetchone()[0], "synthetic first add") + self.released() + + def test_writer_owner_defers_nonblocking_model_fence_and_foreground_finishes(self): + owner = self.writer() + owner.execute("BEGIN IMMEDIATE") + admission, attempted = threading.Event(), threading.Event() + underlying = self.vector._lock + + class AdmittedLock: + def acquire(self, blocking=True): + acquired = underlying.acquire(blocking=blocking) + if acquired and threading.current_thread().name == "synthetic-publish": + admission.set() + return acquired + + def release(self): + underlying.release() + + self.vector._lock = AdmittedLock() + identity = (0, self.vector.EMBEDDING_MODEL, 2) + + def publish(): + with self.vector._foreground_vector_publication(self.conn, identity, 1): + self.conn.execute("INSERT INTO metadata(key,value) VALUES ('synthetic-published','yes')") + + thread, result, done = self.start(publish, "synthetic-publish") + try: + self.assertTrue(admission.wait(2)) + with self.assertRaisesRegex(RuntimeError, "model is busy"): + with self.vector._rebuild_model_fence(identity): + attempted.set() + self.assertFalse(attempted.is_set()) + finally: + owner.rollback() + thread.join(3) + self.assertTrue(done.is_set()) + self.assertEqual(result, [None]) + self.released() + + def test_add_waiting_for_same_engine_update_does_not_own_model(self): + self.seed((1,)) + engine = self.engine() + update_ready, add_waiting = threading.Event(), threading.Event() + underlying = engine._write_lock + + class EngineLock: + def __enter__(self): + if threading.current_thread().name == "synthetic-add": + add_waiting.set() + underlying.acquire() + return self + + def __exit__(self, *_args): + underlying.release() + + engine._write_lock = EngineLock() + original = engine.update.__globals__["update_message"] + + def update(conn, memory_id, **fields): + result = original(conn, memory_id, **fields) + update_ready.set() + self.assertTrue(add_waiting.wait(3)) + self.released() + return result + + with patch.dict(engine.update.__globals__, update_message=update): + update_thread, updated, update_done = self.start(lambda: engine.update(1, "synthetic updated"), "synthetic-update") + self.assertTrue(update_ready.wait(2)) + add_thread, added, add_done = self.start(lambda: engine.add("synthetic added"), "synthetic-add") + update_thread.join(3) + add_thread.join(3) + self.assertTrue(update_done.is_set()) + self.assertTrue(add_done.is_set()) + self.assertIsInstance(updated[0], dict, updated) + self.assertIsInstance(added[0], dict, added) + self.assertEqual(self.ids(), [1, 2]) + self.assertEqual(self.ids("vec_messages_sep"), [1, 2]) + + def test_same_engine_writer_finishes_before_capture_classifies_transaction(self) -> None: + self.seed((1,)) + for operation in ("add", "update"): + with self.subTest(operation=operation): + engine = self.engine() + writer_ready, waiting, release = (threading.Event() for _ in range(3)) + underlying = engine._write_lock + + class EngineLock: + def __enter__(self) -> "EngineLock": + if threading.current_thread().name == "synthetic-capture": + waiting.set() + underlying.acquire() + return self + + def __exit__(self, *_args: object) -> None: + underlying.release() + + engine._write_lock = EngineLock() + self.vector._model = None + loads = [] + + def load() -> object: + self.assertFalse(self.conn.in_transaction) + self.assertTrue(underlying.locked()) + loads.append(True) + self.vector._model = self.model + return self.model + + def writer() -> None: + with engine._write_lock: + self.conn.execute("BEGIN IMMEDIATE") + self.conn.execute("INSERT INTO metadata(key,value) VALUES (?, 'pending')", + ("synthetic-owner-" + operation,)) + writer_ready.set() + self.assertTrue(release.wait(3)) + self.conn.commit() + + with patch.object(self.vector, "get_model", load): + writer_thread, written, writer_done = self.start(writer) + self.assertTrue(writer_ready.wait(2)) + action = (lambda: engine.add("synthetic added")) if operation == "add" else ( + lambda: engine.update(1, "synthetic updated") + ) + mutation_thread, result, done = self.start(action, "synthetic-capture") + try: + self.assertTrue(waiting.wait(2)) + self.assertFalse(done.is_set()) + self.assertEqual(loads, []) + finally: + release.set() + mutation_thread.join(3) + writer_thread.join(3) + self.assertTrue(writer_done.is_set()) + self.assertEqual(written, [None]) + self.assertTrue(done.is_set()) + self.assertIsInstance(result[0], dict, result) + self.assertEqual(loads, [True]) + self.assertFalse(self.conn.in_transaction) + self.released() + + def test_generation_change_during_model_load_rejects_add_and_update_before_source_write(self): + self.seed((1,)) + engine = self.engine() + + def changed(): + self.vector._model_generation += 1 + return self.model + + with patch.object(self.vector, "get_model", changed): + for action in (lambda: engine.add("synthetic rejected"), lambda: engine.update(1, "synthetic rejected")): + with self.assertRaisesRegex(self.vector.VectorPublicationChanged, "changed while loading"): + action() + self.assertEqual(self.conn.execute("SELECT id,content FROM messages").fetchall(), [(1, "synthetic-1")]) + self.assertEqual(self.ids(), []) + self.released() + + def test_generation_change_while_waiting_rejects_before_opening_transaction(self): + captured = (0, self.vector.EMBEDDING_MODEL, 2) + attempted = threading.Event() + underlying = self.vector._lock + + class WaitingLock: + def acquire(self, blocking=True): + attempted.set() + return underlying.acquire(blocking=blocking) + + def release(self): + underlying.release() + + self.vector._lock = WaitingLock() + underlying.acquire() + + def publish(): + with self.vector._foreground_vector_publication(self.conn, captured, 1): + self.fail("Changed generation entered publication") + + thread, results, done = self.start(publish) + try: + self.assertTrue(attempted.wait(2)) + self.assertFalse(done.is_set()) + self.assertFalse(self.conn.in_transaction) + self.vector._model_generation += 1 + finally: + underlying.release() + thread.join(3) + self.assertIsInstance(results[0], self.vector.VectorPublicationChanged) + self.assertIn("model changed", str(results[0])) + self.assertFalse(self.conn.in_transaction) + self.released() + + def test_add_failure_after_source_insert_rolls_back_source_and_vectors(self): + engine = self.engine() + with patch.object(self.vector, "_validate_foreground_vector_target", side_effect=self.vector.VectorPublicationChanged("synthetic target changed")): + with self.assertRaises(self.vector.VectorPublicationChanged): + engine.add("synthetic rollback") + self.assertEqual(self.conn.execute("SELECT count(*) FROM messages").fetchone()[0], 0) + self.assertEqual(self.ids(), []) + self.assertEqual(self.ids("vec_messages_sep"), []) + self.assertFalse(self.conn.in_transaction) + self.released() + + def test_admission_begin_and_commit_failures_release_model(self): + real = self.conn + + class FailedConnection: + def __init__(self, phase): + self.phase = phase + + def __getattr__(self, name): + return getattr(real, name) + + def execute(self, sql, *args): + if self.phase == "begin" and sql == "BEGIN IMMEDIATE": + raise sqlite3.OperationalError("synthetic writer admission") + return real.execute(sql, *args) + + def commit(self): + if self.phase == "commit": + raise sqlite3.OperationalError("synthetic commit failure") + return real.commit() + + for phase in ("begin", "commit"): + with self.subTest(phase=phase): + with self.assertRaises(sqlite3.OperationalError): + self.engine(FailedConnection(phase)).add("synthetic rollback") + self.assertFalse(real.in_transaction) + self.assertEqual(real.execute("SELECT count(*) FROM messages").fetchone()[0], 0) + self.assertEqual(self.ids(), []) + self.released() + + def test_owned_model_load_and_encoding_finish_before_writer(self): + def load(): + self.assertFalse(self.conn.in_transaction) + return self.model + + def encode(texts): + self.assertFalse(self.conn.in_transaction) + self.released() + return [[1., 0.] for _ in texts] + + self.on_encode = encode + with patch.object(self.vector, "get_model", load): + self.engine().add("synthetic outside writer") + self.assertEqual(self.ids(), [1]) + + +class TestBorrowedAdmission(AdmissionFixture): + def actions(self): + engine = self.engine() + return ( + ("add", lambda: engine.add("synthetic rejected")), + ("update", lambda: engine.update(1, "synthetic rejected")), + ("embed_single", lambda: self.vector.embed_single(self.conn, 1, "synthetic-1")), + ("completion", lambda: self.vector.build_vectors(self.conn)), + ("separation", lambda: self.vector.build_separation_vectors(self.conn)), + ) + + def test_busy_capture_defers_every_mutation_without_waiting_or_loading(self): + self.seed((1,)) + self.sentinel() + calls = [] + + class BusyLock: + def acquire(self, blocking=True): + calls.append(blocking) + if blocking: + raise AssertionError("Borrowed caller attempted a blocking model acquisition") + return False + + with patch.object(self.vector, "_lock", BusyLock()), patch.object(self.vector, "get_model", side_effect=AssertionError("Borrowed model load")): + for name, action in self.actions(): + with self.subTest(action=name): + with self.assertRaisesRegex(self.vector.VectorPublicationChanged, "model is busy"): + action() + self.preserved() + self.assertEqual(calls, [False] * 5) + self.assertEqual(self.calls, []) + self.assertEqual(self.conn.execute("SELECT id,content FROM messages").fetchall(), [(1, "synthetic-1")]) + self.assertEqual(self.ids(), []) + self.conn.rollback() + self.released() + + def test_unloaded_capture_defers_every_mutation_and_preserves_sentinel(self): + self.seed((1,)) + self.sentinel() + self.vector._model = None + with patch.object(self.vector, "get_model", side_effect=AssertionError("Borrowed model load")) as loader: + for name, action in self.actions(): + with self.subTest(action=name): + with self.assertRaisesRegex(self.vector.VectorPublicationChanged, "not loaded"): + action() + self.preserved() + self.released() + loader.assert_not_called() + self.assertEqual(self.calls, []) + self.assertEqual(self.conn.execute("SELECT id,content FROM messages").fetchall(), [(1, "synthetic-1")]) + self.conn.rollback() + + def test_borrowed_loaded_snapshot_skips_loader_and_add_remains_rollbackable(self): + self.sentinel() + with patch.object(self.vector, "get_model", side_effect=AssertionError("Borrowed model load")) as loader: + item = self.engine().add("synthetic borrowed") + loader.assert_not_called() + self.preserved() + self.assertEqual(self.ids(), [item["id"]]) + self.assertEqual(self.ids("vec_messages_sep"), [item["id"]]) + self.conn.rollback() + self.assertEqual(self.conn.execute("SELECT count(*) FROM messages").fetchone()[0], 0) + self.assertEqual(self.ids(), []) + self.released() + + def test_borrowed_publication_contention_preserves_caller_transaction(self): + self.sentinel() + held, release = threading.Event(), threading.Event() + + def hold(): + with self.vector._lock: + held.set() + self.assertTrue(release.wait(3)) + + holder = [] + + def encode(texts): + if len(self.calls) == 2: + holder.append(self.start(hold)[0]) + self.assertTrue(held.wait(2)) + return [[1., 0.] for _ in texts] + + self.on_encode = encode + try: + with self.assertRaisesRegex(self.vector.VectorPublicationChanged, "model is busy"): + self.engine().add("synthetic rejected") + self.preserved() + self.assertEqual(self.conn.execute("SELECT count(*) FROM messages").fetchone()[0], 0) + finally: + release.set() + for thread in holder: + thread.join(3) + self.conn.rollback() + self.released() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test-rebuild-foreground-publication-756.py b/tests/test-rebuild-foreground-publication-756.py index 6c71ac5e..5e8b0b7d 100644 --- a/tests/test-rebuild-foreground-publication-756.py +++ b/tests/test-rebuild-foreground-publication-756.py @@ -273,7 +273,7 @@ def __getattr__(self, name): def execute(self, sql, parameters=()): if sql == "BEGIN IMMEDIATE" or sql == "UPDATE messages SET id = id WHERE 0": - case.assertFalse(case.vector._lock.locked()) + case.assertTrue(case.vector._lock.locked()) boundaries.append("writer") if sql == "RELEASE SAVEPOINT rebuild_source": case.assertTrue(case.vector._lock.locked()) @@ -296,13 +296,13 @@ def commit(self): def test_embed_owned_commit_stays_inside_model_fence(self): self._assert_foreground_writer_and_terminal_fence("embed", False) - def test_embed_caller_writer_upgrade_precedes_fence_and_release_stays_inside(self): + def test_embed_caller_fence_precedes_writer_upgrade_and_contains_release(self): self._assert_foreground_writer_and_terminal_fence("embed", True) def test_add_owned_commit_stays_inside_model_fence(self): self._assert_foreground_writer_and_terminal_fence("add", False) - def test_add_caller_writer_upgrade_precedes_fence_and_release_stays_inside(self): + def test_add_caller_fence_precedes_writer_upgrade_and_contains_release(self): self._assert_foreground_writer_and_terminal_fence("add", True) def test_denied_add_savepoint_release_rolls_back_only_add(self): diff --git a/tests/test-rebuild-source-streaming-756.py b/tests/test-rebuild-source-streaming-756.py index ccf10ac2..5ca9d5ff 100644 --- a/tests/test-rebuild-source-streaming-756.py +++ b/tests/test-rebuild-source-streaming-756.py @@ -78,6 +78,7 @@ def setUp(self): self.addCleanup(self.conn.close) self.conn.executescript("CREATE TABLE vec_messages(embedding BLOB); CREATE TABLE vec_messages_sep(embedding BLOB);") self.model = object() + self.vector._model = self.model self.vector.get_model = lambda: self.model self.calls = [] self.on_encode = None diff --git a/tests/test_consolidation_gaps.py b/tests/test_consolidation_gaps.py index 3c92e12c..5344444d 100644 --- a/tests/test_consolidation_gaps.py +++ b/tests/test_consolidation_gaps.py @@ -1,134 +1,60 @@ -"""Regression tests for consolidation gaps (clustering + preferences). - -Verifies that consolidate() calls cluster_messages and extract_preferences — -features that were previously only available via ingest(). -""" -from __future__ import annotations - -import tempfile -import os -from unittest.mock import patch, MagicMock - - -from truememory.engine import TrueMemoryEngine -from truememory.storage import create_db - - -def _make_engine_with_messages(n=20): - """Create an engine with messages in a temp DB (no vectors needed). - - Disables startup consolidation to prevent background threads from - accessing SQLite cross-thread (causes segfault on Python 3.14). - """ - td = tempfile.mkdtemp() - db = os.path.join(td, "test.db") - conn = create_db(db) - for i in range(n): - conn.execute( - "INSERT INTO messages (content, sender, recipient, timestamp, category, modality) " - "VALUES (?, ?, '', '', '', '')", - (f"Message {i} about topic {i % 5}", "alice" if i % 2 == 0 else "bob"), - ) - conn.commit() - conn.close() - - eng = TrueMemoryEngine(db_path=db) - eng._has_consolidation = False - eng._ensure_connection() - return eng, td - - -def test_consolidate_calls_cluster_messages(): - """consolidate() must call cluster_messages when clustering is available.""" - eng, td = _make_engine_with_messages() - eng._has_vectors = True - - with patch("truememory.engine._HAS_CLUSTERING", True), \ - patch("truememory.clustering.cluster_messages", return_value=3) as mock_cm: - result = eng.consolidate() - - assert "cluster_messages" in result - assert "ERROR" not in result["cluster_messages"] - assert "3 clusters" in result["cluster_messages"] - mock_cm.assert_called_once_with(eng.conn) - eng.close() - - -def test_consolidate_calls_extract_preferences(): - """consolidate() must call extract_preferences when personality is available.""" - eng, td = _make_engine_with_messages() - - with patch("truememory.engine._HAS_PERSONALITY", True), \ - patch("truememory.personality.extract_preferences", return_value={}) as mock_ep: - result = eng.consolidate() - - assert "extract_preferences" in result - assert "ERROR" not in result["extract_preferences"] - mock_ep.assert_called_once_with(eng.conn) - eng.close() - - -def test_consolidate_skips_clustering_without_vectors(): - """consolidate() must skip clustering when vectors are unavailable.""" - eng, td = _make_engine_with_messages() - eng._has_vectors = False - - result = eng.consolidate() - assert "cluster_messages" not in result - eng.close() - - -def test_consolidate_handles_cluster_error_gracefully(): - """consolidate() must catch and report clustering errors.""" - eng, td = _make_engine_with_messages() - eng._has_vectors = True - - with patch("truememory.engine._HAS_CLUSTERING", True), \ - patch("truememory.clustering.cluster_messages", side_effect=RuntimeError("boom")): - result = eng.consolidate() - - assert "cluster_messages" in result - assert "ERROR" in result["cluster_messages"] - eng.close() - - -def test_auto_consolidation_threshold_lowered(): - """Default auto-consolidation threshold is 25 (was 100).""" - eng, td = _make_engine_with_messages() - assert eng._auto_consolidate_threshold == 25 - eng.close() - - -def test_startup_consolidation_triggers_when_clusters_empty(): - """On startup, if messages exist but clusters are empty, consolidate fires.""" - eng, td = _make_engine_with_messages(n=30) - eng._has_consolidation = True - - import truememory.engine as _eng_mod - with patch.object(_eng_mod.threading, "Thread") as mock_thread_cls: - mock_thread = MagicMock() - mock_thread_cls.return_value = mock_thread - eng._maybe_startup_consolidate() - - mock_thread_cls.assert_called_once() - assert mock_thread_cls.call_args.kwargs["name"] == "startup-consolidate" - mock_thread.start.assert_called_once() - eng.close() - - -def test_startup_consolidation_skips_when_few_messages(): - """Don't consolidate on startup if fewer messages than threshold.""" - eng, td = _make_engine_with_messages(n=5) - - consolidate_called = False - orig = eng._bg_consolidate - def spy(): - nonlocal consolidate_called - consolidate_called = True - return orig() - - eng._bg_consolidate = spy - eng._has_consolidation = True - eng._maybe_startup_consolidate() - assert not consolidate_called - eng.close() +"""Behavioral coverage for all maintenance layers and startup provenance.""" + +import runpy +from pathlib import Path +from unittest.mock import patch + + +_ROUTING = runpy.run_path(str(Path(__file__).with_name("test-maintenance-engine-routing-753.py"))) + + +class TestConsolidationGaps(_ROUTING["RoutingFixture"]): + def test_clustering_count_matches_real_published_centroids(self): + self.add(5) + self.native.fit = lambda: setattr(self.native, "labels", [0, 0, 1, 1, 2]) + result = self.engine.consolidate() + self.assertIn("3 clusters", result["cluster_messages"]) + self.assertEqual(self.conn.execute("SELECT count(*) FROM cluster_centroids").fetchone()[0], 3) + + def test_preferences_runs_on_the_worker_connection(self): + function = self.modules["truememory.personality"].extract_preferences + with patch.object(self.modules["truememory.personality"], "extract_preferences", wraps=function) as preference: + result = self.engine.consolidate() + self.assertIn("extract_preferences", result) + self.assertNotIn("ERROR", result["extract_preferences"]) + preference.assert_called_once() + self.assertIsNot(preference.call_args.args[0], self.conn) + + def test_missing_vector_index_reports_unavailable_without_skipping_siblings(self): + self.conn.execute("DROP TABLE vec_messages_edge") + self.conn.commit() + result = self.engine.consolidate() + self.assertIn("UNAVAILABLE", result["cluster_messages"]) + self.assertIn("build_summaries", result) + self.assertNotIn("ERROR", result["build_summaries"]) + + def test_cluster_error_is_categorical_and_siblings_still_run(self): + with patch.object(self.cluster, "cluster_messages", side_effect=RuntimeError("synthetic private sentinel")): + result = self.engine.consolidate() + self.assertEqual(result["cluster_messages"], "ERROR (RuntimeError)") + self.assertNotIn("synthetic private sentinel", str(result)) + self.assertNotIn("ERROR", result["detect_episodes"]) + + def test_startup_attempts_initial_small_corpus_once(self): + self.add(5) + self.engine._has_consolidation = True + self.engine._maybe_startup_consolidate() + self.completed() + self.assertEqual(self.state().outcome, "success_empty") + with patch.object(self.coordinator, "request_layers") as request: + self.engine._maybe_startup_consolidate() + request.assert_not_called() + + def test_successful_empty_clusters_do_not_retrigger_all_layers(self): + self.add(30) + self.engine.consolidate() + self.assertEqual(self.conn.execute("SELECT count(*) FROM cluster_centroids").fetchone()[0], 0) + self.engine._has_consolidation = True + with patch.object(self.coordinator, "request_layers") as request: + self.engine._maybe_startup_consolidate() + request.assert_not_called() diff --git a/tests/test_hardening.py b/tests/test_hardening.py index 654845ed..a8950826 100644 --- a/tests/test_hardening.py +++ b/tests/test_hardening.py @@ -188,15 +188,19 @@ class TestAutoConsolidation: def test_auto_consolidation_doesnt_fire_below_threshold(self): eng, td = _make_engine(n_messages=0) + # Establish the initial successful/empty or unavailable attempt. The + # threshold applies after that durable baseline, including on reopen. + eng.consolidate() eng._has_consolidation = True eng._auto_consolidate_threshold = 25 + eng.conn.executemany("INSERT INTO messages(content) VALUES (?)", [("synthetic append",)] * 24) + eng.conn.commit() - import truememory.engine as _eng_mod - with patch.object(_eng_mod.threading, "Thread") as mock_thread_cls: - for i in range(24): + with patch.object(eng._maintenance_coordinator, "request_layers") as request: + for _ in range(3): eng._maybe_auto_consolidate() - mock_thread_cls.assert_not_called() + request.assert_not_called() eng.close() def test_startup_consolidation_no_crash_on_closed_conn(self): diff --git a/tests/test_issue_498_auto_consolidate.py b/tests/test_issue_498_auto_consolidate.py index 901d7f73..4295082f 100644 --- a/tests/test_issue_498_auto_consolidate.py +++ b/tests/test_issue_498_auto_consolidate.py @@ -1,37 +1,49 @@ -"""Regression test for M1 #498: auto-consolidation trigger. - -Consolidation should run automatically after a configurable number of -add() calls, not require manual truememory_consolidate invocation. -""" - -import inspect -import unittest - - -class TestAutoConsolidation(unittest.TestCase): - - def test_auto_consolidate_threshold_exists(self): - from truememory.engine import TrueMemoryEngine - source = inspect.getsource(TrueMemoryEngine.__init__) - self.assertIn("_auto_consolidate_threshold", source) - self.assertIn("TRUEMEMORY_AUTO_CONSOLIDATE_EVERY", source) - - def test_add_calls_maybe_auto_consolidate(self): - from truememory.engine import TrueMemoryEngine - source = inspect.getsource(TrueMemoryEngine.add) - self.assertIn("_maybe_auto_consolidate", source, - "add() should call _maybe_auto_consolidate()") - - def test_maybe_auto_consolidate_checks_threshold(self): - from truememory.engine import TrueMemoryEngine - source = inspect.getsource(TrueMemoryEngine._maybe_auto_consolidate) - self.assertIn("_auto_consolidate_threshold", source) - self.assertIn("_has_consolidation", source) - - def test_bg_consolidate_exists(self): - from truememory.engine import TrueMemoryEngine - self.assertTrue(hasattr(TrueMemoryEngine, "_bg_consolidate")) - - -if __name__ == "__main__": - unittest.main() +"""Durable automatic maintenance thresholds across engine instances.""" + +import runpy +from pathlib import Path +from unittest.mock import patch + + +_ROUTING = runpy.run_path(str(Path(__file__).with_name("test-maintenance-engine-routing-753.py"))) + + +class TestAutoConsolidation(_ROUTING["RoutingFixture"]): + def assert_empty_bootstrap(self) -> None: + maintenance = _ROUTING["MAINTENANCE"] + states = maintenance.read_layer_states( + self.conn, maintenance.engine_layer_specs(self.conn, self.coordinator)) + self.assertEqual(len(states), 8) + self.assertTrue(all(state.outcome == "success_empty" for state in states.values()), states) + self.assertTrue(all(state.attempted_insert_count == 0 for state in states.values()), states) + + def test_default_threshold_is_twenty_five(self): + self.assertEqual(self.engine._auto_consolidate_threshold, 25) + + def test_configured_threshold_uses_committed_source_inserts(self): + self.engine.consolidate() + self.assert_empty_bootstrap() + self.engine._auto_consolidate_threshold = 3 + self.engine._has_consolidation = True + with patch.object(self.coordinator, "request_layers", wraps=self.coordinator.request_layers) as request: + for _ in range(2): + self.engine.add("synthetic committed append") + request.assert_not_called() + self.engine.add("synthetic threshold append") + self.completed() + self.assertTrue(request.called) + self.assertTrue(all(call.kwargs["threshold"] == 3 for call in request.call_args_list)) + + def test_close_and_reopen_does_not_reset_threshold_progress(self): + self.engine.consolidate() + self.assert_empty_bootstrap() + self.add(24) + self.engine.close() + reopened = self.new_engine() + reopened._has_consolidation = True + with patch.object(self.coordinator, "request_layers", wraps=self.coordinator.request_layers) as request: + reopened._maybe_auto_consolidate() + request.assert_not_called() + reopened.add("synthetic twenty fifth") + self.completed() + self.assertTrue(request.called) diff --git a/tests/test_issue_511_write_path_opt.py b/tests/test_issue_511_write_path_opt.py index c2f1cbe4..0d1858a3 100644 --- a/tests/test_issue_511_write_path_opt.py +++ b/tests/test_issue_511_write_path_opt.py @@ -5,7 +5,9 @@ M16: Fallback embed_single inside write lock — removed in favor of skip-and-log """ +import ast import inspect +import textwrap import unittest @@ -30,13 +32,17 @@ class TestM15StyleVecOutsideLock(unittest.TestCase): def test_add_precomputes_style_vec(self): from truememory.engine import TrueMemoryEngine - source = inspect.getsource(TrueMemoryEngine.add) - lock_pos = source.find("self._write_lock") - precompute_pos = source.find("compute_style_vector") - self.assertGreater(precompute_pos, -1, - "add() should reference compute_style_vector for pre-computation") - self.assertLess(precompute_pos, lock_pos, - "compute_style_vector call should appear before _write_lock") + tree = ast.parse(textwrap.dedent(inspect.getsource(TrueMemoryEngine.add))) + calls = [node for node in ast.walk(tree) if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) and node.func.id == "compute_style_vector"] + self.assertEqual(len(calls), 1) + for block in (node for node in ast.walk(tree) if isinstance(node, ast.With)): + self.assertNotIn(calls[0], list(ast.walk(block)), + "Style computation must run outside lock and transaction contexts") + publications = [node for node in ast.walk(tree) if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) and node.func.id == "_update_style_vec"] + self.assertEqual(len(publications), 1) + self.assertLess(calls[0].lineno, publications[0].lineno) def test_style_vec_accepts_precomputed(self): from truememory.personality_style_vec import update_entity_style_vector_incremental diff --git a/tests/test_issue_590_build_vectors_batch.py b/tests/test_issue_590_build_vectors_batch.py index a678b1c8..1f11dc2a 100644 --- a/tests/test_issue_590_build_vectors_batch.py +++ b/tests/test_issue_590_build_vectors_batch.py @@ -73,6 +73,8 @@ def _make_server(monkeypatch, tmp_path): def _seed_messages(ms, n: int) -> None: """Add *n* messages to the database via the server module.""" m = ms._get_memory() + # These focused tests own explicit rebuild maintenance. + m._engine._has_consolidation = False for i in range(n): m.add(f"memory number {i}", user_id="test") diff --git a/tests/test_issue_647_vector_rebuild.py b/tests/test_issue_647_vector_rebuild.py index 3a168421..a85b8530 100644 --- a/tests/test_issue_647_vector_rebuild.py +++ b/tests/test_issue_647_vector_rebuild.py @@ -81,6 +81,8 @@ def _make_server(monkeypatch, tmp_path): def _seed_messages(ms, n: int) -> None: m = ms._get_memory() + # These focused tests own explicit rebuild maintenance. + m._engine._has_consolidation = False for i in range(n): m.add(f"memory number {i}", user_id="test") diff --git a/truememory/engine.py b/truememory/engine.py index 0285b066..6411b587 100644 --- a/truememory/engine.py +++ b/truememory/engine.py @@ -31,6 +31,10 @@ import time import sqlite3 from pathlib import Path +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from truememory.maintenance import LayerResult, MaintenanceCoordinator # ─────────────────────────────────────────────────────────────────────────── # Core modules (always available) @@ -358,18 +362,18 @@ def __init__(self, db_path: str | Path = None, # Coordination flag for background NaN re-embed thread (#485). self._nan_migration_in_progress = False - # Auto-consolidation: run L5 consolidation every N adds (#498) - self._adds_since_consolidation = 0 self._auto_consolidate_threshold = _env_int( "TRUEMEMORY_AUTO_CONSOLIDATE_EVERY", 25, lo=1 ) - self._consolidation_thread: threading.Thread | None = None + self._maintenance_coordinator = None + self._maintenance_handle = None + self._maintenance_pending_reason = "not_connected" # ────────────────────────────────────────────────────────────────────── # Auto-connect (production API) # ────────────────────────────────────────────────────────────────────── - def _ensure_connection(self) -> None: + def _ensure_connection(self, *, _suppress_maintenance: bool = False) -> None: """Open (or create) the database and load extensions if needed. Called automatically by :meth:`add`, :meth:`search`, and other @@ -385,6 +389,8 @@ def _ensure_connection(self) -> None: # through to reconnect so the server self-heals once the FS recovers. try: self.conn.execute("PRAGMA schema_version") + if not _suppress_maintenance: + self._maybe_auto_consolidate() return except sqlite3.Error: logger.warning( @@ -400,6 +406,8 @@ def _ensure_connection(self) -> None: with self._init_lock: if self.conn is not None: + if not _suppress_maintenance: + self._maybe_auto_consolidate() return # Create parent directory if using a real path @@ -581,7 +589,8 @@ def _bg_wrapper(db_path): self._purge_legacy_entity_profile_summaries() self.ready = True - self._maybe_startup_consolidate() + if not _suppress_maintenance: + self._maybe_startup_consolidate() def _purge_legacy_entity_profile_summaries(self) -> None: """Delete legacy ``period='entity_profile'`` summary rows once. @@ -656,25 +665,8 @@ def _purge_legacy_entity_profile_summaries(self) -> None: ) def _maybe_startup_consolidate(self) -> None: - """Trigger background consolidation on startup if data looks stale.""" - if not self._has_consolidation or self.conn is None: - return - try: - msg_count = self.conn.execute("SELECT COUNT(*) FROM messages").fetchone()[0] - if msg_count < self._auto_consolidate_threshold: - return - cluster_count = self.conn.execute( - "SELECT COUNT(*) FROM message_clusters" - ).fetchone()[0] - if cluster_count == 0: - self._consolidation_thread = threading.Thread( - target=self._bg_consolidate, - daemon=True, - name="startup-consolidate", - ) - self._consolidation_thread.start() - except Exception: - pass + """Use durable per-layer provenance, including successful empty work.""" + self._maybe_auto_consolidate() # ────────────────────────────────────────────────────────────────────── # Production CRUD API @@ -724,14 +716,18 @@ def add( pre_embedding = None pre_sep_embedding = None if self._has_vectors: + from truememory.vector_search import VectorPublicationChanged try: from truememory.vector_search import ( _capture_rebuild_model, _encode_with_mps_fallback, _build_sep_text, ) - model, pre_identity = _capture_rebuild_model() + with self._write_lock: + model, pre_identity = _capture_rebuild_model(self.conn) pre_embedding = _encode_with_mps_fallback(model, [content])[0] sep_text = _build_sep_text(sender, recipient, timestamp, content) pre_sep_embedding = _encode_with_mps_fallback(model, [sep_text])[0] + except VectorPublicationChanged: + raise except Exception: logger.debug("Failed to pre-compute embedding during add()", exc_info=True) @@ -743,13 +739,15 @@ def add( except Exception: logger.debug("Failed to pre-compute style vector during add()", exc_info=True) + from contextlib import nullcontext from truememory.rebuild_source import rebuild_transaction - publication_fence = None if pre_embedding is not None: from truememory.vector_search import _foreground_model_fence - publication_fence = _foreground_model_fence(pre_identity) - with self._write_lock, rebuild_transaction(self.conn, write=True, publication_fence=publication_fence): + with self._write_lock, ( + _foreground_model_fence(pre_identity, blocking=not self.conn.in_transaction) + if pre_embedding is not None else nullcontext() + ), rebuild_transaction(self.conn, write=True): msg = { "content": content, "sender": sender, @@ -816,30 +814,54 @@ def add( "metadata": metadata or {}, } + def _get_maintenance_coordinator(self) -> MaintenanceCoordinator: + from truememory.maintenance import get_coordinator + coordinator = self._maintenance_coordinator + if coordinator is not None: + try: + coordinator.status + except RuntimeError: + coordinator = None + if coordinator is None or self._maintenance_handle is not self.conn: + coordinator = get_coordinator(self.db_path) + coordinator.refresh_capabilities() + self._maintenance_coordinator = coordinator + self._maintenance_handle = self.conn + return coordinator + def _maybe_auto_consolidate(self) -> None: - """Trigger background consolidation after threshold adds.""" - self._adds_since_consolidation += 1 - if self._adds_since_consolidation < self._auto_consolidate_threshold: - return - if not self._has_consolidation: + """Probe fixed metadata without waiting on foreground or model work.""" + if not self._has_consolidation or self.conn is None: return - if (self._consolidation_thread is not None - and self._consolidation_thread.is_alive()): + if not self._write_lock.acquire(blocking=False): + self._maintenance_pending_reason = "foreground_busy" return - self._adds_since_consolidation = 0 - self._consolidation_thread = threading.Thread( - target=self._bg_consolidate, - daemon=True, - name="auto-consolidate", - ) - self._consolidation_thread.start() + coordinator = None + eligible = () + try: + if self.conn is None: + return + if self.conn.in_transaction: + self._maintenance_pending_reason = "pending_caller_commit" + return + from truememory.maintenance import engine_layer_specs, plan_layers + coordinator = self._get_maintenance_coordinator() + eligible = plan_layers(self.conn, engine_layer_specs(self.conn, coordinator), + threshold=self._auto_consolidate_threshold) + self._maintenance_pending_reason = "eligible" if eligible else "awaiting_eligibility" + except Exception as error: + self._maintenance_pending_reason = type(error).__name__[:64] + finally: + self._write_lock.release() + if eligible and coordinator is not None: + try: + coordinator.request_layers(threshold=self._auto_consolidate_threshold) + except Exception as error: + self._maintenance_pending_reason = type(error).__name__[:64] def _bg_consolidate(self) -> None: - """Run consolidation in a background thread with its own connection.""" - try: - self.consolidate() - except Exception: - logger.debug("Auto-consolidation failed", exc_info=True) + """Compatibility entry point; the coordinator owns the worker handle.""" + self._maybe_auto_consolidate() def delete(self, memory_id: int) -> bool: """Delete a memory by ID. @@ -848,7 +870,10 @@ def delete(self, memory_id: int) -> bool: """ self._ensure_connection() with self._write_lock: - return delete_message(self.conn, memory_id) + deleted = delete_message(self.conn, memory_id) + if deleted: + self._maybe_auto_consolidate() + return deleted def delete_all(self, user_id: str | None = None) -> bool: """Delete all memories, optionally filtered by user. @@ -1015,147 +1040,51 @@ def delete_all(self, user_id: str | None = None) -> bool: logger.warning("Failed to rebuild FTS index during delete_all", exc_info=True) self.conn.commit() - return deleted + if deleted: + self._maybe_auto_consolidate() + return deleted + + def _apply_manual_maintenance_capabilities(self, results: tuple[LayerResult, ...]) -> None: + completed = {result.layer for result in results + if result.outcome in {"success", "success_empty"}} + if "summaries" in completed: + self._has_consolidation = True + if "clusters" in completed: + self._has_clustering = True def consolidate(self) -> dict[str, str]: - """Run all consolidation layers (L0-L5) under the write lock. - - Returns timing stats for each step. - """ - import time as _time - - self._ensure_connection() - stats: dict[str, str] = {} - - try: - from truememory.consolidation import ( - build_summaries, - detect_contradictions, - build_structured_facts, - ) - except (ImportError, ModuleNotFoundError): - stats["consolidation"] = "SKIPPED (module not available)" - return stats - - try: - from truememory.predictive import build_surprise_index - except (ImportError, ModuleNotFoundError): - build_surprise_index = None - - try: - from truememory.temporal import detect_episodes, detect_landmark_events - except (ImportError, ModuleNotFoundError): - detect_episodes = None - detect_landmark_events = None - - try: - from truememory.personality import build_dunbar_hierarchy - except (ImportError, ModuleNotFoundError): - build_dunbar_hierarchy = None - - _cluster_messages = None - if _HAS_CLUSTERING and self._has_vectors: - try: - from truememory.clustering import cluster_messages as _cm - _cluster_messages = _cm - except (ImportError, ModuleNotFoundError): - pass - - _extract_preferences = None - if _HAS_PERSONALITY: - try: - from truememory.personality import extract_preferences as _ep - _extract_preferences = _ep - except (ImportError, ModuleNotFoundError): - pass - + """Force one pass, preserving caller transactions and shared ownership.""" + from truememory.maintenance import ( + MaintenanceBusyError, format_maintenance_report, maintenance_busy_result, + maintenance_owner, run_engine_maintenance, + ) + self._ensure_connection(_suppress_maintenance=True) with self._write_lock: - if _cluster_messages: + coordinator = self._get_maintenance_coordinator() + coordinator.refresh_capabilities() + borrowed = self.conn.in_transaction + if borrowed or coordinator.path is None: try: - t0 = _time.time() - n = _cluster_messages(self.conn) - stats["cluster_messages"] = f"{n} clusters in {_time.time() - t0:.3f}s" - self._has_clustering = True - except Exception as exc: - stats["cluster_messages"] = f"ERROR: {exc}" - - if _extract_preferences: - try: - t0 = _time.time() - _extract_preferences(self.conn) - stats["extract_preferences"] = f"{_time.time() - t0:.3f}s" - except Exception as exc: - stats["extract_preferences"] = f"ERROR: {exc}" - - try: - t0 = _time.time() - build_summaries(self.conn) - stats["build_summaries"] = f"{_time.time() - t0:.3f}s" - except Exception as exc: - stats["build_summaries"] = f"ERROR: {exc}" - - try: - t0 = _time.time() - detect_contradictions(self.conn) - stats["detect_contradictions"] = f"{_time.time() - t0:.3f}s" - except Exception as exc: - stats["detect_contradictions"] = f"ERROR: {exc}" - - try: - t0 = _time.time() - n = build_structured_facts(self.conn) - stats["structured_facts"] = f"{n} facts in {_time.time() - t0:.3f}s" - except Exception as exc: - stats["structured_facts"] = f"ERROR: {exc}" - - if build_surprise_index: - try: - t0 = _time.time() - build_surprise_index(self.conn) - stats["build_surprise_index"] = f"{_time.time() - t0:.3f}s" - except Exception as exc: - stats["build_surprise_index"] = f"ERROR: {exc}" - - if detect_episodes: - try: - t0 = _time.time() - ep = detect_episodes(self.conn) - stats["detect_episodes"] = f"{ep} episodes in {_time.time() - t0:.3f}s" - except Exception as exc: - stats["detect_episodes"] = f"ERROR: {exc}" - - if detect_landmark_events: - try: - t0 = _time.time() - lm = detect_landmark_events(self.conn) - stats["detect_landmarks"] = f"{lm} events in {_time.time() - t0:.3f}s" - except Exception as exc: - stats["detect_landmarks"] = f"ERROR: {exc}" - - if build_dunbar_hierarchy: + report = run_engine_maintenance(self.conn, coordinator, force=True, + threshold=self._auto_consolidate_threshold, prepare_extensions=False, + allow_caller_transaction=borrowed) + except MaintenanceBusyError: + return maintenance_busy_result() + self._apply_manual_maintenance_capabilities(report.results) + return format_maintenance_report(report) + try: + with maintenance_owner(coordinator.path): + connection = create_db(coordinator.path) try: - t0 = _time.time() - primary = None - try: - row = self.conn.execute( - "SELECT sender, COUNT(*) as cnt FROM messages " - "WHERE sender != '' AND sender IS NOT NULL " - "GROUP BY sender ORDER BY cnt DESC LIMIT 1" - ).fetchone() - if row and row[0] and row[0].strip(): - primary = row[0] - except Exception: - pass - result = build_dunbar_hierarchy(self.conn, primary_entity=primary) - n_rel = len(result) if isinstance(result, dict) else result - stats["dunbar_hierarchy"] = f"{n_rel} relationships in {_time.time() - t0:.3f}s" - except Exception as exc: - stats["dunbar_hierarchy"] = f"ERROR: {exc}" - - self.conn.commit() - - self._has_consolidation = True - return stats + report = run_engine_maintenance(connection, coordinator, force=True, + threshold=self._auto_consolidate_threshold) + finally: + connection.close() + except MaintenanceBusyError: + return maintenance_busy_result() + with self._write_lock: + self._apply_manual_maintenance_capabilities(report.results) + return format_maintenance_report(report) def update(self, memory_id: int, content: str | None = None, **fields) -> dict | None: """Update a memory. @@ -1176,11 +1105,13 @@ def update(self, memory_id: int, content: str | None = None, **fields) -> dict | pre_embedding = None pre_sep_embedding = None if content is not None and self._has_vectors: + from truememory.vector_search import VectorPublicationChanged try: from truememory.vector_search import ( _capture_rebuild_model, _encode_with_mps_fallback, _build_sep_text, ) - model, pre_identity = _capture_rebuild_model() + with self._write_lock: + model, pre_identity = _capture_rebuild_model(self.conn) pre_embedding = _encode_with_mps_fallback(model, [content])[0] row = self.conn.execute( "SELECT sender, recipient, timestamp FROM messages WHERE id = ?", @@ -1192,44 +1123,52 @@ def update(self, memory_id: int, content: str | None = None, **fields) -> dict | timestamp_val = fields.get("timestamp", row[2]) sep_text = _build_sep_text(sender_val, recipient_val, timestamp_val, content) pre_sep_embedding = _encode_with_mps_fallback(model, [sep_text])[0] + except VectorPublicationChanged: + raise except Exception: logger.debug("Failed to pre-compute embedding during update()", exc_info=True) - with self._write_lock: - if content is not None: - fields["content"] = content + source_committed = False + try: + with self._write_lock: + if content is not None: + fields["content"] = content - ok = update_message(self.conn, memory_id, **fields) - if not ok: - return None + ok = update_message(self.conn, memory_id, **fields) + source_committed = ok and not self.conn.in_transaction + if not ok: + return None - if pre_embedding is not None: - from truememory.vector_search import VectorPublicationChanged, _foreground_vector_publication - try: - from truememory.vector_search import serialize_f32, _write_embedder_metadata_no_commit - with _foreground_vector_publication(self.conn, pre_identity, memory_id) as (vec_tbl, sep_tbl): - try: - self.conn.execute(f"DELETE FROM {vec_tbl} WHERE rowid = ?", (memory_id,)) - except Exception: - logger.debug("Failed to delete old vector embedding for message %d", memory_id, exc_info=True) - self.conn.execute( - f"INSERT INTO {vec_tbl}(rowid, embedding) VALUES (?, ?)", - (memory_id, serialize_f32(pre_embedding)), - ) - if pre_sep_embedding is not None: + if pre_embedding is not None: + from truememory.vector_search import VectorPublicationChanged, _foreground_vector_publication + try: + from truememory.vector_search import serialize_f32, _write_embedder_metadata_no_commit + with _foreground_vector_publication(self.conn, pre_identity, memory_id) as (vec_tbl, sep_tbl): try: - self.conn.execute(f"DELETE FROM {sep_tbl} WHERE rowid = ?", (memory_id,)) + self.conn.execute(f"DELETE FROM {vec_tbl} WHERE rowid = ?", (memory_id,)) except Exception: - logger.debug("Failed to delete old sep vector for message %d", memory_id, exc_info=True) + logger.debug("Failed to delete old vector embedding for message %d", memory_id, exc_info=True) self.conn.execute( - f"INSERT INTO {sep_tbl}(rowid, embedding) VALUES (?, ?)", - (memory_id, serialize_f32(pre_sep_embedding)), + f"INSERT INTO {vec_tbl}(rowid, embedding) VALUES (?, ?)", + (memory_id, serialize_f32(pre_embedding)), ) - _write_embedder_metadata_no_commit(self.conn) - except VectorPublicationChanged as exc: - raise VectorPublicationChanged("Source update committed; retry update to publish vectors") from exc - except Exception: - logger.warning("Vector embedding failed for message %d", memory_id, exc_info=True) + if pre_sep_embedding is not None: + try: + self.conn.execute(f"DELETE FROM {sep_tbl} WHERE rowid = ?", (memory_id,)) + except Exception: + logger.debug("Failed to delete old sep vector for message %d", memory_id, exc_info=True) + self.conn.execute( + f"INSERT INTO {sep_tbl}(rowid, embedding) VALUES (?, ?)", + (memory_id, serialize_f32(pre_sep_embedding)), + ) + _write_embedder_metadata_no_commit(self.conn) + except VectorPublicationChanged as exc: + raise VectorPublicationChanged("Source update committed; retry update to publish vectors") from exc + except Exception: + logger.warning("Vector embedding failed for message %d", memory_id, exc_info=True) + finally: + if source_committed: + self._maybe_auto_consolidate() return self.get(memory_id) @@ -1503,12 +1442,18 @@ def ingest(self, data_path: str | Path) -> dict: n_clusters = cluster_messages(self.conn) stats["build_clusters"] = f"{n_clusters} clusters in {time.time() - t0:.3f}s" self._has_clustering = True + if not self.conn.in_transaction: + self._get_maintenance_coordinator().observe_clustering_outcome( + "success_empty" if n_clusters == 0 else "success") except Exception as exc: stats["build_clusters"] = f"ERROR: {exc}" - logger.debug("build_clusters failed", exc_info=True) + self._get_maintenance_coordinator().observe_clustering_outcome( + "unavailable" if isinstance(exc, ImportError) else "failed", type(exc).__name__) self._has_clustering = False else: stats["build_clusters"] = "SKIPPED (clustering or vectors not available)" + self._get_maintenance_coordinator().observe_clustering_outcome( + "unavailable", "ClusteringRuntimeUnavailable" if not _HAS_CLUSTERING else "VectorRuntimeUnavailable") self._has_clustering = False # ── 4. Build entity profiles (L0) ───────────────────────────────── @@ -2553,10 +2498,45 @@ def search_simple(self, query: str, limit: int = 10) -> list[dict]: # Stats / teardown # ────────────────────────────────────────────────────────────────────── + def get_clustering_health(self) -> dict: + """Read existing state without connecting or scheduling maintenance.""" + from truememory.maintenance import clustering_health + if not self._write_lock.acquire(blocking=False): + return clustering_health(pending_reason="foreground_busy") + try: + coordinator = self._maintenance_coordinator if self._maintenance_handle is self.conn else None + return clustering_health(self.conn, coordinator) + finally: + self._write_lock.release() + def get_stats(self) -> dict: """Return ingestion and search statistics.""" self._ensure_connection() stats = dict(self.stats) + if self._write_lock.acquire(blocking=False): + try: + from truememory.maintenance import engine_layer_specs, layer_freshness, read_layer_states + coordinator = self._get_maintenance_coordinator() + maintenance = coordinator.snapshot() + maintenance["last_results"] = [result._asdict() for result in maintenance["last_results"]] + maintenance["pending_reason"] = self._maintenance_pending_reason + maintenance["pending_caller_commit"] = self.conn.in_transaction + states = read_layer_states(self.conn, engine_layer_specs(self.conn, coordinator)) + maintenance["layers"] = {name: { + "outcome": state.outcome, "freshness": layer_freshness(state.source, state, state.dependency), + "coverage": state.successful_coverage, "error_category": state.error_category, + "dependency_error": state.dependency.error_category, "output_count": state.output_count, + "pending_inserts": state.source.insert_count - (state.attempted_insert_count or 0), + } for name, state in states.items()} + stats["maintenance"] = maintenance + except Exception as error: + stats["maintenance"] = {"status": "unavailable", "error_category": type(error).__name__[:64]} + finally: + self._write_lock.release() + else: + stats["maintenance"] = {"status": "pending", "pending_reason": "foreground_busy"} + + stats["clustering"] = self.get_clustering_health() # Add live DB stats if connected. if self.conn: @@ -2579,19 +2559,18 @@ def get_stats(self) -> dict: return stats def close(self): - """Close database connection and join background threads.""" - if self._consolidation_thread is not None and self._consolidation_thread.is_alive(): - try: - self._consolidation_thread.join(timeout=5) - except Exception: - pass - if self.conn: - try: - self.conn.close() - except Exception: - logger.debug("Failed to close database connection", exc_info=True) - self.conn = None - self.ready = False + """Detach this engine; a shared worker retains its own connection.""" + with self._init_lock, self._write_lock: + self._maintenance_coordinator = None + self._maintenance_handle = None + self._maintenance_pending_reason = "not_connected" + if self.conn: + try: + self.conn.close() + except sqlite3.Error: + logger.debug("Failed to close database connection", exc_info=True) + self.conn = None + self.ready = False def __enter__(self): return self diff --git a/truememory/maintenance.py b/truememory/maintenance.py index 325ac2c8..e0717b42 100644 --- a/truememory/maintenance.py +++ b/truememory/maintenance.py @@ -1,7 +1,6 @@ -"""Database maintenance ownership and committed source revision primitives. +"""Database maintenance ownership, source revisions and durable layer scheduling. -The engine scheduler is not routed here yet. No models, polling loop or automatic -builder invocation is introduced by importing this module. +Importing this module loads no models and starts no worker or polling loop. """ import hashlib @@ -9,6 +8,7 @@ import importlib.metadata import importlib.util import json +import logging import os import re import sqlite3 @@ -18,13 +18,16 @@ import uuid import weakref from collections.abc import Callable, Iterator -from contextlib import AbstractContextManager, ExitStack, contextmanager +from contextlib import AbstractContextManager, ExitStack, contextmanager, nullcontext from pathlib import Path from types import ModuleType from typing import NamedTuple from truememory._platform import try_file_lock -from truememory.storage import create_db +from truememory.storage import _prepare_style_maintenance, _style_output_tracking_ready, create_db + + +logger = logging.getLogger(__name__) class MaintenanceUnavailableError(RuntimeError): @@ -40,7 +43,45 @@ class LayerUnavailableError(MaintenanceUnavailableError): class LayerDeferredError(RuntimeError): - """Transient model ownership contention does not establish an attempt baseline.""" + """A narrowly categorized transient condition establishes no attempt baseline.""" + + def __init__(self, message: str, *, category: str = "ModelBusy") -> None: + if category not in {"ModelBusy", "StyleCancelled", "StyleSourceChanged", "StyleWriterBusy"}: + raise ValueError("Unsupported layer deferral category") + self.category = category + super().__init__(message) + + +class StyleAccumulatorUnsupported(LayerUnavailableError): + """Tracked publication cannot replace a newer accumulator format.""" + + +class _StyleSQLInterrupted(LayerDeferredError): + """Only the owned progress callback can attest SQLite's automatic rollback.""" + + def __init__(self, *, automatic_rollback: bool) -> None: + super().__init__("Style work cancelled", category="StyleCancelled") + self.automatic_rollback = automatic_rollback + + +def _style_sqlite_deferral(error: sqlite3.OperationalError, *, cancelled: bool = False) -> str | None: + # SQLite's primary result codes are stable; Python 3.10 exposes neither + # these named constants nor sqlite_errorcode on native exceptions. + code = getattr(error, "sqlite_errorcode", None) + if isinstance(code, int): + primary = code & 255 + if primary in {5, 6}: # SQLITE_BUSY / SQLITE_LOCKED, including extended codes. + return "StyleWriterBusy" + if primary == 9 and cancelled: # SQLITE_INTERRUPT. + return "StyleCancelled" + return None + message = str(error) + if cancelled and message == "interrupted": + return "StyleCancelled" + if (message in {"database is locked", "database table is locked", "database schema is locked"} + or message.startswith(("database table is locked: ", "database schema is locked: "))): + return "StyleWriterBusy" + return None class MaintenanceOwnership(NamedTuple): @@ -104,27 +145,41 @@ def connection_database_path(conn: sqlite3.Connection) -> Path | None: _coordinators: weakref.WeakValueDictionary = weakref.WeakValueDictionary() _held_fds: set[int] = set() _held_paths: dict[Path, int] = {} +_owner_waiters: dict[Path, tuple[int, int, "MaintenanceCoordinator"]] = {} _thread_owners = threading.local() -def _acquire_owner(path: Path) -> int | None: +def _acquire_owner(path: Path, on_busy: Callable[[], None] | None = None) -> int | None: with _registry_lock: if path in _held_paths: + if on_busy is not None: + on_busy() return None fd = try_file_lock(Path(str(path) + ".maintenance.lock")) if fd is not None: _held_fds.add(fd) _held_paths[path] = fd + elif on_busy is not None: + on_busy() return fd def _release_owner(fd: int) -> None: + notify = None with _registry_lock: os.close(fd) _held_fds.discard(fd) for path, claimed_fd in tuple(_held_paths.items()): if claimed_fd == fd: del _held_paths[path] + waiter = _owner_waiters.get(path) + if waiter is not None and waiter[0] == fd: + del _owner_waiters[path] + notify = waiter[2] + else: + notify = _coordinators.get(path) + if notify is not None: + notify._owner_released() @contextmanager @@ -169,13 +224,14 @@ def maintenance_owner(db_path: str | os.PathLike[str] | None) -> Iterator[Mainte def _after_fork() -> None: - global _registry_lock, _coordinators, _held_fds, _held_paths, _thread_owners + global _registry_lock, _coordinators, _held_fds, _held_paths, _owner_waiters, _thread_owners # The child must not prolong its parent's ownership by retaining an # inherited descriptor. Inherited coordinator objects reject child use. for fd in _held_fds: os.close(fd) _held_fds = set() _held_paths = {} + _owner_waiters = {} _thread_owners = threading.local() _coordinators = weakref.WeakValueDictionary() _registry_lock = threading.Lock() @@ -203,6 +259,13 @@ def __init__(self, path: Path | None) -> None: self._done.set() self._status = "pending" self._error_category: str | None = None + self._notification_generation = 0 + self._pending: tuple[int, int] | None = None + self._last_report = None + self._capability_epoch = 0 + self._capability_versions: tuple[tuple[str, str | None], ...] | None = None + self._extension_failure: LayerDependency | None = None + self._clustering_warning: tuple | None = None def _check_process(self) -> None: if self._pid != os.getpid(): @@ -214,6 +277,109 @@ def status(self) -> tuple[str, str | None]: with self._mutex: return self._status, self._error_category + def snapshot(self) -> dict: + self._check_process() + with self._mutex: + return {"status": self._status, "error_category": self._error_category, + "active": self._active, "pending_generations": int(self._pending is not None), + "capability_epoch": self._capability_epoch, + "last_results": self._last_report.results if self._last_report is not None else ()} + + def refresh_capabilities(self, *, reset_extension: bool = True) -> None: + """Refresh three package records at open/manual setup, never per add.""" + self._check_process() + versions = tuple((module, _installed_dependency(module, distribution)) for module, distribution in ( + ("numpy", "numpy"), ("hdbscan", "hdbscan"), ("sqlite_vec", "sqlite-vec"), + )) + with self._mutex: + self._capability_epoch += 1 + if reset_extension or versions != self._capability_versions: + self._extension_failure = None + self._capability_versions = versions + + def capability_snapshot(self) -> tuple[int, tuple, "LayerDependency | None"]: + self._check_process() + with self._mutex: + return self._capability_epoch, self._capability_versions or (), self._extension_failure + + def _publish_extension_evidence(self, epoch: int, dependency: "LayerDependency | None") -> None: + with self._mutex: + if epoch == self._capability_epoch: + self._extension_failure = dependency + + def observe_clustering_outcome(self, outcome: str, error_category: str | None = None) -> None: + """Warn once per failure transition, only after an actual work attempt.""" + self._check_process() + if outcome in {"success", "success_empty"}: + with self._mutex: + self._clustering_warning = None + return + if outcome not in {"unavailable", "failed"}: + return + category = _clustering_error_category(error_category) or "ClusteringUnavailable" + missing = _missing_clustering_dependencies() + key = (outcome, category, missing) + with self._mutex: + if key == self._clustering_warning: + return + self._clustering_warning = key + guidance = _clustering_install_guidance(missing) + logger.warning("Clustering %s (%s).%s", outcome, category, + " " + guidance if guidance else "") + + def clustering_failure(self) -> tuple[str, str] | None: + """Last failure observed in this process, cleared by successful work.""" + self._check_process() + with self._mutex: + return self._clustering_warning[:2] if self._clustering_warning is not None else None + + def _reserve_locked(self) -> None: + self._active = True + self._done.clear() + self._cancel.clear() + self._status = "starting" + self._error_category = None + + def request_layers(self, *, threshold: int = 25) -> bool: + """Coalesce a real foreground wake; no callback captures its engine.""" + self._check_process() + if type(threshold) is not int or threshold < 1: + raise ValueError("Maintenance threshold must be a positive integer") + with self._mutex: + if self.path is None: + self._status = "pending_in_memory" + return False + if self._active and self._cancel.is_set(): + return False + self._notification_generation += 1 + self._pending = (self._notification_generation, threshold) + pending = self._take_pending_locked() + return self._launch_layers(pending) if pending is not None else False + + def _take_pending_locked(self) -> tuple[int, int] | None: + if self._active or self._pending is None: + return None + pending, self._pending = self._pending, None + self._reserve_locked() + return pending + + def _launch_layers(self, pending: tuple[int, int]) -> bool: + def work(conn: sqlite3.Connection, cancel: threading.Event) -> "MaintenanceReport": + return run_engine_maintenance(conn, self, threshold=pending[1], cancel=cancel) + return self._launch(work, pending) + + def _owner_released(self) -> None: + self._check_process() + with self._mutex: + pending = self._take_pending_locked() + if pending is not None: + try: + self._launch_layers(pending) + except Exception: + # The launch path records a bounded failure. Notification must + # not replace a synchronous owner's unrelated operation error. + pass + def request(self, work: Callable[[sqlite3.Connection, threading.Event], None]) -> bool: """Start one owned worker, or report busy/in-memory pending without retry.""" self._check_process() @@ -223,11 +389,21 @@ def request(self, work: Callable[[sqlite3.Connection, threading.Event], None]) - return False if self._active: return False - self._active = True - self._done.clear() - self._cancel.clear() - self._status = "starting" - self._error_category = None + self._reserve_locked() + return self._launch(work) + + def _launch(self, work: Callable, pending: tuple[int, int] | None = None) -> bool: + def busy() -> None: + # Registry -> coordinator ordering closes the release/admission + # race. No coordinator-mutex holder acquires the registry lock. + with self._mutex: + self._active = False + self._status = "cancelled" if self._cancel.is_set() else "busy" + if pending is not None and self._pending is None and not self._cancel.is_set(): + self._pending = pending + if self._pending is not None and self.path in _held_paths: + _owner_waiters[self.path] = (_held_paths[self.path], self._pending[0], self) + self._done.set() fd = None handoff = None @@ -235,12 +411,8 @@ def request(self, work: Callable[[sqlite3.Connection, threading.Event], None]) - try: # Serializing descriptor acquisition with fork registration makes # every inherited owner descriptor visible to the child cleanup. - fd = _acquire_owner(self.path) + fd = _acquire_owner(self.path, on_busy=busy) if fd is None: - with self._mutex: - self._active = False - self._status = "busy" - self._done.set() return False handoff = {"fd": fd, "started": False} @@ -266,7 +438,10 @@ def run() -> None: if started: # start() can be interrupted after launching the target. That # worker owns cleanup until it really finishes, even on error. - self._cancel.set() + with self._mutex: + if self._active and self._thread is worker: + self._pending = None + self._cancel.set() raise if fd is not None: _release_owner(fd) @@ -275,7 +450,14 @@ def run() -> None: self._thread = None self._status = "failed" self._error_category = "worker_start" - self._done.set() + successor = self._take_pending_locked() + if successor is None: + self._done.set() + if successor is not None: + try: + self._launch_layers(successor) + except Exception: + pass raise def _run(self, fd: int, work: Callable[[sqlite3.Connection, threading.Event], None]) -> None: @@ -287,11 +469,15 @@ def _run(self, fd: int, work: Callable[[sqlite3.Connection, threading.Event], No try: if not self._cancel.is_set(): conn = create_db(self.path) - work(conn, self._cancel) + report = work(conn, self._cancel) if conn.in_transaction: conn.rollback() raise RuntimeError("Maintenance work left an unfinished transaction") outcome = "cancelled" if self._cancel.is_set() else "success" + if isinstance(report, MaintenanceReport): + outcome, error_category = maintenance_report_status(report, self._cancel) + with self._mutex: + self._last_report = report finally: try: if conn is not None: @@ -307,14 +493,29 @@ def _run(self, fd: int, work: Callable[[sqlite3.Connection, threading.Event], No _release_owner(fd) with self._mutex: self._active = False + self._thread = None self._status = outcome self._error_category = error_category - self._done.set() + if self._cancel.is_set(): + self._pending = None + pending = self._take_pending_locked() + if pending is None: + self._done.set() + if pending is not None: + try: + self._launch_layers(pending) + except Exception: + pass # Launch already recorded the categorical failure. def cancel(self) -> None: """Signal a phase boundary; never release ownership of active work.""" self._check_process() - self._cancel.set() + with _registry_lock, self._mutex: + self._pending = None + self._cancel.set() + waiter = _owner_waiters.get(self.path) + if waiter is not None and waiter[2] is self: + del _owner_waiters[self.path] def wait(self, timeout: float | None = None) -> bool: """Wait for this worker if requested; report whether teardown completed.""" @@ -369,6 +570,7 @@ class LayerSpec(NamedTuple): read_coverage: Callable[[sqlite3.Connection], str] | None = None borrowed_publication_guard: Callable[[sqlite3.Connection], AbstractContextManager[Callable[[], None]]] | None = None connection: sqlite3.Connection | None = None + execution_guard: Callable[[sqlite3.Connection, bool], AbstractContextManager[None]] | None = None class LayerState(NamedTuple): @@ -468,6 +670,29 @@ def read_layer_states(conn: sqlite3.Connection, specs: tuple[LayerSpec, ...]) -> return states +def _read_layer_run_baseline( + conn: sqlite3.Connection, specs: tuple[LayerSpec, ...], +) -> dict[str, LayerState] | None: + """Return no style observation only for a read lock with unchanged ownership. + + This boundary is used before running diagnostics and for cancelled reads, + never around builders or publication transactions. + """ + borrowed = conn.in_transaction + try: + return read_layer_states(conn, specs) + except sqlite3.OperationalError as error: + if (len(specs) == 1 and specs[0].layer == "style_vectors" and conn.in_transaction == borrowed + and _style_sqlite_deferral(error) == "StyleWriterBusy"): + return None + raise + + +def _style_baseline_busy_result(spec: LayerSpec, borrowed: bool) -> LayerResult: + return LayerResult(spec.layer, spec.result_key, "deferred", None, 0.0, + "StyleWriterBusy", False, "unverified", borrowed) + + @contextmanager def layer_read_snapshot( conn: sqlite3.Connection, layer: str, dependency: LayerDependency, @@ -562,10 +787,14 @@ def _owned_transaction( raise RuntimeError("Maintenance runner requires a clean transaction boundary") conn.execute("BEGIN") completed = False + automatic_rollback = False try: yield conn.execute("COMMIT") completed = True + except _StyleSQLInterrupted as error: + automatic_rollback = error.automatic_rollback + raise finally: if not completed: if conn.in_transaction: @@ -578,7 +807,10 @@ def _owned_transaction( if on_rollback is not None: on_rollback() elif on_rollback is not None: - raise _LayerRollbackFailed("Output transaction ended without a confirmed rollback") + if automatic_rollback: + on_rollback() + else: + raise _LayerRollbackFailed("Output transaction ended without a confirmed rollback") @contextmanager @@ -645,7 +877,7 @@ def _read_layer_row(conn: sqlite3.Connection, layer: str) -> tuple | None: def _restore_deferred_attempt( conn: sqlite3.Connection, layer: str, previous: tuple | None, running: tuple, - owner: MaintenanceOwnership, + owner: MaintenanceOwnership, *, protect_rollback: bool = False, ) -> None: """Restore only this owner's unchanged diagnostic, after output rollback.""" if conn.in_transaction: @@ -653,7 +885,7 @@ def _restore_deferred_attempt( current_owner = getattr(_thread_owners, "owners", {}).get(owner.path) if current_owner != owner or owner.process_id != os.getpid(): raise MaintenanceBusyError("Maintenance ownership changed before diagnostic restoration") - with _owned_transaction(conn): + with _owned_transaction(conn, on_rollback=(lambda: None) if protect_rollback else None): conn.execute("UPDATE maintenance_layers SET layer=layer WHERE 0") current = _read_layer_row(conn, layer) if (current != running or current is None or current[2] != "running" @@ -700,11 +932,16 @@ def record_attempt(state: LayerState, outcome: str, category: str | None) -> Non _record_attempt(conn, state, outcome, owner.generation, category) # Validate uniqueness before any builder is invoked. - read_layer_states(conn, specs) + if _read_layer_run_baseline(conn, specs) is None: + return (_style_baseline_busy_result(specs[0], borrowed),) for spec in specs: if cancel is not None and cancel.is_set(): break - state = read_layer_states(conn, (spec,))[spec.layer] + observed = _read_layer_run_baseline(conn, (spec,)) + if observed is None: + results.append(_style_baseline_busy_result(spec, borrowed)) + continue + state = observed[spec.layer] if state.dependency.deferred: results.append(LayerResult(spec.layer, spec.result_key, "deferred", state.output_count, 0.0, state.dependency.error_category, False, state.successful_coverage, borrowed)) @@ -728,16 +965,32 @@ def record_attempt(state: LayerState, outcome: str, category: str | None) -> Non # This diagnostic commits before the read/compute snapshot starts. previous_row = running_row = None if not borrowed: - with _owned_transaction(conn): - previous_row = _read_layer_row(conn, spec.layer) - conn.execute( - "INSERT INTO maintenance_layers(layer,builder_version,outcome,run_generation,error_category) " - "VALUES (?,?,'running',?,NULL) ON CONFLICT(layer) DO UPDATE SET " - "builder_version=excluded.builder_version,outcome='running'," - "run_generation=excluded.run_generation,error_category=NULL", - (spec.layer, state.dependency.builder_version, owner.generation), - ) - running_row = _read_layer_row(conn, spec.layer) + diagnostic_rolled_back = False + + def confirm_diagnostic_rollback() -> None: + nonlocal diagnostic_rolled_back + diagnostic_rolled_back = True + + try: + with _owned_transaction(conn, on_rollback=( + confirm_diagnostic_rollback if spec.layer == "style_vectors" else None + )): + previous_row = _read_layer_row(conn, spec.layer) + conn.execute( + "INSERT INTO maintenance_layers(layer,builder_version,outcome,run_generation,error_category) " + "VALUES (?,?,'running',?,NULL) ON CONFLICT(layer) DO UPDATE SET " + "builder_version=excluded.builder_version,outcome='running'," + "run_generation=excluded.run_generation,error_category=NULL", + (spec.layer, state.dependency.builder_version, owner.generation), + ) + running_row = _read_layer_row(conn, spec.layer) + except sqlite3.OperationalError as error: + if (spec.layer != "style_vectors" or not diagnostic_rolled_back + or _style_sqlite_deferral(error) != "StyleWriterBusy"): + raise + results.append(LayerResult(spec.layer, spec.result_key, "deferred", state.output_count, + time.monotonic() - started, "StyleWriterBusy", False, state.successful_coverage)) + continue count = None coverage = state.successful_coverage built = output_started = rolled_back = False @@ -756,7 +1009,9 @@ def confirm_rollback() -> None: guard = factory(conn) if factory is not None else None with ExitStack() as publication: transaction = _borrowed_transaction if borrowed else _owned_transaction - with transaction(conn, on_rollback=confirm_rollback): + with transaction(conn, on_rollback=confirm_rollback), ( + spec.execution_guard(conn, not borrowed) if spec.execution_guard is not None else nullcontext() + ): output_started = True state = read_layer_states(conn, (spec,))[spec.layer] if state.dependency.deferred: @@ -789,9 +1044,21 @@ def confirm_rollback() -> None: if output_started and not rolled_back: raise _LayerRollbackFailed("Deferred output has no confirmed rollback") from error if not borrowed: - _restore_deferred_attempt(conn, spec.layer, previous_row, running_row, owner) + try: + if spec.layer == "style_vectors": + _restore_deferred_attempt(conn, spec.layer, previous_row, running_row, owner, + protect_rollback=True) + else: + _restore_deferred_attempt(conn, spec.layer, previous_row, running_row, owner) + except sqlite3.OperationalError as restore_error: + if (spec.layer != "style_vectors" or conn.in_transaction + or _style_sqlite_deferral(restore_error) != "StyleWriterBusy"): + raise + # Another writer still owns SQLite. Keep the + # committed running row for later owner recovery; + # do not claim restoration or retry under this owner. results.append(LayerResult(spec.layer, spec.result_key, "deferred", baseline_state.output_count, - time.monotonic() - started, "ModelBusy", built, baseline_state.successful_coverage, borrowed)) + time.monotonic() - started, error.category, built, baseline_state.successful_coverage, borrowed)) if cancel is not None and cancel.is_set(): break continue @@ -814,6 +1081,157 @@ def confirm_rollback() -> None: return tuple(results) +def prepare_style_maintenance( + conn: sqlite3.Connection, *, _on_safe_failure: Callable[[sqlite3.OperationalError], None] | None = None, +) -> None: + """Explicit lazy enrollment; preserve a caller's final commit/rollback.""" + borrowed = conn.in_transaction + try: + read_source_revision(conn) + except sqlite3.OperationalError as error: + if _on_safe_failure is not None and conn.in_transaction == borrowed: + _on_safe_failure(error) + raise + _prepare_style_maintenance(conn, _on_safe_failure=_on_safe_failure) + + +def style_layer_spec( + conn: sqlite3.Connection, *, cancel: threading.Event | None = None, connection_owned: bool = False, +) -> LayerSpec: + """Describe private style work without installing schema or reading profiles.""" + if type(connection_owned) is not bool: + raise ValueError("Connection ownership opt-in must be a boolean") + parameters = { + "accumulator_version": 1, "hash_version": 2, "dimension": 256, + "ngrams": [3, 4, 5], "sum_format": "<256d", + } + + def resolve_dependency() -> LayerDependency: + ready = _style_output_tracking_ready(conn) + return make_layer_dependency(1, parameters, available=ready, + error_category=None if ready else "StyleTrackingUnprepared") + + def check_cancel() -> None: + if cancel is not None and cancel.is_set(): + raise LayerDeferredError("Style work cancelled", category="StyleCancelled") + + @contextmanager + def execution_guard(connection: sqlite3.Connection, owned: bool) -> Iterator[None]: + style = importlib.import_module("truememory.personality_style_vec") + progress = connection_owned and owned and cancel is not None + interrupted_by_callback = False + + def progress_handler() -> int: + nonlocal interrupted_by_callback + if cancel.is_set(): + interrupted_by_callback = True + return 1 + return 0 + + if progress: + connection.set_progress_handler(progress_handler, 1000) + try: + check_cancel() + yield + except _LayerCancelled as error: + raise LayerDeferredError("Style work cancelled", category="StyleCancelled") from error + except style._StyleSourceChanged as error: + raise LayerDeferredError("Style source changed", category="StyleSourceChanged") from error + except sqlite3.OperationalError as error: + category = _style_sqlite_deferral(error, cancelled=cancel is not None and cancel.is_set()) + if category == "StyleCancelled": + code = getattr(error, "sqlite_errorcode", None) + native_interrupt = (isinstance(code, int) and (code & 255) == 9) or ( + code is None and not hasattr(sqlite3, "SQLITE_INTERRUPT") and str(error) == "interrupted" + ) + raise _StyleSQLInterrupted(automatic_rollback=bool( + progress and interrupted_by_callback and native_interrupt and not connection.in_transaction + )) from error + if category == "StyleWriterBusy": + raise LayerDeferredError("Style writer busy", category="StyleWriterBusy") from error + raise + finally: + # Exit before the runner rolls back or commits. Borrowed handlers + # cannot be recovered through SQLite's API, so never replace them. + if progress: + connection.set_progress_handler(None, 0) + + def compatible(connection: sqlite3.Connection) -> None: + # A custom future schema can be empty. Do not downgrade its declared + # accumulator format merely because no row demonstrates it yet. + default = next(row[4] for row in connection.execute("PRAGMA table_info(entity_style_vectors)") + if row[1] == "accumulator_version") + if default is not None: + literal = default.strip(" ()'\"") + if literal not in {"0", "1", "NULL", "null"}: + raise StyleAccumulatorUnsupported("Unsupported style accumulator default") + if connection.execute( + "SELECT 1 FROM entity_style_vectors WHERE accumulator_version > 1 LIMIT 1" + ).fetchone(): + raise StyleAccumulatorUnsupported("Unsupported style accumulator version") + + def build(connection: sqlite3.Connection) -> object: + style = importlib.import_module("truememory.personality_style_vec") + source = read_source_revision(connection) + schema, query, rows = style._capture_style_source(connection, check_cancel) + result, stored_rows = style._compute_entity_style_vectors(rows, check_cancel=check_cancel) + check_cancel() + # The runner owns final commit. Upgrade before both format and source + # validation; a stale reader cannot replace a newer generation. + connection.execute("UPDATE messages SET sender=sender WHERE 0") + style._publish_style_vectors(connection, schema, query, rows, stored_rows, + check_cancel=check_cancel, before_replace=compatible) + if read_source_revision(connection) != source: + raise style._StyleSourceChanged("Style source changed before publication") + connection.execute("INSERT INTO metadata(key,value) VALUES ('style_vec_hash_version','2') " + "ON CONFLICT(key) DO UPDATE SET value=excluded.value") + return result + + return LayerSpec("style_vectors", "style_vectors", resolve_dependency, build, + lambda connection: connection.execute("SELECT count(*) FROM entity_style_vectors").fetchone()[0], + connection=conn, execution_guard=execution_guard) + + +def run_style_maintenance( + conn: sqlite3.Connection, *, force: bool = False, threshold: int = 25, + cancel: threading.Event | None = None, allow_caller_transaction: bool = False, + connection_owned: bool = False, +) -> LayerResult: + """Explicit tracked style rebuild, separate from all eight public adapters.""" + if type(allow_caller_transaction) is not bool: + raise ValueError("Caller transaction opt-in must be a boolean") + if type(connection_owned) is not bool: + raise ValueError("Connection ownership opt-in must be a boolean") + borrowed = conn.in_transaction + if borrowed and not allow_caller_transaction: + raise RuntimeError("Maintenance runner requires a clean transaction boundary") + if type(threshold) is not int or threshold < 1: + raise ValueError("Maintenance threshold must be a positive integer") + with maintenance_owner(connection_database_path(conn)): + started = time.monotonic() + + def preparation_failure(error: sqlite3.OperationalError) -> None: + if _style_sqlite_deferral(error) == "StyleWriterBusy": + raise LayerDeferredError("Style preparation writer busy", category="StyleWriterBusy") from error + + try: + prepare_style_maintenance(conn, _on_safe_failure=preparation_failure) + except LayerDeferredError as error: + return LayerResult("style_vectors", "style_vectors", "deferred", None, + time.monotonic() - started, error.category, False, "unverified", borrowed) + spec = style_layer_spec(conn, cancel=cancel, connection_owned=connection_owned) + results = run_layers(conn, (spec,), force=force, threshold=threshold, cancel=cancel, + allow_caller_transaction=allow_caller_transaction) + if results: + return results[0] + observed = _read_layer_run_baseline(conn, (spec,)) + if observed is None: + return _style_baseline_busy_result(spec, borrowed) + state = observed[spec.layer] + return LayerResult(spec.layer, spec.result_key, "deferred", state.output_count, 0.0, + "StyleCancelled", False, state.successful_coverage, borrowed) + + def nonvector_layer_specs() -> tuple[LayerSpec, ...]: """The original six standard-library adapters, with unchanged defaults.""" definitions = ( @@ -860,12 +1278,15 @@ def _installed_dependency(module: str, distribution: str) -> str | None: return None -def _cluster_schedule_dependency(conn: sqlite3.Connection) -> LayerDependency: +def _cluster_schedule_dependency( + conn: sqlite3.Connection, *, versions: dict[str, str | None] | None = None, +) -> LayerDependency: parameters = {"min_cluster_size": 10, "min_samples": 5, "metric": "euclidean", "cluster_selection_method": "eom"} - versions = {module: _installed_dependency(module, distribution) for module, distribution in ( - ("numpy", "numpy"), ("hdbscan", "hdbscan"), ("sqlite_vec", "sqlite-vec"), - )} + if versions is None: + versions = {module: _installed_dependency(module, distribution) for module, distribution in ( + ("numpy", "numpy"), ("hdbscan", "hdbscan"), ("sqlite_vec", "sqlite-vec"), + )} parameters["versions"] = versions def unavailable(category: str) -> LayerDependency: @@ -1042,3 +1463,252 @@ def dunbar_dependency() -> LayerDependency: return tuple(specs[name] for name in ( "clusters", "summaries", "contradictions", "structured_facts", "surprise", "episodes", "landmarks", "dunbar", )) + + +class MaintenanceReport(NamedTuple): + results: tuple[LayerResult, ...] + preferences: str + + +def _dependency_parameters(dependency: LayerDependency) -> dict | None: + if dependency.deferred: + return None + return json.loads(dependency.key)["parameters"] + + +def engine_layer_specs( + conn: sqlite3.Connection, coordinator: MaintenanceCoordinator | None, *, evidence: tuple | None = None, +) -> tuple[LayerSpec, ...]: + """Small live probes plus one bounded worker-capability evidence record.""" + if evidence is None: + if coordinator is None: + raise ValueError("Maintenance capability evidence is required") + evidence = coordinator.capability_snapshot() + _, versions, failure = evidence + + def resolve() -> LayerDependency: + dependency = _cluster_schedule_dependency(conn, versions=dict(versions)) + if (failure is not None and not dependency.deferred + and _dependency_parameters(dependency) == _dependency_parameters(failure)): + return failure + return dependency + + return tuple(spec._replace(resolve_dependency=resolve) if spec.layer == "clusters" else spec + for spec in all_layer_specs(conn)) + + +def _clustering_error_category(value: str | None) -> str | None: + if value is None: + return None + return value if re.fullmatch(r"[A-Za-z][A-Za-z0-9_]{0,63}", value) else "UnknownError" + + +def _missing_clustering_dependencies() -> tuple[str, ...]: + """Only an absent top-level module warrants missing-package guidance.""" + missing = [] + for module, distribution in (("numpy", "numpy"), ("hdbscan", "hdbscan"), ("sqlite_vec", "sqlite-vec")): + try: + absent = importlib.util.find_spec(module) is None + except (ImportError, ValueError): + absent = False + if absent: + missing.append(distribution) + return tuple(missing) + + +def _clustering_install_guidance(missing: tuple[str, ...]) -> str | None: + if "hdbscan" in missing: + return "Install the optional clustering extra: pip install 'truememory[clustering]'." + return None + + +def clustering_health( + conn: sqlite3.Connection | None = None, coordinator: MaintenanceCoordinator | None = None, + *, pending_reason: str | None = None, +) -> dict: + """Inspect existing evidence without opening a connection or loading models. + + Callers serialize their existing connection. Package probes inspect only + top-level specs and distribution metadata; installed packages do not prove + that their native imports or the database's published clusters are usable. + """ + health = { + "status": "degraded", "state": "unknown", "availability": "unknown", "outcome": None, + "freshness": "unknown", "coverage": "unverified", "output_count": None, + "last_error": None, "dependency_error": None, "missing_dependencies": [], + "process_error": None, + "guidance": None, "pending_reason": pending_reason or "not_connected", + "pending_caller_commit": False, + } + owned = False + try: + missing = _missing_clustering_dependencies() + versions = tuple((module, None if distribution in missing else _installed_dependency(module, distribution)) + for module, distribution in ( + ("numpy", "numpy"), ("hdbscan", "hdbscan"), ("sqlite_vec", "sqlite-vec"), + )) + health["missing_dependencies"] = list(missing) + health["guidance"] = _clustering_install_guidance(missing) + if missing or any(version is None for _, version in versions): + health.update(state="degraded", availability="unavailable", dependency_error="DependencyMissing") + elif pending_reason is not None: + health["state"] = "deferred" + if conn is None or pending_reason is not None: + return health + health["pending_reason"] = None + health["pending_caller_commit"] = conn.in_transaction + process_failure = coordinator.clustering_failure() if coordinator is not None else None + if process_failure is not None: + health["process_error"] = process_failure[1] + owned = not conn.in_transaction + if owned: + conn.execute("BEGIN") + failure = coordinator.capability_snapshot()[2] if coordinator is not None else None + spec = next(spec for spec in engine_layer_specs(conn, coordinator, evidence=(0, versions, failure)) + if spec.layer == "clusters") + state = read_layer_states(conn, (spec,))["clusters"] + dependency = state.dependency + freshness = layer_freshness(state.source, state, dependency) + health.update( + availability="deferred" if dependency.deferred else "available" if dependency.available else "unavailable", + outcome=state.outcome, freshness=freshness, coverage=state.successful_coverage, + output_count=state.output_count, last_error=_clustering_error_category(state.error_category), + dependency_error=_clustering_error_category(dependency.error_category), + ) + if dependency.deferred: + health.update(state="deferred", pending_reason="model_busy") + elif not dependency.available or state.outcome in {"failed", "unavailable"} or process_failure is not None: + health["state"] = "degraded" + elif conn.in_transaction and not owned: + health.update(state="pending", pending_reason="pending_caller_commit") + elif freshness != "current" or state.outcome not in {"success", "success_empty"}: + health.update(state="pending", pending_reason="maintenance_pending") + elif state.successful_coverage != "complete": + health.update(state="degraded", pending_reason="coverage_unverified") + else: + health.update(state="ready", status="ok") + return health + except Exception as error: + health.update(state="unknown", last_error=_clustering_error_category(type(error).__name__), + pending_reason="inspection_unavailable") + return health + finally: + if owned and conn is not None: + try: + if conn.in_transaction: + conn.rollback() + except sqlite3.Error as error: + health.update(status="degraded", state="unknown", + last_error=_clustering_error_category(type(error).__name__), + pending_reason="inspection_rollback_failed") + + +def _prepare_worker_extensions(conn: sqlite3.Connection, coordinator: MaintenanceCoordinator) -> tuple: + coordinator.refresh_capabilities(reset_extension=False) + epoch, versions, previous_failure = coordinator.capability_snapshot() + failed = False + enabled = False + try: + if dict(versions).get("sqlite_vec") is None: + failed = True + else: + module = importlib.import_module("sqlite_vec") + conn.enable_load_extension(True) + enabled = True + module.load(conn) + except Exception: + failed = True + finally: + if enabled: + conn.enable_load_extension(False) + dependency = _cluster_schedule_dependency(conn, versions=dict(versions)) + failure = previous_failure if failed else None + if failed and not dependency.deferred: + parameters = _dependency_parameters(dependency) + if parameters is not None and "identity_sha256" in parameters: + failure = make_layer_dependency(dependency.builder_version, parameters, + available=False, error_category="VectorExtensionUnavailable") + elif dependency.error_category == "VectorExtensionUnavailable": + failure = dependency + coordinator._publish_extension_evidence(epoch, failure) + return epoch, versions, failure + + +def run_engine_maintenance( + conn: sqlite3.Connection, coordinator: MaintenanceCoordinator, *, threshold: int = 25, + force: bool = False, cancel: threading.Event | None = None, + prepare_extensions: bool = True, allow_caller_transaction: bool = False, +) -> MaintenanceReport: + """Use only the supplied owned worker/original explicitly borrowed handle.""" + evidence = _prepare_worker_extensions(conn, coordinator) if prepare_extensions else None + specs = engine_layer_specs(conn, coordinator, evidence=evidence) + results = run_layers(conn, specs, threshold=threshold, force=force, cancel=cancel, + allow_caller_transaction=allow_caller_transaction) + for result in results: + if result.layer == "clusters" and result.attempted: + # RELEASE only publishes into the caller's still-rollbackable + # transaction. It cannot clear a previously observed failure. + if not (result.pending_caller_commit and result.outcome in {"success", "success_empty"}): + coordinator.observe_clustering_outcome(result.outcome, result.error_category) + preferences = "SKIPPED (no maintenance attempt)" + if cancel is not None and cancel.is_set(): + preferences = "CANCELLED" + elif force or any(result.attempted for result in results): + started = time.monotonic() + try: + importlib.import_module("truememory.personality").extract_preferences(conn) + except (ImportError, AttributeError): + preferences = "UNAVAILABLE (DependencyMissing)" + except Exception as error: + preferences = "ERROR (" + type(error).__name__[:64] + ")" + else: + preferences = f"{time.monotonic() - started:.3f}s" + if conn.in_transaction and allow_caller_transaction: + preferences += " (pending caller commit)" + return MaintenanceReport(results, preferences) + + +def maintenance_report_status(report: MaintenanceReport, cancel: threading.Event | None = None) -> tuple[str, str | None]: + if cancel is not None and cancel.is_set(): + return "cancelled", "Cancelled" + for outcome in ("failed", "unavailable", "deferred", "abandoned"): + for result in report.results: + if result.outcome == outcome: + return outcome, result.error_category + if report.preferences.startswith(("ERROR", "UNAVAILABLE")): + return "failed", "PreferencesUnavailable" + if any(result.outcome not in {"success", "success_empty", "current"} for result in report.results): + return "pending", None + if any(result.coverage != "complete" for result in report.results): + return "completed_with_limits", None + return "success", None + + +_MAINTENANCE_RESULT_KEYS = ( + "cluster_messages", "build_summaries", "detect_contradictions", "structured_facts", + "build_surprise_index", "detect_episodes", "detect_landmarks", "dunbar_hierarchy", "extract_preferences", +) + + +def maintenance_busy_result() -> dict[str, str]: + return dict.fromkeys(_MAINTENANCE_RESULT_KEYS, "BUSY (maintenance owner active)") + + +def format_maintenance_report(report: MaintenanceReport) -> dict[str, str]: + counts = {"clusters": "clusters", "structured_facts": "facts", "episodes": "episodes", + "landmarks": "events", "dunbar": "relationships"} + output = {} + for result in report.results: + if result.outcome in {"success", "success_empty"}: + prefix = f"{result.output_count} {counts[result.layer]} in " if result.layer in counts else "" + value = prefix + f"{result.elapsed_seconds:.3f}s" + if result.coverage != "complete": + value += " (" + result.coverage + ")" + else: + label = "ERROR" if result.outcome == "failed" else result.outcome.upper() + value = label + (" (" + result.error_category + ")" if result.error_category else "") + if result.pending_caller_commit: + value += " (pending caller commit)" + output[result.result_key] = value + output["extract_preferences"] = report.preferences + return output diff --git a/truememory/mcp_server.py b/truememory/mcp_server.py index 66a1cb71..6270b185 100644 --- a/truememory/mcp_server.py +++ b/truememory/mcp_server.py @@ -920,9 +920,9 @@ def _build_model_server_health() -> dict: def _build_health_payload() -> dict: """Return the `stats['health']` dict exposed by `truememory_stats`. - Each subsystem reports `{status, last_error, ...}`. ``status`` is one of - ``"ok"`` or ``"degraded"`` — the latter means a writer (F05 / F06 / F08) - stored a non-None error during the current process lifetime. + Every subsystem retains its ``ok`` / ``degraded`` status contract. + Clustering's separate ``state`` distinguishes unknown, pending and deferred + inspection; package presence never certifies a current published index. """ # Reranker — written by F06's _set_reranker. with _reranker_error_lock: @@ -948,8 +948,27 @@ def _build_health_payload() -> dict: eg_err = _encoding_gate_last_error eg_count = _encoding_gate_degradation_count + # Never construct Memory or open its database for this health observation. + try: + memory = _memory + if memory is None: + from truememory.maintenance import clustering_health + cluster_info = clustering_health() + else: + cluster_info = memory._engine.get_clustering_health() + except Exception as error: + category = type(error).__name__ + cluster_info = { + "status": "degraded", "state": "unknown", "availability": "unknown", "outcome": None, + "freshness": "unknown", "coverage": "unverified", "output_count": None, + "last_error": category if re.fullmatch(r"[A-Za-z][A-Za-z0-9_]{0,63}", category) else "UnknownError", + "dependency_error": None, "process_error": None, "missing_dependencies": [], "guidance": None, + "pending_reason": "inspection_unavailable", "pending_caller_commit": False, + } + return { "model_server": model_server_info, + "clustering": cluster_info, "reranker": { "status": "ok" if reranker_err is None else "degraded", "last_error": reranker_err, @@ -1569,8 +1588,10 @@ def truememory_status(status_id: int = 0) -> str: Returns a JSON object with: - ``rebuild``: tier-switch re-embedding progress (if any). - ``degradation``: per-subsystem health — model server, reranker, - HyDE LLM, encoding gate, and sqlite-vec. Each entry has a - ``status`` of ``"ok"`` or ``"degraded"`` plus diagnostic detail. + HyDE LLM, encoding gate, sqlite-vec, and clustering. Clustering + includes availability, freshness, coverage and the last attempt. + Every status is ``ok`` or ``degraded``; clustering's separate state + distinguishes ``unknown``, ``pending`` and ``deferred`` readiness. When everything is healthy every subsystem shows ``"ok"``. When a component silently degrades (reranker OOM, HyDE outage, diff --git a/truememory/personality_style_vec.py b/truememory/personality_style_vec.py index 26e0bd89..3734ecea 100644 --- a/truememory/personality_style_vec.py +++ b/truememory/personality_style_vec.py @@ -26,7 +26,7 @@ import re import sqlite3 import struct -from collections.abc import Iterator +from collections.abc import Callable, Iterator from contextlib import contextmanager from datetime import datetime, timezone @@ -49,6 +49,10 @@ def __init__(self, reason: str) -> None: super().__init__(f"Style accumulator rebuild required: {reason}") +class _StyleSourceChanged(sqlite3.OperationalError): + """A verified source mismatch, distinct from unexpected SQLite failures.""" + + def compute_style_vector(text: str) -> list[float]: """Compute a 256-d L2-normalized char-n-gram hash vector from text. @@ -165,30 +169,36 @@ def _l0_validate_source( schema: tuple[tuple, ...], query: str, source_rows: list[tuple], + check_cancel: Callable[[], None] | None = None, ) -> None: """Compare exact relevant source values while holding writer ownership.""" error = "L0 source changed during computation; retry the rebuild" if _l0_source_schema(conn) != schema: - raise sqlite3.OperationalError(error) + raise _StyleSourceChanged(error) current = conn.execute(query) try: for expected in source_rows: + if check_cancel is not None: + check_cancel() row = current.fetchone() if row is None or tuple(row) != expected: - raise sqlite3.OperationalError(error) + raise _StyleSourceChanged(error) if current.fetchone() is not None: - raise sqlite3.OperationalError(error) + raise _StyleSourceChanged(error) finally: current.close() def _compute_entity_style_vectors( rows: list[tuple], + *, check_cancel: Callable[[], None] | None = None, ) -> tuple[dict[str, list[float]], list[tuple]]: from collections import defaultdict by_sender: dict[str, list[str]] = defaultdict(list) for sender, content, _timestamp, ordinary in rows: + if check_cancel is not None: + check_cancel() if ordinary: by_sender[sender.lower()].append(content) @@ -198,6 +208,8 @@ def _compute_entity_style_vectors( for sender, contents in by_sender.items(): total = [0.0] * DIM for content in contents: + if check_cancel is not None: + check_cancel() vec = _validated_style_vector(compute_style_vector(content)) for index in range(DIM): total[index] += vec[index] @@ -293,6 +305,48 @@ def _style_append_transaction(conn: sqlite3.Connection) -> Iterator[None]: conn.execute("RELEASE SAVEPOINT truememory_style_append") +def _capture_style_source( + conn: sqlite3.Connection, check_cancel: Callable[[], None] | None = None, +) -> tuple[tuple[tuple, ...], str, list[tuple]]: + schema = _l0_source_schema(conn) + query = ( + f"SELECT sender, content, timestamp, {_l0_directive_filter(schema)} AS ordinary " + "FROM messages WHERE sender != '' ORDER BY sender, timestamp" + ) + rows = [] + cursor = conn.execute(query) + try: + for row in cursor: + if check_cancel is not None: + check_cancel() + rows.append(tuple(row)) + finally: + cursor.close() + return schema, query, rows + + +def _publish_style_vectors( + conn: sqlite3.Connection, schema: tuple[tuple, ...], query: str, + rows: list[tuple], stored_rows: list[tuple], *, + check_cancel: Callable[[], None] | None = None, + before_replace: Callable[[sqlite3.Connection], None] | None = None, +) -> None: + """Publish only inside the transaction owned by the selected entry point.""" + _l0_validate_source(conn, schema, query, rows, check_cancel) + _ensure_style_accumulator_schema(conn) + if before_replace is not None: + before_replace(conn) + if check_cancel is not None: + check_cancel() + conn.execute("DELETE FROM entity_style_vectors") + conn.executemany( + """INSERT INTO entity_style_vectors + (entity, vector, message_count, updated_at, vector_sum, accumulator_version) + VALUES (?, ?, ?, ?, ?, ?)""", + stored_rows, + ) + + def build_entity_style_vectors(conn: sqlite3.Connection) -> dict[str, list[float]]: """Batch-build style vectors for every entity (sender) in the database. @@ -313,25 +367,11 @@ def build_entity_style_vectors(conn: sqlite3.Connection) -> dict[str, list[float A changed source raises ``sqlite3.OperationalError`` without retrying. """ with _l0_transaction(conn): - schema = _l0_source_schema(conn) - query = ( - f"SELECT sender, content, timestamp, {_l0_directive_filter(schema)} AS ordinary " - "FROM messages WHERE sender != '' " - "ORDER BY sender, timestamp" - ) - rows = [tuple(row) for row in conn.execute(query)] + schema, query, rows = _capture_style_source(conn) result, stored_rows = _compute_entity_style_vectors(rows) with _l0_transaction(conn, write=True): - _l0_validate_source(conn, schema, query, rows) - _ensure_style_accumulator_schema(conn) - conn.execute("DELETE FROM entity_style_vectors") - conn.executemany( - """INSERT INTO entity_style_vectors - (entity, vector, message_count, updated_at, vector_sum, accumulator_version) - VALUES (?, ?, ?, ?, ?, ?)""", - stored_rows, - ) + _publish_style_vectors(conn, schema, query, rows, stored_rows) return result diff --git a/truememory/storage.py b/truememory/storage.py index c1f2d1ab..dc5b6d9e 100644 --- a/truememory/storage.py +++ b/truememory/storage.py @@ -18,6 +18,7 @@ import os import sqlite3 import time +from collections.abc import Callable from pathlib import Path log = logging.getLogger(__name__) @@ -374,6 +375,80 @@ def _messages_fts_trigger_definitions(conn: sqlite3.Connection | None = None) -> "surprise", "episodes", "landmarks", "dunbar", ) +_STYLE_OUTPUT_INVALIDATE_SQL = """UPDATE maintenance_layers SET + outcome='pending',full_rebuild_required=1,successful_coverage='unverified', + attempted_epoch=NULL,attempted_revision=NULL,attempted_dependency=NULL, + attempted_insert_count=NULL,run_generation=NULL,error_category=NULL + WHERE layer='style_vectors'""" +_STYLE_OUTPUT_TRIGGERS = { + "truememory_style_output_" + operation.lower(): + "CREATE TRIGGER truememory_style_output_" + operation.lower() + + " AFTER " + operation + " ON entity_style_vectors BEGIN " + + _STYLE_OUTPUT_INVALIDATE_SQL + "; END" + for operation in ("INSERT", "UPDATE", "DELETE") +} + + +def _style_output_tracking_ready(conn: sqlite3.Connection) -> bool: + expected = {name: " ".join(sql.split()) for name, sql in _STYLE_OUTPUT_TRIGGERS.items()} + installed = {name: " ".join(sql.strip().rstrip(";").split()) for name, sql in conn.execute( + "SELECT name,sql FROM sqlite_master WHERE type='trigger' AND name IN (?,?,?)", tuple(expected), + )} + return installed == expected and conn.execute( + "SELECT 1 FROM maintenance_layers WHERE layer='style_vectors'").fetchone() is not None + + +def _prepare_style_maintenance( + conn: sqlite3.Connection, *, _on_safe_failure: Callable[[sqlite3.OperationalError], None] | None = None, +) -> None: + """Explicitly enroll style output; ordinary schema initialization stays lazy.""" + expected = {name: " ".join(sql.split()) for name, sql in _STYLE_OUTPUT_TRIGGERS.items()} + + def installed() -> dict[str, str]: + return {name: " ".join(sql.strip().rstrip(";").split()) for name, sql in conn.execute( + "SELECT name,sql FROM sqlite_master WHERE type='trigger' AND name IN (?,?,?)", + tuple(expected), + )} + + owned = not conn.in_transaction + try: + if _style_output_tracking_ready(conn): + return + conn.execute("BEGIN IMMEDIATE" if owned else "SAVEPOINT truememory_style_tracking") + except sqlite3.OperationalError as error: + # No preparation writes precede admission. A failed admission is safe + # only while the original transaction boundary remains unchanged. + if _on_safe_failure is not None and conn.in_transaction == (not owned): + _on_safe_failure(error) + raise + try: + conn.execute("UPDATE maintenance_layers SET layer=layer WHERE 0") + conn.execute("INSERT OR IGNORE INTO maintenance_layers(layer) VALUES ('style_vectors')") + current = installed() + if current != expected: + for name, sql in _STYLE_OUTPUT_TRIGGERS.items(): + if current.get(name) != expected[name]: + conn.execute("DROP TRIGGER IF EXISTS " + name) + conn.execute(sql) + # Missing/replaced triggers make even v1 output unverified. This + # metadata-only invalidation also supersedes a failed attempt. + conn.execute(_STYLE_OUTPUT_INVALIDATE_SQL) + conn.execute("COMMIT" if owned else "RELEASE SAVEPOINT truememory_style_tracking") + except BaseException as error: + if conn.in_transaction: + if owned: + conn.rollback() + else: + conn.execute("ROLLBACK TO SAVEPOINT truememory_style_tracking") + conn.execute("RELEASE SAVEPOINT truememory_style_tracking") + if (_on_safe_failure is not None and isinstance(error, sqlite3.OperationalError) + and conn.in_transaction == (not owned)): + _on_safe_failure(error) + # An ended transaction, failed rollback or failed savepoint release + # cannot attest that preparation left the previous generation intact. + raise + + _SCHEMA_SQL += _DUNBAR_OWNERSHIP_SQL + ";\n" diff --git a/truememory/vector_search.py b/truememory/vector_search.py index 507e4bb9..965ff0ab 100644 --- a/truememory/vector_search.py +++ b/truememory/vector_search.py @@ -739,13 +739,23 @@ def _encode_with_mps_fallback(model, texts, **kwargs): """ -def _capture_rebuild_model() -> tuple[object, tuple[int, str, int]]: +def _capture_rebuild_model(conn: sqlite3.Connection | None = None) -> tuple[object, tuple[int, str, int]]: + """Caller transactions can borrow a cached model, but cannot wait or load.""" + if conn is not None and conn.in_transaction: + if not _lock.acquire(blocking=False): + raise VectorPublicationChanged("Embedding model is busy; retry outside the caller transaction") + try: + if _model is None: + raise VectorPublicationChanged("Embedding model is not loaded; load it outside the caller transaction") + return _model, (_model_generation, EMBEDDING_MODEL, _embedding_dim) + finally: + _lock.release() with _lock: before = (_model_generation, EMBEDDING_MODEL) model = get_model() with _lock: if before != (_model_generation, EMBEDDING_MODEL): - raise RuntimeError("Embedding model changed while loading rebuild model") + raise VectorPublicationChanged("Embedding model changed while loading rebuild model") return model, (_model_generation, EMBEDDING_MODEL, _embedding_dim) @@ -767,8 +777,9 @@ class VectorPublicationChanged(RuntimeError): @contextmanager -def _foreground_model_fence(identity: tuple[int, str, int]) -> Iterator[None]: - if not _lock.acquire(blocking=False): +def _foreground_model_fence(identity: tuple[int, str, int], *, blocking: bool = False) -> Iterator[None]: + """Blocking admission is permitted only before an owned writer starts.""" + if not _lock.acquire(blocking=blocking): raise VectorPublicationChanged("Embedding model is busy; retry vector publication") try: if identity != (_model_generation, EMBEDDING_MODEL, _embedding_dim): @@ -815,7 +826,7 @@ def _foreground_vector_publication( ) -> Iterator[tuple[str, str]]: from truememory.rebuild_source import rebuild_transaction - with rebuild_transaction(conn, write=True, publication_fence=_foreground_model_fence(identity)): + with _foreground_model_fence(identity, blocking=not conn.in_transaction), rebuild_transaction(conn, write=True): yield _validate_foreground_vector_target(conn, identity, message_id) @@ -856,7 +867,7 @@ def _build_streamed_vectors( save_manifest(conn, key, empty) _clear_build_in_progress(conn, table) return 0 - model, identity = _capture_rebuild_model() + model, identity = _capture_rebuild_model(conn) with rebuild_transaction(conn, write=True, publication_fence=_rebuild_model_fence(identity)): manifest = load_manifest(conn, key) resumable = (manifest is not None and not manifest.complete @@ -1191,7 +1202,7 @@ def embed_single(conn: sqlite3.Connection, message_id: int, content: str) -> Non message_id: The ``messages.id`` of the row being embedded. content: The text to embed. """ - model, identity = _capture_rebuild_model() + model, identity = _capture_rebuild_model(conn) embedding = _encode_with_mps_fallback(model, [content])[0] normed = _normalize_for_cosine(embedding) if normed is None: