semantic search: strut keeps the graph's vectors (the pass, the index reconcile, typed search through the domain indexes, the no-embed mark)

AGENTS.md

@@ -126,9 +126,10 @@ strut/
 │   │   ├── claims-writer.ts # ClaimsWriter: the claim graph's invariants — ≥1 check per claim, edits SUPERSEDE (successor claim carries ABOUT + checks; successor check takes over TESTS), retire = timestamp, detach = deleted edge, last check / last subject refused
 │   │   ├── claim-schema-upgrade.ts # one-shot standalone mirror of jarvis migration 124 (Claim re-keyed on id, Epistemic/Thing); runs before the ontology seed, only with STRUT_GRAPH_SEED_ONTOLOGY
 │   │   ├── exact-key-migration.ts # one-shot boot re-key of existing StrutWorkflow/-Version/StrutStep/-Version nodes from the sanitized name to the exact (hex) key — ledger-stamped, idempotent, every namespace, soft-deleted nodes included; ref_ids, edges and the Schema nodes untouched
-│   │   ├── embeddings.ts  # local all-MiniLM-L6-v2 via transformers.js, tokenized like sentence-transformers (256 incl. specials); NULL-scan backfill
-│   │   ├── search.ts      # the read surface: hybrid search (RRF + title boost + usage tiebreak), get/neighbors/counts, ontology, namespaces
-│   │   ├── backend.ts     # openGraphBackend(): cached per config; runs seed + backfill on first open
+│   │   ├── index-reconcile.ts # boot step (plans/semantic-search.md §2): a live Schema's `index` that is a non-empty STRICT SUBSET of the bundled library's (`fixtures/jarvis-ontology.ts`) is widened to the library's list — jarvis's own values, written directly (its Schema API re-embeds a whole type in one request and rebuilds the shared keyword index per type). Once per fixture `version` (a `Migration` ledger row), so a list jarvis narrows on purpose is never fought over; the pass then sees every node of the type as stale
+│   │   ├── embeddings.ts  # local all-MiniLM-L6-v2 via transformers.js, tokenized like sentence-transformers (256 incl. specials), 2 intra-op threads; + the EMBEDDING PASS (plans/semantic-search.md §1): a timer in the serving process that keeps `Data_Bank` + `text_embeddings` + the domain label current on EVERY node of a Schema-marked type, whoever wrote it (stakgraph's ingest, jarvis, an import) — stale = the stored text ≠ the writer's composition from the Schema's `index` fields, or no vector — strips the no-embed types (`embed: false`: `Turn` via its Schema, `StrutToolCall` / `StrutTurn` in code), pages by `node_key` (an index seek per page, never a scan), works 15 s a minute, sweeps hourly (code types first), embeds NEWCOMERS by `date_added_to_graph` between sweeps. `GET /graph/embeddings` is its status; `STRUT_GRAPH_EMBEDDINGS=off` turns it off with the embedder
+│   │   ├── search.ts      # the read surface: hybrid search (RRF + title boost + usage tiebreak), get/neighbors/counts, ontology, namespaces. Routing (`routeIndexes`): a typed search reads the requested types' DOMAIN vector/fulltext indexes with an over-fetch (`overfetch`: k from cap×10 by ×4 to 2,000 — the database cannot filter a vector query, so `type` is a WHERE after the nearest-k cut), an untyped one the global `text_embeddings_vector_index`
+│   │   ├── backend.ts     # openGraphBackend(): cached per config; runs the migrations, the seeds and the Schema `index` reconcile on first open, then STARTS the embedding pass (nothing waits on it; `close()` stops it)
 │   │   ├── query.ts       # readQuery(): read-only raw Cypher for the chat builder's graph_query — keyword pre-check + READ tx, streamed row cap, tx timeout, strings/vectors compacted; a chat tool, deliberately not a step
 │   │   ├── test-util.ts   # live-test helpers (wipe, canonical graph snapshot) — only ever point at a throwaway Neo4j
 │   │   └── fixtures/      # Python-produced MiniLM golden vectors + jarvis sanitize_node_key parity cases
@@ -296,7 +297,7 @@ GATEWAY_IMAGE=stakgraph-gateway:v1.6.2 docker compose -f docker-compose.yml \
 | `STRUT_AUTO_RESUME` | `1` (persistent run store) | Boot-time auto-resume of runs cut off by a crash/restart (RUN_CONTROL_SPEC §5.3): on whenever the run store outlives the process — anything but `MemoryRunStore`, whatever the workspace's kind, so the lab's graph workspace + `FileRunStore` counts — the newest root run per workflow with a log but no summary, unless paused/cancelling, older than 7 days, or already resumed 5 times. `0` disables. |
 | `NEO4J_URI` / `NEO4J_HOST` | (unset) / `localhost:7687` | Graph backend connection — same names and defaults as mcp's own Neo4j client: `NEO4J_URI` wins, else `bolt://<NEO4J_HOST>`; `NEO4J_USER`/`NEO4J_PASSWORD` default `neo4j`/`testtest`; optional `NEO4J_DATABASE`. The `graph/*` lib steps read these via the secrets capability (secret store → env) and need nothing configured for a local Neo4j; `openGraphBackendFromEnv` stays opt-in (null when neither is set). |
 | `STRUT_GRAPH_NAMESPACE` | `default`   | jarvis namespace every Strut node is written into |
-| `STRUT_GRAPH_EMBEDDINGS` | (on)       | `off` disables the local MiniLM embedder (vectors stay NULL; search is fulltext-only) |
+| `STRUT_GRAPH_EMBEDDINGS` | (on)       | `off` disables the local MiniLM embedder and the embedding pass (vectors stay NULL; search is fulltext-only) |
 | `STRUT_GRAPH_SEED_ONTOLOGY` | (off)   | `1` seeds the bundled jarvis ontology (153 schemas + edge schemas + indexes, add-only) on first open, so a standalone Neo4j can host jarvis-typed data (Document, EvalSet, Concept, …) with no jarvis process. No-op on a jarvis-seeded DB. Also turns on the one-shot `Claim` schema upgrade (the standalone mirror of jarvis migration 124) — never run against a jarvis-hosted graph. |
 | `STRUT_VERIFY_BUDGET_USD` | `10` | Verify-pass spend cap PER VERIFIED RUN: once the pass's checks have reported this much, remaining checks presumed paid (an `agent`/`llm` step anywhere in the check closure, or an unresolvable one) are skipped (`lastVerify: { skipped: "budget" }`) and the claim stays `unknown` — never `supported`. Checks that report no cost never count. |
 | `STRUT_VERIFY_BUDGET_USD_PER_DAY` | `100` | The same cap PER SUBJECT PER (UTC) DAY, computed from the run store alone: the cost of today's runs under `check:<id>` tagged with that subject. A harness that verifies many candidates raises it, or sets its paid checks to `manual`. |

plans/jarvis-graph-compat.md

@@ -300,20 +300,16 @@ vector) to prove parity with jarvis's encoder.
 compatible: a jarvis backfill would simply find nothing NULL. Never
 invent a pending-marker property.
 
-**Crash-safe backfill sweep** (replaces the durability the queue gave
-jarvis): a crash between the node MERGE and the vector write leaves
-`text_embeddings` NULL permanently. At every boot of the graph
-backend, sweep and heal:
-
-```cypher
-MATCH (n:Domain_strut)
-WHERE n.Data_Bank IS NOT NULL AND n.text_embeddings IS NULL
-RETURN n.ref_id, n.Data_Bank
-```
-
-→ embed in batches → `SET n.text_embeddings` (matching jarvis's own
-NULL-scan backfill idiom, `migration.py:857-861`). Idempotent, cheap
-when clean.
+**The pass** (replaces the durability the queue gave jarvis, and the
+boot-time `Domain_strut` NULL-scan backfill this plan first specified):
+a crash between the node MERGE and the vector write leaves
+`text_embeddings` NULL — and so does every node another writer puts in
+the graph. `EmbeddingPass` (`embeddings.ts`; `plans/semantic-search.md`
+§1) runs on a timer inside the serving process, started when the graph
+backend opens: every node of a Schema-marked type, whoever wrote it, is
+stale when its stored `Data_Bank` is not the writer's composition from
+the Schema's `index` fields or its vector is missing, and is rewritten
+the way the writer would have written it. Idempotent, cheap when clean.
 
 **Do not implement**: `edge_text_embeddings`; metaphone3.
 Per-property `vector_index` embeddings (`{stem}_embeddings`) ARE

plans/semantic-search.md

@@ -1,6 +1,23 @@
 # Semantic search without jarvis — strut keeps the graph's vectors
 
-> **Status (2026-10-11): PROPOSED, nothing built.** Written after a seeded
+> **Status (2026-10-11): the strut side is BUILT** — §1 (`src/graph/embeddings.ts`,
+> `EmbeddingPass`), §2 (`src/graph/index-reconcile.ts`), §3 (`routeIndexes` /
+> `overfetch` in `search.ts`), strut's half of §4 (`embed` in the core-key
+> mirror, the resolver, the writer, the two registry entries), `GET
+> /graph/embeddings`. Not yet: the jarvis PR (§4), the stakgraph PR (§5), the
+> fixture re-dump after the jarvis PR lands (until then a test sets `Turn`'s
+> mark by hand), the swarm38 validation (after the mcp pin). Where the build
+> differs from the text below: the reconcile only WIDENS (a string `index`
+> equal to the library's one-item list is left as it is — fewer writes to
+> jarvis's nodes, same meaning); the pass addresses a node by `elementId`,
+> never by `ref_id` or `(node_key, namespace)` (stakgraph writes both, but
+> nothing here should depend on it); the newcomer tick is one range seek over
+> `Data_Bank` reading every marked type's index fields at once, not one query
+> per type; a tick logs only when it did something, and a page that fails
+> moves the sweep to the next type (the next sweep retries); the per-field
+> `vector_index` vectors are filled by the sweep, not by the newcomer tick;
+> `GET /graph/embeddings` has no web client. Before this it was PROPOSED,
+> written after a seeded
 > `explore` run on swarm38 never used the graph (job run `1791735838490` →
 > explore run `1791735843683`, a question about hive's GitHub integration:
 > 0 graph calls, 16 file reads), and revised twice the same day. First

specs/API.md

@@ -495,6 +495,7 @@ injected its own `sessions` capability.
 | Method | Path | Response |
 | ------ | ---- | -------- |
 | GET    | `/graph/nodes/:ref_id` | `{ ref_id, node_type, name, properties }` — the node as the graph holds it NOW (vectors and internal stamps stripped); 404 when it is gone; 501 on a deployment with no graph |
+| GET    | `/graph/embeddings` | the embedding pass's status (`plans/semantic-search.md` §1; a debugging read): `{ enabled: true, sweep: { n, startedAt, finishedAt, phase, type, cursor }, current, last, skipped, newcomers: { since, embedded }, lastError, ticks }` — `current` / `last` are per type `{ seen, stale, embedded, removed, errors }` for the sweep in flight / the last complete one; `skipped` is `[{ type, reason: "no_schema" \| "no_index" \| "no_embed" \| "invalid_label" }]`. `{ enabled: false }` with `STRUT_GRAPH_EMBEDDINGS=off`; 501 on a deployment with no graph |
 
 What the run view opens when a node in a step's **Nodes** list is clicked: a
 run's events say which nodes a step touched (`nodes`, above), never their

src/createStrut.test.ts

@@ -473,6 +473,7 @@ describe("createStrut", () => {
     const res = await strut.app.request("/graph/nodes/n-1");
     assert.equal(res.status, 501);
     assert.match(((await res.json()) as { error: string }).error, /no graph/);
+    assert.equal((await strut.app.request("/graph/embeddings")).status, 501);
   });
 
   it("exposes /steps with registered types", async () => {

src/createStrut.ts

@@ -2000,6 +2000,16 @@ export async function createStrut<TServices = unknown>(
     return c.json({ ref_id: node.ref_id, node_type: node.node_type ?? null, name: node.name ?? null, properties: node.properties });
   });
 
+  // The embedding pass (plans/semantic-search.md §1) as the graph backend
+  // reports it: the sweep's position, per-type counts, the types it skips
+  // and why, the newcomer mark, the last error. `{ enabled: false }` with
+  // embeddings off. Read-only — a debugging read. 501 without a graph.
+  app.get("/graph/embeddings", async (c) => {
+    const graph = opts.graph ?? workspace.graph;
+    if (!graph) return c.json({ error: "this deployment has no graph" }, 501);
+    return c.json(graph.embeddings ? graph.embeddings.status() : { enabled: false });
+  });
+
   // ── Secrets ──────────────────────────────────────────────────────────────
   // Deployment-scoped credential store behind `ctx.services.secrets`. Values
   // are write-only over the API: GET returns NAMES + metadata only, never the

src/graph/backend.ts

@@ -10,15 +10,20 @@
  *   2c. `migrateExactKeys` — one-shot re-key of the workspace nodes from the
  *       sanitized name to the exact one (hex);
  *   3. `seedStrutDomain` — schema meta-graph, constraints, indexes (§4);
- *   4. `backfillEmbeddings` — heal any NULL vectors left by a crash (§2).
+ *   3b. `reconcileIndexLists` — stale Schema `index` lists widened to the
+ *       bundled library's, once per fixture version (semantic-search.md §2);
+ *   4. the `EmbeddingPass` is STARTED (semantic-search.md §1): a timer in
+ *      this process that embeds every marked node whoever wrote it, strips
+ *      the no-embed types, and keeps up with newcomers. Nothing waits on it.
  *
  * Consumers (the `graph/*` lab steps, a future `Neo4jWorkspaceStore` and
  * run projector) call `openGraphBackend(cfg)` and share the instance.
  */
 import { Bolt, graphConfigFromEnv, type GraphConfig } from "./bolt.js";
 import { upgradeClaimSchema, type ClaimSchemaUpgradeReport } from "./claim-schema-upgrade.js";
 import { EdgeWriter } from "./edge-writer.js";
-import { MiniLMEmbedder, backfillEmbeddings, type BackfillReport } from "./embeddings.js";
+import { EmbeddingPass, MiniLMEmbedder, type EmbeddingPassOptions } from "./embeddings.js";
+import { reconcileIndexLists, type IndexReconcileReport } from "./index-reconcile.js";
 import { NodeWriter, type Embedder } from "./node-writer.js";
 import { seedJarvisOntology, type OntologySeedReport } from "./ontology-seed.js";
 import { SchemaResolver } from "./schema-resolver.js";
@@ -40,6 +45,9 @@ export interface GraphBackendOptions {
    *  host jarvis-typed data (Document, EvalSet, Concept, …) with no jarvis
    *  process. Env: `STRUT_GRAPH_SEED_ONTOLOGY=1`. */
   seedOntology?: boolean;
+  /** Tuning for the embedding pass (page sizes, the tick, `autoStart`) —
+   *  tests. Not part of the cache key. */
+  pass?: EmbeddingPassOptions;
 }
 
 export interface GraphBackend {
@@ -58,7 +66,10 @@ export interface GraphBackend {
   readonly exactKeyMigration: ExactKeyMigrationReport | undefined;
   readonly claimSchemaUpgrade: ClaimSchemaUpgradeReport | undefined;
   readonly ontologySeed: OntologySeedReport | undefined;
-  readonly backfill: BackfillReport | undefined;
+  readonly indexReconcile: IndexReconcileReport | undefined;
+  /** The pass, running on its timer; undefined with embeddings off or the
+   *  boot skipped. `status()` is `GET /graph/embeddings`. */
+  readonly embeddings: EmbeddingPass | undefined;
   close(): Promise<void>;
 }
 
@@ -108,7 +119,7 @@ export function openGraphBackendFromEnv(
 export async function closeGraphBackends(): Promise<void> {
   const all = [...cache.values()];
   cache.clear();
-  await Promise.all(all.map((p) => p.then((b) => b.bolt.close()).catch(() => undefined)));
+  await Promise.all(all.map((p) => p.then((b) => b.close()).catch(() => undefined)));
 }
 
 async function open(cfg: GraphConfig, opts: GraphBackendOptions): Promise<GraphBackend> {
@@ -123,7 +134,7 @@ async function open(cfg: GraphConfig, opts: GraphBackendOptions): Promise<GraphB
     let exactKeyMigration: ExactKeyMigrationReport | undefined;
     let claimSchemaUpgrade: ClaimSchemaUpgradeReport | undefined;
     let ontologySeed: OntologySeedReport | undefined;
-    let backfill: BackfillReport | undefined;
+    let indexReconcile: IndexReconcileReport | undefined;
     if (!opts.skipBoot) {
       // Legacy names first, so the seed below extends the renamed Schema
       // nodes instead of creating twins beside them.
@@ -159,9 +170,17 @@ async function open(cfg: GraphConfig, opts: GraphBackendOptions): Promise<GraphB
         console.warn(`[graph] re-keyed workspace nodes on their exact names: ${JSON.stringify(exactKeyMigration)}`);
       }
       seed = await seedStrutDomain(bolt);
-      if (embedder) backfill = await backfillEmbeddings(bolt, embedder);
+      // Not gated on `seedOntology`: a jarvis-hosted graph is the one that
+      // needs it (its Schemas predate jarvis's own widening).
+      indexReconcile = await reconcileIndexLists(bolt);
+      if (indexReconcile.status === "reconciled") {
+        console.warn(`[graph] widened stale Schema index lists to the bundled library's: ${JSON.stringify(indexReconcile.widened)}`);
+      }
     }
     const schemas = new SchemaResolver(bolt);
+    // The pass starts now and runs in the background; the old awaited
+    // backfill held the boot for as long as the sweep took.
+    const embeddings = embedder && !opts.skipBoot ? new EmbeddingPass(bolt, embedder, schemas, opts.pass) : undefined;
     const backend: GraphBackend = {
       cfg,
       bolt,
@@ -176,9 +195,11 @@ async function open(cfg: GraphConfig, opts: GraphBackendOptions): Promise<GraphB
       exactKeyMigration,
       claimSchemaUpgrade,
       ontologySeed,
-      backfill,
+      indexReconcile,
+      embeddings,
       async close() {
         for (const [k, p] of cache) if ((await p.catch(() => null)) === backend) cache.delete(k);
+        await embeddings?.stop();
         await bolt.close();
       },
     };