semantic-search plan: compare-based stale test, node_key cursor, domain-routed typed search, no-embed mark via jarvis

[Evanfeenstra]

Second review of plans/semantic-search.md (the plan merged in #138), done against the code it touches, then a third pass reading jarvis-backend end to end. Still PROPOSED, nothing built; the choices to confirm are under "Rulings".

What the review found

• stakgraph's ingest rewrites Data_Bank on every re-sync. ast/src/lang/graphs/neo4j/queries/nodes.rs puts Data_Bank = name into $properties and runs ON MATCH SET node += $properties. A stale test keyed on _search_fields_used (the previous plan) calls such a node clean, with a vector that no longer matches the stored text. Now Problem 5.
• LIMIT-without-cursor scans are quadratic during catch-up (the 448k Turn removals would visit nodes on the order of a billion times) and make every clean check a scan of the whole type, roughly four full-graph reads an hour, forever.
• Global index + over-fetch starves rare types. Endpoint is 642 of ~330k vectors globally and a few hundred of ~40k in Domain_codeartifact, whose index already exists on swarm38 and in strut's seed.
• The vector removal was a standing tug of war with jarvis's worker, which keeps embedding every Turn written through its API.

What changed

• Ruling 4: stale = stored Data_Bank differs from the text the writer composes from the index fields, or the vector is missing. One invariant that also covers edited and filled-in fields (the "comparing cursor" the plan deferred). The decision is made in strut through buildSearchText, so there is one composition.
• §1: the pass pages by a node_key cursor (index-backed on every seeded type; the pass runs the same IF NOT EXISTS statement at open). Sweep hourly, code types first, no-embed removals last; a per-minute newcomer tick on date_added_to_graph; intraOpNumThreads: 2 on the ONNX session since the pass runs inside the serving process; GET /graph/embeddings reporting the sweep, per-type counts and the types skipped and why.
• Ruling 5 / §3: typed searches route through the domain vector indexes, untyped ones through text_embeddings_vector_index; over-fetch only for the residual. domain_all_vector_index is dropped (never exists on 5.19).
• Ruling 2 / §4: the no-embed mark goes on the Schema node as embed: false through a jarvis PR; strut's own two types keep the mark in code. Until it deploys, the pass removes what jarvis adds.
• §5: a stakgraph PR moves Data_Bank to ON CREATE only, like date_added_to_graph. strut is correct without either PR; they remove waste.
• Ruling 3 / §2: the index reconcile is stamped per fixture version, so a narrowing jarvis makes on purpose is not undone at every strut boot.
• Tests, validation, "not doing" and open questions follow.

§4 verified against jarvis-backend 04317241

• Boot path (gunicorn.conf.py → setup_jarvis): Migration.run(), then SchemaHelper.run() → create_schema for missing types, then _synchronize_schemas → _get_missing_schema_updates, which sends a live Schema only the SCHEMA_KNOWN_PROPERTIES keys it lacks (plus absent attributes) through schema_crud.update_schema. A new library key lands at the next boot; an existing index is never touched. That is the crud function, not the route: sync_indexes_for_schema (keyword index drop/recreate) runs only on a PUT, and index_changed only when index is in the update. So the mark lands with no rebuild and no re-embed, and strut's direct index write triggers nothing.
• Why the key must be in jarvis's list first: _validate_schema_properties refuses any key outside SCHEMA_KNOWN_PROPERTIES; _assert_is_valid_schema requires a type string for anything else. And an older jarvis reads embed: false on a Schema node as a required attribute of type false (assert_is_valid_schema_of_type), so every node create of that type through its API would fail. The key therefore reaches a node only through jarvis's own sync; strut never writes it.
• Enqueue sites the PR gates: the create path in schema_node_helper.py, the v1 update path in node_service.py, migrate_existing_data_bank_index, the v1 bulk edge path. get_schema is uncached, so the check is free and jarvis composes with a widened index from its next write.
• The startup backfills are tracker-gated one-shots; the worker drains Redis only; ensure_concept_domain_labels_on_startup is jarvis's own per-boot precedent for the label half of the pass; date_added_to_graph is epoch ms in jarvis too (now_epoch_ms), closing that open question.