Match business questions to code with native concept search

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## Unreleased
 
+- Match engineering business vocabulary through bounded synonyms and witnessed
+  Markdown-to-code links, with typed approximate-match provenance. Retain bounded
+  document and Python docstring excerpts in resource details for prose search.
+- Reuse valid disposable FTS indexes on reopen; validate FTS with a writable
+  handle before opening the query connection read-only.
+- Add optional offline corpus-LSA embeddings to `ask`/`search` and MCP discovery/
+  search. Exact lookup is unchanged; semantic corpus limits fail explicitly.
+
 - Resolve Python calls through annotated receivers and unit-of-work member chains,
   injected constructor fields, and source return contracts. Preserve unknown
   invocations as bounded source-call inventories and optional inferred receiver

COMPATIBILITY.md

@@ -1,5 +1,24 @@
 # Compass compatibility
 
+Concept recall adds optional `conceptMatches` to `compass.query/1` with closed
+methods `synonym`, `document_link`, and `semantic_lsa`. Strict consumers must
+update to the matching enum/field manifest and fingerprint. Absence means no
+concept recall; approximate matching never promotes structural confidence.
+CLI `ask`/non-exact `search` and MCP bounded discovery/search enable synonym and
+document-link recall by default; `--semantic-search` / `semantic_search: true`
+explicitly enable offline corpus-LSA fallback per request. Exact lookup retains
+its existing meaning. Semantic corpus limits return typed errors, not no-match.
+
+Graph/1 resource details gain optional `content`, a source-backed excerpt of at
+most 4 KiB for document/rationale resources. Document identity, edge direction,
+and reference provenance remain unchanged. Closed resource-details decoders
+must admit this additive field. Markdown normalization preserves its bounded
+prose, and Python rationale extraction retains docstring excerpts under AST
+cache v15. Disposable JSON query-index format v2 adds prose terms; older caches
+rebuild automatically. Existing immutable store indexes/graphs must be newly
+published to obtain prose postings; history is never rewritten. Native names
+and existing reference edges remain queryable in older graphs.
+
 Compass is an independent native product. Its compatibility contract is defined
 by the shipped `compass` CLI, documented file and protocol formats, native tests,
 and migration notes. Compass does not execute, import, check out, or test against

MIGRATION.md

@@ -7,6 +7,15 @@ layout remains visible and clearly owned.
 
 ## Graph rebuilds and query resolution
 
+To enable full bounded document/prose recall, rebuild with
+`compass extract <root> --force`. Document/rationale resources now carry optional
+`details.data.content` (up to 4 KiB), and strict query consumers must admit
+optional `conceptMatches` and its closed match-method enum. AST cache v15 and
+JSON query-index format v2 invalidate older disposable entries automatically.
+Rebuild existing SQLite sidecars from the new graph to get prose postings;
+historical realizations retain their original content and indexes. Default
+concept recall needs no credentials; corpus-LSA search remains explicit opt-in.
+
 Rebuild Python graphs with `compass extract <root> --code-only --force` to obtain
 receiver-chain resolution and source-call inventories. AST cache version 14
 recomputes older entries automatically. Add `--inference-level max` to retain

SECURITY.md

@@ -127,3 +127,13 @@ have independent bounds. Membership comes from recorded directed structural
 relationships; inferred/deferred records, calls and references are excluded.
 Source failures and budget exhaustion remain explicit. Missing digests never
 become verified merely because an owner-to-member relationship is present.
+
+Optional concept semantic search (`--semantic-search` / MCP `semantic_search`)
+learns bounded corpus-LSA embeddings from the selected local graph. It uses no
+network, credentials, runtime model download, or external vector service. Names
+and bounded document/docstring excerpts stay in the graph and engine memory;
+this option does not upload source. Graph publication may now retain up to 4 KiB
+of source-backed prose per document/rationale resource, so exported artifacts
+include those excerpts as well as existing names and provenance. Apply the same
+repository disclosure policy to these graph artifacts. Corpus size/deadline
+failures remain explicit and do not publish a partially built semantic index.

crates/compass-cli/src/code_query_commands.rs

@@ -266,9 +266,17 @@ fn execute(
         open_with_engine(&graph, program.as_deref(), &cache, engine)
             .map_err(|error| error.to_string())?
     }
-    .with_deadline(deadline);
+    .with_deadline(deadline)
+    .with_semantic_search(args.iter().any(|arg| arg == "--semantic-search"));
     let limits = limits(args, page_scale)?;
     let include_heuristic = args.iter().any(|arg| arg == "--include-heuristic");
+    if args.iter().any(|arg| arg == "--semantic-search") && !matches!(operation, "ask" | "search") {
+        return Err("--semantic-search is supported by compass ask and search".to_owned());
+    }
+    if args.iter().any(|arg| arg == "--semantic-search") && args.iter().any(|arg| arg == "--exact")
+    {
+        return Err("--semantic-search cannot be combined with --exact".to_owned());
+    }
     let relations = impact_relations(operation, args)?;
     let (response, question, operands) = match operation {
         "ask" => {

crates/compass-cli/src/help.rs

@@ -176,7 +176,7 @@ const PAGES: &[Page] = &[
         "ask",
         "Route a natural-language question to a typed code-graph query",
         ["compass ask <QUESTION> [OPTIONS]"],
-        "Arguments:\n  <QUESTION>                    Natural-language code-graph question\n\nOptions:\n  --graph <PATH>                Typed graph [default: compass-out/graph.json]\n  --at <REV>                    Use an immutable trusted revision graph\n  --program <PATH>              Optional Program IR enrichment\n  --cache <DIR>                 Query-index cache directory\n  --engine <default|json|store> Graph storage engine [default: default]\n  --max-depth <N>               Traversal radius\n  --max-nodes <N>               Node bound\n  --max-edges <N>               Edge bound\n  --max-paths <N>               Path bound\n  --max-candidates <N>          Candidate bound [default: 64]\n  --include-heuristic           Include heuristic evidence\n  --format <text|agent-json|json> Output format [default: text]\n  --text-budget <N>             Approximate tokens per text page [default: 2000]\n  --cursor <TOKEN>              Continue the same text result (text only)\n  --timeout-ms <N>              Query deadline in milliseconds [default: 60000; max: 600000]\n  --brief                       Emit compass.query.agent-view.brief/1 (agent-json only)\n\nExamples:\n  compass ask \"who calls PaymentService.charge?\"\n  compass ask \"what does CheckoutController.create call?\" --format json\n  compass ask \"path from CheckoutController.create to PaymentGateway.charge\"\n  compass ask \"who calls PaymentService.charge?\" --at HEAD~2 --format json\n\nNotes:\n  High-confidence callers, callees, impact, and path questions route to the matching typed operation. Contradictory or low-confidence input falls back to bounded symbol search. --at is mutually exclusive with --graph, --program, --cache, and --engine. The response uses compass.query/1."
+        "Arguments:\n  <QUESTION>                    Natural-language code-graph question\n\nOptions:\n  --graph <PATH>                Typed graph [default: compass-out/graph.json]\n  --at <REV>                    Use an immutable trusted revision graph\n  --program <PATH>              Optional Program IR enrichment\n  --cache <DIR>                 Query-index cache directory\n  --engine <default|json|store> Graph storage engine [default: default]\n  --max-depth <N>               Traversal radius\n  --max-nodes <N>               Node bound\n  --max-edges <N>               Edge bound\n  --max-paths <N>               Path bound\n  --max-candidates <N>          Candidate bound [default: 64]\n  --semantic-search             Optional offline corpus-LSA fallback [default: off]\n  --include-heuristic           Include heuristic evidence\n  --format <text|agent-json|json> Output format [default: text]\n  --text-budget <N>             Approximate tokens per text page [default: 2000]\n  --cursor <TOKEN>              Continue the same text result (text only)\n  --timeout-ms <N>              Query deadline in milliseconds [default: 60000; max: 600000]\n  --brief                       Emit compass.query.agent-view.brief/1 (agent-json only)\n\nExamples:\n  compass ask \"who calls PaymentService.charge?\"\n  compass ask \"what does CheckoutController.create call?\" --format json\n  compass ask \"path from CheckoutController.create to PaymentGateway.charge\"\n  compass ask \"who calls PaymentService.charge?\" --at HEAD~2 --format json\n\nNotes:\n  High-confidence callers, callees, impact, and path questions route to the matching typed operation. Contradictory or low-confidence input falls back to bounded symbol search. --at is mutually exclusive with --graph, --program, --cache, and --engine. The response uses compass.query/1."
     ),
     page!(
         "call-graph",
@@ -320,7 +320,7 @@ const PAGES: &[Page] = &[
         "search",
         "Search typed code symbols by name",
         ["compass search <QUERY> [OPTIONS]"],
-        "Arguments:\n  <QUERY>                       Symbol name or qualified name\n\nOptions:\n  --exact                       Exact ID or normalized name only; no lexical fallback\n  --file <PATH>                 Exact stored source path (requires --exact)\n  --line <N>                    Declaration start line (requires --exact and --file)\n  --kind <KIND>                 Stored node kind, e.g. function (requires --exact)\n  --graph <PATH>                Typed graph [default: compass-out/graph.json]\n  --program <PATH>              Optional Program IR enrichment\n  --cache <DIR>                 Query-index cache directory\n  --engine <default|json|store> Graph storage engine [default: default]\n  --max-candidates <N>          Candidate bound [default: 64]\n  --format <text|agent-json|json> Output format [default: text]\n  --text-budget <N>             Approximate tokens per text page [default: 2000]\n  --cursor <TOKEN>              Continue the same text result (text only)\n  --timeout-ms <N>              Query deadline in milliseconds [default: 60000; max: 600000]\n  --brief                       Emit compass.query.agent-view.brief/1 (agent-json only)\n\nExamples:\n  compass search PaymentService\n  compass search checkout --format json\n  compass search PaymentService --text-budget 800\n\nNotes:\n  Exact mode retains all matches. Candidate bounds apply before filters; truncation never proves uniqueness or absence. Exact text results do not automatically widen bounds.\n  Search uses the versioned compass.query/1 response contract in all formats. Text output is paged: the footer carries a cursor that continues the same deterministic result at the same page budget."
+        "Arguments:\n  <QUERY>                       Symbol name or qualified name\n\nOptions:\n  --exact                       Exact ID or normalized name only; no lexical fallback\n  --file <PATH>                 Exact stored source path (requires --exact)\n  --line <N>                    Declaration start line (requires --exact and --file)\n  --kind <KIND>                 Stored node kind, e.g. function (requires --exact)\n  --graph <PATH>                Typed graph [default: compass-out/graph.json]\n  --program <PATH>              Optional Program IR enrichment\n  --cache <DIR>                 Query-index cache directory\n  --engine <default|json|store> Graph storage engine [default: default]\n  --max-candidates <N>          Candidate bound [default: 64]\n  --semantic-search             Optional offline corpus-LSA fallback [default: off]\n  --format <text|agent-json|json> Output format [default: text]\n  --text-budget <N>             Approximate tokens per text page [default: 2000]\n  --cursor <TOKEN>              Continue the same text result (text only)\n  --timeout-ms <N>              Query deadline in milliseconds [default: 60000; max: 600000]\n  --brief                       Emit compass.query.agent-view.brief/1 (agent-json only)\n\nExamples:\n  compass search PaymentService\n  compass search checkout --format json\n  compass search PaymentService --text-budget 800\n\nNotes:\n  Exact mode retains all matches. Candidate bounds apply before filters; truncation never proves uniqueness or absence. Exact text results do not automatically widen bounds.\n  Search uses the versioned compass.query/1 response contract in all formats. Text output is paged: the footer carries a cursor that continues the same deterministic result at the same page budget."
     ),
     page!(
         "callers",

crates/compass-cli/tests/code_query_cli.rs

@@ -2816,3 +2816,54 @@ fn impact_relation_filter_is_typed_and_reports_excluded_contacts() -> Result<(),
     assert!(bad.stderr.contains("unknown impact relationship"));
     Ok(())
 }
+
+#[test]
+fn concept_search_discloses_synonyms_and_keeps_exact_mode_strict() -> Result<(), Box<dyn Error>> {
+    let dir = tempfile::tempdir()?;
+    let path = support::write_typed_graph(dir.path())?;
+    let mut graph = GraphDocument::load(&path)?;
+    let target = graph
+        .nodes
+        .iter_mut()
+        .find(|node| node.name == "Target")
+        .ok_or("missing Target")?;
+    target.name = "AuthzGate".into();
+    target.qualified_name = "Fixture.AuthzGate".into();
+    let target_id = target.id.clone();
+    std::fs::write(&path, serde_json::to_vec(&graph)?)?;
+    let execute = |command: &str, format: &str, extra: &[&str]| {
+        let mut args = vec![
+            OsString::from(command),
+            OsString::from("authorization"),
+            OsString::from("--graph"),
+            path.as_os_str().to_owned(),
+            OsString::from("--format"),
+            OsString::from(format),
+        ];
+        args.extend(extra.iter().map(OsString::from));
+        run(Frontend::Compass, args)
+    };
+    let found = execute("search", "json", &[]);
+    assert_eq!(found.code, 0, "{}", found.stderr);
+    let response: Value = serde_json::from_str(&found.stdout)?;
+    assert!(response["conceptMatches"].as_array().is_some_and(|items| {
+        items
+            .iter()
+            .any(|item| item["nodeId"] == target_id && item["method"] == "synonym")
+    }));
+    let text = execute("ask", "text", &["--semantic-search"]);
+    assert_eq!(text.code, 0, "{}", text.stderr);
+    assert!(
+        text.stdout.contains("Approximate") && text.stdout.contains("via synonym"),
+        "{}",
+        text.stdout
+    );
+    let exact = execute("search", "json", &["--exact"]);
+    assert_eq!(exact.code, 0, "{}", exact.stderr);
+    let response: Value = serde_json::from_str(&exact.stdout)?;
+    assert_eq!(response["nodes"], serde_json::json!([]));
+    let rejected = execute("search", "json", &["--exact", "--semantic-search"]);
+    assert_ne!(rejected.code, 0);
+    assert!(rejected.stderr.contains("cannot be combined with --exact"));
+    Ok(())
+}

crates/compass-files/src/cache.rs

@@ -13,7 +13,7 @@ use sha2::{Digest, Sha256};
 use crate::{FileError, StatHashIndex, file_hash, io_error, write_bytes_atomic, write_json_atomic};
 
 /// Changes whenever cached extraction semantics change, even if the wire encoding does not.
-pub const AST_CACHE_VERSION: &str = "14";
+pub const AST_CACHE_VERSION: &str = "15";
 /// Portable cache encoding version used in the on-disk namespace.
 pub const CACHE_ENCODING_VERSION: u32 = 1;
 const MESSAGEPACK_EXTENSION: &str = "msgpack";

crates/compass-graph/src/snapshot.rs

@@ -3330,7 +3330,7 @@ fn build_term_postings(graph: &GraphDocument) -> BTreeMap<String, Vec<String>> {
 }
 
 fn searchable_node_terms(node: &NodeRecord) -> BTreeSet<String> {
-    let mut terms = BTreeSet::new();
+    let mut terms = compass_model::search::document_search_terms(node);
     terms.extend(search_terms(&node.name));
     terms.extend(search_terms(&node.qualified_name));
     terms.extend(compass_model::search::identifier_search_terms(&node.name));

crates/compass-graph/src/v1.rs

@@ -1833,7 +1833,7 @@ fn normalize_v1_with_mode(
 /// and derive stable identities. `compass.graph/1` keeps its established wire
 /// shape: document nodes publish as `Resource(document)` and express Markdown
 /// structure through source-backed nodes, meaningful names, qualified names,
-/// containment, and reference edges.
+/// containment, reference edges, and an optional bounded content excerpt.
 fn downgrade_document_details_for_graph_v1(nodes: &mut [NodeRecord]) {
     for node in nodes {
         let details = match node.details.take() {
@@ -1866,6 +1866,7 @@ fn downgrade_document_details_for_graph_v1(nodes: &mut [NodeRecord]) {
             resource_kind: ResourceKind::Document,
             uri: details.uri,
             media_type,
+            content: details.content,
         }));
     }
 }
@@ -3697,7 +3698,12 @@ fn insert_raw_evidence(attributes: &mut Map<String, Value>, evidence: &Provenanc
 
 fn insert_raw_node_details(attributes: &mut Map<String, Value>, details: &NodeDetails) {
     match details {
-        NodeDetails::File(_) | NodeDetails::Resource(_) => {}
+        NodeDetails::File(_) => {}
+        NodeDetails::Resource(details) => {
+            insert_optional_string(attributes, "resource_content", details.content.as_ref());
+            insert_optional_string(attributes, "uri", details.uri.as_ref());
+            insert_optional_string(attributes, "media_type", details.media_type.as_ref());
+        }
         NodeDetails::Document(details) => insert_raw_document_details(attributes, details),
         NodeDetails::Symbol(details) => {
             if let Some(inventory) = &details.call_sites {
@@ -5365,6 +5371,7 @@ fn node_details(
                             .map(|slug| format!("#{slug}"))
                     }),
                     media_type: optional_string(attributes, "media_type"),
+                    content: optional_string(attributes, "resource_content"),
                 }))
             }
         }

crates/compass-graph/tests/markdown_identity.rs

@@ -413,3 +413,41 @@ fn markdown_explicit_link_evidence_survives_absolute_path_aliases() -> Result<()
     assert!(edge.relationship_site.is_some());
     Ok(())
 }
+
+#[test]
+fn published_markdown_retains_bounded_prose_and_trusted_round_trip() -> Result<(), Box<dyn Error>> {
+    let dir = tempfile::tempdir()?;
+    let root = dir.path();
+    let path = root.join("README.md");
+    fs::write(
+        &path,
+        format!(
+            "# Operations\n\n{} Account suspension is implemented here.\n",
+            "Context ".repeat(100)
+        ),
+    )?;
+    let raw = Engine::default().extract(&path)?;
+    let graph = normalize_document_v1(
+        &build_from_extraction(&raw, true, Some(root)),
+        root,
+        "sha256:test",
+        None,
+    )?;
+    let paragraph = graph.nodes.iter().find(|node| matches!(&node.details, Some(NodeDetails::Resource(details)) if details.content.as_ref().is_some_and(|text| text.contains("Account suspension")))).ok_or("prose missing")?;
+    assert!(!paragraph.name.contains("Account suspension"));
+    assert!(compass_model::search::document_search_terms(paragraph).contains("suspension"));
+    let round_trip: compass_model::code_graph::GraphDocument =
+        serde_json::from_slice(&serde_json::to_vec(&graph)?)?;
+    assert_eq!(round_trip, graph);
+    let recomposition = compass_graph::extraction_from_v1(&graph);
+    let raw = recomposition
+        .nodes
+        .iter()
+        .find(|node| node.id == paragraph.id)
+        .ok_or("missing recomposition record")?;
+    assert!(
+        raw.string("resource_content")
+            .contains("Account suspension")
+    );
+    Ok(())
+}

crates/compass-graph/tests/store_snapshot.rs

@@ -373,6 +373,7 @@ fn immutable_snapshot_indexes_semantic_markdown_table_nodes() -> Result<(), Box<
         resource_kind: ResourceKind::Document,
         uri: Some("#owners".to_owned()),
         media_type: Some("text/markdown".to_owned()),
+        content: None,
     }));
 
     let mut row = node("table-row");
@@ -384,6 +385,7 @@ fn immutable_snapshot_indexes_semantic_markdown_table_nodes() -> Result<(), Box<
         resource_kind: ResourceKind::Document,
         uri: None,
         media_type: Some("text/markdown".to_owned()),
+        content: None,
     }));
     document.nodes.extend([table, row]);