Haystack live retrieval and a local MCP surface

README.md

@@ -8,7 +8,7 @@ Two workspaces, never mixed. **Live research** uses actual EIA RSS, Federal Regi
 
 1. Start the local app (setup below). Source checks run on opening the live workspace, then while the tab is visible and auto-check is enabled. The polling floor is 60 seconds, with longer provider Cache-Control/Retry-After respected. These are publication feeds, not tick streams.
 2. Inspect source status, capture times and publication dates. First-seen documents are labelled new to this installation, not newly published. Errors preserve old evidence with a stale-source warning.
-3. Filter the inbox or select documents. Ask a focused question using **Evidence retrieval only** or **AI brief with citations**. Retrieval uses SQLite FTS5/BM25 over the latest captured document versions.
+3. Filter the inbox or select documents. Ask a focused question using **Evidence retrieval only** or **AI brief with citations**. Retrieval runs a Haystack 2.x BM25 pipeline over the latest captured document versions.
 4. Inspect the returned passages, exact citations and version diffs. AI output is a draft interpretation, not verified truth. No model call occurs on background source refresh.
 5. Export the complete job/evidence JSON. Jobs persist their stages, result and errors; cancellation suppresses acceptance of in-flight results. Restarted unfinished jobs become `interrupted`, never falsely `completed`.
 
@@ -67,6 +67,43 @@ python -m infodesk.app
 
 Open **http://127.0.0.1:8000/**. The local database defaults to `data/desk.sqlite3` and survives restarts. Override its path with `INFODESK_DB`. There is no automatic `.env` loader; set variables in your shell.
 
+### Connect in Cursor
+
+The same local tools are available over MCP stdio. GitHub Pages has no MCP server and does not pretend to.
+
+```json
+{
+  "mcpServers": {
+    "info-desk": {
+      "command": "python",
+      "args": ["-m", "infodesk.mcp_server"],
+      "cwd": "/absolute/path/to/info-desk",
+      "env": {
+        "INFODESK_DB": "/absolute/path/to/info-desk/data/desk.sqlite3"
+      }
+    }
+  }
+}
+```
+
+Point `command` at the virtualenv interpreter that has `pip install -e ".[dev]"` installed. After the local app has captured live documents, `search_sources` reuses that Haystack BM25 index. `get_claim` reads the recorded claim table. `propose_write` inserts a pending draft only.
+
+List the same tools from the running local API:
+
+```powershell
+curl -s http://127.0.0.1:8000/api/mcp/tools
+```
+
+#### MCP checklist
+
+| Check | What is actually true |
+|---|---|
+| Auth | Loopback FastAPI + stdio. No API key, login, or hosted MCP. Trusted-Host still rejects unexpected Host headers. |
+| Tool schema | Extra fields (`approve`, unknown names) and missing required fields return a visible 422/tool error. |
+| Writes need `approve()` | `propose_write` cannot approve itself. `POST /api/drafts/{id}/decide` or `store.approve()` writes the note. |
+| 429 / timeout | `search_sources` surfaces existing live-check 429/timeout details. It does not invent a successful fetch. |
+| Eval / harness | Recorded harness, Pages replay JSON and retrieval eval stay on their existing paths. MCP is local/live only. |
+
 To preview only the public replay:
 
 ```powershell
@@ -101,13 +138,17 @@ The default model request timeout is an uncalibrated resource budget, not a late
 |---|---|---|
 | `sources.py`, `tools.py` | Allowlisted stored excerpts | Capture metadata and controlled reads; editorial additions excluded from evidence |
 | `extract.py`, `claims.py` | Curated claim rules + analysed source text | Claim cells, named quantities and attribution; absence is not denial |
+| `haystack_retrieval.py` | Latest captured `live_chunks` rows | Haystack `Document` + `InMemoryBM25Retriever` Pipeline; lexical ranking only |
+| `mcp_tools.py`, `mcp_server.py` | Live store + recorded claims | stdio/HTTP MCP: `search_sources`, `get_claim`, `propose_write`; no self-approve |
 | `interpret.py` | Evidence-backed findings; optional Ollama | Deterministic memo and bounded finding ordering |
 | `validate.py`, `policy.py` | Quantities, licenses, configured instruction-pattern checks | Release guard; no model-controlled database write |
 | `store.py` | SQLite transactions and conditional status updates | Persistent run snapshots, deduplication by complete draft identity, atomic approval and note insertion |
 | `workbench.py` | The same analysis service | Shared payload for local API and generated replay |
 | `public/` | Replay JSON or local API | Responsive source selector, claim inspector, review queue, exports and visible evaluations |
 
-**Why this scope:** the live workspace demonstrates retrieval-augmented generation with lexical search, versioned evidence and bounded model output. The recorded workspace remains a fixed-rule comparison, not RAG or a general extraction benchmark. Neither independently verifies the publisher's claims.
+Live ask retrieval is: **captures (SQLite ledger) → Haystack `Document` → BM25 retriever → existing citation layer**. Haystack does not write the ledger, approve drafts, or call the model. Citations, claims, `approve()`, and allowlisted connectors stay in Python. This is local lexical BM25, not embeddings, semantic search, or a production service. GitHub Pages still serves the recorded case; Haystack runs only on the live localhost path. Captures are flat paragraph windows, so AutoMergingRetriever is unused.
+
+**Why this scope:** the live workspace demonstrates retrieval-augmented generation with lexical Haystack BM25, versioned evidence and bounded model output. The recorded workspace remains a fixed-rule comparison, not RAG or a general extraction benchmark. Neither independently verifies the publisher's claims.
 
 The local API binds to loopback, rejects unexpected Host headers and cross-origin writes, and accepts only registered source IDs. It is **single-user**. Do not expose it publicly without authentication, per-user authorization, request limits and a production storage design.
 
@@ -122,6 +163,8 @@ The local API binds to loopback, rejects unexpected Host headers and cross-origi
 | `POST /api/drafts/{id}/decide` | `{"decision":"approve"}` or `{"decision":"reject"}` |
 | `GET /api/db` | Current local ledger state |
 | `GET /api/harness` | Isolated regression evaluation, not mutations to your ledger |
+| `GET /api/mcp/tools` | List local MCP tools and input schemas |
+| `POST /api/mcp/tools/{name}` | Invoke one listed tool; `propose_write` still needs human `approve()` |
 
 All supplied public-source cases currently produce `hold` or `verify_first`; the UI correctly prevents approval. Positive approval, rejection, replay and concurrency paths are exercised with clearly labelled SYNTHETIC tests. Exporting a memo never approves a note.
 

evals/README.md

@@ -6,7 +6,7 @@ Run from the repository root:
 python -m infodesk.evaluate_retrieval --output test-results/retrieval-evaluation.json
 ```
 
-This exercises the actual SQLite FTS5/BM25 retriever using an isolated in-memory database. It makes no network requests or model calls and never modifies the live database. CI saves the full result as an artifact.
+This exercises the live Haystack BM25 retriever using captured versions loaded into an isolated in-memory store. It makes no network requests or model calls and never modifies the live database. CI saves the full result as an artifact. The committed baseline JSON below was measured against SQLite FTS5/BM25 on 10 September 2026; CI reruns the same annotations through Haystack and does not treat that file as a pass/fail gate.
 
 ## Dataset and scoring
 

public/index.html

@@ -51,7 +51,7 @@
         <section id="panel-evaluations" role="tabpanel" aria-labelledby="tab-evaluations" hidden><div class="review-layout"><div><div class="section-heading"><h3>Regression cases</h3><span id="eval-score" class="badge"></span></div><p class="muted">Assertions inspect the action, evidence and SQLite writes. Attack and retry faults are synthetic. These curated cases are not a general AI accuracy benchmark.</p><div id="eval-cases"></div></div><aside class="memo"><h3>This analysis, step by step</h3><p id="trace-mode" class="muted"></p><ol id="trace"></ol><p id="engine-note"></p><details><summary>Database snapshot at analysis time</summary><pre id="db"></pre></details><a href="harness.json">Download recorded evaluation report ↗</a></aside></div></section>
       </section>
     </section>
-    <section id="how-it-works" class="architecture"><p class="eyebrow">Under the hood</p><h2>From a source update to a reviewable answer.</h2><p>The live pipeline collects official records, keeps document versions and retrieves passages with SQLite FTS5 / BM25. A local model writes a short answer with exact source quotations. Failed checks stay failed; no recording is substituted for a live result.</p><div class="table-scroll"><table><thead><tr><th>Component</th><th>Depends on</th><th>Produces / enforces</th></tr></thead><tbody><tr><th>Live connectors</th><td>EIA RSS, Federal Register API, OFAC HTML</td><td>Conditional HTTP requests, provider cooldowns, response limits and visible source errors</td></tr><tr><th>Version store</th><td>Validated source documents</td><td>Append-only captures, fingerprints and diffs; publication and fetch dates stay separate</td></tr><tr><th>Retrieval</th><td>Latest stored versions + user-selected scope</td><td>BM25-ranked passages with document/version IDs; no-match abstention</td></tr><tr><th>AI brief</th><td>Question + retrieved passages + structured-output model</td><td>Quote and numeric validation, visible failures, reviewable interpretation — no automatic publication</td></tr><tr><th>Job worker</th><td>Local API + SQLite checkpoints</td><td>Responsive progress, one-job backpressure, cancellation and restart recovery to an interrupted state</td></tr></tbody></table></div><p class="muted">The separate recorded case remains a reproducible regression lab. Live coverage is publisher summaries, abstracts and license titles, not every full document. Lexical retrieval can miss relevant passages, and citation matching cannot prove semantic correctness. Public multi-user hosting requires authentication and a deployed backend.</p></section>
+    <section id="how-it-works" class="architecture"><p class="eyebrow">Under the hood</p><h2>From a source update to a reviewable answer.</h2><p>The live pipeline collects official records, keeps document versions and retrieves passages with a Haystack BM25 pipeline. A local model writes a short answer with exact source quotations. Failed checks stay failed; no recording is substituted for a live result.</p><div class="table-scroll"><table><thead><tr><th>Component</th><th>Depends on</th><th>Produces / enforces</th></tr></thead><tbody><tr><th>Live connectors</th><td>EIA RSS, Federal Register API, OFAC HTML</td><td>Conditional HTTP requests, provider cooldowns, response limits and visible source errors</td></tr><tr><th>Version store</th><td>Validated source documents</td><td>Append-only captures, fingerprints and diffs; publication and fetch dates stay separate</td></tr><tr><th>Retrieval</th><td>Latest stored versions + Haystack BM25 Pipeline</td><td>Lexical BM25-ranked passages with document/version IDs; no-match abstention. Haystack does not write the ledger or call the model</td></tr><tr><th>AI brief</th><td>Question + retrieved passages + structured-output model</td><td>Quote and numeric validation, visible failures, reviewable interpretation — no automatic publication</td></tr><tr><th>Job worker</th><td>Local API + SQLite checkpoints</td><td>Responsive progress, one-job backpressure, cancellation and restart recovery to an interrupted state</td></tr></tbody></table></div><p class="muted">The separate recorded case remains a reproducible regression lab. Live coverage is publisher summaries, abstracts and license titles, not every full document. Lexical retrieval can miss relevant passages, and citation matching cannot prove semantic correctness. Public multi-user hosting requires authentication and a deployed backend.</p></section>
   </main>
   <footer><span>Info Desk · Built by Jordi</span><a href="https://github.com/coder058/info-desk">Source, setup & tests ↗</a></footer>
 </body>

pyproject.toml

@@ -13,13 +13,16 @@ dependencies = [
   "httpx>=0.27",
   "beautifulsoup4>=4.12",
   "defusedxml>=0.7",
+  "haystack-ai==2.31.0",
+  "mcp==2.2.0",
 ]
 
 [project.optional-dependencies]
 dev = ["pytest>=8.0", "httpx>=0.27"]
 
 [project.scripts]
 info-desk = "infodesk.app:main"
+info-desk-mcp = "infodesk.mcp_server:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]

src/infodesk/app.py

@@ -19,6 +19,7 @@
 from .workbench import analyse
 from .live_store import LiveStore
 from .live_service import LiveRequest, LiveService
+from .mcp_tools import ToolError, bind_store, call_tool, list_tools
 
 ROOT = Path(__file__).resolve().parents[2]
 PUBLIC = ROOT / "public"
@@ -39,6 +40,7 @@ def create_app(db_path: str | Path | None = None) -> FastAPI:
     async def lifespan(app):
         app.state.store = LiveStore(db_path or os.environ.get("INFODESK_DB", str(ROOT / "data" / "desk.sqlite3")))
         app.state.live = LiveService(app.state.store)
+        bind_store(app.state.store)
         try:
             yield
         finally:
@@ -175,6 +177,23 @@ def harness():
         # Each evaluation uses an isolated in-memory database, never the user's ledger.
         return run_harness()
 
+    @app.get("/api/mcp/tools")
+    def mcp_tools():
+        return {
+            "tools": list_tools(),
+            "transport": ["stdio", "http"],
+            "approval": "propose_write cannot approve. Human approve() writes the note.",
+            "pages": "Recorded GitHub Pages has no MCP server.",
+        }
+
+    @app.post("/api/mcp/tools/{name}")
+    def mcp_call(name: str, body: dict):
+        try:
+            return call_tool(store(), name, body)
+        except ToolError as exc:
+            status = 404 if str(exc).startswith("Unknown tool") else 422
+            raise HTTPException(status, str(exc)) from exc
+
     @app.get("/")
     def index():
         return FileResponse(PUBLIC / "index.html")

src/infodesk/evaluate_retrieval.py

@@ -8,6 +8,8 @@
 from pathlib import Path
 from time import perf_counter
 
+import haystack
+
 from .live_store import LiveStore
 from .sources import ROOT, load_sources
 
@@ -76,7 +78,7 @@ def evaluate(dataset_path, repeats):
         'dataset_sha256': hashlib.sha256(raw).hexdigest(),
         'corpus_sha256': {sid: hashlib.sha256(s['body'].encode()).hexdigest() for sid, s in sources.items()},
         'runtime': {'python': platform.python_version(), 'sqlite': sqlite3.sqlite_version,
-                    'platform': platform.platform()},
+                    'haystack': haystack.__version__, 'platform': platform.platform()},
         'summary': {'cases': len(rows), 'answerable': len(answerable), 'unanswerable': len(negative),
             'mean_evidence_recall_at_returned_limit': mean([r['evidence_recall'] for r in answerable]),
             'top1_hit_rate': mean([r['top1_relevant'] for r in answerable]),
@@ -88,7 +90,8 @@ def evaluate(dataset_path, repeats):
             'Corpus is the supplied dated excerpts, not current live connector coverage or independently verified facts.',
             'No answer generation or semantic correctness is scored. Context returned for an unanswerable question is not necessarily a wrong answer.',
             'No retrieval confidence or safe abstention can be inferred from non-empty search results.',
-            'Local in-memory retrieval timing excludes network, persistence I/O and model inference.'],
+            'Local in-memory retrieval timing excludes network, persistence I/O and model inference.',
+            'Live ranking is Haystack lexical BM25 over captured versions, not embeddings or a semantic index.'],
         'cases': rows,
     }
 

src/infodesk/haystack_retrieval.py

@@ -0,0 +1,78 @@
+"""Lexical BM25 retrieval over already-captured live document versions.
+
+Haystack ranks passages only. It does not write the SQLite ledger, approve
+drafts, or call a model. Captures are flat paragraph windows, so this module
+does not invent hierarchy metadata or use AutoMergingRetriever.
+"""
+import os
+
+# Keep CI and local tests offline; BM25 does not need provider credentials.
+os.environ.setdefault("HAYSTACK_TELEMETRY_ENABLED", "False")
+
+from haystack import Document, Pipeline
+from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+
+# GUESS: six passages bound CPU-model context. Relevance is not calibrated.
+PASSAGE_LIMIT = 6
+
+
+def documents_from_captures(rows):
+    """Map ledger chunks to Haystack Documents. Title is searchable; body stays exact."""
+    documents = []
+    for row in rows:
+        title = row["title"] or ""
+        body = row["body"]
+        content = f"{title}\n{body}".strip() if title and title != body else body
+        documents.append(Document(
+            id=str(row["chunk_id"]),
+            content=content,
+            meta={
+                "chunk_id": row["chunk_id"],
+                "body": body,
+                "version_id": row["version_id"],
+                "document_id": row["document_id"],
+                "source_id": row["source_id"],
+                "url": row["url"],
+                "title": row["title"],
+                "received_at": row["received_at"],
+                "published_at": row["published_at"],
+                "text_hash": row["text_hash"],
+                "coverage": row["coverage"],
+            },
+        ))
+    return documents
+
+
+def retrieve_passages(question, rows, top_k=PASSAGE_LIMIT):
+    """Run a Haystack 2.x BM25 Pipeline and return the existing passage dicts."""
+    if not rows:
+        return []
+    store = InMemoryDocumentStore()
+    store.write_documents(documents_from_captures(rows))
+    pipeline = Pipeline()
+    pipeline.add_component(
+        "retriever",
+        InMemoryBM25Retriever(document_store=store, top_k=top_k),
+    )
+    result = pipeline.run({"retriever": {"query": question}})
+    passages = []
+    for document in result["retriever"]["documents"]:
+        if document.score is None or document.score <= 0:
+            continue
+        meta = document.meta
+        passages.append({
+            "chunk_id": meta["chunk_id"],
+            "body": meta["body"],
+            "version_id": meta["version_id"],
+            "document_id": meta["document_id"],
+            "source_id": meta["source_id"],
+            "url": meta["url"],
+            "title": meta["title"],
+            "received_at": meta["received_at"],
+            "published_at": meta["published_at"],
+            "text_hash": meta["text_hash"],
+            "coverage": meta["coverage"],
+            "rank": document.score,
+        })
+    return passages

src/infodesk/live_service.py

@@ -229,7 +229,7 @@ def _run(self, job_id, kind, request):
             if kind == "ask":
                 started = time.perf_counter()
                 result["passages"] = self.store.retrieve(request.question, request.source_ids, request.document_ids)
-                self.store.job_step(job_id, f"Retrieved {len(result['passages'])} passages with SQLite FTS5 / BM25",
+                self.store.job_step(job_id, f"Retrieved {len(result['passages'])} passages with Haystack BM25",
                                     (time.perf_counter()-started)*1000)
                 if not result["passages"]:
                     result["answer"] = {"statements": [], "insufficient_evidence": True,