HUB-043: make the corpus queryable as one knowledge base

backlog.md

@@ -1191,7 +1191,11 @@ were retrieved, the budget, the reserve, and that raising or omitting
 
 ### HUB-042 — Search engines exhaust under sustained job volume
 
-**Status:** 🟡 MITIGATED 2026-08-13 by widening the engine pool and surfacing
+**Status:** ✅ DONE 2026-08-13 — ADR-002 stages 1 and 2 deployed and verified.
+Pacing and pre-crawl ranking cut wasted crawls (36 fetched for 4 kept became
+17 for 17 on the worst topic), and a keyed Serper fallback now covers a fully
+blocked engine pool, proven by stopping SearXNG and watching a job complete on
+`{"serper": 4}`. Previously: 🟡 MITIGATED 2026-08-13 by widening the engine pool and surfacing
 suspension. Not closed: the underlying quota problem is unsolved, and one
 follow-up is now open (see the calibration note below).
 
@@ -1257,6 +1261,148 @@ SearXNG. Brave Search API is rejected pending an operator licensing decision —
 its terms restrict storing API results, and this system exists to build a
 persistent corpus.
 
+### HUB-043 — Retrieval is job-scoped; the corpus cannot be queried as one base
+
+**Status:** ✅ DONE — implemented and verified 2026-08-13. See
+`docs/CURRENT_STATE.md`, "One retrieval path".
+
+`ScopedRetrievalService.retrieve(job_id, topic)` took a job id as its first
+argument, so hybrid dense+BM25 fusion, per-source caps and the FTS5 needle
+channel ran **only** during one job's report synthesis. `/query` and `/rag`
+used a separate, simpler, dense-only path.
+
+The system therefore had two retrieval implementations and the better one
+could not see the corpus: 679 documents sat physically in one index and
+logically in 62 silos. The exact-term channel that HUB-017 measured lifting
+hit@4 from `0.6923` to `1.0` — the one that recovered a DOI no reranking
+could reach — was unavailable to every corpus-wide query.
+
+This was the gap between the system and the stated goal of a searchable,
+cross-referenced knowledge base, and a prerequisite for HUB-044 through
+HUB-046.
+
+**Approach.** Make `job_id` an optional filter rather than a required
+argument, unscope the lexical channel, and route `/query` and `/rag` through
+the same service. This *deletes* a duplicate implementation rather than
+adding one. Note it deliberately crosses the PRD boundary that held `/query`
+and `/rag` unchanged — that boundary protected a regression surface which is
+now the thing to fix.
+
+**Acceptance:** one retrieval path; corpus-wide queries use dense+BM25+RRF;
+job-scoped report synthesis is byte-identical to today on a retried report.
+
+**Outcome.** All three met, with the third criterion corrected: "byte-identical
+synthesis" is not checkable because synthesis is LLM-driven. The deterministic
+thing underneath it was checked instead — retrieval fingerprinted over the
+ordered `(document_id, chunk_index, score, channels, rrf_score)` list for the
+six largest jobs (360 chunks) under the deployed image and the new one, all
+six digests identical.
+
+Two things the analysis had not anticipated:
+
+- **The filters had to move, not just widen.** `topic_filter`/`tags_filter`
+  were Qdrant payload conditions. Left there, they would have narrowed the
+  dense channel while the lexical channel searched the whole corpus. They now
+  resolve to a document scope from `job_sources`, which is also the more
+  correct source: a page found by two jobs on different topics belongs to
+  both, and only `job_sources` records the second.
+- **Unscoping made row width matter.** `documents_for_job` read whole rows;
+  corpus-wide that meant decoding 44 MB of markdown per query to obtain three
+  identity columns. Both accessors are now projected.
+
+Live: "how does reciprocal rank fusion combine rankings" now draws 64 chunks
+from 33 sources across jobs; "kubernetes observability tracing" 65 from 42.
+Unblocks HUB-044 through HUB-046.
+
+### HUB-044 — No retrieval-breadth metric; evidence concentrates on few sources
+
+**Status:** 🔴 OPEN — measured, unmeasured.
+
+A synthesis run selected 15 chunks drawn from only **7 of 22 available
+sources**. Breadth is bought expensively during acquisition and then partly
+discarded at retrieval, and nothing tracks it.
+
+Graph-Aware Late Chunking (arXiv 2603.22633) argues ranking metrics
+systematically undervalue breadth: content-similarity methods scored the
+highest MRR while always retrieving from a single document section, and
+structure-aware methods reached up to 15.6x more sections. It proposes
+coverage metrics (SecCov@k, CS Recall) alongside MRR/Recall@k.
+
+**Approach.** Track source coverage at k next to recall in the retrieval
+benchmark. Without this number, changes to chunking or retrieval cannot be
+judged — which is why it precedes HUB-045.
+
+### HUB-045 — Chunk embeddings lose their document context
+
+**Status:** 🔴 OPEN — deferred behind HUB-044, which would prove whether
+chunking is actually the bottleneck.
+
+Chunking is a fixed 800/100 recursive split applied identically to API
+reference tables, Q&A pages, marketing copy and academic PDFs, and each chunk
+is embedded in isolation so it loses the context of its surrounding passage.
+
+Late Chunking (arXiv 2409.04701) embeds the whole document once with a
+long-context model and pools per chunk afterwards, reporting better retrieval
+without retraining and without changing chunk boundaries — a low-risk swap at
+the embedding step. **Precondition:** the embedding model's context window
+must cover typical documents; that is the load-bearing assumption to verify
+first.
+
+Adaptive Chunking (arXiv 2603.25333) is the more thorough alternative —
+per-document metrics selecting among chunkers, correctness 62–64% → 72% — but
+adds five metrics and multiple splitters, which is disproportionate until
+HUB-044 shows chunking is the limit.
+
+### HUB-046 — Cross-document linking: bridge entities, not shared vocabulary
+
+**Status:** 🔴 OPEN — supersedes the mechanism half of HUB-040.
+
+Nothing connects a document to any other: no entity linking, no cross-job
+deduplication, no path for "what do we know about X across everything". The
+one cross-document mechanism that exists pairs evidence spans by **shared
+vocabulary**, which optimises for combinability rather than relatedness and
+produced zero genuine conflicts across 450 sampled pairs.
+
+Entity-centered Cross-document RE (arXiv 2210.16541) connects documents
+through *bridge entities* that co-occur with both targets, filtering out the
+noisy surrounding text that lexical overlap admits. Sequential Cross-Document
+Coreference (arXiv 2104.08413) supplies the cost shape: incremental
+mention-to-cluster scoring rather than exhaustive pairwise comparison, linear
+rather than quadratic.
+
+**Take the ideas, not the models** — both assume labelled supervision this
+corpus does not have. Entity keys are also what deduplication and cross-job
+aggregation both need, so this is the cheapest first step toward either.
+
+### HUB-047 — No end-to-end retrieval evaluation set
+
+**Status:** 🔴 OPEN — the gap the backlog never named.
+
+There are benchmarks for exact-term recall, claim support and source
+screening, but nothing measuring whether the knowledge base answers real
+questions well. Every retrieval decision to date has been judged on a proxy.
+
+**Approach.** A held-out question set with known-good source documents,
+scored on answer correctness and source coverage. HUB-044 is its first
+metric. This is what would let HUB-045 and HUB-048 be decided by measurement
+rather than argument.
+
+### HUB-048 — Knowledge-graph go/no-go, decided by measurement
+
+**Status:** 🔴 OPEN — replaces the open-ended evaluation in HUB-027.
+
+RAG vs. GraphRAG (arXiv 2502.11371) benchmarks both under a unified protocol
+and finds GraphRAG's advantage is task- and dataset-dependent rather than
+universal, with graph construction adding nontrivial LLM preprocessing cost —
+material on one workstation with a 9B local model.
+
+**Approach.** Run that protocol on this corpus: existing hybrid retrieval
+against a minimal graph prototype, on representative aggregation queries, and
+let the measured delta decide. Do not build the graph first.
+
+**Blocked by HUB-043 and HUB-047**: comparing against a corpus-wide baseline
+requires one to exist, and deciding by measurement requires an evaluation set.
+
 ### HUB-025 — Add scheduled research jobs
 
 Add recurring jobs only after the durable worker, idempotency, and notification paths are complete.
@@ -1336,6 +1482,12 @@ document store unless the adopted use case requires shared transactional access.
 
 ### HUB-027 — Evaluate a knowledge graph layer
 
+**Superseded 2026-08-13 by HUB-048**, which replaces the open-ended
+evaluation with a measured protocol. Note the stated precondition is **not
+met**: hybrid retrieval exists for jobs but not corpus-wide (HUB-043), so
+"hybrid retrieval first" has not actually happened for the queries a graph
+would serve.
+
 Do not add a graph until hybrid retrieval, reranking, citation validation, and a retrieval evaluation set are in place.
 
 **Revisit trigger:** measured failures are primarily relationship/multi-hop failures that hybrid retrieval cannot solve.
@@ -1388,6 +1540,10 @@ HUB-017 ✅, HUB-018 ✅, HUB-019 ✅, HUB-020 ✅, HUB-021 ✅, HUB-022 ✅, HU
 
 ### Milestone 5 — Expansion only when earned
 
+HUB-043 🔴 (corpus-wide retrieval — the highest-value open item), HUB-044 🔴
+(retrieval-breadth metric), HUB-045 🔴 (late chunking), HUB-046 🔴
+(bridge-entity cross-document linking), HUB-047 🔴 (end-to-end retrieval
+evaluation set), HUB-048 🔴 (knowledge-graph go/no-go by measurement).
 HUB-024 ✅ (2026-08-13 — deployed and measured: 25 sources / 23 domains
 against the 6/6 single-query baseline, zero off-topic acquisition; design and
 citations in `PRDs/hub-024-query-planning.md`). HUB-025 through HUB-030 remain

docs/ADR-002-search-provider-strategy.md

@@ -1,7 +1,7 @@
 # ADR-002 — Search provider strategy for research acquisition
 
 Date: 2026-08-13
-Status: Stage 1 Accepted and deployed 2026-08-13; stage 2 deferred pending an operator spend decision
+Status: Accepted — stages 1 and 2 deployed and verified 2026-08-13
 Item: HUB-042 (`backlog.md`)
 
 ## Context
@@ -156,6 +156,32 @@ survived the screen. The retained domains are `tokio.rs`, `docs.rs`,
 The post-crawl screen is not redundant — it remains the guard for documents
 whose snippet flatters them — but it now has little left to remove.
 
+## Stage 2 outcome (2026-08-13)
+
+Serper deployed as a fallback behind SearXNG, keyed from `SERPER_API_KEY` and
+inert when unset.
+
+**Verified by taking SearXNG away.** With the container stopped, a job
+recorded `search_providers: {"serper": 4}` — all four facet queries served by
+the fallback — retained 17 sources and completed its report, drawing on
+`kubernetes.io`, `docs.cloud.google.com`, `datadoghq.com` and `signoz.io`.
+With SearXNG restored the next job recorded `{"searxng": 3}`, so primacy
+returns automatically and the paid path stays insurance rather than default.
+The key appears in no log line.
+
+**The live contract exposed a defect no mock could.** Serper returns dates as
+`"Oct 31, 2019"`, which neither ISO nor RFC-2822 parsing accepts, so every
+Serper result parsed as undated. Any job setting `freshness_days` would have
+rejected all of them as `stale_or_undated` — silently, since the results
+would simply be absent. `parse_source_date` now accepts human-readable
+formats after the existing two, leaving previously-parsing shapes untouched.
+
+Two earlier attempts to force the fallback failed instructively and are worth
+recording: pointing `SEARXNG_ENGINES` at a nonexistent engine did nothing
+because SearXNG falls back to its category defaults, and overriding
+`SEARXNG_URL` in `.env` did nothing because `docker-compose.yml` hardcodes it.
+Stopping the container is the honest simulation.
+
 ## Acceptance
 
 - A run of comparable volume to 2026-08-13 (roughly 30 jobs, 150+ queries)

docs/CURRENT_STATE.md

@@ -2,6 +2,61 @@
 
 Last verified: 2026-08-13 on the local Windows 11 workstation.
 
+## One retrieval path: the corpus is queryable as one base (HUB-043, 2026-08-13)
+
+Hybrid dense+BM25+RRF retrieval used to run **only** inside a research job's
+report synthesis. `/query` and `/rag` used a second, dense-only
+implementation, so 679 documents collected across 62 jobs sat physically in
+one index and logically in 62 silos, and the FTS5 needle channel that lifted
+exact-term hit@4 from `0.6923` to `1.0` was unreachable from any corpus-wide
+query.
+
+`ScopedRetrievalService.retrieve` now takes `job_id=None` to mean the whole
+corpus. The job id is a filter, not a mode: the same fusion, per-source caps
+and needle channel run either way. The dense-only path in `app/query.py` is
+deleted, not deprecated.
+
+- **`/query`'s `topic_filter` and `tags_filter` survive, relocated.** They
+  were Qdrant payload conditions; they are now resolved to a document scope
+  from `job_sources` (`documents_matching`). The lexical channel has no
+  payload to filter on, so a payload filter would have narrowed one channel
+  and not the other. Verified against live metadata: "Kubernetes pod
+  autoscaling and resource management" → 122 documents, tag `llm` → 9, an
+  unmatched topic → 0 (not the whole corpus).
+- **Job-scoped retrieval is provably unchanged.** Retrieval was fingerprinted
+  over the ordered `(document_id, chunk_index, score, channels, rrf_score)`
+  list for the six largest jobs — 360 selected chunks — under the deployed
+  image and the new one. All six SHA-256 digests match exactly. (The earlier
+  acceptance wording, "report synthesis is byte-identical", was not
+  checkable: synthesis is LLM-driven. This is the deterministic thing
+  underneath it.)
+- **Corpus scope sends no identity filter.** Enumerating all 679 ids would be
+  equivalent but grows an unbounded query with the corpus. An *empty* list
+  remains an error in both `search_evidence` and `search_chunks` — that means
+  a caller expected a scope and lost it, and silently widening to the corpus
+  is exactly the failure the guard exists to catch.
+- **Retrieval rows are projected, not whole.** The corpus holds 44 MB of
+  markdown; `SELECT *` would have decoded all of it on every query.
+  `documents_for_job` and `all_documents` now select identity and title only.
+- **Scores report the ranking that produced them.** Under fusion `/query`
+  returns the RRF score rather than a cosine — a similarity that contradicts
+  the ordering would be worse. `hub_retrieval_score` still observes only
+  candidates with a real cosine, and query latency moved off
+  `hub_embedding_duration_seconds` (which timed the whole retrieval under a
+  name meaning one part of it) onto `hub_retrieval_duration_seconds`.
+
+Live proof against the real corpus, new image on the deployed Qdrant and
+Ollama with a copy of the document store: "how does reciprocal rank fusion
+combine rankings" selected 64 chunks from **33 sources**, "kubernetes
+observability tracing" 65 from **42 sources**, both channels contributing and
+fusing. Under the old code these queries could reach one job's sources at
+most.
+
+Prerequisite fact confirmed before deploying: all 68,072 Qdrant points carry
+a `document_id` and every one resolves in SQLite, so corpus-wide retrieval
+drops nothing. (33 retained documents have no Qdrant chunks — deduplicated
+sources — and remain lexically reachable.)
+
 ## Live report generated through the deployed judge gate (2026-08-13)
 
 The judge pivot had been proven structurally but no report had ever been
@@ -251,6 +306,32 @@ per report.
 Findings across the ten retries ranged 3–40, tracking corpus quality rather
 than corpus size.
 
+## Keyed search fallback deployed (ADR-002 stage 2, 2026-08-13)
+
+Serper runs behind SearXNG, engaging only when a query returns nothing — what
+a fully blocked engine pool looks like. Keyed from `SERPER_API_KEY` and inert
+when unset, so acquisition is unchanged without it. It returns URLs and
+snippets only; this stack does its own crawling, so providers bundling page
+content would be paid for output it discards.
+
+**Verified by removing SearXNG.** With the container stopped a job recorded
+`search_providers: {"serper": 4}`, retained 17 sources and completed its
+report on `kubernetes.io`, `docs.cloud.google.com` and `datadoghq.com`. With
+SearXNG restored the next job recorded `{"searxng": 3}` — primacy returns
+automatically. The key reaches no log line and is excluded from the config
+repr.
+
+Failure is legible rather than silent: a quota error or transport failure
+returns no results and is logged with its cause, unlike the CAPTCHAs this
+exists to survive. Fallback URLs pass through the unchanged source policy and
+SSRF vetting.
+
+**The live contract exposed a defect mocks could not.** Serper returns dates
+as `"Oct 31, 2019"`; neither ISO nor RFC-2822 parsing accepts that, so every
+result parsed as undated and any job setting `freshness_days` would have
+silently rejected all of them. `parse_source_date` now handles human-readable
+formats.
+
 ## Search stage 1: pacing and pre-crawl ranking (ADR-002, 2026-08-13)
 
 Two free changes, both previously untried, decided in
@@ -984,6 +1065,8 @@ SQLite FTS5 needle channel under deterministic reciprocal rank fusion (`k=60`):
 - `REPORT_HYBRID_RETRIEVAL` (default true) disables the channel; with it off, or when
   no needle term matches, candidate ordering is byte-identical to dense-only.
   `/query` and `/rag` remain dense-only per the PRD regression boundaries.
+  *(Superseded 2026-08-13 by HUB-043: that boundary protected the regression
+  surface that had become the defect. Both endpoints now use this channel.)*
 
 Phase 6 (local reranking) is not entered: hybrid recall on the measured manifest is
 `1.0`, so precision is not the bottleneck.

research-hub/app/clients.py

@@ -324,25 +324,30 @@ def search_evidence(
         canonical_urls: list[str] | None = None,
         document_ids: list[str] | None = None,
     ) -> list[dict]:
-        """Return full internal candidates restricted to retained source identity."""
+        """Return full internal candidates, optionally scoped to source identity.
+
+        ``None`` means the whole collection (HUB-043): a corpus-wide query is
+        scoped by the corpus itself, and enumerating every id as a filter
+        would be equivalent but grows without bound. An *empty* list stays an
+        error -- it means a caller expected a scope and lost it, and silently
+        widening to the corpus is the failure this guard exists to catch.
+        """
         from qdrant_client.models import FieldCondition, Filter, MatchAny
 
         conditions = []
-        if canonical_urls:
-            conditions.append(FieldCondition(
-                key="canonical_url", match=MatchAny(any=canonical_urls)
-            ))
-        if document_ids:
-            conditions.append(FieldCondition(
-                key="document_id", match=MatchAny(any=document_ids)
-            ))
-        if not conditions:
-            raise ValueError("evidence search requires retained source scope")
+        for key, values in (
+            ("canonical_url", canonical_urls), ("document_id", document_ids),
+        ):
+            if values is None:
+                continue
+            if not values:
+                raise ValueError("evidence search requires retained source scope")
+            conditions.append(FieldCondition(key=key, match=MatchAny(any=values)))
         hits = self._client.search(
             collection_name=self.collection,
             query_vector=vector,
             limit=limit,
-            query_filter=Filter(must=conditions),
+            query_filter=Filter(must=conditions) if conditions else None,
         )
         excluded = {
             "text", "source_url", "source_title", "canonical_url",