HUB-043: make the corpus queryable as one knowledge base - #12
Merged
Conversation
The live Serper contract returns dates as 'Oct 31, 2019', which neither ISO nor RFC-2822 parsing accepts. Every Serper result therefore parsed as undated, and any job setting freshness_days would have rejected all of them as stale_or_undated -- a defect the mocked tests could not expose, found on the first real API call. Adds explicit human-readable formats after the existing two, so shapes that already parsed are untouched. 340 tests green in-container. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With SearXNG stopped, a job served all four facet queries from Serper, retained 17 sources and completed. With SearXNG restored the next job went back to searxng, so primacy returns automatically and the paid path stays insurance. The key reaches no log line. Records the two failed attempts to force the fallback, both instructive: SEARXNG_ENGINES pointed at a nonexistent engine does nothing because SearXNG uses category defaults, and overriding SEARXNG_URL in .env does nothing because docker-compose.yml hardcodes it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six items, each grounded in a fetched arXiv abstract and in measurements from this session. HUB-043 is the highest-value: ScopedRetrievalService.retrieve takes job_id as its first argument, so hybrid dense+BM25+RRF and the FTS5 needle channel run only inside one job's report synthesis while /query and /rag use a separate dense-only path. Two retrieval implementations, and the better one cannot see the corpus -- 530 documents physically in one index and logically in 49 silos. Also corrects HUB-027: its stated precondition 'hybrid retrieval first' is not actually met, because hybrid retrieval exists for jobs and not for the corpus-wide queries a graph layer would serve. HUB-048 replaces its open-ended evaluation with the RAG-vs-GraphRAG measured protocol. HUB-047 names a gap the backlog never had: benchmarks exist for exact-term recall, claim support and source screening, but nothing measures whether the knowledge base answers real questions well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-043) Hybrid dense+BM25+RRF retrieval ran only inside a research job's report synthesis. /query and /rag used a second, dense-only implementation, so 679 documents collected across 62 jobs sat physically in one index and logically in 62 silos, and the FTS5 needle channel that lifted exact-term hit@4 from 0.6923 to 1.0 was unreachable from any corpus-wide query. ScopedRetrievalService.retrieve now takes job_id=None to mean the whole corpus. The job id is a filter, not a mode: the same fusion, per-source caps and needle channel run either way. The dense-only path is deleted, not deprecated. /query's topic_filter and tags_filter survive but relocate. They were Qdrant payload conditions; left there they would have narrowed the dense channel while the lexical channel searched the whole corpus. They now resolve to a document scope from job_sources, which is also the more correct source: a page found by two jobs on different topics belongs to both, and only job_sources records the second. Unscoping made row width matter: documents_for_job read whole rows, which corpus-wide meant decoding 44MB of markdown per query to obtain three identity columns. Both accessors are now projected. Corpus scope sends no identity filter, since enumerating every id would be equivalent but grows without bound. An empty list stays an error in both search_evidence and search_chunks -- that means a caller expected a scope and lost it, and silently widening is the failure the guard exists to catch. Under fusion /query reports the RRF score rather than a cosine: a similarity that contradicts the ordering would be worse. hub_retrieval_score still observes only candidates with a real cosine, and query latency moves off hub_embedding_duration_seconds -- which timed the whole retrieval under a name meaning one part of it -- onto hub_retrieval_duration_seconds. Verified: job-scoped retrieval fingerprinted over the ordered (document_id, chunk_index, score, channels, rrf_score) list for the six largest jobs (360 chunks) under the deployed image and the new one -- all six digests identical. Live against the real corpus, "how does reciprocal rank fusion combine rankings" draws 64 chunks from 33 sources across jobs. All 68,072 Qdrant points carry a document_id resolving in SQLite, so nothing is dropped. 357 tests pass in a throwaway container. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hybrid dense+BM25+RRF retrieval ran only inside a research job's report synthesis.
/queryand/ragused a second, dense-only implementation, so 679 documents collected across 62 jobs sat physically in one index and logically in 62 silos — and the FTS5 needle channel that lifted exact-term hit@4 from0.6923to1.0was unreachable from any corpus-wide query.ScopedRetrievalService.retrievenow takesjob_id=Noneto mean the whole corpus. The job id is a filter, not a mode. The dense-only path is deleted, not deprecated.Two things the analysis had not anticipated
topic_filter/tags_filterwere Qdrant payload conditions. Left there, they would have narrowed the dense channel while the lexical channel searched the whole corpus. They now resolve to a document scope fromjob_sources, which is also the more correct source: a page found by two jobs on different topics belongs to both, and onlyjob_sourcesrecords the second.documents_for_jobread whole rows; corpus-wide that meant decoding 44 MB of markdown per query to obtain three identity columns. Both accessors are now projected.Verification
(document_id, chunk_index, score, channels, rrf_score)list for the six largest jobs — 360 selected chunks — under the deployed image and the new one. All six SHA-256 digests identical. (The original acceptance wording, "report synthesis is byte-identical", was not checkable: synthesis is LLM-driven. This is the deterministic thing underneath it.)document_idand every one resolves in SQLite, so corpus-wide retrieval drops nothing.llm→ 9, unmatched topic → 0 (not the whole corpus).Unblocks HUB-044 through HUB-046.
🤖 Generated with Claude Code