From 1fc1218eec4fb03c710ac9cd0d5bf4dad86544c7 Mon Sep 17 00:00:00 2001 From: funsaized Date: Thu, 13 Aug 2026 03:50:49 -0400 Subject: [PATCH] docs(backlog): open HUB-024 with a prior-art-grounded planning methodology MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The revisit trigger tripped: a job issues exactly one SearXNG query, so a report is built from whatever that phrasing surfaced. The narrowness is on the acquisition side (hybrid retrieval already measures hit@4 1.0), and it caps HUB-034 cross-source pair drafting, which can only find disagreements between sources that were actually crawled. Design (PRDs/hub-024-query-planning.md), grounded in 16 arXiv abstracts fetched and read 2026-08-13: - Breadth is emergent, not a fixed count: candidate facets are admitted by marginal distinctness (embedding cosine below a threshold against the admitted set), so a narrow topic admits one facet and behaves exactly as today while a broad topic admits as many angles as it has. This lifts ScoreGate's threshold-not-top-K cardinality principle from chunk selection to query planning; Adaptive-RAG-style complexity routing falls out of single-facet collapse rather than needing a trained classifier. - Rounds are gap-driven (KiRAG): each round names what is still uncovered from a per-facet coverage summary, and only those gaps become queries. - Stopping is saturation-first (KAIR, instanced on canonical-URL identity), coverage-second, with hard budget caps as backstop only (AdaRankLLM argues learned necessity over fixed iteration budgets). - Traps avoided, each from a paper's own stated limitation: fixed depth x breadth (Static-DRA), expecting breadth to fix report quality (DeepWeb-Bench: retrieval is 12-14% of errors, derivation/calibration exceed 70%), redundant tool calls (HotelQuEST). Acceptance measures corpus breadth (distinct domains, represented sources) against the single-query baseline — not report quality, which stays the judge gate's job. Ships behind REPORT_QUERY_PLANNING=false; judge calls per report stay bounded by existing drafting caps. Implementation not started. Also refreshes the staged next-session prompt, which said P3 was fully closed. Co-Authored-By: Claude Fable 5 --- PRDs/hub-024-query-planning.md | 199 ++++++++++++++++++++++++++++++ PRDs/prompt-backlog-post-pivot.md | 20 ++- backlog.md | 62 ++++++++-- 3 files changed, 268 insertions(+), 13 deletions(-) create mode 100644 PRDs/hub-024-query-planning.md diff --git a/PRDs/hub-024-query-planning.md b/PRDs/hub-024-query-planning.md new file mode 100644 index 0000000..5eb04f1 --- /dev/null +++ b/PRDs/hub-024-query-planning.md @@ -0,0 +1,199 @@ +# HUB-024 — Adaptive query planning and iterative research + +Status: design, operator-opened 2026-08-13 (revisit trigger tripped — reports +are built from whatever a single search phrasing surfaced). +Prior art reviewed: 16 arXiv abstracts, fetched and read 2026-08-13 (citations +at the end; every claim below is attributed to a fetched abstract or marked as +this design's own choice). + +## Problem + +A research job issues exactly ONE SearXNG query. Everything downstream — +retrieval, span selection, cross-document pair drafting, the judge gate — can +only work with the documents that one phrasing happened to surface. The +narrowness is on the **acquisition** side, not the retrieval side: hybrid +dense+FTS5 retrieval already measures hit@4 `1.0` on the exact-term manifest +(HUB-017), so the corpus is searched well; it is *assembled* narrowly. + +This also caps the cross-source machinery deployed in HUB-034: pair drafting +can only find disagreements between sources that were actually crawled, and a +single query tends to return sources that agree with each other. + +## The methodology, and why breadth is not a constant + +The operator's constraint — *no arbitrary fixed sub-query count* — matches the +prior art's own finding. Static-DRA (2512.03887) makes Depth and Breadth +user-tunable parameters and its own limitation is exactly that: fixed +parameters cannot respond to what the evidence turns out to need. The modern +alternative is **cardinality by threshold, not by count** — ScoreGate +(2606.14269) replaces fixed top-K chunk selection with a score-fusion cutoff, +and reports better efficiency at equal quality. This design lifts that +principle from chunk selection up to query planning. + +### 1. Breadth is emergent (marginal-distinctness admission) + +One bounded local-LLM call proposes candidate facet queries for the topic — +distinct *information needs*, not paraphrases. Each candidate is embedded with +the already-deployed `nomic-embed-text` and admitted only if its maximum +cosine similarity to the already-admitted set is below `PLAN_FACET_DISTINCT` +(design default 0.85). Admission stops when the next candidate adds no +distinct retrieval intent. + +Consequences, which are the point: a narrow factual topic admits one facet and +the job behaves exactly as it does today (no planning overhead, no extra +searches); a broad topic admits as many facets as it genuinely has angles. The +cap (`PLAN_MAX_FACETS`) exists as a safety rail against a pathological +planner, never as the mechanism that decides breadth. + +Complexity routing supports this: Adaptive-RAG (2403.14403) selects among +no-retrieval / single-step / multi-step per query complexity and reports +efficiency and accuracy gains over uniform pipelines. Here the collapse of all +candidates into one facet *is* the complexity signal — no separate classifier +to train, and no cost paid on simple topics. + +### 2. Rounds are gap-driven, not depth-numbered + +After each round's ingestion, one bounded call reads a **coverage summary** +(per facet: retained document count, distinct domains) and names the facets +still uncovered and the sub-questions still unanswered. Only those become the +next round's queries. This is KiRAG's mechanism (2502.18397): reason over what +was retrieved to identify knowledge gaps explicitly, and let each gap drive +the next retrieval. REPAIR (2601.04618) makes the same move with reasoning +plans as feedback signals into adaptive retrieval, and Tree of Reviews +(2404.14464) structures it as expand / reject / accept decisions per node with +pruning — this design keeps the expand/reject decision but stays flat (a +per-facet round list), because a tree buys depth control we do not need and +costs an orchestration layer we would have to test. + +### 3. Stopping: saturation first, coverage second, budget last + +- **Novelty saturation (primary).** Stop when a round's yield of *new + canonical URLs* falls below `PLAN_NOVELTY_MIN` (design default: fewer than + 20% of that round's retained documents are new, or fewer than 2 new + documents). KAIR (2601.16462) progressively updates anchoring indices across + iterative retrieval rounds; saturation in what a round newly anchors is the + natural stop signal, and canonical-URL identity is this system's cheapest + and most reliable instance of it (canonical URLs and content hashes already + form stable document IDs). +- **Coverage (secondary).** Stop when every admitted facet has at least one + retained document and the gap pass names nothing further. +- **Budget rails (backstop).** Hard caps on rounds, searches, crawls, and + wall-clock, per job and configurable. AdaRankLLM (2604.15621) argues for + learned necessity signals over fixed iteration budgets; this design follows + that for the *decision* while keeping hard caps as a safety net, because an + unbounded crawl loop on a workstation is an availability problem, not a + quality one. + +### 4. What the prior art says NOT to do + +- **Do not expect breadth to fix report quality.** DeepWeb-Bench (2605.21482) + evaluated nine frontier models and found retrieval failures account for only + 12–14% of errors while derivation and calibration failures exceed 70%. + Acceptance for this item is therefore measured as *corpus breadth* + (distinct domains, represented sources), not as report quality; the judge + gate remains the quality guard. +- **Do not let rounds multiply tool calls carelessly.** HotelQuEST + (2602.23949) found LLM agents beat traditional retrievers on accuracy but at + substantially higher cost from redundant tool calls and routing that fails + to match query complexity to capability. Canonical-URL dedup *across* facets + before crawling is mandatory, not an optimization. +- **Do not adopt fixed depth × breadth.** (Static-DRA's own stated limitation.) + +## Scope + +In scope: research-job acquisition only (`app/research.py` planning + the +worker's search/crawl loop), job progress provenance, config surface, tests +and a deterministic breadth benchmark. + +Out of scope: `/query` and `/rag` (PRD regression boundaries, unchanged); +retrieval ranking (HUB-017 measured `1.0` on its manifest); the claim gate and +its v4 seal (untouched — this changes what gets crawled, not how claims are +judged); any change to judge-call volume per report. + +## Config surface (all default OFF/inert until measured) + +| Variable | Default | Meaning | +|---|---|---| +| `REPORT_QUERY_PLANNING` | `false` | Master switch; off = today's single-query behavior, byte-identical | +| `PLAN_FACET_DISTINCT` | `0.85` | Max cosine similarity for admitting a new facet | +| `PLAN_MAX_FACETS` | `8` | Safety rail, not the breadth mechanism | +| `PLAN_MAX_ROUNDS` | `3` | Budget backstop | +| `PLAN_NOVELTY_MIN` | `0.2` | New-canonical-URL fraction below which rounds stop | +| `PLAN_SEARCH_BUDGET` / `PLAN_CRAWL_BUDGET` | `12` / `40` | Per-job hard caps | + +## Acceptance criteria + +- On a fixed topic set, planned jobs retain **more distinct domains and more + represented sources** per report than the single-query baseline on the same + topics; the comparison is recorded, not asserted. +- A topic whose candidates collapse to one facet issues exactly one search: + no planning overhead on simple topics, and the resulting job is equivalent + to today's path. +- Every sub-query inherits the full HUB-020/021 source policy (allowed/blocked + domains, per-domain limit, freshness) and SSRF vetting (HUB-006); canonical + URLs are deduplicated across facets and rounds before crawling, so no + document is fetched twice. +- Judge calls per report stay bounded by the existing drafting caps — + breadth must not increase metered cost. +- Worker semantics unchanged: leases, heartbeats, timeouts, bounded retries, + idempotent re-ingestion. +- The plan is auditable: admitted facets, per-round queries, new-document + yield, and the stop reason (`saturation` / `coverage` / `budget`) are + recorded in job progress. +- `REPORT_QUERY_PLANNING=false` reproduces current behavior exactly. + +## Load-bearing risk + +The planner is a generative component sitting upstream of acquisition: a bad +plan quietly narrows or skews the corpus rather than failing loudly. Mitigation +— the single-facet collapse path is the identity function on today's behavior, +every rail is a hard cap, and the stop reason is recorded per job so a +degenerate plan is visible in provenance rather than inferred from a thin +report. + +## Open thread (design-review checkpoint before shipping) + +None of the reviewed abstracts establishes how a saturation threshold +transfers across topic domains — whether `PLAN_NOVELTY_MIN = 0.2` means the +same thing for a niche clinical question as for a broad engineering topic. +Treat the first measurement as calibration of that threshold, not as +validation of the design. + +## Citations (fetched from arXiv 2026-08-13) + +Primary mechanism: +- 2502.18397 — KiRAG: Knowledge-Driven Iterative Retriever for Enhancing + Retrieval-Augmented Generation (https://arxiv.org/abs/2502.18397) — + gap identification drives the next retrieval. +- 2606.14269 — ScoreGate: Adaptive Chunk Selection for RAG via Dual-Score + Statistical Fusion (https://arxiv.org/abs/2606.14269) — adaptive cardinality + by threshold instead of fixed top-K; lifted here to planning. +- 2601.16462 — Finding What Matters: Anchoring Context Knowledge with Evolving + Indices for Iterative Retrieval (https://arxiv.org/abs/2601.16462) — + round-over-round saturation as a stop signal. + +Supporting evidence: +- 2403.14403 — Adaptive-RAG (https://arxiv.org/abs/2403.14403) — route by + complexity instead of a uniform pipeline. +- 2604.15621 — AdaRankLLM (https://arxiv.org/abs/2604.15621) — learned + necessity signals over fixed iteration budgets. +- 2601.04618 — REPAIR (https://arxiv.org/abs/2601.04618) — reasoning plans as + feedback into adaptive retrieval. +- 2404.14464 — Tree of Reviews (https://arxiv.org/abs/2404.14464) — per-node + expand / reject / accept with pruning. + +Failure modes to avoid: +- 2605.21482 — DeepWeb-Bench (https://arxiv.org/abs/2605.21482) — retrieval is + 12–14% of errors; derivation/calibration exceed 70%. +- 2602.23949 — HotelQuEST (https://arxiv.org/abs/2602.23949) — redundant tool + calls and complexity-mismatched routing dominate cost. +- 2512.03887 — Static-DRA (https://arxiv.org/abs/2512.03887) — fixed Depth and + Breadth parameters cannot adapt to what the evidence needs. + +Read but not adopted (recorded so a later pass need not re-read them): +2410.20286 (Quam — recall via document-similarity graph, single-query scope), +2607.15283 (biomedical question-type routing, predefined categories), +2511.03214 (LGM — concept meta-relations for ambiguous terms), +2412.12559 (EXIT — post-retrieval extractive compression), +2601.06551 (L-RAG — entropy gating to skip retrieval), +2506.21506 (Mind2Web 2 — agent-as-judge evaluation of agentic search). diff --git a/PRDs/prompt-backlog-post-pivot.md b/PRDs/prompt-backlog-post-pivot.md index d452ea9..1ec7978 100644 --- a/PRDs/prompt-backlog-post-pivot.md +++ b/PRDs/prompt-backlog-post-pivot.md @@ -47,13 +47,21 @@ main; a new set means new operator-annotated cases, never reuse of the consumed v4 set). The gate is not trusted again until the new final passes. MiniMax reports no finer version granularity; that string is the trigger. -## 3. P3 discipline +## 3. HUB-024 — adaptive query planning (opened 2026-08-13, not implemented) -HUB-024 through HUB-030 stay closed behind their revisit triggers (see -backlog): open one ONLY if its trigger has actually tripped (e.g. repeated -user need for multi-angle synthesis, sustained scheduled-research demand). -If nothing tripped and the live proof is recorded, end the pass cleanly — -do not manufacture work. +The revisit trigger tripped and the operator opened it; the design is +`PRDs/hub-024-query-planning.md` (grounded in fetched arXiv prior art). +Implement only after the live proof above, because planning changes what gets +crawled and a broken plan is easiest to see against a known-good baseline +report. Build order: marginal-distinctness facet admission (breadth emerges +from a similarity threshold — never a fixed count), canonical-URL dedup across +facets, per-job budget rails, then gap-driven rounds with saturation stopping. +Ship behind `REPORT_QUERY_PLANNING=false` and measure distinct domains and +represented sources against the single-query baseline before enabling it. + +HUB-025 through HUB-030 stay closed behind their revisit triggers; open one +ONLY if its trigger actually tripped. If nothing tripped and the live proof is +recorded, end the pass cleanly — do not manufacture work. ## Standing constraints diff --git a/backlog.md b/backlog.md index e5d4a5a..ca810c7 100644 --- a/backlog.md +++ b/backlog.md @@ -846,11 +846,55 @@ the deployed image. ## P3 — Optional expansion after sustained usage -### HUB-024 — Add query planning and iterative research - -Add query decomposition, follow-up searches based on evidence gaps, stopping criteria, and budget controls. This is the work that would justify the “deep research” label. - -**Revisit trigger:** users repeatedly need broader or multi-angle synthesis than a single search query produces. +### HUB-024 — Adaptive query planning and iterative research + +**Status:** 🔴 OPEN — revisit trigger tripped and item opened by the operator +2026-08-13. Design: `PRDs/hub-024-query-planning.md` (grounded in 16 arXiv +abstracts fetched and read 2026-08-13; citations in the PRD). + +**Trigger record:** a research job issues exactly one SearXNG query, so the +retained corpus for a report contains only what that phrasing surfaced. The +narrowness is on the acquisition side, not retrieval (hybrid retrieval already +measures hit@4 `1.0` on the exact-term manifest, HUB-017). It also caps the +cross-source machinery deployed in HUB-034: pair drafting can only find +disagreements between sources that were crawled, and one query tends to return +sources that agree. + +**Methodology (breadth is emergent, never a fixed count):** + +- **Marginal-distinctness admission.** One bounded local-LLM call proposes + candidate facet queries; each is embedded with the deployed + `nomic-embed-text` and admitted only if its max cosine similarity to the + admitted set is below `PLAN_FACET_DISTINCT`. Breadth is whatever survives + the threshold — a narrow topic admits one facet and behaves exactly as + today; `PLAN_MAX_FACETS` is a safety rail, not the mechanism. (ScoreGate's + threshold-not-top-K principle lifted from chunk selection to planning; + Adaptive-RAG's complexity routing falls out of single-facet collapse.) +- **Gap-driven rounds.** After each round, one bounded call reads a per-facet + coverage summary (retained documents, distinct domains) and names what is + still uncovered; only those gaps become the next round's queries (KiRAG). +- **Stopping — saturation first, coverage second, budget last.** Stop when a + round's yield of new canonical URLs falls below `PLAN_NOVELTY_MIN` (KAIR's + round-over-round saturation, instanced on this system's stable canonical-URL + document identity); or when every facet is covered and no gap is named; with + hard per-job caps on rounds, searches, crawls, and wall-clock as the + backstop only. + +**Prior-art traps the design explicitly avoids:** fixed depth × breadth +parameters (Static-DRA's own limitation); expecting breadth to raise report +quality (DeepWeb-Bench: retrieval is 12–14% of errors, derivation/calibration +exceed 70% — so acceptance measures corpus breadth, not report quality, and +the judge gate stays the quality guard); redundant tool calls (HotelQuEST — +canonical-URL dedup across facets before crawling is mandatory). + +**Acceptance criteria:** more distinct domains and represented sources per +report than the single-query baseline on a fixed topic set, recorded not +asserted; a single-facet topic issues exactly one search; every sub-query +inherits source policy and SSRF vetting with canonical dedup across facets and +rounds; judge calls per report stay bounded by existing drafting caps; worker +lease/retry/idempotency semantics unchanged; plan provenance (facets, queries, +new-document yield, stop reason) recorded in job progress; +`REPORT_QUERY_PLANNING=false` reproduces current behavior exactly. ### HUB-025 — Add scheduled research jobs @@ -983,7 +1027,9 @@ HUB-017 ✅, HUB-018 ✅, HUB-019 ✅, HUB-020 ✅, HUB-021 ✅, HUB-022 ✅, HU ### Milestone 5 — Expansion only when earned -HUB-024 through HUB-030 — all deferred behind explicit revisit triggers; none tripped. +HUB-024 🔴 (trigger tripped 2026-08-13; opened with a researched design — see +`PRDs/hub-024-query-planning.md`). HUB-025 through HUB-030 remain deferred +behind their explicit revisit triggers; none tripped. ### Recommended order for the remaining open work (2026-08-12, post-pivot) @@ -993,6 +1039,8 @@ HUB-024 through HUB-030 — all deferred behind explicit revisit triggers; none 4. **HUB-034** — ✅ done 2026-08-12 (operator-authorized; judge deployed as the only gate, NLI stack decommissioned, FastAPI/Starlette upgraded, deploy fully verified). 5. **HUB-032** — ✅ done 2026-08-12 (v4 final gate-side + HUB-034 report-side pair drafting and disclaimer logic). -The pivot sequence is complete; Milestone 4 is closed. Remaining open work is P3 (revisit triggers, none tripped). +The pivot sequence is complete; Milestone 4 is closed. + +6. **HUB-024** — adaptive query planning and iterative research (opened 2026-08-13, trigger tripped; design in `PRDs/hub-024-query-planning.md`, implementation not started). **Exit condition:** each expansion is justified by measured usage or a documented limitation, not by architectural possibility.