Repository navigation
Injection quality: chunk-boundary refinement — AST path never clamps, no long-function splitting (gated on #95) #97
Description
Split out of #54 (Injection quality Tier 3), item 2.
State, verified against main at e706eb3 (v0.16.3) — partly overtaken, core ask still open
Token-window clamping now exists and did not when #54 was filed:
max_chunk_tokens(src/index/parser.rs:40) splits oversized chunkschunk_transcriptsplits bodies over 100 lines at paragraph breaks (:634)
But the original ask is still open. max_chunk_tokens is referenced only in with_chunking
(:68-72) and chunk_by_lines (:725-746) — never in the AST path — and the field's own doc says
"line/markdown chunks are split". So:
- a long function from AST chunking is still emitted whole (no long-function splitting)
- there is still no cross-chunk caller/callee capture
The work
Chunker redesign touching indexing, re-index and eval.
⚠️ Two gates
- Gated on Injection quality: run a fresh calibrated eval — committed results predate all 14 quality features #95 (fresh calibrated eval). Chunk boundaries are a retrieval-quality bet, and the
committed numbers predate all 14 quality features, so there is no current evidence that chunking
is what limits quality. - Reconcile with in-flight chunking work first. There is active work on a
fix/chunk-fallback-collisionsbranch. Whoever picks this up should build on that rather than
redesign around it — otherwise the two collide in exactly the code both touch.
Activity
scbrown commentedon Oct 12, 2026
Rechecked against current
mainat94bfc9353d9d5f7a256e673c489aa691c3155b6a. Both requested capabilities remain open:src/index/parser/semantic.rs; it emits the entire AST node body. The token-window limit is still consumed by line-based chunking, so long AST functions are not split.ChunkEdgeTypenow supports seven structural/similarity relations, but no caller/callee relation. Those existing edges do not fulfill this issue's call-capture requirement.The prerequisite #95 remains open. The current evaluation preregistration approves deterministic offline L0 retrieval/ablation measurements; a full L1 agent study needs separate approval. The prior calibration artifact is a one-task result, so it is not evidence that a chunker redesign is the next retrieval-quality bottleneck.
Next step: obtain a reproducible current-source, held-out evaluation under #95, then use those measurements to select and assess long-function splitting and call capture. The earlier same-line chunk-identity collision fix is already present in semantic extraction; preserve its span/ID behavior when splitting is introduced. This triage does not claim #95 or #97 is implemented.
scbrown commentedon Oct 21, 2026
2026-10-21 disposition: keep the chunker change gated on usable #95 evidence
The current semantic extractor still pushes the selected AST node body into a chunk (
content: node_content.to_string()). That bounded source check establishes the long-node behavior; it is not a proof about every extraction path or measured retrieval quality.The #95 partial report covers 330/1,320 cells, includes 34 invalid header outcomes, and does not justify a chunker redesign. Retain this issue open. Owner: Gennaro. Resume after the header/dedup measurement contracts and usable paired held-out evidence under #95 establish that chunk boundaries are a bottleneck. Then test long-function splitting and caller/callee capture against real span/ID and collision controls, with an unchanged baseline and explicit reindex/rollback scope.
No new call-edge capability, completed redesign, benchmark, shared reindex or runtime change is claimed by this triage. Existing same-line identity/collision handling must survive any later splitting implementation.
scbrown commentedon Oct 23, 2026
2026-10-23 triage — owner: Gennaro. Keep open, blocked on usable #95 evidence and a measured chunk-boundary hypothesis.
At frozen main
fddcdb7b1bdf885528dbf0b432958125afe9c29d, semantic extraction still stores the selected AST node body as chunk content. This bounded read confirms that path's whole-node behavior; it is not an exhaustive chunker audit. #204 repaired opt-in scoring paths, not chunk boundaries, and produced no new efficacy evidence.Next step: Gennaro reconciles #95's header/dedup outcomes and scoring convention before any further bounded evaluation. If paired held-out results then identify long-node boundaries as a bottleneck, prepare an isolated long-function/call-context experiment with span/ID/collision positives and refusal controls, an unchanged baseline, and explicit reindex/rollback scope. Reconcile any live chunking proposal at that boundary; the historical branch name alone is not current custody proof. No redesign, caller/callee capability, reindex or runtime improvement is claimed here.
scbrown commentedon Oct 24, 2026
2026-10-24: Still gated on #95; its measurement-contract repairs (#204 and #210) are merged, but the bounded current-runtime evaluation and acceptance gates remain open.