feat(search): yield one hit per unit colgrep matched in a file

AGENTS.md

@@ -114,6 +114,7 @@ reports, `OBS-*` observation reports from a reviewer pass.
 | `OBS-H` | `__reports__/repo_health/01-observation_review_v0.md` |
 | `OBS-C` | `__reports__/consistency/01-observation_review_v0.md` |
 | `OBS-D` | `__reports__/dev_plugin/02-observation_review_v0.md` |
+| `OBS-M` | `__reports__/match_lines/00-observation_markdown_match_lines_v0.md` |
 | `PH` | `__reports__/repo_health/00-findings_launch_placeholders_v0.md` |
 | `CI` | `__reports__/repo_health/03-findings_ci_matrix_v0.md` |
 | `RI` | `__reports__/repo_health/04-findings_remote_install_v0.md` |

README.md

@@ -12,7 +12,7 @@ One directory installs as a **Claude Code plugin**, an **[Agent Plugins 1.0](htt
 
 | Dependency | Why | Install |
 |:--|:--|:--|
-| `colgrep` ≥ 1.6 | does the indexing and ranking | `cargo install colgrep` or see the [next-plaid README](https://github.com/lightonai/next-plaid) |
+| `colgrep` ≥ 1.8.4 | does the indexing and ranking | `cargo install colgrep` or see the [next-plaid README](https://github.com/lightonai/next-plaid) |
 | `uv` | `uvx` fetches the server from PyPI and caches it on first launch | [docs.astral.sh/uv](https://docs.astral.sh/uv/) |
 | Python ≥ 3.11 | fetched automatically by `uv` if missing | — |
 

__reports__/match_lines/00-observation_markdown_match_lines_v0.md

@@ -0,0 +1,77 @@
+# Observation: a search hit on a long markdown file shows the file's first lines, not the lines that matched (v0)
+
+date: 2026-10-24 · observed from: the Murmuration lab (wave 8, explorer-w08-b and the PI) · colgrep 1.6.2, colgrep-mcp 0.5.2
+
+## In one paragraph
+
+colgrep indexes a markdown file as one unit (`unit_type: document`, lines 1 to the end of the
+file). Its **text** renderer still shows the lines inside that unit that match the query: on a
+319-line note it printed lines 137–139, where the answer is. Its **`--json`** output, which this
+server reads, carries only the unit (its whole `code`) and a score, with no match location. So
+`search` can only show the unit's first `snippet_lines` lines (`tools_search.py::hit_from_raw`,
+`code_lines[:snippet_lines]`), and `expand` reads the unit from its first line (`max_lines`
+default 200). For a long note the hit is a path and a title; the agent must open the file. In the
+Murmuration lab this is why workers read notes whole: 152 of 175 whole reads of markdown in four
+waves, and documents are 14.3% of what its workers pay to carry context.
+
+## Evidence
+
+| Probe | What came back |
+|:--|:--|
+| `colgrep -n 3 -k 3 "PI decision on w05-c1's ESCALATE" __statera__/wave-05.md` (text) | the unit header (`1 # Wave 05`), then `137 - 16:16 (the closing commit), at close:`, `138   - **PI decision on w05-c1's ESCALATE:**`, `139 …`: the matching lines |
+| same query, `--json -k 1` | one hit: keys `unit`, `score`; `unit.line` 1, `unit.end_line` 319, `unit_type` `document`, `code` 25,228 characters; no field for the matching lines |
+| MCP `search` on the lab's `__statera__/` (query about the same decision) | four hits, each `unit_type: document`, `hit_id` `…:1-51`, `…:1-26`, …, `snippet` = the file's first three lines (`# Hand-backs of …`); the note that holds the answer is not in the top four |
+| MCP `expand` on `__statera__/wave-07.md` (603 lines; explorer-w08-b) | `truncated: true` at line 200; the answer at line 470 is out of reach without `max_lines` |
+
+Code: `server/colgrep_mcp/tools_search.py`, `hit_from_raw`:
+`snippet = "\n".join(code_lines[:snippet_lines])`. The server already holds the whole `code`,
+the query, and the optional `pattern`, so it can find the matching lines itself.
+
+## Two places to fix it
+
+```mermaid
+flowchart LR
+  Q[query, pattern] --> CG[colgrep --json]
+  CG -->|unit.code, score| S[colgrep-mcp hit_from_raw]
+  S -->|today: first N lines| H[hit snippet]
+  S -.->|A: window around the match| H2[match snippet + match_hit_id]
+  CG -.->|B: upstream: match lines in JSON| S
+```
+
+**A. In this server (no upstream change).** When building a hit:
+1. With `pattern`: find the first regex match in `code`; else score each line of `code` by the
+   query's terms (case-folded, stop words out) and take the densest window.
+2. Return the window as `snippet` (keep `snippet_lines`), plus `match_line` and a narrowed
+   `match_hit_id` (`<file>:<a>-<b>`) that `expand` reads directly. Keep `hit_id` as the unit.
+3. For `document` units, let `expand` accept a hit id whose range is inside the unit (it already
+   reads `[line, end_line]` off disk), so an agent reads 20 lines, not 200.
+
+Cost: a few dozen lines in `tools_search.py`, a field in `SearchHit`, tests. Risk: the window
+picks the wrong lines for a purely semantic match with no shared words; fall back to the first
+lines and say so (`match_line: null`).
+
+**B. Upstream in colgrep.** Ask for the text renderer's match lines in `--json` (for example
+`matches: [{line, text}]` per hit). Then A becomes a pass-through and agrees with the CLI.
+
+**C. Also upstream, separable.** A section-level unit for markdown (one unit per heading)
+would make ranking itself section-aware: today a 600-line note competes as one unit, so the right
+note may rank below short files that share words (the probe above).
+
+## What to test
+
+- A 300+ line markdown file with the answer past line 200: `search` snippet shows it, and
+  `expand(match_hit_id)` returns under 30 lines.
+- `pattern` given: the window holds the first regex match.
+- Code units: unchanged behaviour (the function's first lines are usually right), or the window
+  only when the match is past `snippet_lines`.
+- A semantic-only hit: `match_line: null`, first lines as today.
+
+## For the skill, meanwhile
+
+`colgrep-search` could say: for a note or report, a hit tells you the file; then `grep -n` the
+heading or term in that one file and `sed -n 'a,bp'` the lines, rather than `expand`.
+
+## Pointers
+
+Murmuration lab (local): `__reports__/wave-08/explorer-w08-b/00-findings_v0.md` (23 questions,
+top-k rates, unit sizes), synthesis open item 43.

__reports__/match_lines/README.md

@@ -0,0 +1,7 @@
+# Reports — match_lines
+
+## Round 00
+- `00-observation_markdown_match_lines_v0.md` — a hit on a long markdown file shows the file's first lines: colgrep's text renderer finds the matching lines, its `--json` does not carry them, so `hit_from_raw` takes `code_lines[:snippet_lines]`; fix A in this server (match window, `match_hit_id`), B upstream (match lines in `--json`), C upstream (section-level markdown units) (latest)
+
+## Status
+Opened 2026-10-24 from the Murmuration lab's wave 8. Not started.

__reports__/match_units/00-findings_upstream_colgrep_contract_v0.md

@@ -0,0 +1,108 @@
+# Match Units — Findings: what colgrep should report so the server can check, not infer (v0)
+
+Date: 2026-11-01
+
+---
+type: findings
+topic: match_units
+date: 2026-11-01
+version: v0
+prior-version: none
+key-metric: hits whose location is inferred rather than checked: 35 of 110 (32 %) (prior: N/A, delta: N/A)
+decision-required: confirm
+---
+
+## Headline Result
+
+metric: share of `search` hits whose `location_verified` rests on an inference instead of a check against the unit's own content
+value: 35 / 110 (32 %)
+unit: hits, colgrep 1.8.5+cs.1, 5 queries on a scratch copy of Murmuration's `__statera__`
+prior: N/A (first run)
+direction: new
+
+Every non-best unit of a file is in this class. The server has no content to check it against, because colgrep sends `code` only for the best unit. Ask #1 below would bring the number to 0.
+
+## Results Tables
+
+### The asks, most valuable first
+
+Source lines are at fork commit `7f1e855` (`install/markdown-sections`, upstream v1.8.5 plus one fork commit).
+
+| # | Ask | colgrep today | Server workaround today | Removes |
+|:--|:--|:--|:--|:--|
+| 1 | Add a content hash of each unit to every `matches` entry | `UnitMatch` carries `line`, `end_line`, `name` and `score` only (`colgrep/src/index/mod.rs` `UnitMatch`). `absorb_matches` already holds each unit's `code` when it merges (`index/mod.rs:3497-3510`) | Non-best units are trusted when the best unit is span-verified, the index was refreshed, and the file's mtime is no later than the spawn time | The whole mtime inference, plus all three accepted gaps below |
+| 2 | Keep the best unit at `matches[0]` | `absorb_matches` sorts every entry by score, then by line ascending. On a tie, `matches[0]` may not be the unit whose `code` is in `unit` | `adapter.unit_matches` re-picks the leader by line count and display name | The tie heuristic, and the unnamed-tie gap |
+| 3 | Report the pre-search update outcome under `--json` (refreshed, skipped because the index is locked, or rebuilt) | `quiet = json \|\| files_only` (`commands/search.rs:1127`) also hides "Index is being updated by another process, searching existing index..." (`search.rs:1273-1281`) | None possible: the lock-skip is silent. `index_updated` keys on "Building index", so it misses incremental re-indexing | The lock gap, and a wrong `index_updated` |
+| 4 | Collapse results per file across all search paths before taking the top `-k` | `cmd_search` collapses per path, then concatenates and takes the top k (`search.rs:543-613`) | `search` dedupes on (file, line, end_line, name). `find_files` over-fetches with `limit*3` | The dedupe, and `find_files`' threefold over-fetch |
+
+### What not to ask
+
+| Candidate | Why not |
+|:--|:--|
+| Make `unit.line..end_line` the best unit's own range | It is widened over the file's matches on purpose (`collapse_by_file`, `index/mod.rs:4807`), and other consumers may read it as the file's hit span. `matches[0]` already gives the true range |
+| Include full `code` for every `matches` entry | It multiplies the JSON payload by the number of matched units, which can be 30 per markdown file, and the server keeps only `limit` units. A hash is a few bytes |
+| Move verification against disk into colgrep | colgrep cannot know what the server reads after it exits. The check belongs where the read happens; colgrep only has to state what it indexed |
+
+### Gaps the server accepts today
+
+These are measured by the adversarial review of `match_units`. Probes were in that session's scratchpad and are not committed.
+
+| Gap | How it shows | Window | Closed by |
+|:--|:--|:--|:--|
+| Another process holds the index lock | colgrep searches the old index silently. A file edited before the call keeps its later units trusted | Concurrent indexing of the same project | #1, or #3 |
+| A file replaced with its mtime and size preserved (`cp -p`, `rsync -t`, or `os.utime` after a same-size edit) | colgrep's change scan skips it (ns-mtime and size equal), and so does the server's mtime rule. Later units come back off by one under the right name | Tools that preserve metadata | #1 |
+| An exact score tie between two unnamed units of equal length | The wrong unit leads, giving a duplicate `hit_id` and a missing unit | Exact float ties only | #2 |
+
+## Observations
+
+| Signal | Baseline / Expected | Observed [source] | Interpretation |
+|:--|:--|:--|:--|
+| Best units verified by exact span | Most, if colgrep 1.8 ranges are exact | 75 / 75, with 0 whole-file reads [verifier's real-binary probe] | colgrep's own unit ranges are exact. Only the widened `unit` range was ever wrong (R05 D1) |
+| Non-best units trusted by a name-on-first-line rule | Close to 100 % on unedited notes | 22 / 35. The 13 failures were all continuation chunks of long sections [first verifier pass] | Names don't identify ranges. A heuristic was the wrong tool, which is why the rule moved to index freshness |
+| Non-best units trusted by the index-freshness rule | Close to 100 % on unedited notes | 35 / 35 [recheck] | Sound wherever colgrep's change detection is sound, and no sounder |
+| Lock-skip signal under `--json` | Present on stderr | Absent: stderr `''`, exit 0, results returned [implementer probe with `.lock` held] | A correctness signal is being silenced together with progress chatter |
+| Multi-path collapse | Global | Per path [`search.rs:543-613`] | `-k` counts results, not distinct files, so `-k limit` cannot replace the over-fetch |
+
+## Charts & Visualizations
+
+```mermaid
+flowchart LR
+  subgraph today["colgrep 1.8.5 today"]
+    U["unit: best unit's code,<br/>range widened over the file"] --> S1["server: span-check the best unit"]
+    M["matches[]: line, end_line, name, score"] --> S2["server: infer the others<br/>(index refreshed + mtime ≤ spawn)"]
+  end
+  subgraph asked["with asks 1-3"]
+    U2["unit: unchanged"] --> T1["server: span-check the best unit"]
+    M2["matches[]: + hash, best first"] --> T2["server: hash-check each kept unit's span"]
+    E["stderr: update outcome,<br/>even when quiet"] --> T3["server: index_updated, stale-index note"]
+  end
+```
+*Caption: where `location_verified` comes from today, and with asks #1–#3. Ask #4 changes only how many results `-k` admits.*
+
+```
+hits on statera (110)      checked ███████████████████████████████████████ 75
+                           inferred ██████████████████ 35   ← 0 with ask #1
+```
+*Caption: 110 hits over 5 queries, split by how the location was established.*
+
+## Contradictions & Surprises
+
+- colgrep's `--json` mode is the one every agent integration uses, and it is also the one mode that hides the lock-skip warning.
+- The original "colgrep reports wrong line numbers" finding (R05 D1, 1.6.2) is explained by the widening in `collapse_by_file`, not by bad unit ranges. Every unit range checked here was exact.
+
+## Steering Questions
+
+- [now] Open #2 and #3 upstream first, as two separate single-concern PRs from the CrackingShells fork? Both are a few lines, and outside PRs of that shape get merged.
+- [now] For #1, which hash? colgrep already hashes file contents for change detection (`compute_update_plan`, `index/mod.rs:3296+`). Reusing that algorithm avoids a new dependency. The server must hash spans the same way, including the `\n`-only line rule.
+- [next run] When a colgrep release with #1 exists, should the server prefer the hash and keep the mtime rule only for older binaries? Or should the soft minimum move to that release and the mtime rule go?
+- [next run] If #4 lands, switch `find_files` to `-k limit` and drop the dedupe. The step 5 test in `__roadmap__/match_units/unit_pipeline.md` already describes the argv check.
+- [later] Does #3's update outcome belong on stderr, or in a JSON envelope behind a flag? The bare-array output is what existing consumers parse.
+
+## Pointers
+
+- `__roadmap__/match_units/` — the campaign, its leaves and gates
+- `__reports__/match_units/00-knowledge_transfer_v0.md` — how these gaps were found
+- `__reports__/match_lines/00-observation_markdown_match_lines_v0.md` (OBS-M) — why the per-unit hits exist
+- `server/colgrep_mcp/adapter.py` `unit_matches`; `server/colgrep_mcp/tools_search.py` `hits_from_raw`, `_do_search`
+- [CrackingShells/colgrep-mcp#18](https://github.com/CrackingShells/colgrep-mcp/pull/18)
+- Upstream: [lightonai/next-plaid](https://github.com/lightonai/next-plaid); fork: CrackingShells/next-plaid, branches `feat/markdown-sections` and `install/markdown-sections`