[danyaberezun]

Problem

The palette's # scope matched spec bodies by substring: every typed word somewhere in the file, the first matching line as the hit, no ranking beyond the matcher's content tier. The agent had only spec_grep (regex / substring) to find the spec that governs a change. Neither could answer a question such as rename chat title with the section that decides it (agent › Session titles). The requirement was one engine for both surfaces, so the user and the agent get the same answer:

• local, with no model and no download;
• section hits that jump to their heading;
• a path to the agent for questions that share no words with the specs.

Seventh PR of the global-search stack; stacked on #933.

Approach

• One word-based engine in pi-spec-graph/core. core/ is the only code both the agent (vanilla pi, the CLI's bundled extension) and the host can reach.
  • Sections: by ATX heading, fence-aware, with file line numbers.
  • Tokens: identifier parts plus the joined form (renameSession → renamesession, rename, session), English stopwords, and a conservative stemmer (Porter step 1 + Snowball -ly + final e, so renaming ↔ rename while isolation ≠ isolate).
  • Scoring: BM25F (spec title + id ×3, heading trail ×2, body ×1), times the squared idf-weighted share of query words the section holds (every word beats some), doubled for an exact word-start phrase.
  • Partial words: a word absent from the corpus completes to the terms it starts, or to the stem a partial -ing extends, so the palette's half-typed last word still matches.
  • Phrasings: several are fused by reciprocal rank (k 60).
  • Rejected: pi's BM25 ranker (not exported), minisearch / Orama (a dependency outside the module's allowed set), and a local embedding model (transformers.js needs native onnxruntime-node, ~300 MB, and breaks bun build --compile). No model was the user's call: paraphrases go to the agent, whose model reformulates them.
• Cheap enough to keep warm. An append-only vocabulary with sorted typed-array postings per section brings this repo's 2.25 MB of specs to ~9 MB of heap (37 MB with per-section Maps), ~160 ms to index and 2–10 ms per query. indexSections(entries, previous) re-tokenizes only files whose content changed. Both the tool and the host keep at most four roots.
• Benchmark as the gate. A frozen copy of 15 real specs (*.md.txt, so the repo's indexer never sees a duplicate id) and 20 questions are the gate. A separate agent wrote the questions without seeing the engine.
  • The engine places the expected section in the top 3 for 20/20 (gate ≥ 16).
  • Five paraphrase misses are recorded, ungated: 0/5, as designed.
  • bun run --cwd packages/spec-graph bench:search prints the same report for a live tree (this repo: 17/20; its scratch task-specs quote the very questions).
• The agent: spec_search.
  • Params: query (a string, or an array of phrasings), limit (8, max 25), type / tag / parent.
  • Output: path:line Spec › Section — snippet, with (n/m words) on a partial match.
  • The skill and the always-on rule now name it for finding the governing section; spec_grep stays for exact patterns. No call is mandatory.
  • It joins the subagent builtins' spec read tools and the plan reviewer's tools.
• The wire: section hits, protocol v96.
  • Sections are a new SearchKind with SearchHit.section { heading, line, matched, terms }, rather than a spec hit with an extra field. The separate kind gives them their own total / state / limit and keeps them out of All at the request level: the web never asks All for them.
  • The spec module builds a root's section index off the request path. It answers indexing until the index is ready, so the client's bounded polling picks it up, and snapshot rebuilds re-sync it.
  • The line-substring content spec hits are gone.
• The palette: Sections, then «Ask the agent».
  • # lists title matches, then Sections (Spec › Heading with the spec name muted, the best line highlighted, an n/m words badge).
  • The last row is always «Ask the agent: “…”», with ⌥↵ from any row. This is variant B of the design mockup: a fixed place keeps muscle memory, where a "weak results" heuristic misfires both ways.
  • Asking starts a new chat in the context workspace (on Project Home, the project's Default) with the question and the note «Find the answer in this project's specs and cite the sections.». The current chat is not interrupted.
  • ↵ opens the spec in the Specs tool scrolled to the heading; ⌘I previews it at the section. The rendered Markdown view now honours ResourceReveal.

Changes

• packages/spec-graph:
  • core/tokenize.ts, core/sections.ts, core/search.ts; indexSections / searchSpecs and their types exported from pi-spec-graph/core.
  • core/bench/: the frozen corpus, questions.json, the gate test and the bench:search live script.
  • tools/search.ts (spec_search); the bounded section-index cache in tools/shared.ts.
  • The skill's Tools and Working-with-specs lines, and one always-on rule line.
  • core/SPEC.md (a Ranked section search record), tools/SPEC.md, SPEC.md.
• packages/contracts: PROTOCOL_VERSION = 96 = SPEC_SECTIONS_PROTOCOL_VERSION; section in SEARCH_KINDS; SearchHitSection; the v95 pin now asserts >=.
• packages/server:
  • spec: searchSpecSections, plus the section index scheduled off-request and re-synced with the snapshot. SpecSectionHit is re-exported as a type so search never imports the package.
  • search: searchSpecs keeps title / id / path; the new searchSections provider receives the query as typed; ProviderInput.text; the dead lineSnippet is removed.
  • host/search.ts wiring; host/reviewerRole.ts.
• packages/resource-renderers: MarkdownPreview honours reveal: it scrolls the last block whose source-line stamp is at or before the line, waiting for the document to render. The non-review branch now stamps lines too.
• apps/web:
  • search: the section item kind, the Sections group, titleLead, the words badge, PaletteFile.line, the Ask row + askEligible + ⌥↵ + footer hint, and the PaletteAsk export. scopeKinds: # asks for spec + section, All never asks for section.
  • shell: PaletteAsk → startChatInTab (via enterDefaultWorkspace on Project Home); a section hit → requestSpecView(…, line); the preview gets reveal.
  • store: specRequest.line.
  • panels/SpecsPanel: reveals the line with no renderer override.
  • chat/tools: spec_search renders in SpecToolCard with path links.
  • Seven en.json keys.
• packages/pi-subagents: spec_search added to the builtin read tools.
• Specs: goal-and-requirements (the Specs capability), architecture (what core/ backs), module-spec-graph, submodule-spec-graph-core, submodule-spec-graph-tools, module-contracts, submodule-server-search, submodule-server-spec, submodule-web-search (new Spec sections / Ask the agent sections), shell, store, panels, resource-renderers, pi-subagents, planReview. README: the Specs bullet.
• e2e:

  • search-palette.spec.ts: the #SPECGRAPHPROBE case now expects a section row. A new scenario covers:

    • words in any order, the badge, and the Ask row last;
    • ⌘I and ↵ both land on the heading;
    • ⌥↵ opens a new chat with the question and the note;
    • All shows neither sections nor the Ask row.

    A second new scenario asks from Project Home: ⌥↵ enters the project's Default workspace and starts the chat there.

    A spec the test writes is searched only once the Specs tree shows it. A shard's host outlives the per-test reset, and its 5 s spec snapshot can otherwise answer for an earlier test's worktree at the same path.

  • spec-tools.live.spec.ts: a spec_search case.

  • workflows/routing.live.spec.ts: the forbidden-tool list.

Deferred: a second (embedding) ranker fused behind searchSpecs, Russian or other-language queries (through the agent meanwhile), sections in the All list, an on-disk index, and non-spec documents.

Screenshots

Same scenario and viewport on both builds: a workspace carrying six of this repository's specs, the palette in #.

Before (the base, #933): #rename chat title matched each spec's frontmatter title: line — one row per file, unranked:

After — the same query: ranked Sections, Spec › Heading, the best line highlighted, n/m words:

Before / after — a paraphrase that shares no words with its section (#who keeps my chats when I reload); after, «Ask the agent» is the last row (selected here, ⌥↵ from any row):

After — ⌘I previews the rendered spec at the section, and ↵ opens it in the Specs tool scrolled to the heading:

Testing

• bun run test → 22/22 tasks (web 2002 pass; server 1837 pass, 4 skip; pi-spec-graph 107 pass, 3 skip, including the benchmark gate; contracts, resource-renderers, pi-subagents, shared, … green)
• frozen benchmark (core/bench) → 20/20 expected sections in the top 3 (gate ≥ 16); paraphrase misses 0/5, as designed; bun run --cwd packages/spec-graph bench:search on this repository → 17/20
• bun run e2e (full no-agent suite, 8 shards, 650 tests) → all passed
• bun run e2e:full e2e/spec-tools.live.spec.ts → 2 passed (a real agent calls spec_search and gets SPEC.md:7 Sample Project › Goal)
• bun run e2e e2e/search-palette.spec.ts e2e/specs-panel.spec.ts → 15 passed; the new scenario was checked by sabotage (reveal forced to line 1 → it fails at the preview's toBeInViewport)
• bun run e2e e2e/search-palette.spec.ts --repeat-each=3 → 39 passed (the two ask scenarios alone, --repeat-each=5 → 10 passed)
• bun run lint, bun run typecheck (22 packages), check:deps, check:boundaries, check:spec-surface, check:i18n, check:compiler → green
• heap probe: indexing this repository's 2.25 MB of specs → +9.1 MB, ~160 ms; queries 1.5–11 ms. The unit test's bound on that 2 MB index is a 20 s blowup guard, not a speed gate: the first CI run took 2.85 s for it on a loaded runner and failed a 2 s bound.
• CI on 004bc58f2 (after linking feat: spec section search — rank spec sections by words for the palette's # scope and the agent's spec_search, and ask the agent from # #939 into stack #807, which is what dispatches CI for a stacked PR) → 16/16 jobs green, including unit tests, both e2e shards, binary and desktop smoke + e2e (4 shards each), the Windows binary smoke and vanilla-pi package parity

Related issues

Part of the global-search stack (#804 → #805 → #806 → #887 → #888 → #933 → this).

Checklist

• Fast gates pass: bun run lint, bun run typecheck, bun run test
• E2E suite passes for app-affecting changes (bun run e2e, or bun run e2e:full when touching agent behavior)
• Before/after screenshots are included for frontend changes, or marked not applicable
• Relevant SPEC.md / top-level specs updated to reflect any boundary, contract, or behavior change
• I have read the Contributing guide and agree to the Code of Conduct

[jetbrains-air]

Approved — ready to merge.

[jetbrains-air]

Non-blocking:
Problem: The new Markdown reveal effect reapplies the retained request on every mount, even when its tick is unchanged, overriding the saved reading position.
Failure scenario: Open a section from #, scroll farther down the spec, switch to a chat tab, then return. The preview remounts and jumps back to the original heading instead of restoring where the reader left off.
Suggested fix: Track the handled reveal tick across mounts, or consume the request at the tab owner after a successful reveal, so ordinary tab restoration preserves subsequent scrolling.

[jetbrains-air]

Approved — ready to merge.

[jetbrains-air]

Approved — ready to merge.

[jetbrains-air]

Non-blocking:
Problem: A reveal before the first stamped block never scrolls the document; the observer cannot find a target even after rendering finishes.
Failure scenario: A spec has a blank line after frontmatter and introductory prose before its first heading. Searching that prose returns line 6, while the first rendered paragraph starts at line 7. If the spec is already open and scrolled down, selecting its preamble hit leaves it at the old position instead of revealing the result.
Suggested fix: When rendered blocks exist but the requested line precedes them all, reveal the first block, retaining the observer for documents that have not rendered yet.

[jetbrains-air]

Approved — ready to merge.

[jetbrains-air]

Non-blocking:
Problem: The fence opener accepts backtick-containing info strings, so a standalone inline code span is mistaken for an unclosed fenced block.
Failure scenario: With ## Syntax, a line containing ```literal```, and then ## Shutdown, both #shutdown and spec_search("shutdown") return Syntax at its earlier line; the real Shutdown heading is never indexed. The installed Markdown parser renders Shutdown as a heading.
Suggested fix: Reject backtick fence openers whose remaining info string contains a backtick, as the existing indivisibleSpans scanner does.

[jetbrains-air]

Approved — ready to merge.

[jetbrains-air]

Non-blocking:
Problem: Fence recognition misses code blocks opened inside list items, corrupting subsequent section boundaries.
Failure scenario: A spec has a list item beginning with a fenced Markdown example, an indented ## Example inside it, its indented closing fence, then ## Boundary. The closing fence becomes a new opener, so Boundary disappears as a section. Both # and spec_search return Boundary's text under Example and jump into the code sample. This reproduces with the pinned Markdown parser recognizing Boundary as a real heading.
Suggested fix: Account for list/container indentation when recognizing fence openers and closers, keeping sample headings excluded without swallowing subsequent document sections.