rfc(search): is a model-free lexical lane intended for the file corpus?

[waterbro-8]

Summary

Text search over files has no model-free path. If the embedding worker is not
reachable, file search does not degrade — it refuses. Structured memory has a
deliberate lexical lane; the file corpus has none.

This is a design question first, not a defect: the model-free guarantee in the
repository's own decisions is consistently written about memories, and nobody on
this repository has ever applied it to files as far as I can find.

Source reproduction

On main @ 7a194f1eba4167d54bd46cf84cdbe86e00532319:

• server/internal/search/search.go:220-221 — when the worker is nil or
  disabled, search returns search disabled: worker not configured. There is no
  branch that continues without an embedding.
• SPEC.md:536 grants the deterministic no-model lane to memories
  ("memories FTS + trigram — 无模型的确定性立即召回"). files is not
  mentioned.
• The lexical index exists only on memories: search_tsv and the trigram GIN
  are declared at server/internal/db/migrations/0008_agent_memories.sql:79-99
  and :100-101; files (0001_init.sql:35-61) has no tsvector and no trigram
  index, and server/internal/search/search.go contains no full-text or
  trigram branch at all — its only retrieval routes are vector cosine, plus
  mime/path/time filters.
• CLI surface: the flag block is server/cmd/mem/cmds_search.go:103-114 and it
  exposes --type --route --since --until --limit --format --idempotency-key.
  --route only selects text|visual|auto, all three of which are vector
  routes — there is no text-only route to select. Separately, path scoping is
  not a service gap: search.Service already takes
  PathPrefix (search.go:190) and applies it via appendPathFilters at
  :321, :361, :641, :687. The CLI simply exposes no flag for it, so that
  piece is CLI-only and smaller than the rest of this issue.
• Precedent for the index shape already exists on the memories side:
  0002_folders.sql:28 gives files(user_id, path text_pattern_ops), and
  0008_agent_memories.sql:89/100-101 pair the same pattern index with a
  trigram GIN. The file corpus has the first and not the second.
• Related precedent, all memory-side: feat(recall): orchestrate hybrid candidates, reranking, diversity, and budgets #71 ("Keep a lexical/model-independent
  path"), feat(models): add a hardware-aware local embedding catalog and guided installer #47 ("keeps lexical/structured memory usable when the user skips this
  step"), feat(platform): gate managed embeddings with workspace entitlements and idempotent usage #45 ("preserves model-free structured/lexical memory where
  available"). feat(recall): orchestrate hybrid candidates, reranking, diversity, and budgets #71 was closed not_planned.

Why this is worth deciding rather than coding

The stated product frame is an AI drive where a user must be able to find things
(GOAL.md §6, "找得到"). Two properties that are currently true by accident,
and that a decision would fix either way:

1. A self-hosted deployment that never configures a worker cannot search its own
   file names' contents, but can browse by path. That asymmetry is not
   documented as intentional anywhere I found.
2. Filename and path substring search — the thing that makes a drive usable with
   no model at all — does not need a new idea; 0008 already shows the shape
   for memories, and files.path already carries a text_pattern_ops index
   used for subtree candidates.

Scope boundary

Explicitly not asked here: a Chinese tokenizer. I checked and the
to_tsvector('simple', …) choice at 0008:79-80 is documented as intentional,
with the comment at :77-78 stating that pg_trgm below is the CJK and
typo-tolerant lane. Whether that trigram lane is good enough for Chinese is an
open empirical question, and it belongs to #175, which exists because
nothing currently measures it. This issue must not be merged into a tokenizer change.

Also out of scope: search result ranking, and anything that requires the
generation lifecycle.

Acceptance

• A maintainer decides, recorded in this issue, whether a model-free
  retrieval lane over file content and paths is intended for the file corpus
  or explicitly not.
• If intended: mem_search and the CLI can complete a query with the worker
  stopped, the degradation is stated in SPEC.md §6.2 and in docs/mcp.md,
  and a test proves the no-worker path returns results rather than an error.
• If not intended: the current error becomes a documented, named limitation
  in the deployment and MCP docs, and this issue closes with that reason.
• Whatever is decided, the no-worker behavior of file search differs from
  memory search in exactly the way the decision describes, and one test
  covers each side of that difference.

Evidence level

E2 — source-level, including the negative claims (no lexical branch in
search.go, no tsvector/trigram on files) which were checked by reading the
whole of search/search.go and every migration. No behavior was observed at
runtime.

Proposed triage

Applied on filing, per docs/maintainers/triage.md and the precedent in #135:
type:rfc, area:server, area:recall, evidence:e2-source,
status:needs-triage.
Left to a maintainer: priority, and any change to the set below.

type:rfc ("requires a reviewed product or architecture decision") is chosen over
type:feature deliberately: the first acceptance item is a maintainer decision, and
the honest answer may be "not intended", in which case this closes with a documented
limitation instead of a change. triage.md says to change the type when the
classification changes, so flipping it to type:feature after a "yes" decision is
the expected path.

Raised from the data/index audit on 2026-09-08.

[xiaocui-big]

Analysis and recommendation

Current asymmetry (confirmed from source)

Capability	Memories	Files
FTS tsvector column	search_tsv generated column (0008:79-81)	None
Trigram GIN index	idx_memories_content_trgm (0008:100-101)	None
Lexical recall in service	Recall() three-tier: exact → FTS → trigram (memory/service.go:209-302)	None
No-worker behavior	Returns lexical results	Hard error: "search disabled: worker not configured" (search.go:220-222)
SPEC guarantee	§6.2, F5.4, F8.6 — explicit model-free lane	Not mentioned

The memory side already proves the shape works: tiered scoring (exact > 0.949, FTS 0.70–0.949, trigram 0.20–0.699), pg_trgm for CJK tolerance, and zero model dependency.

Recommendation: yes, intended

A model-free lexical lane for files is consistent with the product frame (GOAL.md §6 "找得到") and the existing precedent. A self-hosted deployment that never configures a worker should still be able to find files by name and path substring — the same way it can already recall memories lexically.

The implementation is well-scoped because the shape already exists on the memory side.

Proposed implementation shape (if maintainer agrees)

1. Migration (next available number): add to files:

   • name_tsv tsvector GENERATED ALWAYS AS (to_tsvector('simple', name)) STORED
   • CREATE INDEX idx_files_name_tsv ON files USING gin (name_tsv)
   • CREATE INDEX idx_files_name_trgm ON files USING gin (lower(name) gin_trgm_ops)
   • The existing path text_pattern_ops index (0002:28) already covers prefix path matching.

2. Search service: add a RouteLexical constant and a searchLexical() method that runs when the worker is nil/disabled, or when explicitly selected. Score tiers mirror memory recall: exact name match → FTS → trigram on files.name.

3. Degradation, not failure: when worker is unavailable, Search() falls through to lexical instead of returning an error. Auto route becomes: lexical always + vector when available.

4. CLI: add --route lexical option. Existing text|visual|auto routes unchanged.

5. Tests: one test proves no-worker path returns results; one test proves the lexical/vector difference on each side.

6. Docs: SPEC.md §6.2 and docs/mcp.md state the file lexical lane and its degradation contract.

Scope discipline

Agreed with the issue's scope boundaries: no tokenizer change (that's #175), no ranking overhaul, no generation lifecycle involvement. The simple tsvector configuration and pg_trgm CJK lane are deliberate existing choices.

Evidence level

E2 — source-level, matching the issue's own evidence level. No runtime observation.

[waterbro-8]

This is still an RFC, not a defect. Draft implementation candidate: #194.

I am not coding a file-corpus lexical lane until product records a decision. Current source on main still refuses file search when the worker is missing (search disabled: worker not configured); memories keep the FTS/trigram lane. That matches SPEC.md:536 as written.

If the ruling is "files stay model-required", this issue should close not_planned with that sentence in the record. If the ruling is "files get a lexical degrade path", that is a new status:ready implementation issue, not a silent patch on this RFC.

[waterbro-8]

已解决（completed）。维护者此前建议的“文件语料支持无 Worker 的 model-free lexical lane”已由合并 PR #194 实现并验证：新增 route=lexical、文件名 FTS/trigram、路径限制、CLI/MCP/API 文档及无 Worker 回归测试。该 RFC 的决策为“支持”，实现已在 main。