Repository navigation
Two-character Han filename substrings can miss indexed files (coordination with #111) #137
Description
What happens
On the v0.6.1 baseline, searching 輔導 can miss an indexed filename such as 2026學生情緒輔導紀錄.pdf: the unspaced Han name becomes one FTS token. Expected: a literal two-character/internal filename match should find the file and rank direct filename hits before content-only or semantic matches.
Related draft: #85, currently at 3e7070e. I opened the draft before the issue-first discussion requested in CONTRIBUTING. This issue asks whether this bounded fix fits your plans; the PR will stay in draft until you agree to proceed.
Proposed scope and coordination
- Find two-character Han words inside filenames #85 adds literal basename matching for queries containing at least two consecutive Han characters, using existing normalized filename rows. All query components must match; exact filename/stem hits precede partial names. Kind filters, rename/delete handling, and deduplication remain covered.
- No reindexing, schema/model changes, Traditional/Simplified conversion, or single-character substring matching.
- Fix Han full-text matching and preserve tokenizer-budgeted text tails #111 by @shimon-ai-dev handles Han full-text tokenization, index migration, and tokenizer-budgeted text tails. Its implementation and body-search tests are not copied into Find two-character Han words inside filenames #85. This PR can be reviewed independently of Fix Han full-text matching and preserve tokenizer-budgeted text tails #111, while preserving room to integrate the two later.
- Interface localization stays in Add English and Traditional Chinese interface localization #34; Add Simplified Chinese macOS localization using the shared framework #112's localization additions remain separate.
Validation and tradeoff
- Earlier same-index, synthetic filename evaluation: the expected first result improved from 4/12 to 12/12. All 126 upstream synthetic/multilingual/code evaluation outputs were byte-identical before/after. This is fixture evidence, not a general Chinese semantic-accuracy claim.
- Earlier full suite: 101 tests passed, including actual model reference vectors.
- Latest test-only update on macOS 27.0.1: all 10
HanFilenameTestspassed using the Xcode build system. The added regression checks exact filename > partial filename > content-only ranking, no duplicate files, typing and submitted-query modes, with vectors disabled. Its standalone body keyword already works on v0.6.1 and does not require Fix Han full-text matching and preserve tokenizer-budgeted text tails #111. - Cost on the earlier 20,000-name fixture:
輔導median CLI time increased from 15.18 to 59.02 ms; Latinannualstayed about 40 ms. These measurements include process startup/index opening and were not rerun for the test-only update. Reproduction commands and detailed results are in Find two-character Han words inside filenames #85.
System
Mac; DigUp v0.6.1 baseline, Apple M2 Pro, 16 GB. Current focused validation: macOS 27.0.1. No Windows/Linux runtime validation is claimed.
Logs
Only synthetic filenames and fixtures are used in the published tests/evals; no personal search/index logs are attached.
Would you accept this filename-only scope for review, or prefer it to be coordinated differently with #111?
Activity
ARahim3 commentedon Oct 21, 2026
Thanks @MaxTheHaters. The miss is real and I want it fixed where it comes from: the keyword index's tokenizer treats an unspaced Chinese name as one word, so the fix is a tokenizer that indexes Chinese, Japanese and Korean by character pairs or triples, for file names and documents at once, in one schema change. A filename-only pass beside it, at the query cost you measured, isn't something I'd carry once that's in, so #85 stays closed and this closes with it. When I start the tokenizer work I'll open an issue for it, and help on that would be welcome.