feat(core): add CJK substring search support

[Aryan-Pardeshi]

Summary

Closes #1294.

• Add dependency-free overlapping CJK bigram token generation for Chinese, Japanese, and Korean text.
• Persist auxiliary search tokens through SQLite FTS5 and PostgreSQL row/chunk indexes without changing display text.
• Render contiguous CJK terms as adjacency phrases while preserving existing Latin search behavior and ranking paths.
• Cover migrations, accepted-note writes, reindex behavior, boolean/relaxed/count/retrieval paths, and MCP integration.
• Document the required upgrade command: basic-memory reindex --full --search.

Verification

• Focused repository/service/integration suite: 159 passed.
• Task 4 regression sweep: 858 passed, 36 skipped (PostgreSQL unavailable locally).
• Ruff checks and formatting clean for changed Python files.
• Full SQLite suite was attempted but stalled in this environment; PostgreSQL parity requires Docker.

An overlapping maintainer PR (#1373) is currently open for this issue. Please compare implementations and close/supersede whichever branch is not selected.

[CLAassistant]

All committers have signed the CLA.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9152cf35a1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve gaps between independently tokenized CJK runs

When CJK runs are separated by Latin text, punctuation, or a field boundary, appending every run's bigrams without a separator makes their tokens adjacent in the FTS stream. For example, cjk_search_tokens("适者 latin 者生存") produces 适者 者生 生存, so both SQLite and PostgreSQL incorrectly satisfy the phrase query for the contiguous substring 适者生存; the regression test avoids this by manually inserting a gap token rather than exercising this function. Insert a nonmatching positional separator between runs and fields so only genuinely contiguous text satisfies adjacency.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Keep non-CJK operands in mixed CJK matches

When a query mixes CJK and non-CJK terms under the default AND semantics (for example, 适者生存 backendmarker or the explicit 适者生存 AND backendmarker), this top-level OR lets any row matching only the CJK phrase through even if backendmarker is absent. The analogous PostgreSQL condition has the same behavior; the current test's “CJK only” row happens to contain nonadjacent auxiliary tokens, so it does not cover an exact CJK-only match. Combine the auxiliary phrase with the remaining query operands according to the original Boolean semantics instead of treating it as an alternative to the whole query.

Useful? React with 👍 / 👎.