diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a60f1c..789de97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,9 @@ The project publishes 0.x prerelease versions; a stable release line is not yet ### Added +- Advertise the model-free lexical file-search route in the MCP `mem_search` + schema and verify `tools/list` plus route/filter forwarding through `tools/call`. + - `mem doctor` — a read-only diagnosis of why the CLI cannot talk to a working server (`#112`). It reports four checks in a fixed order: reachability of the configured server URL, whether a credential exists, the workspace the server @@ -28,6 +31,10 @@ The project publishes 0.x prerelease versions; a stable release line is not yet deployment path (`deploy/compose`, `docs/DEPLOYMENT.md`), instead of telling somebody to log in against a server that is not running yet. Hosts that already have a configuration keep the previous, shorter hint. +- File search gains a model-free lexical route (`route=lexical`). FTS + trigram + over `files.name` — same tier shape as memory Recall — so a deployment with + no embedding worker can still find files by name. CLI: `mem search "query" + --route lexical`. ### Changed diff --git a/SPEC.md b/SPEC.md index d1e1b55..277caa8 100644 --- a/SPEC.md +++ b/SPEC.md @@ -146,7 +146,7 @@ | F5.1 | `mem context "..."` → 返回有大小预算的文件/结构化记忆证据包 | | F5.2 | 每条 evidence 必须含 source kind/id、稳定 citation、内容哈希、片段和 locator | | F5.3 | mem 只走 recall → context pack；回答与行动由调用方 Agent 完成 | -| F5.4 | `source=all|file|memory`；结构化记忆在无 Worker、无模型时也必须可立即召回 | +| F5.4 | `source=all|file|memory`；结构化记忆在无 Worker、无模型时也必须可立即召回；文件词法路由（`route=lexical`）同样无需 Worker | | F5.5 | 联合召回单路失败但仍有证据时返回 `200 + partial=true + warnings[]`；无幸存证据时返回 `502 context_unavailable` | ### F5A · 结构化 Agent 记忆 @@ -534,6 +534,9 @@ embeddings_face ( - `folders (user_id, path)` UNIQUE — 路径唯一性约束 - `memories (workspace_id, idempotency_key_sha256)` UNIQUE — 不落明文幂等键的幂等写入 - `memories` FTS + trigram — 无模型的确定性立即召回 +- `files` FTS + trigram — 显式指定 `route=lexical` 的文件名无模型词法召回； + `auto` 只融合 text/visual，不自动回退到 lexical，worker 不可用时仍报错。 + 仅搜索 `files.name`，路径只用于筛选，不检索文件正文或路径片段。 - `embeddings_* (embedding)` — pgvector HNSW - `file_entities (entity_id)` — 反查"和某人有关的所有文件" diff --git a/docs/MIGRATION_SEQUENCE.md b/docs/MIGRATION_SEQUENCE.md new file mode 100644 index 0000000..7ed41df --- /dev/null +++ b/docs/MIGRATION_SEQUENCE.md @@ -0,0 +1,65 @@ +# Pending migration deployment sequence + +These draft changes are cumulative, not independently deployable: + +| Order | Draft / original PR | Migration | Required predecessor | +| --- | --- | --- | --- | +| 1 | #194 / #183 | 0024 file lexical lane | released/main schema 23 | +| 2 | #197 / #180 | 0025 HNSW indexes | #194, schema 24 | +| 3 | #195 / #185 | 0026 data-plane hygiene | #197, schema 25 | + +The PR base chain is `main` → `codex/fix-pr-183` → `codex/fix-pr-180` +→ `codex/fix-pr-185`. Successor branches must include their predecessor schema and source. Local +repair branches are rebuilt on current main and replay the original authored +changes; published commit identities remain in the original PR history. +Keep this order when retargeting after a predecessor merges. + +On 2026-09-10, main `2986fe38175f54d99f15dd38a498708c6ecd88cd` and published +tags `v0.1.0` / `v0.1.1` contain only migrations 0001–0023. This does not prove +that a private deployment never applied a draft. Consequently migration +numbers and SQL identities are retained, not renumbered on an assumption. + +HNSW DDL and shipping text-query planner acceptance are separate evidence: +creating indexes does not prove that a text query uses them. An index-only +successor must describe that partial scope and leave the broader #173 acceptance +open. #195 still requires the predecessor schema 25 regardless of query strategy. +This document does not approve a product decision or query-strategy change, +waive a review gate, or authorize deployment. #176's model-free file-lane RFC +also requires a maintainer decision before this draft is made review-ready. + +Goose startup remains strict: no `WithAllowMissing` or equivalent option is +enabled. A database that already applied 26 while omitting 24/25 will correctly +fail startup against the cumulative schema. Stop and obtain an operator-owned +recovery plan for such a database; do not edit its migration history, renumber +its SQL, or apply lower versions out of order to manufacture a pass. + +## Migration 0024 operational boundary + +Adding the stored generated `search_tsv` column rewrites existing `files` rows, +and its two indexes are built without `CONCURRENTLY`. Schedule a maintenance +window sized for the file corpus and expect table locks to block other access. +The migration indexes filenames only; `PathPrefix` remains a filter, not path +substring retrieval. Downgrading 0024 removes the derived column/indexes and +requires deploying server code that does not query the lexical route. + +Goose runs this migration transactionally, so an ordinary failure rolls back +its DDL. If an operator has applied some statements manually, `IF NOT EXISTS` +does not prove that an existing column or index has the correct definition. +Inspect both `goose_db_version` and the actual schema/index definitions before +an operator-owned recovery; do not mark an unverified partial schema applied. + +## Regression evidence + +`TestMigrationFilesContiguous` rejects embedded numeric gaps without a DB. +`scripts/verify.sh integration` creates a separate, owned `_test` database and +runs `TestMigrationUpgradeSequence`. It applies real Goose migrations to 23, +seeds a file with duplicate text chunks, then advances one version at a time +to the branch's declared head (24, 25, or 26). Each step checks full applied +history and preserved data; subsequent steps check lexical backfill, valid +HNSW DDL, and deduplication/unique rejection. Finally the ordinary production +`DB.Migrate` startup path must accept the resulting history unchanged. + +The dedicated test uses `MEM_MIGRATION_SEQUENCE_TEST_DB`, refuses a database +that already has Goose history, and must not target any developer or production +database. The existing owned-database runner performs cleanup. These are real +database tests over synthetic fixtures, not retrieval-quality or latency proof. diff --git a/docs/mcp.md b/docs/mcp.md index 5fd17a2..6999b3d 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -90,6 +90,18 @@ coordinates, timezone-free timestamps and control characters are rejected by the HTTP API. The metadata is persisted server-side and is not included in an enrichment-model prompt. +### Model-free file search + +Use `mem_search` with `route=lexical` (or `mem search "query" --route lexical`) +to search filenames without an embedding worker or managed provider. `scope` +restricts the virtual-folder subtree; it does not make paths or file contents +searchable. `route=auto` fuses only the text and visual embedding routes and +does not fall back to lexical when the worker is unavailable. + +Lexical scoring uses name substrings, simple full-text matching, then tolerant +trigram matching. It scores the files remaining after workspace, path, MIME +and time filters; no indexed candidate-pruning or latency guarantee is claimed. + ### Reviewing file annotations Use `mem_info` (or `mem info --format json`) to read pending @@ -138,7 +150,7 @@ The canonical product surface is: | `mem_checkpoint_list` | List newest-first bounded checkpoint summaries for one task | | `mem_checkpoint_get` | Get one immutable checkpoint and its full handoff payload | | `mem_resume` | Restore the current task head or a selected historical checkpoint, including resolved and missing evidence | -| `mem_search` | Natural-language search (text / visual / auto fuse); ranked files + snippets | +| `mem_search` | Natural-language search (text / visual / auto fuse); ranked files + snippets. `route=lexical` is model-free (FTS + trigram over file names, no worker needed) | | `mem_context` | Build an evidence-backed context pack for the calling Agent | | `mem_related` | Top-K files related to a `file_id` by embedding similarity | | `mem_face` | Person clusters: `action=list` / `name` / `merge` | @@ -295,7 +307,8 @@ same logical request should supply and retain a stable key so a committed result can replay without another provider invocation or charge. A `504` means the provider outcome is uncertain: do not automatically retry, and do not invent a new key. `mem_context` with `source=memory` stays lexical and -model-independent. +model-independent. `mem_search` with `route=lexical` is likewise model-free: +it uses FTS + trigram over file names and works without a configured worker. Its target output is structured for an Agent to consume: diff --git a/scripts/verify.sh b/scripts/verify.sh index eed61b0..b8a45ad 100755 --- a/scripts/verify.sh +++ b/scripts/verify.sh @@ -4,7 +4,7 @@ set -euo pipefail REPO_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)" MODE="${1:-unit}" -EXPECTED_MIGRATION_HEAD=23 +EXPECTED_MIGRATION_HEAD=24 MIGRATION_ROLLBACK_TARGET=11 MODEL_TEXT_CANONICAL_BASE=15 WORKSPACE_AI_PROFILE_BASE=16 @@ -304,6 +304,16 @@ run_migration_round_trip() { MEM_TEST_TARGET_DB="$MEM_TEST_DB" testdb assert-unsafe-derived-text-scrubbed } +run_migration_sequence() { + log "Strict populated migration upgrades from released head 23 to $EXPECTED_MIGRATION_HEAD" + ( + cd "${REPO_ROOT}/server" + MEM_MIGRATION_SEQUENCE_TEST_DB="$MEM_TEST_DB" \ + go test -count=1 -v ./internal/db -run '^TestMigrationUpgradeSequence$' + ) + assert_migration_version "$EXPECTED_MIGRATION_HEAD" +} + run_migrations_up() { ( cd "${REPO_ROOT}/server" @@ -385,6 +395,7 @@ run_postgres_tests() { run_integration() { validate_test_database + with_fresh_test_database migration_sequence run_migration_sequence with_fresh_test_database migration run_migration_round_trip with_fresh_test_database integration run_postgres_integration } diff --git a/server/cmd/mem-mcp/server_test.go b/server/cmd/mem-mcp/server_test.go index ba317b0..8ca3146 100644 --- a/server/cmd/mem-mcp/server_test.go +++ b/server/cmd/mem-mcp/server_test.go @@ -162,6 +162,74 @@ func TestMCP_ToolsCallRoundTrip(t *testing.T) { } } +func TestMCP_LexicalSearchSchemaAndForwarding(t *testing.T) { + requests := make(chan map[string]any, 1) + fake := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost || r.URL.Path != "/v1/search" { + t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path) + } + var body map[string]any + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + t.Errorf("decode search: %v", err) + } + requests <- body + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"results":[]}`)) + })) + defer fake.Close() + reg := tools.New() + if err := builtin.RegisterAll(reg, apiclient.New(fake.URL, "test-token")); err != nil { + t.Fatal(err) + } + srv, buf := newTestServer(reg) + in := strings.NewReader(`{"jsonrpc":"2.0","id":1,"method":"tools/list"}` + "\n" + + `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"mem_search","arguments":{"query":"budget","route":"lexical","scope":"/Work","type":"text","limit":3}}}` + "\n") + if err := srv.serve(in); err != nil { + t.Fatal(err) + } + responses := readResponses(t, buf) + if len(responses) != 2 { + t.Fatalf("want list and call responses, got %d", len(responses)) + } + t.Run("exported schema advertises all routes", func(t *testing.T) { + for _, item := range responses[0]["result"].(map[string]any)["tools"].([]any) { + tool := item.(map[string]any) + if tool["name"] != "mem_search" { + continue + } + schema := tool["inputSchema"].(map[string]any) + route := schema["properties"].(map[string]any)["route"].(map[string]any) + enum := route["enum"].([]any) + want := map[string]bool{"text": true, "visual": true, "auto": true, "lexical": true} + for _, value := range enum { + delete(want, value.(string)) + } + if len(enum) != 4 || len(want) != 0 { + t.Fatalf("mem_search route enum = %v; missing %v", enum, want) + } + return + } + t.Fatal("mem_search missing from tools/list") + }) + t.Run("lexical call forwards route and filters", func(t *testing.T) { + if responses[1]["error"] != nil { + t.Fatalf("RPC error: %v", responses[1]["error"]) + } + if result := responses[1]["result"].(map[string]any); result["isError"] != false { + t.Fatalf("tool error: %v", result) + } + select { + case body := <-requests: + if body["query"] != "budget" || body["route"] != "lexical" || + body["scope"] != "/Work" || body["type"] != "text" || body["limit"] != float64(3) { + t.Fatalf("forwarded search = %#v", body) + } + default: + t.Fatal("lexical request was not forwarded") + } + }) +} + func TestMCP_ToolErrorSurfacedInContent(t *testing.T) { // memd returns 404 fake := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { diff --git a/server/cmd/mem/cmds_search.go b/server/cmd/mem/cmds_search.go index 47704ec..d686449 100644 --- a/server/cmd/mem/cmds_search.go +++ b/server/cmd/mem/cmds_search.go @@ -101,7 +101,7 @@ func newSearchCmd() *cobra.Command { }, } cmd.Flags().StringVar(&typ, "type", "", "mime prefix filter: image|text|application|audio|video") - cmd.Flags().StringVar(&route, "route", "", "search route: text|visual|auto (default auto)") + cmd.Flags().StringVar(&route, "route", "", "search route: text|visual|auto|lexical (default auto)") cmd.Flags().StringVar(&since, "since", "", "YYYY-MM-DD inclusive lower bound on timeline_at") cmd.Flags().StringVar(&until, "until", "", "YYYY-MM-DD inclusive upper bound on timeline_at") cmd.Flags().IntVar(&limit, "limit", 0, "max results (default 10, max 100)") diff --git a/server/internal/api/api.go b/server/internal/api/api.go index e8c6d07..d959ed9 100644 --- a/server/internal/api/api.go +++ b/server/internal/api/api.go @@ -1334,9 +1334,9 @@ func (s *Server) handleSearch(w http.ResponseWriter, r *http.Request) { return } switch req.Route { - case "", search.RouteAuto, search.RouteText, search.RouteVisual: + case "", search.RouteAuto, search.RouteText, search.RouteVisual, search.RouteLexical: default: - writeError(w, http.StatusBadRequest, "bad_route", "route must be auto, text, or visual") + writeError(w, http.StatusBadRequest, "bad_route", "route must be auto, text, visual, or lexical") return } scope, err := pathx.Normalize(req.Scope) diff --git a/server/internal/api/managed_embeddings.go b/server/internal/api/managed_embeddings.go index a4e54f3..d9cd6dc 100644 --- a/server/internal/api/managed_embeddings.go +++ b/server/internal/api/managed_embeddings.go @@ -161,8 +161,9 @@ func (s *Server) managedSearcher( s.Search == nil { return nil, nil, entitlement.ErrEntitlementUnavailable } - // A visual-only query does not invoke the managed text embedding provider. - if query.Route == search.RouteVisual { + // A visual-only or lexical query does not invoke the managed text embedding + // provider. + if query.Route == search.RouteVisual || query.Route == search.RouteLexical { return s.Search, nil, nil } spec, err := s.Search.EmbeddingSpec(r.Context(), query.UserID) diff --git a/server/internal/api/managed_embeddings_test.go b/server/internal/api/managed_embeddings_test.go index 46dfd5b..508d538 100644 --- a/server/internal/api/managed_embeddings_test.go +++ b/server/internal/api/managed_embeddings_test.go @@ -573,3 +573,26 @@ func TestReadinessIsDeploymentModeAwareAndPlanIndependent(t *testing.T) { } }) } + +func TestLexicalSearchBypassesManagedEmbeddingReservation(t *testing.T) { + searchFake := &managedSearchFake{spec: "openai:text-embedding-3-small"} + usageFake := &managedEntitlementFake{reserveErr: errors.New("must not reserve lexical search")} + server := &Server{ + Search: searchFake, DeploymentMode: "saas", + ManagedEmbeddingProvider: searchFake.spec, Entitlements: usageFake, + } + // There is deliberately no paid plan, idempotency key, or model context. + request := httptest.NewRequest(http.MethodPost, "/v1/search", nil) + query := search.Query{UserID: uuid.New(), Route: search.RouteLexical, Text: "notes"} + searcher, executor, err := server.managedSearcher(request, "search.query", nil, query) + if err != nil || executor != nil || searcher != searchFake { + t.Fatalf("lexical dispatch: searcher=%T executor=%v err=%v", searcher, executor, err) + } + if _, err := searcher.Search(request.Context(), query); err != nil { + t.Fatal(err) + } + if searchFake.searchCalls != 1 || searchFake.embeddingCalls != 0 || usageFake.reserveCalls != 0 { + t.Fatalf("lexical dispatch invoked model policy: search=%d embedding=%d reserve=%d", + searchFake.searchCalls, searchFake.embeddingCalls, usageFake.reserveCalls) + } +} diff --git a/server/internal/db/migration_sequence_test.go b/server/internal/db/migration_sequence_test.go new file mode 100644 index 0000000..5908cf3 --- /dev/null +++ b/server/internal/db/migration_sequence_test.go @@ -0,0 +1,132 @@ +package db + +import ( + "context" + "database/sql" + "errors" + "os" + "strconv" + "strings" + "testing" + "time" + + "github.com/google/uuid" + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgconn" + "github.com/pressly/goose/v3" +) + +func contiguousMigrationHead(t *testing.T) int { + t.Helper() + entries, err := migrationsFS.ReadDir("migrations") + if err != nil { + t.Fatal(err) + } + for i, entry := range entries { + prefix, _, ok := strings.Cut(entry.Name(), "_") + version, err := strconv.Atoi(prefix) + if !ok || err != nil || version != i+1 { + t.Fatalf("migration sequence gap: want %04d, got %q; include predecessors before deployment", i+1, entry.Name()) + } + } + return len(entries) +} + +func TestMigrationFilesContiguous(t *testing.T) { + contiguousMigrationHead(t) +} + +// The runner supplies a NEW, owned database, distinct from the shared +// MEM_TEST_DB integration fixture. Never roll back or renumber deployed DDL. +func TestMigrationUpgradeSequence(t *testing.T) { + head := contiguousMigrationHead(t) + dsn := os.Getenv("MEM_MIGRATION_SEQUENCE_TEST_DB") + if dsn == "" { + t.Skip("MEM_MIGRATION_SEQUENCE_TEST_DB not set; requires a fresh owned test database") + } + cfg, err := pgx.ParseConfig(dsn) + if err != nil { + t.Fatal(err) + } + if !strings.HasSuffix(cfg.Database, "_test") { + t.Fatalf("refusing non-test database %q", cfg.Database) + } + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + defer cancel() + sqldb, err := sql.Open("pgx", dsn) + if err != nil { + t.Fatal(err) + } + defer sqldb.Close() + var existing sql.NullString + if err := sqldb.QueryRowContext(ctx, "SELECT to_regclass('goose_db_version')::text").Scan(&existing); err != nil { + t.Fatal(err) + } + if existing.Valid { + t.Fatal("sequence regression requires a fresh database; refusing an existing migration history") + } + goose.SetBaseFS(migrationsFS) + if err := goose.SetDialect("postgres"); err != nil { + t.Fatal(err) + } + var userID, fileID uuid.UUID + for version := 23; version <= head; version++ { + // Match production's strict Goose behavior: no AllowMissing option. + if err := goose.UpToContext(ctx, sqldb, "migrations", int64(version)); err != nil { + t.Fatalf("upgrade to %d: %v", version, err) + } + actual, err := goose.GetDBVersionContext(ctx, sqldb) + if err != nil || actual != int64(version) { + t.Fatalf("migration head = %d, want %d, err=%v", actual, version, err) + } + var applied int + if err := sqldb.QueryRowContext(ctx, "SELECT count(DISTINCT version_id) FROM goose_db_version WHERE is_applied AND version_id BETWEEN 1 AND $1", version).Scan(&applied); err != nil || applied != version { + t.Fatalf("applied history has %d of %d predecessors, err=%v", applied, version, err) + } + if version == 23 { + if err := sqldb.QueryRowContext(ctx, "INSERT INTO users(email,password_hash) VALUES($1,'test') RETURNING id", uuid.NewString()+"@example.test").Scan(&userID); err != nil { + t.Fatal(err) + } + if err := sqldb.QueryRowContext(ctx, "INSERT INTO files(user_id,name,path,size,sha256,mime,storage_key) VALUES($1,'migration-sequence.txt','/fixture',0,'fixture','text/plain','test://sequence') RETURNING id", userID).Scan(&fileID); err != nil { + t.Fatal(err) + } + if _, err := sqldb.ExecContext(ctx, "INSERT INTO embeddings_text(file_id,chunk_index,chunk_text,embedding) SELECT $1,0,'populated duplicate',array_fill(0.1::real,ARRAY[768])::vector FROM generate_series(1,2)", fileID); err != nil { + t.Fatal(err) + } + } + if version >= 24 { + var lexical bool + if err := sqldb.QueryRowContext(ctx, "SELECT search_tsv @@ plainto_tsquery('simple','migration-sequence.txt') FROM files WHERE id=$1", fileID).Scan(&lexical); err != nil || !lexical { + t.Fatalf("populated lexical backfill: %v, err=%v", lexical, err) + } + } + if version >= 25 { + var indexes int + if err := sqldb.QueryRowContext(ctx, `SELECT count(*) FROM pg_index i JOIN pg_class c ON c.oid=i.indexrelid JOIN pg_am am ON am.oid=c.relam + WHERE i.indisvalid AND am.amname='hnsw' AND c.relname IN + ('idx_embeddings_text_embedding_hnsw','idx_embeddings_visual_embedding_hnsw','idx_embeddings_face_embedding_hnsw')`).Scan(&indexes); err != nil || indexes != 3 { + t.Fatalf("valid HNSW indexes=%d, err=%v", indexes, err) + } + } + var chunks int + wantChunks := 2 + if version >= 26 { + wantChunks = 1 + } + if err := sqldb.QueryRowContext(ctx, "SELECT count(*) FROM embeddings_text WHERE file_id=$1", fileID).Scan(&chunks); err != nil || chunks != wantChunks { + t.Fatalf("preserved chunks=%d, want=%d, err=%v", chunks, wantChunks, err) + } + t.Logf("PASS: strict Goose upgrade to %d; complete history and populated data preserved", version) + } + if head >= 26 { + _, err := sqldb.ExecContext(ctx, "INSERT INTO embeddings_text(file_id,chunk_index,chunk_text) VALUES($1,0,'duplicate')", fileID) + var pgErr *pgconn.PgError + if !errors.As(err, &pgErr) || pgErr.Code != "23505" { + t.Fatalf("expected duplicate rejection 23505, got %v", err) + } + } + // The real startup path must accept the upgraded history unchanged. + if err := (&DB{url: dsn}).Migrate(ctx); err != nil { + t.Fatalf("production startup after sequential upgrade: %v", err) + } +} diff --git a/server/internal/db/migrations/0024_files_lexical_search.sql b/server/internal/db/migrations/0024_files_lexical_search.sql new file mode 100644 index 0000000..f6a46c2 --- /dev/null +++ b/server/internal/db/migrations/0024_files_lexical_search.sql @@ -0,0 +1,26 @@ +-- +goose Up +-- Model-free lexical lane for the file corpus. Mirrors the FTS + trigram +-- shape already established for memories (0008) so that filename (name column) +-- substring search works without an embedding worker. + +-- +goose StatementBegin +ALTER TABLE files + ADD COLUMN IF NOT EXISTS search_tsv tsvector GENERATED ALWAYS AS ( + to_tsvector('simple', coalesce(name, '')) + ) STORED; +-- +goose StatementEnd + +-- +goose StatementBegin +CREATE INDEX IF NOT EXISTS idx_files_search_tsv + ON files USING gin (search_tsv); +-- +goose StatementEnd + +-- +goose StatementBegin +CREATE INDEX IF NOT EXISTS idx_files_name_trgm + ON files USING gin (lower(name) gin_trgm_ops); +-- +goose StatementEnd + +-- +goose Down +DROP INDEX IF EXISTS idx_files_name_trgm; +DROP INDEX IF EXISTS idx_files_search_tsv; +ALTER TABLE files DROP COLUMN IF EXISTS search_tsv; diff --git a/server/internal/search/lexical_test.go b/server/internal/search/lexical_test.go new file mode 100644 index 0000000..1453459 --- /dev/null +++ b/server/internal/search/lexical_test.go @@ -0,0 +1,244 @@ +package search + +import ( + "context" + "os" + "strings" + "testing" + "time" + + "github.com/google/uuid" + "github.com/jackc/pgx/v5/pgxpool" + + memdb "github.com/PeterGuy326/mem/server/internal/db" +) + +func TestLexicalSearchWithoutWorker(t *testing.T) { + dsn := os.Getenv("MEM_TEST_DB") + if dsn == "" { + t.Skip("MEM_TEST_DB not set; skipping lexical search PostgreSQL test") + } + config, err := pgxpool.ParseConfig(dsn) + if err != nil { + t.Fatalf("parse MEM_TEST_DB: %v", err) + } + if !strings.HasSuffix(config.ConnConfig.Database, "_test") { + t.Fatalf( + "refusing to modify non-test database %q; MEM_TEST_DB must end in _test", + config.ConnConfig.Database, + ) + } + ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second) + defer cancel() + database, err := memdb.Open(ctx, dsn) + if err != nil { + t.Fatal(err) + } + t.Cleanup(database.Close) + if err := database.Migrate(ctx); err != nil { + t.Fatal(err) + } + + userID := uuid.New() + if _, err := database.Pool.Exec(ctx, ` + INSERT INTO users (id, email, password_hash) + VALUES ($1, $2, 'test-only') + `, userID, "lexical-"+uuid.NewString()+"@example.test"); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cleanupCancel() + database.Pool.Exec(cleanupCtx, `DELETE FROM users WHERE id = $1`, userID) + }) + + now := time.Now().UTC().Truncate(time.Second) + files := []struct { + name string + path string + mime string + }{ + {"quarterly_report.pdf", "/Work/Reports", "application/pdf"}, + {"meeting_notes.md", "/Work/Notes", "text/markdown"}, + {"meeting_notes.md", "/Work/NotesExtra", "text/markdown"}, + {"literal_scope.txt", "/Work/100%_done", "text/plain"}, + {"literal_scope.txt", "/Work/100XXdone", "text/plain"}, + {"photo_beach.jpg", "/Photos", "image/jpeg"}, + {"年度总结.docx", "/Work", "application/vnd.openxmlformats-officedocument.wordprocessingml.document"}, + } + for _, f := range files { + if _, err := database.Pool.Exec(ctx, ` + INSERT INTO files ( + user_id, name, path, size, sha256, mime, storage_key, + index_status, created_at, updated_at + ) VALUES ( + $1, $2, $3, 1, $4, $5, $6, + 'ready', $7, $7 + ) + `, userID, f.name, f.path, strings.Repeat("b", 64), f.mime, + "test/"+uuid.NewString(), now); err != nil { + t.Fatal(err) + } + } + + // Service with nil worker — the key precondition for this test. + service := New(database.Pool, nil) + + t.Run("lexical route works without worker", func(t *testing.T) { + hits, err := service.Search(ctx, Query{ + UserID: userID, + Text: "report", + Route: RouteLexical, + Limit: 10, + }) + if err != nil { + t.Fatalf("lexical search failed: %v", err) + } + if len(hits) == 0 { + t.Fatal("lexical search returned no results for 'report'") + } + found := false + for _, h := range hits { + if h.Name == "quarterly_report.pdf" { + found = true + if h.Source != RouteLexical { + t.Errorf("hit source = %q, want %q", h.Source, RouteLexical) + } + break + } + } + if !found { + t.Errorf("expected quarterly_report.pdf in results, got %+v", hits) + } + }) + + t.Run("text route fails without worker", func(t *testing.T) { + _, err := service.Search(ctx, Query{ + UserID: userID, + Text: "report", + Route: RouteText, + Limit: 10, + }) + if err == nil { + t.Fatal("text route should fail without worker") + } + if !strings.Contains(err.Error(), "worker not configured") { + t.Fatalf("unexpected error: %v", err) + } + }) + + t.Run("auto route fails without worker", func(t *testing.T) { + _, err := service.Search(ctx, Query{ + UserID: userID, + Text: "report", + Route: RouteAuto, + Limit: 10, + }) + if err == nil { + t.Fatal("auto route should fail without worker") + } + if !strings.Contains(err.Error(), "worker not configured") { + t.Fatalf("unexpected error: %v", err) + } + }) + + t.Run("substring match for CJK filename", func(t *testing.T) { + hits, err := service.Search(ctx, Query{ + UserID: userID, + Text: "总结", + Route: RouteLexical, + Limit: 10, + }) + if err != nil { + t.Fatalf("lexical CJK search failed: %v", err) + } + if len(hits) == 0 { + t.Fatal("lexical search returned no results for CJK query '总结'") + } + }) + + t.Run("path filter applies to lexical", func(t *testing.T) { + hits, err := service.Search(ctx, Query{ + UserID: userID, + Text: "notes", + Route: RouteLexical, + PathPrefix: "/Work/Notes", + Limit: 10, + }) + if err != nil { + t.Fatalf("lexical path-filtered search failed: %v", err) + } + if len(hits) != 1 || hits[0].Name != "meeting_notes.md" || hits[0].Path != "/Work/Notes" { + t.Fatalf("path filter should return only the matching subtree: %+v", hits) + } + }) + t.Run("trigram typo retains weak matches without similarity prefilter", func(t *testing.T) { + hits, err := service.Search(ctx, Query{UserID: userID, Route: RouteLexical, Text: "quaterly", Limit: 10}) + if err != nil { + t.Fatal(err) + } + for _, hit := range hits { + if hit.Name == "quarterly_report.pdf" { + if hit.Score < 0.20 || hit.Score >= 0.70 { + t.Fatalf("typo should be ranked in the trigram tier: %+v", hit) + } + return + } + } + t.Fatalf("trigram-only typo was dropped: %+v", hits) + }) + + t.Run("literal authorized subtree", func(t *testing.T) { + hits, err := service.Search(ctx, Query{ + UserID: userID, Route: RouteLexical, Text: "literal_scope", Limit: 10, + AllowedPaths: []string{"/Work/100%_done"}, + }) + if err != nil { + t.Fatal(err) + } + if len(hits) != 1 || hits[0].Path != "/Work/100%_done" { + t.Fatalf("literal authorization scope leaked or lost results: %+v", hits) + } + }) + + t.Run("MIME and time filters", func(t *testing.T) { + hits, err := service.Search(ctx, Query{UserID: userID, Route: RouteLexical, Text: "beach", Type: "image"}) + if err != nil || len(hits) != 1 || hits[0].Name != "photo_beach.jpg" { + t.Fatalf("image filter: hits=%+v err=%v", hits, err) + } + after := now.Add(time.Second) + before := now.Add(-time.Second) + for _, q := range []Query{ + {Type: "audio"}, {Since: &after}, {Until: &before}, + {AllowedPaths: []string{"/Private"}}, + {AllowedPaths: []string{""}}, + } { + q.UserID, q.Route, q.Text = userID, RouteLexical, "beach" + hits, err := service.Search(ctx, q) + if err != nil || len(hits) != 0 { + t.Fatalf("filter %+v: hits=%+v err=%v", q, hits, err) + } + } + }) + + t.Run("other owner cannot retrieve files", func(t *testing.T) { + hits, err := service.Search(ctx, Query{UserID: uuid.New(), Route: RouteLexical, Text: "quarterly_report"}) + if err != nil || len(hits) != 0 { + t.Fatalf("other-owner search: hits=%+v err=%v", hits, err) + } + }) + +} + +// Query validation belongs to the exported service entry point and must happen +// before any database or worker access, including the model-free route. +func TestLexicalSearchRejectsEmptyQuery(t *testing.T) { + for _, text := range []string{"", " ", "\n\t"} { + _, err := New(nil, nil).Search(context.Background(), Query{ + UserID: uuid.New(), Route: RouteLexical, Text: text, + }) + if err == nil || !strings.Contains(err.Error(), "query is empty") { + t.Fatalf("empty query %q returned %v", text, err) + } + } +} diff --git a/server/internal/search/search.go b/server/internal/search/search.go index c6ff4d0..5f0152a 100644 --- a/server/internal/search/search.go +++ b/server/internal/search/search.go @@ -149,10 +149,12 @@ var ErrReplayReferenceUnavailable = errors.New("managed embedding replay referen // "text" -> ANN over embeddings_text (Ollama / OpenAI text embedder) // "visual" -> ANN over embeddings_visual via CLIP text encoder // "auto" -> both routes in parallel, merged + deduped by file_id (default) +// "lexical" -> model-free FTS + trigram over files.name (no worker needed) const ( - RouteText = "text" - RouteVisual = "visual" - RouteAuto = "auto" + RouteText = "text" + RouteVisual = "visual" + RouteAuto = "auto" + RouteLexical = "lexical" ) // Hit is one search result. @@ -217,8 +219,10 @@ func (s *Service) Search(ctx context.Context, q Query) ([]Hit, error) { if q.SnippetChars > 16_000 { q.SnippetChars = 16_000 } - if s.worker == nil || !s.worker.Enabled() { - return nil, fmt.Errorf("search disabled: worker not configured") + if q.Route != RouteLexical { + if s.worker == nil || !s.worker.Enabled() { + return nil, fmt.Errorf("search disabled: worker not configured") + } } switch q.Route { @@ -228,8 +232,10 @@ func (s *Service) Search(ctx context.Context, q Query) ([]Hit, error) { return s.searchVisual(ctx, q, text) case "", RouteAuto: return s.searchAuto(ctx, q, text) + case RouteLexical: + return s.searchLexical(ctx, q, text) default: - return nil, fmt.Errorf("unknown route %q (expected text|visual|auto)", q.Route) + return nil, fmt.Errorf("unknown route %q (expected text|visual|auto|lexical)", q.Route) } } @@ -715,6 +721,77 @@ func (s *Service) runVisualANN(ctx context.Context, q Query, vec []float32) ([]H return s.scanHits(ctx, sql, args, RouteVisual, q.SnippetChars) } +// searchLexical is the model-free file recall lane. It uses the same +// three-tier shape as memory Recall (name substring → FTS → trigram) so a +// deployment with no worker can still find files by name. +func (s *Service) searchLexical(ctx context.Context, q Query, text string) ([]Hit, error) { + // Search rejects empty queries before dispatch. Keep the numeric trigram + // threshold here: a pg_trgm % prefilter uses a different similarity measure + // and would discard valid word_similarity matches. This query scores the + // filtered file set; index-backed candidate pruning is a separate change. + args := []any{q.UserID} + where := []string{"f.user_id = $1"} + args, where = appendPathFilters(args, where, q.PathPrefix, q.AllowedPaths) + args, where = appendMIMEFilter(args, where, q.Type) + if q.Since != nil { + args = append(args, *q.Since) + where = append(where, fmt.Sprintf("COALESCE(f.timeline_at, f.created_at) >= $%d", len(args))) + } + if q.Until != nil { + args = append(args, *q.Until) + where = append(where, fmt.Sprintf("COALESCE(f.timeline_at, f.created_at) <= $%d", len(args))) + } + args = append(args, text) + textArg := len(args) + args = append(args, q.Limit) + limitArg := len(args) + + sql := fmt.Sprintf(` + WITH candidates AS ( + SELECT f.id AS file_id, f.name, f.path, f.mime, f.sha256, + f.summary, f.timeline_at, f.created_at, + strpos(lower(f.name), lower($%d)) > 0 AS name_contains, + f.search_tsv @@ plainto_tsquery('simple', $%d) AS fts_match, + ts_rank_cd( + f.search_tsv, + plainto_tsquery('simple', $%d) + )::double precision AS fts_rank, + word_similarity( + lower($%d), + lower(f.name) + )::double precision AS trigram_score + FROM files f + WHERE %s + ), + ranked AS ( + SELECT candidates.*, + CASE + WHEN name_contains THEN 1.0::double precision + WHEN fts_match THEN LEAST( + 0.949::double precision, + 0.70::double precision + 0.24::double precision * fts_rank + ) + ELSE LEAST( + 0.699::double precision, + 0.20::double precision + 0.49::double precision * trigram_score + ) + END AS score + FROM candidates + WHERE name_contains + OR fts_match + OR trigram_score >= 0.12 + ) + SELECT 'lexical:' || r.file_id::text, r.file_id, r.name, r.path, r.mime, + r.sha256, -1, r.score::real, r.name, r.summary, + r.timeline_at, r.created_at + FROM ranked r + ORDER BY r.score DESC, r.created_at DESC, r.file_id + LIMIT $%d + `, textArg, textArg, textArg, textArg, strings.Join(where, " AND "), limitArg) + + return s.scanHits(ctx, sql, args, RouteLexical, q.SnippetChars) +} + // scanHits is the common cursor → []Hit loop. Tags every hit with its source route. func (s *Service) scanHits(ctx context.Context, sql string, args []any, route string, snippetChars int) ([]Hit, error) { rows, err := s.pool.Query(ctx, sql, args...) diff --git a/server/internal/tools/builtin/builtin.go b/server/internal/tools/builtin/builtin.go index 9c760e0..552347f 100644 --- a/server/internal/tools/builtin/builtin.go +++ b/server/internal/tools/builtin/builtin.go @@ -475,7 +475,7 @@ func registerSearch(reg *tools.Registry, c *apiclient.Client) error { "query": {Type: "string", Description: "Free-form natural-language query, e.g. \"2012 photos with Xiao Ming\""}, "scope": {Type: "string", Description: "Optional virtual-folder scope, e.g. /Projects/mem"}, "type": {Type: "string", Description: "MIME prefix filter: image|text|application|audio|video"}, - "route": {Type: "string", Description: "Search route: text|visual|auto (default auto fuses both)", Enum: []string{"text", "visual", "auto"}}, + "route": {Type: "string", Description: "Search route: text|visual|auto|lexical (default auto fuses text and visual; lexical needs no model)", Enum: []string{"text", "visual", "auto", "lexical"}}, "since": {Type: "string", Description: "YYYY-MM-DD lower bound on timeline_at"}, "until": {Type: "string", Description: "YYYY-MM-DD upper bound on timeline_at"}, "limit": {Type: "integer", Description: "Max results (default 10, max 100)", Default: 10},