How well does a search find what a person means? npm run bench measures it: six public repositories pinned by
commit, 372 questions written the way people type them, each asked four ways.
npm run bench:fetch # the repositories, into ~/.cache/semantic-grep-bench/repos
npm run bench -- --profile words # no model
npm run bench -- --profile laptop --models /path/to/gguf --llama-server /path/to/llama-server
CUDA_VISIBLE_DEVICES=0 npm run bench -- --profile beefy --models /path/to/gguf --compare bench/results/laptop-2026-11-03.jsonEach question is asked as the plugin asks it:
| mode | what | in the picker |
|---|---|---|
words |
SQLite FTS5, BM25: pieces with every word of the question (stop words left out) | on every key |
meaning |
the question's vector against every piece's, cosine | after the pause |
fused |
both lists fused by reciprocal rank | :SemanticGrep with vim.ui.select, the library's search |
rerank |
the fused list, its first pieces reordered by the profile's reranker | once the list settles |
A result answers a question when it is in the labelled file and its lines overlap the labelled range (or one of the
question's also places). For each mode: recall@1/5/10 (the share of questions answered within the first 1, 5,
10 results), MRR (the mean of 1/rank of the first answer, 0 past 30 results) and the p50/p95 latency of the
whole search in milliseconds, including embedding the question and the rerank. A rerank the model could not give
(it timed out, failed, or is resting a minute after a failure) keeps the fused order, as in the picker, and counts
in rerank all the same; the run says how many were applied (reranks:). The local profiles give the reranker 20 s
where the plugin's default is 3 s, so that the rerank is measured, not its timeout: its p95 says whether it fits a
picker's wait, and the timeoutMs to set in setup(). Per repository, overall, and by the
style of question. Each run is saved as JSON (bench/results/<profile>-<date>.json, with every question's rank and
time; a run saved under its default name never overwrites an earlier one), and --compare FILE prints the change against an earlier one, and
whether it asked the same questions.
Flags: --repos hono,flask, --split dev|test, --modes words,fused, --questions FILE, --out FILE, --fresh
(index again; otherwise a run reuses the index and vectors in the cache), --parsers none (cut pieces by indentation, not
Neovim's tree-sitter), --profiles FILE (your own profiles), --cache DIR.
:SemanticGrep files lists files, not pieces, so it is scored by files too: a question is answered by its labelled file
(or a file in its also), anywhere in it. Each mode is asked as the files view asks it (Meaning.files), and the
saved run has the same numbers for the files (files, and fileModes per repository; fileRank per question):
| mode | the pieces a file is ranked by |
|---|---|
words |
the 60 with any of the question's words, by BM25 (a sentence's words are seldom all in one piece) |
meaning |
the 60 closest in meaning, not cut by the model's band |
fused |
both lists fused by reciprocal rank: meaning's, and the pieces with every word, as the pieces view fuses them |
rerank |
the fused 60, their first reordered by the profile's reranker: what the files view does when there is one |
A file scores by its pieces among those 60: the one at rank r is worth 1/(5 + r); its best piece counts in full,
its next one half, the one after a quarter, and so on: the best piece first, and a small bonus for several
(filesOf in src/meaning.mjs). The 5, the half and the 60 were chosen on the dev questions (121) with the laptop's
model, from a grid of 1/(k + r) for k 0 to 60, the next piece's share 0 to 0.5, 30 to 100 pieces, and meaning's
band from the model's to none:
on dev, fused |
R@1 | R@5 | R@10 | MRR |
|---|---|---|---|---|
| the files of the pieces list, each where it first comes | 0.645 | 0.901 | 0.950 | 0.758 |
| the best piece alone (the share 0) | 0.653 | 0.917 | 0.975 | 0.769 |
| k 60, a tenth of the next | 0.653 | 0.942 | 0.975 | 0.762 |
| k 5, half the next (chosen) | 0.694 | 0.942 | 0.975 | 0.787 |
| the same with meaning cut at the model's band (0.06) | 0.686 | 0.917 | 0.950 | 0.778 |
For the words alone, any word rather than every word took file recall@5 from 0.18 to 0.86 on dev; fusing meaning
with the any-word list instead of the every-word one was a little worse (0.934 against 0.942), so the fused list keeps
the picker's.
Results on the test split (not tuned on), 2026-11-03: words and
laptop over all six repositories (251 questions), beefy over hono,
cobra and fd (113 questions; the three the rerankers were
chosen on, as its 8B model had to run on the CPU here). cloud has no numbers yet, for want of a key (see Profiles),
nor the laptop with a reranker: none fits its wait (profiles.md).
| profile, mode | file R@1 | R@5 | R@10 | MRR | p50 ms | p95 ms |
|---|---|---|---|---|---|---|
| words | 0.558 | 0.892 | 0.960 | 0.702 | 0.9 | 2.7 |
| laptop meaning | 0.622 | 0.916 | 0.972 | 0.745 | 17 | 23 |
| laptop fused (the default) | 0.637 | 0.912 | 0.968 | 0.757 | 18 | 23 |
| beefy words | 0.558 | 0.920 | 0.991 | 0.719 | 1.4 | 3.6 |
| beefy meaning | 0.673 | 0.947 | 0.991 | 0.799 | 223 | 282 |
| beefy fused | 0.681 | 0.973 | 0.991 | 0.812 | 221 | 292 |
| beefy rerank | 0.717 | 0.973 | 0.991 | 0.833 | 370 | 450 |
File recall@5 per repository (test):
| hono | telescope | flask | cobra | fd | vue-docs | |
|---|---|---|---|---|---|---|
| words | 0.964 | 0.762 | 0.885 | 0.932 | 0.878 | 0.955 |
| laptop fused | 0.893 | 0.810 | 0.846 | 0.977 | 0.976 | 0.977 |
| beefy fused | 1.000 | 0.955 | 0.976 |
- The target is met: the default (no setup: the laptop's EmbeddingGemma 2 on 4 CPU threads, no reranker) has the
right file among the first five for 91% of the
testquestions (recall@5 0.912), in 23 ms at p95, against the 0.85 and 300 ms asked. A first index costs more than a search: about 108 s a thousand pieces on 4 threads (profiles.md). - Words find files better than pieces: a file has many pieces, and any of a sentence's words in them points at it:
0.892 at recall@5 without any model, against 0.131 for the words' pieces. Meaning still adds most at the top
(recall@1 0.558 to 0.637). On hono and flask the words alone do better at recall@5 than fused; fusing meaning with
the any-word list instead was worse on
dev, so that is left for more questions to decide. - The rerank adds a little for files, at the top: beefy's reranker lifts the first file (recall@1 0.681 to 0.717, MRR 0.812 to 0.833) and leaves recall@5 where it was (0.973): a file's several pieces already say much of what a reranked piece would. It stays off on the laptop, where it would cost 2 s a search.
- beefy's latency here is not its own: the GPUs were taken by other work during this run, so its Qwen3-Embedding-8B ran on 8 CPU threads (the reranker on a GPU); its quality is the same as on a GPU, and on one its search took 114 / 215 ms (profiles.md).
bench/corpus.json, fetched by bench/fetch.mjs (one shallow fetch of each commit, not
vendored):
| name | repository | language | files | pieces |
|---|---|---|---|---|
| hono | honojs/hono | TypeScript | 572 | 4186 |
| telescope | nvim-telescope/telescope.nvim | Lua | 103 | 1558 |
| flask | pallets/flask | Python (and its docs) | 227 | 1589 |
| cobra | spf13/cobra | Go | 64 | 1021 |
| fd | sharkdp/fd | Rust | 57 | 871 |
| vue-docs | vuejs/docs | Markdown: documentation | 437 | 1902 |
Pieces are cut with the tree-sitter parsers of the Neovim on the machine (50 here), as the plugin cuts them; with others the pieces differ a little, the labels (lines) do not.
bench/questions.jsonl: 372 questions, one JSON line each:
{"id": "flask-025", "repo": "flask", "query": "env var values parsed as json for config", "file": "docs/config.rst", "from": 603, "to": 606, "style": "vague", "split": "test"}22 pieces of each repository were drawn at random (seeded; at most a sixth of them tests, one per file; no lockfiles,
changelogs or data). For each, Claude read the piece in its file and wrote 3 to 5 searches a developer would type to
land there, without its identifiers, file names or literal strings, and labelled each with the lines that answer it
(narrowed where only part of the piece does, with other places that answer as well in also). Pieces nobody would
search for (bare imports, a licence header, a frontmatter, a benchmark file the same as its siblings) were left out:
16 of 132. Styles: vague (short, typed before knowing the names: "fake current time in tests"), plain (a plain
description: "error when more than one flag from an exclusive group is set") and multi-hop (a purpose with a context,
or a symptom: "why does my completion still show descriptions I disabled with an environment variable").
A third of the pieces, with all their questions, are dev (121 questions) and the rest test (251): tune on
--split dev, report --split test. The pieces were drawn before anything was tuned on them.
A sample of 37 questions (one in ten, seeded) was read by hand against its labelled lines: all are realistic
searches and answered by their label; one (telescope-037, "picker listing ctags entries…") is labelled with a block
that holds that picker among two others, a few name the thing tested ("test that …") rather than the code, and a
handful of thin pieces (an old helper kept for compatibility, a link-only docs section) are easy to miss for any
search. Recall here is a floor: a search can find a right answer the labels do not list. Some questions use the
plain word a file is named by ("config", "yaml", "computed": 38 of 372), and one names an identifier (hono-003,
"createElement"); words alone owe some of their vague score to them.
The plugin keeps each search you made and the piece you opened for it (see the README, "It learns"). Exported, they are a private questions file, outside this repository and outside the project, and run through the same bench:
npm run bench:export -- --root ~/code/app # ~/.cache/semantic-grep-bench/private/app-picks.jsonl
npm run bench -- --profile laptop --questions ~/.cache/semantic-grep-bench/private/app-picks.jsonlrepo is then the project's path, and the run is saved in the cache, not in bench/results. The bench indexes the
project afresh in its own data folder, so what the plugin learned does not answer for it. These are searches whose
piece was found (else you would not have opened it): they tell whether what works keeps working, not what is missed.
bench/profiles.json; each entry is the plugin's embedding and rerank config, so a
profile that measures well is one to copy into setup() (with the timeoutMs its p95 asks for). A local profile's serve says how to start its models:
when nothing answers at its baseUrl, the bench starts llama-server (--llama-server, $LLAMA_SERVER or the one
on the PATH) with the GGUF found under --models (or $SEMANTIC_GREP_BENCH_MODELS), at nice 19, and stops it at
the end. A server already answering there is used as it is.
- words: no model.
- laptop: CPU only (llama.cpp with
CUDA_VISIBLE_DEVICES="", 4 threads), about 16 GB of RAM: EmbeddingGemma 2 Q8_0, no reranker (see profiles.md). laptop-d1 is the one before: EmbeddingGemma 300M and the d1-2B decision model (system-one, top 15), the baseline below. - beefy: one GPU of 16 GB, about 10 GB of it: Qwen3-Embedding-8B
Q5_K_M and the Qwen3-Reranker-0.6B cross-encoder
(
rerank,llama-server --rerank, top 20, reading 1200 characters of each piece). Choose the card withCUDA_VISIBLE_DEVICES. beefy-4b is the one before (Qwen3-Embedding-4B, the reranker at top 30), the baseline. - cloud: Voyage
voyage-code-3embeddings andrerank-2.5, the key in$VOYAGE_API_KEY. Voyage's/v1/embeddingstakes OpenAI's shape; the prompts it adds forinput_typeare set asqueryPrefix/documentPrefix. Also cloud-openai (text-embedding-3-largeat 1024 dimensions, Coherererank-v3.5) and cloud-gemini (gemini-embedding-001through Gemini's OpenAI-compatible endpoint, Jina's reranker). A key is named, never written:apiKeyEnv. - openrouter: everything through OpenRouter, the key in
$OPENROUTER_API_KEY:qwen/qwen3-embedding-8bembeddings and Voyage'smongodb/rerank-3(top 10, 800 characters). Run it withOPENROUTER_API_KEY=… npm run bench -- --profile openrouter.
The cloud profiles are configured, checked as valid configs with their keys named by environment variable
(test/bench.test.mjs), and the /rerank answers of llama.cpp, Cohere, Jina and Voyage are tested against fakes
(test/rerank.test.mjs); embeddings use the same OpenAI-shaped /v1/embeddings the local profiles do. They have not
been measured yet, for want of a key. With one:
VOYAGE_API_KEY=… npm run bench -- --profile cloud, and add the numbers below.
The openrouter profile is the one hosted profile measured so far: on hono's 28 test questions its fused list has
MRR 0.671 against beefy's 0.638, and its reranked list 0.780 against 0.645
(run: 20 of the 28 reranked before a
burst met OpenRouter's tokens-a-minute limit and rested the model; the others kept the fused order). Voyage's
voyage-code-4 through OpenRouter, with the input_type it needs, was measured on fd's 41 test questions and not
kept: pieces fused MRR 0.303 and files 0.666, against beefy's 0.656 and 0.868
(run); without input_type it is 0.259.
See profiles.md.
--notes WRITER asks the questions with a note on every piece, written by that writer of
bench/writers.json as the plugin's notes writes them (cached by piece, in
<cache>/notes/), and saves what writing them took (notes: time, tokens and dollars a thousand pieces). On the
test questions of flask, cobra and fd (137), with the laptop's embedding model, fused MRR of the pieces / of the
files, 2026-11-03 (runs; docs/notes.md has the rest):
| notes | pieces | files | a thousand pieces |
|---|---|---|---|
| none | 0.544 | 0.725 | |
| Qwen3.5-2B (the laptop's) | 0.582 | 0.729 | 28 min on 4 CPU threads |
| Qwen3-4B-Instruct-2507 (beefy's¹) | 0.571 | 0.753 | 3.4 min on a GPU |
| Qwen3-30B-A3B (dropped) | 0.538 | 0.721 | $0.04 |
| Claude Haiku 5.5 (the cloud's) | 0.605 | 0.760 | $0.10 |
| Claude Sonnet 5.5 | 0.633 | 0.773 | $1.92 |
¹ Measured with the laptop's embedding model only, as every row here; with the beefy profile's own models (Qwen3-Embedding-8B, Qwen3-Reranker-0.6B) the notes' gain is not measured (notes.md).
The baseline, 2026-11-03, on this change's code at its commit 2f455a2 with nothing uncommitted (the commits after it
add only what a run records: these runs have no servers field yet) (Node 25.2, 50 tree-sitter parsers);
laptop and beefy (now laptop-d1 and beefy-4b) built their indexes afresh and started their own llama.cpp servers from the profiles' serve
entries (EmbeddingGemma 300M Q8_0 and d1-2B Q8_0 on the CPU; Qwen3-Embedding-4B Q8_0 and Qwen3-Reranker-0.6B Q8_0 on
one GPU); the runs, with every question's rank and time, are
words, laptop and
beefy. Machine: 12 cores, 125 GB, two RTX 5060 Ti 16 GB, shared with other
work; one run at a time, every model at nice 19. The words' latency is the words run's; within the other runs it is
1 to 2 ms, the machine busy with their models.
Overall, 372 questions:
| R@1 | R@5 | R@10 | MRR | p50 ms | p95 ms | |
|---|---|---|---|---|---|---|
words (the words run) |
0.097 | 0.140 | 0.142 | 0.115 | 0.2 | 0.7 |
| laptop meaning | 0.446 | 0.755 | 0.825 | 0.575 | 14 | 18 |
| laptop fused | 0.452 | 0.761 | 0.831 | 0.582 | 14 | 18 |
| laptop rerank (d1-2B, CPU) | 0.511 | 0.782 | 0.831 | 0.625 | 2633 | 4497 |
| beefy meaning | 0.460 | 0.766 | 0.820 | 0.585 | 17 | 21 |
| beefy fused | 0.484 | 0.774 | 0.831 | 0.605 | 18 | 23 |
| beefy rerank (Qwen3-Reranker-0.6B, GPU) | 0.489 | 0.812 | 0.855 | 0.625 | 84 | 261 |
MRR on the test split alone (251 questions; dev in brackets, 121):
| words | meaning | fused | rerank | |
|---|---|---|---|---|
| laptop | 0.105 (0.136) | 0.600 (0.523) | 0.606 (0.533) | 0.622 (0.630) |
| beefy | 0.105 (0.136) | 0.593 (0.567) | 0.610 (0.593) | 0.630 (0.615) |
MRR per repository:
| hono | telescope | flask | cobra | fd | vue-docs | |
|---|---|---|---|---|---|---|
| words | 0.111 | 0.079 | 0.134 | 0.158 | 0.047 | 0.168 |
| laptop meaning | 0.609 | 0.488 | 0.524 | 0.580 | 0.576 | 0.692 |
| laptop fused | 0.590 | 0.500 | 0.544 | 0.599 | 0.573 | 0.701 |
| laptop rerank | 0.672 | 0.539 | 0.605 | 0.605 | 0.653 | 0.695 |
| beefy meaning | 0.524 | 0.516 | 0.522 | 0.605 | 0.598 | 0.739 |
| beefy fused | 0.565 | 0.540 | 0.567 | 0.617 | 0.589 | 0.748 |
| beefy rerank | 0.556 | 0.563 | 0.548 | 0.688 | 0.652 | 0.741 |
MRR by style (vague 119, plain 137, multi-hop 116 questions):
| vague | plain | multi-hop | |
|---|---|---|---|
| words | 0.206 | 0.119 | 0.017 |
| laptop fused | 0.628 | 0.676 | 0.426 |
| laptop rerank | 0.682 | 0.714 | 0.461 |
| beefy fused | 0.627 | 0.687 | 0.484 |
| beefy rerank | 0.641 | 0.716 | 0.502 |
A first build of all six (11,127 pieces): 16.6 minutes on 4 CPU threads (EmbeddingGemma 300M), 5.4 minutes on the GPU (Qwen3-Embedding-4B); four pieces a request either way.
What it says:
- Words alone find little of what people type: recall@10 0.14, and 0.017 on multi-hop questions. A piece must have every word of the question, and people's words are rarely the code's. In the picker the words are what shows on every key; meaning is what finds.
- Meaning does the work, and the small model holds up: EmbeddingGemma 300M on a CPU (MRR 0.575) is within 0.01 of Qwen3-Embedding-4B on a GPU (0.585); fusing the words adds 0.01 to 0.02.
- The rerank helps, at a price: d1-2B lifts MRR from 0.582 to 0.625 and recall@1 from 0.45 to 0.51, but takes
2.6 s (p50) to 4.5 s (p95) on 4 CPU threads, past the plugin's default
timeoutMsof 3 s; 104 of its 372 answers were flat (no piece stood out), and the search's order was kept. The cross-encoder on a GPU gives the same MRR in 84 ms (p95 261 ms), and adds most to recall@5 (0.774 to 0.812). - A cross-encoder needs to read the piece: Qwen3-Reranker-0.6B reading 160 characters of each piece made the list
worse; tried on the
devquestions in one-off runs (not kept), its MRR was 0.488 at 160 characters, 0.595 at 500 and 0.615 at 1200, against the fused list's 0.593. 1200 is the default forapi = 'rerank'now (charsin the entry changes it); thetestsplit, not tuned on, shows the same gain (0.610 to 0.630). - Multi-hop questions are the hardest for every mode (fused MRR 0.43 to 0.48 against 0.63 to 0.69).
- 21 to 24 questions had nothing to rerank: meaning keeps only the pieces within 0.1 of the best, and those lists were one piece long (a right one for all but 3 or 4 of them).
Not measured yet: the cloud profiles (no key). Every candidate model since, and the profiles that came of it, are in profiles.md.