vbcdd-text-search

A CLI indexer and search tool for a directory of HTML, Markdown, and text files. It builds an SQLite full-text index, optionally adds embedding vectors from a local llama-server, and fuses the two result lists for natural language questions.

Warning: Large Language Model-Generated Code

This code functions as of the time of writing, yet it has received little to no human review. It must not be regarded as reliable: bugs and stability issues are to be expected.

The tool exists to search the offline Arch Wiki without the MediaWiki navigation chrome, but the root profiles make it general: any directory of supported files can be indexed.

How it works

Two cache tiers keep the expensive work incremental.

1. Mirror. Each source file is extracted into an ordered list of heading and paragraph blocks, stored as one JSON file per source under $XDG_CACHE_HOME/vbcdd/text-search/<root>/. A content hash of the source and the profile version decide whether re-extraction is needed.
2. Index. Blocks are packed into heading-scoped chunks with a token budget. Every chunk is identified by a hash over (profile_version, source, chunk header, chunk text), so an unchanged document is skipped. Chunks feed an FTS5 table and, when embed runs, a vec1 vector table.

Lexical search keeps the content words of a query and drops the function words plus the terms that cover a large share of the root. What remains becomes an FTS5 expression ranked with bm25(), with a boost for a chunk whose header carries every term. A page with more matching chunks ranks ahead of a page with fewer, and the best chunk score breaks the tie. It joins the terms with AND first and retries with OR when the AND pass returns nothing. Vector search embeds the query with the EmbeddingGemma query prefix and runs an exact flat cosine scan. Hybrid search fuses the two lists with reciprocal rank fusion.

Requirements

• Go 1.27 or newer.
• SQLite runs as WebAssembly through ncruces/go-sqlite3, so no system SQLite library is required.
• Vector search needs a local llama-server build that serves EmbeddingGemma. Every other command works without it.

Build

go build ./cmd/vbcdd-text-search

The command is written to ./vbcdd-text-search.

Quick start

Create a manifest with the assistant:

./vbcdd-text-search init

On a terminal it asks for the source directory and where to keep the manifest: the working directory (./.vbcdd/text-search/manifest.toml) or the user config directory ($XDG_CONFIG_HOME/vbcdd/text-search/, falling back to ~/.config when the variable is empty). Flags (--root, --id, --extractor, --include, --url, --config, --local, --force, --yes) make it non-interactive.

Every command that needs the manifest resolves it in this order: the --config flag, then ./.vbcdd/text-search/manifest.toml, then the user config directory. config also accepts a positional path. stats and version read no manifest.

A manifest looks like this:

[[roots]]
id = "en"
path = "/usr/share/doc/arch-wiki/html/en"
extractor = "html"
body_selector = ".mw-parser-output"
title_rule = "h1#firstHeading"
tokenizer = "porter"
max_tokens = 400
overlap_tokens = 40
include = ["*.html", "**/*.html"]

[embedding]
url = "http://127.0.0.1:8080"
batch_size = 32

./vbcdd-text-search config --check
./vbcdd-text-search extract          # build the mirror
./vbcdd-text-search index            # build the FTS5 index
./vbcdd-text-search search 'partial upgrade'

The index lives at $XDG_DATA_HOME/vbcdd/text-search/index.db and the mirror at $XDG_CACHE_HOME/vbcdd/text-search/. Both paths can be overridden with --db and --cache.

Commands

Command	Purpose
config --check [manifest]	Validate the manifest and print the resolved roots.
init	Create a manifest, interactively or from flags.
extract	Build or refresh the text mirror.
chunks --stats	Chunk the mirror and print token percentiles; touches no database.
index	Extract as needed, chunk, and upsert into SQLite.
stats	Print document, chunk, and vector counts.
search	Run a lexical, vector, hybrid, or automatic query.
embed	Embed the indexed chunks with the configured endpoint.
eval	Report recall@k and MRR over a query set.
version	Print the SQLite version and the registered modules.

Common search flags: --mode lexical|vector|hybrid|auto, --limit, --root, --raw-match, --min-coverage, --json, --color and --no-color. The command exits 0 when at least one result is returned and 1 when none are.

Colour turns on by itself when standard output is a terminal and off when it is a pipe, so a redirected or captured run stays plain. --color forces it on and --no-color forces it off; NO_COLOR in the environment disables colour unless --color is given explicitly. The path is cyan, the line number yellow, the matching text bold red, and the match count dim. --json never contains colour.

Result format

One result is one line: the absolute source path, the 1-based line of the matching block, and a single-line snippet in parentheses with the matched text in brackets.

/usr/share/doc/arch-wiki/html/en/Fzf.html:299 (Try this to [fuzzy-search] through all available packages, with package info shown in a preview window…)

The line is the source line of the block whose chunk matched, not the line of the click in the rendered page. The HTML extractor recovers it by matching the extracted block text against a tag-stripped copy of the file, starting at the article body so a table of contents cannot shadow a heading. The extractor records that line in the mirror and the index, so a change to the extractor output shape bumps mirror.Schema and makes the next index run rebuild the chunks.

When a page contributes more than three matching chunks, the result collapses to the path and the count, and no sample is shown:

/usr/share/doc/arch-wiki/html/en/Pacman.html [57 matches]

Manifest fields

Field	Meaning
roots[].id	Mirror directory name and --root value; defaults to the base name of path.
roots[].path	Directory to index.
roots[].extractor	html, markdown, text, pdf, docx, epub, command, or auto by extension.
roots[].body_selector	A small CSS subset that narrows the HTML body.
roots[].title_rule	title for the <title> element, otherwise a CSS subset.
roots[].tokenizer	porter (default) or trigram (opt-in; the index grows about 6x).
roots[].profile_version	Hashed into every chunk ID; bump it to force a full re-index and re-embed.
roots[].max_tokens, overlap_tokens	Packing budget and sentence overlap.
roots[].include	Glob patterns; ** crosses directories.
roots[].command	Shell command for the command extractor; {path} is substituted.
embedding.url	llama-server base URL.
embedding.query_prefix, document_prefix	Task prefixes applied by the client.
embedding.batch_size	Inputs per /v1/embeddings request.

The body_selector uses a deliberately small selector language: a tag, an optional #id, any number of .class names, and descendant combinators. A full CSS engine is not a dependency.

Embedding server

Start llama-server with the embedding flag. Without it, /v1/embeddings answers HTTP 501.

llama-server -hf ggml-org/embeddinggemma-300M-qat-q4_0-GGUF \
  --embeddings --host 127.0.0.1 --port 8080 -c 2048

Then:

./vbcdd-text-search embed --limit 1000   # a resumable pass
./vbcdd-text-search embed                 # the rest

embed is a ledger, not a single pass. One row per chunk records its hash, the model fingerprint and a timestamp. A run selects only chunks without a vector for the current fingerprint, commits every 64 rows, and resumes where it stopped after an interrupt or a reboot.

Model identity

Every vector is stored with a fingerprint of the model that produced it. When the GGUF path in GET /props is readable, the fingerprint is the SHA-256 of the file; otherwise it is a hash of model_path, model_ftype, n_params, n_embd and build_info from /props and /v1/models. embed and search --mode vector|hybrid compare the stored fingerprint with the server and exit non-zero on a mismatch instead of mixing vectors from two models. Pass --reembed to replace the vectors after a model change.

The client applies the EmbeddingGemma prefixes: task: search result | query: for queries and title: {title} | text: for documents. A start-up self-check embeds one sample raw and once with the query prefix and fails when the two vectors are identical, which indicates a server that applies its own prefix (do not start llama-server with --embd-gemma-default).

Fallback

--mode auto, the default, probes GET /health and then checks how many chunks carry a fresh vector for the stored model fingerprint. When the probe fails, the server was started without --embeddings, the fingerprint is missing or mismatched, or fresh-vector coverage sits below --min-coverage (default 0.9), the query runs lexical only. The mode line on stderr names the real reason, and the exit code stays 0. Coverage counts vectors whose ledger hash and fingerprint match the current chunk, restricted to --root when it is set. An explicit --mode vector or --mode hybrid ignores the gate and exits non-zero against a dead or misconfigured server instead of degrading in silence.

Measured costs

Measured on the reference host over the English Arch Wiki (arch-wiki-docs, 2490 HTML files, about 28 MB of mirror JSON and 19 M characters of text):

Operation	Result
Extract the mirror, cold	about 4.6 s
Extract the mirror, warm	about 0.4 s
Chunk the mirror	about 0.6 s
Build the FTS5 index, cold	about 10 s
Build the FTS5 index, warm	about 1.2 s
Chunks	34 440; mean 127, median 92, p90 306 tokens
Index size	41 MB
Lexical query	under a millisecond

The token figures use the in-process estimator, not /tokenize, so indexing never depends on the embedding server. The estimator packs slightly smaller chunks than the model tokenizer would; the exact cap is not part of a chunk hash.

The full embedding pass costs about 1.5 h for English at the throughput measured for EmbeddingGemma Q4_0 on CPU (about 970 tokens/s). On a 37-page subset where the vector index was completed, recall@10 rose from 0.143 under lexical search to 0.952 under vector and hybrid search, and MRR from 0.143 to 0.886. The subset is too small to set a default; TODO.md records the full-corpus comparison. vec1 runs in flat mode, so vector queries are exact: about 41 milliseconds over 20 000 vectors, extrapolating to about 200 milliseconds over 100 000. vec1_train is deliberately not used, because a single OPQ training run took 13 minutes in WebAssembly, and the published plan for SQLite lists SIMD support on WebAssembly as unfinished.

Notes

• HTML is parsed in process with the golang.org/x/net/html package, because the chunk header needs the heading hierarchy. PDF, DOCX, and EPUB use the external pdftotext and pandoc commands.
• Prose without markup is split with uax29/v2/sentences, which is deterministic and needs no model to host.
• Every lexical query term is quoted unless --raw-match is given, so operators such as OR and NEAR are literal by default. Function words and terms that cover too much of the root leave the query first. Terms join with AND, and a shell-quoted argument that contains whitespace stays one phrase; an empty AND result is retried with OR over the individual words.
• chunk_trigram holds only roots whose tokenizer is trigram; search queries it automatically for those roots, so no flag is needed.
• One index process at a time; the tool relies on SQLite for locking and does not start or stop llama-server.

Development

go test ./...
go vet ./...

The tests cover the manifest loader, the glob matcher, the HTML extractor, the Markdown extractor, the chunker, the SQLite schema, module registration, and a lexical plus vector round trip.

License

Released into the public domain under the Unlicense.