Skip redundant regex work when scoring search passages

[admin-raintree]

Search scoring now does less regex work per passage, and its scores and cleaned text are unchanged.

Problem

After #168, scoring was the largest cost in broad context queries. _search_score calls _term_count 9 times per passage, and _term_count rebuilt its pattern string on every call. _clean_passage ran four uncompiled re.sub passes over each field, even on plain text.

Change

All changes are in src/docpull/pack_tools.py:

• _term_count compiles each term's pattern once and caches it with lru_cache.
• _clean_passage uses precompiled patterns. Each one is skipped when the character it needs is absent: ]( for links, -, * or + for list markers, and # for headings.
• _clean_passage collapses whitespace with " ".join(text.split()). On every Unicode code point, str.split() and regex \s treat the same characters as whitespace, so the result is identical.

Checks

• New test test_search_scoring_matches_pre_fast_path_values pins cleanup output and scores recorded from the previous implementation. It covers markdown links, list and heading markers, c++, café and Unicode whitespace.
• Local fuzzing, not committed: 200,000 random markdown-like strings gave the same _clean_passage output in the old and new versions.
• uv run pytest tests/test_pack_tools.py tests/test_context_query.py tests/test_web_search.py -q: 53 passed.
• ruff check, ruff format and mypy pass on the changed files. bun run validate:full was not run.

Benchmark

On a synthetic project of 3,000 documents with 52,000 matching passages, the context query "retry backoff" took a median of 1.35s, versus 1.87s with main's scoring code. Both versions returned identical payloads, and each timing is the median of 3 runs.

The synthetic passages contain no - or #, so every cleanup pattern was skipped. Real documentation usually contains hyphens, so the speedup there will be smaller.

🤖 Generated with Claude Code