Add an offline discovery index of downloadable search shards

[admin-raintree]

This PR adds the client and builder for a discovery index. Users download search shards once and then search them offline, so no query is sent to a third party. Hosting the shards and the scheduled crawl that builds them will come in a later PR.

What's new

• docpull index build --project … --out DIR --name NAME
  • Reads each project's locked run from documents.jsonl.
  • Writes NAME.sqlite: an FTS5 table over title, headings, and URL, plus content hash, fetch time, and source.
  • Adds the shard's sha256, size, document count, and domains to DIR/manifest.json.
  • Shards hold no page bodies. A test checks that body text never appears in the shard file.
• docpull index pull [MANIFEST_URL]
  • The manifest URL must be HTTPS and is never defaulted. Pass it as an argument or set DOCPULL_INDEX_URL.
  • Downloads go through UrlValidator and the existing DNS-pinned _ValidatedResolver, with redirects refused.
  • The manifest is capped at 1 MB. The total shard size is checked against --max-bytes (default 500 MB) before any download.
  • Each shard is size-capped to its manifest entry and installed atomically only if its sha256 matches. Shards that are already current are skipped.
• docpull index list shows installed shards.
• docpull search --backend index, plus serve --compat --search-backend index:
  • Searches installed shards with bm25, weighting title over headings over URL, and applies domain filters.
  • Never opens a network connection. A test makes socket use fail to prove it.
  • --include-content still fetches hits through the local pipeline.
• Shards install to DOCPULL_INDEX_DIR, or $XDG_DATA_HOME/docpull/index when unset. --index-dir overrides it on index, search, and serve.

Not included

• Signatures: shards are trusted through the HTTPS manifest and sha256 only. ed25519 signing from provenance.py can be added once a published key exists.
• Hosting, seed list, scheduled crawl: these come in the next PR.
• MCP tool and SDK export: search_web has neither yet, so they should be added together.
• Cross-shard ranking: bm25 scores are only roughly comparable across shards. A ponytail: comment notes this.

Checks

• tests/test_discovery_index.py covers:
  • build output, headings that skip code fences, and no bodies in the shard
  • verified pull, skipping current shards, the size limit, and unknown shard names
  • rejecting a tampered shard
  • refusing http:// and private hosts
  • offline ranking and domain filters
  • the CLI
• Live checks:
  • _fetch_bytes downloaded https://example.com/ through the pinned resolver and enforced its cap.
  • docpull index build over a 3,000-document project produced a 737 KB shard.
  • docpull search --backend index and POST /search on serve --compat --search-backend index returned results.
• bun run validate:full passes: 1,068 Python tests pass and 18 are skipped. Lint, mypy, the JS SDK, and the reproducible build also pass.

🤖 Generated with Claude Code