Add an offline discovery index of downloadable search shards

docs/CHANGELOG.md

@@ -19,6 +19,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `--include-content` fetches each result through the local pipeline.
   `docpull serve --compat --search-backend ...` also serves Tavily and Exa
   `/search` and Firecrawl `/v1/search` and `/v2/search` shapes.
+- Build, download, and search an offline discovery index with
+  `docpull index build|pull|list` and `docpull search --backend index`. Shards
+  are SQLite files of page URLs, titles, and headings from synced projects, with
+  no page bodies. `pull` needs an explicit HTTPS manifest URL. It verifies each
+  shard's sha256 and enforces a total download size limit. Searches run offline.
 
 ### Removed
 

docs/alternatives.md

@@ -23,7 +23,7 @@ tool.
 | Self-host a browser-based crawler with LLM-oriented Markdown output | Crawl4AI | Open-source, self-hosted, browser-driven; a good fit when pages need a real browser and you want to run the infrastructure yourself |
 | Point agent code written for a hosted extract API at a local fetcher | DocPull `serve --compat` | Accepts Tavily extract, Exa contents, and Firecrawl scrape request shapes and answers from the local pipeline; see [Local Extract API](cli-recipes.md#local-extract-api) |
 | Use a hosted web-to-LLM extraction service | Firecrawl, Jina Reader, hosted extraction APIs | Useful when you want an API service to manage crawling/extraction infrastructure |
-| Search the live web before building a pack | DocPull `search` with a SearXNG instance or Marginalia | `docpull search` queries a backend you choose and can fetch each result through the local pipeline; no paid provider is involved. See [Web Search](cli-recipes.md#web-search) |
+| Search the live web before building a pack | DocPull `search` with a SearXNG instance, Marginalia, or an offline discovery index | `docpull search` queries a backend you choose and can fetch each result through the local pipeline; no paid provider is involved. See [Web Search](cli-recipes.md#web-search) |
 
 ## Where DocPull Is Strongest
 

docs/cli-recipes.md

@@ -56,8 +56,8 @@ Minimal GitHub Actions step:
 
 ## Web Search
 
-`docpull search` queries a search backend you choose explicitly. DocPull has no
-index of its own and never calls a paid search provider for this command.
+`docpull search` queries a search backend you choose explicitly. It never calls a
+paid search provider.
 
 ```bash
 # Self-hosted SearXNG with `json` enabled under search.formats in settings.yml
@@ -72,13 +72,47 @@ docpull search "static site generators" --backend marginalia --max-results 5 --j
 | --- | --- | --- |
 | `searxng` | Run a SearXNG instance and pass `--searxng-url` or set `DOCPULL_SEARXNG_URL` | Metasearch across the engines your instance enables; the configured URL is trusted, so it may be local |
 | `marginalia` | None; set `DOCPULL_MARGINALIA_KEY` for your own key | The shared `public` key is often rate limited; results carry Marginalia's license, CC-BY-NC-SA 4.0 for free keys |
+| `index` | Download shards with `docpull index pull` | Searches installed discovery shards offline; no query leaves the machine. See [Discovery Index](#discovery-index) |
 
 Each query returns at most 20 results. `--include-domain` and `--exclude-domain`
 filter results by host. One include domain also narrows the upstream query.
 `--include-content` fetches every result through the same local pipeline as
 `docpull sync`, with HTTPS-only fetching, private-network blocking, and robots.txt
 handling, and attaches the Markdown.
 
+## Discovery Index
+
+A discovery index is a set of SQLite shards. Each shard lists page URLs, titles,
+headings, content hashes, and fetch times from synced DocPull projects. Shards
+hold no page bodies. You download shards once, then search them offline with
+`--backend index`. Use `--include-content` to fetch hits through the local
+pipeline.
+
+```bash
+# Maintainer: build a shard from one or more synced projects
+docpull index build --project ./python-docs --name python-docs --out ./published
+
+# User: download and verify shards, then search offline
+docpull index pull https://index.example/v1/manifest.json
+docpull search "asyncio timeouts" --backend index
+docpull index list
+```
+
+`build` writes `<name>.sqlite` and adds its sha256, size, and domains to
+`manifest.json` in the output directory. Shard URLs in the manifest are relative
+to the manifest.
+
+`pull` requires an HTTPS manifest URL. Pass it as an argument or set
+`DOCPULL_INDEX_URL`. DocPull has no default index URL, so nothing downloads
+unless you ask. Downloads use the same private-network blocking as page fetches
+and refuse redirects. Each shard must match its manifest sha256 before it is
+installed. `pull` skips shards that are already current. Before downloading,
+`pull` checks the total size against `--max-bytes`, which defaults to 500 MB.
+
+Shards install to `DOCPULL_INDEX_DIR`, or `$XDG_DATA_HOME/docpull/index` when
+that is unset. Pass `--index-dir` to `index`, `search`, or `serve` to use
+another directory.
+
 ## Local Extract API
 
 `docpull serve --compat` accepts the request shapes of common hosted extract
@@ -96,8 +130,8 @@ docpull serve --compat --port 8765
 | `POST /contents` | Exa contents: `urls` or `ids`, `text` (bool or `maxCharacters`) | `results[].text`, `statuses[]` |
 | `POST /v1/scrape`, `POST /v2/scrape` | Firecrawl scrape: `url`, `formats: ["markdown"]` | `success`, `data.markdown`, `data.metadata` |
 
-Start the server with `--search-backend searxng` or `--search-backend marginalia`
-to also serve search shapes. Those are Tavily or Exa `POST /search`, chosen by the
+Start the server with `--search-backend searxng`, `marginalia`, or `index` to
+also serve search shapes. Those are Tavily or Exa `POST /search`, chosen by the
 request fields, and Firecrawl `POST /v1/search` and `/v2/search`. A client can
 pin a flavor by setting its base URL to `http://127.0.0.1:8765/tavily`,
 `/exa`, or `/firecrawl`. Without a search backend, the search routes return

docs/surface-contract.md

@@ -42,7 +42,8 @@ DocPull targets capability alignment, not 1:1 flag parity. MCP should not mirror
 | List configured sources | Not a primary CLI command | `docpull.mcp.sources` internals, not public SDK | `list_sources` | MCP-focused |
 | List cached/indexed sources | Not a primary CLI command | Local filesystem/search helpers | `list_indexed` | MCP-focused |
 | Project lifecycle | `docpull init`, `add`, `install`, `deps`, `sources`, `sync`, `diff`, `status`, `history`, `review`, `release context-pack`, `watch` with explicit crawl bounds | `docpull.project` helpers | Not exposed yet | Core-aligned |
-| Search the web | `docpull search QUERY --backend searxng\|marginalia` with domain filters and `--include-content` | `search_web`, `SearchResponse`, `SearchHit` in `docpull.web_search` | Not exposed | CLI-first |
+| Search the web | `docpull search QUERY --backend searxng\|marginalia\|index` with domain filters and `--include-content` | `search_web`, `SearchResponse`, `SearchHit` in `docpull.web_search` | Not exposed | CLI-first |
+| Offline discovery index | `docpull index build\|pull\|list` | `build_shard`, `pull`, `search_index` in `docpull.discovery_index` | Not exposed | CLI-first |
 | Search cached sources | SQLite/local search helpers where applicable | `search_sqlite_documents` | `grep_docs` | Adapted |
 | Query locked context with budgets, version filters, deduplication, and precise citations | `docpull query` | `query_context`, `ContextQueryError` | `context_query` | Core-aligned |
 | Read cached source by path/range | Filesystem responsibility | Filesystem responsibility | `read_doc` | MCP-specific |

src/docpull/cli.py

@@ -1687,6 +1687,10 @@ def main(argv: list[str] | None = None) -> int:
         from .graph import run_graph_cli
 
         return run_graph_cli(raw_argv[1:])
+    if raw_argv and raw_argv[0] == "index":
+        from .discovery_index import run_index_cli
+
+        return run_index_cli(raw_argv[1:])
     if raw_argv and raw_argv[0] == "search":
         from .web_search import run_search_cli