Share reproducible pack recipes through a GitHub registry

[admin-raintree]

This PR shares packs as recipes: pinned public sources plus the content hash of each page the publisher saw. A recipe never contains fetched content. Anyone can reproduce a recipe locally and see exactly which pages changed since it was published.

It stacks on #169 because it reuses discovery_index._fetch_bytes. The base moves to main when #169 merges.

Commands

• docpull registry export --name N --version V [--description D] [--out FILE]

  • Writes the project's sources, their pinned discovered_urls, and a {url: content_hash} map per source from the locked run. It prints the recipe's sha256.
  • Refuses sources that use auth.
  • Warns when a source has no pinned URLs, because consumers could then fetch a different page set.

• docpull add name@version (or name@latest):

  1. Reads index.json from the registry, then downloads the recipe and checks it against the sha256 in the index. Downloads reuse the HTTPS-only, DNS-pinned, no-redirect fetcher from Add an offline discovery index of downloadable search shards #169.
  2. Validates the recipe.
  3. Adds the sources with discovery off, so the pinned page set is fetched.
  4. Saves the recipe to .docpull/registry/.
  5. Syncs and prints how many pages match, changed, went missing, or are new.

  Drift is reported but never fails the command.

• docpull registry verify name@version [--json] repeats the comparison after a later sync, without fetching.

• Registry location: defaults to https://raw.githubusercontent.com/raintree-technology/docpull-registry/main/. Set DOCPULL_REGISTRY_URL to use another registry.

Security

• Registry recipes are untrusted input, so they may only contain HTTPS crawl sources (auto, html, pdf, markdown, github, sitemap). A recipe therefore can't point add at local files (for example a dataset path) or at typed specs.
• Recipes are validated with extra="forbid", and the served name@version must match the request.
• The index is capped at 1 MB. Recipes are capped at 10 MB, about 90,000 pages.
• Plain alias, URL, and typed-spec add calls behave as before. A test checks that react, an HTTPS URL, npm:react@18.3.1, and owner/repo@v1 never reach the registry.

Not included

• Signed indexes: trust is HTTPS plus GitHub PR review plus a sha256 for each entry in the index.
• registry list and search: add once there are more packs.
• MCP and SDK exports.

Checks

• tests/test_registry.py:
  • export hashes, and no page content in the recipe
  • refusing auth sources and bad names
  • sha256 mismatch, unknown pack or version, and latest
  • local-path, typed, and http:// sources rejected
  • add syncs and reports 1 match and 1 change for a changed page
  • name collisions
  • registry verify --json
  • unchanged handling of other add inputs
• Live: exported a real recipe from an 11-page Python asyncio docs project. A 3,000-page project exports to 320 KB.
• bun run validate:full passes: 1,077 Python tests pass and 18 are skipped. Lint, mypy, the JS SDK, and the reproducible build also pass.

🤖 Generated with Claude Code