Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
are SQLite files of page URLs, titles, and headings from synced projects, with
no page bodies. `pull` needs an explicit HTTPS manifest URL. It verifies each
shard's sha256 and enforces a total download size limit. Searches run offline.
- Share reproducible packs as recipes. `docpull registry export` writes a pack's
public sources, pinned page URLs, and per-page content hashes, with no fetched
content. `docpull add name@version` downloads a recipe from the registry,
checks its sha256, adds and syncs the sources, and reports which pages still
match. `docpull registry verify` repeats the check after a later sync.

### Removed

Expand Down
34 changes: 34 additions & 0 deletions docs/cli-recipes.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,40 @@ Shards install to `DOCPULL_INDEX_DIR`, or `$XDG_DATA_HOME/docpull/index` when
that is unset. Pass `--index-dir` to `index`, `search`, or `serve` to use
another directory.

## Shared Pack Recipes

A recipe lists a pack's public sources, its pinned page URLs, and the content hash
of every page the publisher synced. It contains no fetched content. When you add
a recipe, DocPull fetches every page locally and reports which pages still match
the publisher's hashes.

```bash
# Consumer: add, sync, and verify a published pack
docpull add stripe-api@2026-10
docpull registry verify stripe-api@2026-10 --json

# Publisher: export a recipe from a synced project
docpull registry export --name stripe-api --version 2026-10 --description "Stripe API reference"
```

`docpull add name@version`:
1. Downloads the recipe through the registry's `index.json`, checking the
recipe's sha256 against its index entry.
2. Adds the recipe's sources with their pinned URLs.
3. Saves the recipe under `.docpull/registry/`.
4. Runs `docpull sync`.
5. Prints how many pages match, changed, went missing, or are new.

Live pages change, so drift is reported, not treated as an error. Use
`name@latest` for the newest version.

Recipes may only contain HTTPS crawl sources. Export refuses sources that use
`auth`, so private content never goes into a registry.

The registry is a public GitHub repository, read over HTTPS. Publishing a recipe
means opening a pull request that adds the recipe file and its `index.json`
entry. Set `DOCPULL_REGISTRY_URL` to use another registry.

## Local Extract API

`docpull serve --compat` accepts the request shapes of common hosted extract
Expand Down
1 change: 1 addition & 0 deletions docs/surface-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ DocPull targets capability alignment, not 1:1 flag parity. MCP should not mirror
| Project lifecycle | `docpull init`, `add`, `install`, `deps`, `sources`, `sync`, `diff`, `status`, `history`, `review`, `release context-pack`, `watch` with explicit crawl bounds | `docpull.project` helpers | Not exposed yet | Core-aligned |
| Search the web | `docpull search QUERY --backend searxng\|marginalia\|index` with domain filters and `--include-content` | `search_web`, `SearchResponse`, `SearchHit` in `docpull.web_search` | Not exposed | CLI-first |
| Offline discovery index | `docpull index build\|pull\|list` | `build_shard`, `pull`, `search_index` in `docpull.discovery_index` | Not exposed | CLI-first |
| Shared pack recipes | `docpull add name@version`, `docpull registry export\|verify` | `export_recipe`, `fetch_recipe`, `install_recipe`, `verify_recipe` in `docpull.registry` | Not exposed | CLI-first |
| Search cached sources | SQLite/local search helpers where applicable | `search_sqlite_documents` | `grep_docs` | Adapted |
| Query locked context with budgets, version filters, deduplication, and precise citations | `docpull query` | `query_context`, `ContextQueryError` | `context_query` | Core-aligned |
| Read cached source by path/range | Filesystem responsibility | Filesystem responsibility | `read_doc` | MCP-specific |
Expand Down
4 changes: 4 additions & 0 deletions src/docpull/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -1687,6 +1687,10 @@ def main(argv: list[str] | None = None) -> int:
from .graph import run_graph_cli

return run_graph_cli(raw_argv[1:])
if raw_argv and raw_argv[0] == "registry":
from .registry import run_registry_cli

return run_registry_cli(raw_argv[1:])
if raw_argv and raw_argv[0] == "index":
from .discovery_index import run_index_cli

Expand Down
16 changes: 15 additions & 1 deletion src/docpull/project.py
Original file line number Diff line number Diff line change
Expand Up @@ -407,7 +407,11 @@ def run_init_cli(argv: list[str] | None = None) -> int:

def run_add_cli(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="docpull add", description="Add a source to a DocPull project")
parser.add_argument("sources", nargs="+", help="HTTPS URL, bundled alias, or typed source spec")
parser.add_argument(
"sources",
nargs="+",
help="HTTPS URL, bundled alias, typed source spec, or registry pack name@version",
)
parser.add_argument("--name", help="Source name")
parser.add_argument("--type", choices=SOURCE_TYPES, default="auto")
parser.add_argument("--discover", action="store_true", help="Discover and store source URLs now")
Expand All @@ -424,6 +428,16 @@ def run_add_cli(argv: list[str] | None = None) -> int:
)
parser.add_argument("--auth-header-name", help="Header name for --auth header-env")
args = parser.parse_args(argv)
if len(args.sources) == 1 and _infer_typed_source_type(args.sources[0]) is None:
from .registry import REF, run_add_recipe_cli

if REF.fullmatch(args.sources[0].strip()):
if args.name or args.type != "auto" or args.discover or args.auth:
Console().print(
"[red]Project error:[/red] registry packs take no --name, --type, --discover, or --auth"
)
return 1
return run_add_recipe_cli(args.sources[0].strip())
console = Console()
try:
auth = _auth_ref_from_cli(
Expand Down
Loading
Loading