Add cited research answers and async job routes

[admin-raintree]

DocPull now answers questions from the web with cited evidence, with no hosted research API involved. This stacks on #165 (local search). The base retargets to main when #165 merges.

docpull research QUESTION --backend searxng|marginalia and docpull.research:

1. It searches with the chosen backend from Add local-first web search through SearXNG or Marginalia #165 and fetches up to --max-sources results (default 5, at most 10) through the local pipeline, with HTTPS-only fetching, private-network blocking, and robots.txt handling.
2. It splits each page with the docpull query passage splitter, which keeps paragraphs and code fences whole, and scores the passages with the same _search_score.
3. It keeps the best --max-passages (default 8), at most three per source. Each passage keeps its exact char_start/char_end span. Sources are numbered S1, S2, … in order of first use.
4. The answer is those passages, each ending in its [S#] citation.

Optional model rewrite. --llm-url and --llm-model call any OpenAI-compatible /chat/completions endpoint, such as Ollama or llama.cpp, at temperature 0, with instructions to answer only from the numbered evidence.

• The rewrite is kept only if it cites at least one source and every citation names a real source. Otherwise the cited passages are returned with a warning saying why.
• An endpoint outside localhost requires --allow-remote-llm, because it may cost money. --llm-api-key-env NAME sends a bearer key.
• If the model fails or times out, the cited passages are returned instead.

Compatible routes. docpull serve --compat --search-backend ..., plus the optional --llm-* flags, adds:

• Exa POST /answer, which returns answer and citations[], with passage text when text: true.
• Tavily POST /search with include_answer, which fills answer.

Surface. research moves from PRUNED_CLI_COMMANDS to the public CLI. The surface contract, CLI recipes, and changelog are updated.

Async job routes (second commit). The compatible server now runs the async APIs:

Create	Poll	Notes
Firecrawl POST /v1/crawl, /v2/crawl	GET /v{1,2}/crawl/{id}, DELETE cancels	Local discovery and fetching. limit defaults to 50 and is capped at 500. Firecrawl's regex includePaths and excludePaths are applied to the URL path (DocPull's own crawler uses globs, so they are applied after discovery). Status values are scraping, completed, failed, and cancelled.
Parallel POST /v1/tasks/runs	GET /v1/tasks/runs/{run_id}, GET .../result?timeout=	Run IDs start with trun_. /result waits up to the timeout (600 s max) and returns 408 while the run is still active. output.basis[0].citations[] carries each source's passages as excerpts.
Tavily POST /research	GET /research/{request_id}	Returns 201 on create, then 202 with pending/in_progress while running, and 200 with content and sources[] when done.

• Jobs are background asyncio tasks that live in memory, so they are lost when the server stops. Finished jobs expire after 24 hours, and the store holds at most 200 jobs.
• A task done-callback marks each job finished. That covers a job cancelled before its task starts, which previously never completed: the test for it hung until I fixed this.
• Job routes use the same API-key check as the other routes.

Not covered yet. No live end-to-end run happened, because it needs a working search backend and a local model.

Validation. The bun run validate:full results below are for the research commit. For the job commit, I ran ruff, mypy on the changed modules, and the 31 compat, search, research, and job tests. I didn't run full local validation because another session had uncommitted changes in the same checkout, so CI is the full check for this commit.

• bun run validate:full passes:
  • ruff, biome, metadata and schema checks
  • mypy across 140 files
  • 1,065 Python tests passed, 18 skipped
  • all 14 JS SDK tests
  • reproducible build
• tests/test_research.py and tests/test_compat_jobs.py cover:
  • exact spans and dense source IDs
  • extractive citations, fetch-failure and no-match warnings
  • a fake model server: accepted, unknown, and missing citations
  • the remote-endpoint guard
  • CLI JSON output
  • both compatible answer routes
  • the crawl lifecycle, cancellation, and failure
  • the task-run lifecycle and result timeout
  • the research lifecycle
  • job-route auth and regex path filters

🤖 Generated with Claude Code