feat: hflow serve — an HTTP API over a workspace, plus corpus-identity fixes

[kstonekuan]

Serves a workspace over HTTP, and fixes three ways corpus identity was coupled to things that are not the data.

Split out of #87 so the API can be reviewed and merged without waiting on a decision about the frontend. #87 now stacks on top of this and contains only the React app.

hflow serve — the workspace API

A new wheel, hflow-server, and a hflow serve subcommand. One process over one data root: the catalog as a faceted episode listing with server-compiled filters, curation SQL with previews and pinned manifests, ingest runs and their live per-task state, the registered pipeline, and media resolved back to files under the data root.

The API is the product surface, not an implementation detail of a frontend. Every endpoint publishes a typed response schema — _contract.py owns every payload, so all 26 endpoints are typed and none returns a bare object — and the whole surface is described at /api/openapi.json. Nothing is reachable only from a browser, so a UI is a client: it can be built, replaced, or duplicated without forking this package. The wheel ships no assets; HFLOW_UI_ASSETS or a packaged hflow_server/static/ supplies one, and with neither, / serves a page pointing at the schema.

Separate wheel on purpose: pipeline workers install hflow into every task venv and should never carry a web server. hflow serve imports it lazily, so the SDK does not depend on it.

serve is deliberately distinct from up. up brings up the runtime that processes episodes (an Airflow stack in Docker); serve reads the data root and can trigger a run on a runtime, executing nothing itself. Either is useful without the other — you can browse a bucket corpus with no Airflow anywhere, and CI can ingest with no web server. They are not merged because folding the read plane into Airflow's own FastAPI api-server would inherit Airflow's pinned dependency tree, which is the exact coupling user-venv-init exists to prevent.

Offline by design: no CDN, no telemetry, no outbound call. FastAPI's docs pages are disabled because they load Swagger from jsdelivr; the schema is served instead.

Identity fixes

episode_id hashes the canonical bytes, so anything folded into those bytes decides whether re-ingesting an unchanged recording dedupes.

• The release number was in there. compute_pipeline_version folded hflow.__version__, and the MCAP header's library string carried it too, so every release minted a new pipeline_version, a new episode_id for byte-identical input, and a new version for every step referencing the hflow module. A CLI-only patch invalidated a whole corpus. Replaced by hflow.behavior.TRANSFORM_BEHAVIOR_VERSION, bumped deliberately when the transform would write different bytes.
• Message order was decided by append order. The sort keyed on log_time alone; a stable sort then settled ties by the order the transform happened to append. On a 20s two-camera episode 1350 of 3200 messages share a timestamp — reversing the tie order moved episode_id from 8046ef0b563a7c01 to 3192361fc21d6305. Now (log_time, topic): a property of the data, not of the code.
• RESAMPLE_POLICY_VERSION was invisible. It decides derived-channel samples, and no step hash can see it (the taught idiom reaches hflow.resample.to_grid through the module, and a module contributes only its name). Now folded into pipeline_version — but only for episodes that have derived channels, so bumping it never churns a corpus that resampled nothing.

TRANSFORM_BEHAVIOR_VERSION goes 1 → 2 for the ordering change. Pre-v1, nothing published.

Also

• ingest_dag_topology describes the generated DAGs as data, so a client can draw them without parsing rendered Python. tests/test_runtime_topology.py reads the rendered source back and checks every task id and edge the description claims.
• A per-episode catalog re-read removed. The labels and media quarantine gate re-synced a bucket mirror and re-scanned every episodes parquet file once per episode. Reading every quarantined episode in one pass took a 50-episode batch on a 2000-file catalog from 96.8 ms to 2.0 ms per episode.
• AirflowClient gains task_instances() and a batch-count argument on ingest().

Verification

638 tests pass; ruff check, ruff format --check and ty check clean.

🤖 Generated with Claude Code