Dump the files collection to a flattened relational DuckDB database — Closes #123

README.md

@@ -1240,3 +1240,52 @@ cfdb sync 4dn hubmap
 - `--api-url` - cfdb API base URL (default: `http://localhost:8000`, env: `CFDB_API_URL`)
 - `--api-key` - API key for sync endpoint (env: `SYNC_API_KEY`)
 - `--debug` / `-d` - Enable debugpy debugging
+
+### DuckDB dump
+
+`cfdb dump duckdb` flattens the materialized `files` collection into a single relational DuckDB file, so corpus-level questions — counts by assembly and consortium, every contact map from a given tissue, `output_type` coverage — become SQL anyone can run on a laptop instead of aggregation pipelines run by whoever holds Mongo credentials. It requires the `duckdb` extra: `pip install "cfdb[duckdb] @ git+https://github.com/abdenlab/cfdb.git"` (or `uv sync --extra dev` in a checkout).
+
+```bash
+# Dump everything
+cfdb dump duckdb cfdb.duckdb
+
+# One or more consortia, plus a Parquet export of every table
+cfdb dump duckdb cfdb.duckdb -s 4dn -s encode --to-parquet ./parquet
+```
+
+**Options:**
+- `--database-url` - MongoDB connection string (default: `mongodb://127.0.0.1:27017`, env: `DATABASE_URL`)
+- `--database-name` - Database holding the `files` collection (default: `cfdb`, env: `DATABASE_NAME`)
+- `--submission` / `-s` - Restrict to these consortia (`4dn`, `encode`, `hubmap`); repeatable, omit for all
+- `--to-parquet` - Also `EXPORT DATABASE` every table as Parquet into this directory
+- `--force` - Overwrite the output file if it exists
+
+**Schema.** The schema of record is [`src/cfdb/dump/schema.sql`](src/cfdb/dump/schema.sql) — a checked-in artifact in the same spirit as `schema.graphql`, versioned by its `schema_version` header and designed in [#123](https://github.com/abdenlab/cfdb/issues/123). The tables mirror C2M2 where C2M2 has a table (`file`, `collection`, `biosample`, `subject`, and the six junctions, all keyed by natural `(id_namespace, local_id)` composites, with `submission` as a non-key column on every entity); enrichment with a cross-consortium slot lives in promoted common columns (`genome_assembly`, `output_type`, `experiment_type`, `analyte_class`, …); the per-consortium residue lives in 1:1 `*_extra_*` extension tables, which omit verbatim-promoted fields but keep any field whose promotion is derived rather than verbatim (`replicate_info` survives because the replicate columns are regex-extracted from it). Three views carry the joins analysts would otherwise retype: `v_file` (one row per file with term names and extension columns inlined), `v_file_biosample` (the file × collection × biosample × subject fan-out grain), and `v_subject_demographics` (sex/race/age coalesced across the C2M2 columns and HuBMAP's enrichment, with a `*_source` column per value so provenance survives the coalesce).
+
+**Common-column mapping.** Each promoted column and the upstream field that fills it, per consortium. A column populated by one consortium today is still a common column, not a 4DN or ENCODE column: it names a slot any consortium could fill, which is what makes the next DCC's ingest a mapping exercise rather than a schema change.
+
+`file`:
+
+| Column | 4DN source | ENCODE source | HuBMAP source |
+|---|---|---|---|
+| `genome_assembly` | `genome_assembly` | `File assembly` | dataset `genome_assembly` |
+| `genome_annotation` | — | `Genome annotation` | — |
+| `output_type` | `file_type` | `Output type` | — |
+| `output_type_detail` | `file_type_detailed` | `File format type` | — |
+| `biological_replicates` | parsed from `replicate_info` | `Biological replicate(s)` | — |
+| `technical_replicates` | parsed from `replicate_info` | `Technical replicate(s)` | — |
+| `assay_info` | `assay_info` | — | — |
+| `condition` | `condition` | — | — |
+
+`collection`:
+
+| Column | 4DN source | ENCODE source | HuBMAP source |
+|---|---|---|---|
+| `lab` | `lab` | `Lab` | — |
+| `experiment_type` | `experiment_type` | assay | `dataset_type` |
+| `experiment_target` | — | `Experiment target` / `Targets` | — |
+| `analyte_class` | — | `Library made from` | `analyte_class` |
+
+Enrichment with no common slot today — ENCODE's `annotation_type`, `organism`, `software_used`, `encyclopedia_version`, `life_stage`, `age`, `age_units` among it — stays in the extension tables under its upstream name, and promoting such a field later is a schema change the `schema_version` header records.
+
+**Loader policies.** Entity rows repeated across documents are first-write-wins, and every divergent later copy is counted and sampled into `dump_conflict` — divergence is reported, never silently resolved. Junction rows take the union over all copies instead, so a relationship present in only one copy is never dropped. A `(id_namespace, local_id)` key appearing under two submissions aborts the dump: the natural keys are only globally unique because id_namespaces are DCC-scoped, and the loader asserts that rather than trusting it (duplicate file keys abort likewise). String-typed numbers from the C2M2 TSV path (`size_in_bytes` on 4DN/HuBMAP) and unparseable timestamps coerce to NULL with a counter, a document failing model validation is skipped with a counter, and every counter plus per-table row counts land in `dump_stat`, next to `dump_info` (schema version, generation time, cfdb version/commit, source database, submissions, file count) — so a dump is self-describing about which code produced it and where the source disagreed with itself. Counters measure occurrences, not entities: a divergent or uncoercible value on an entity embedded by many documents counts once per embedding document, so a large count can be one bad shared entity rather than many. Foreign keys are declared and enforced, so a dump that violates referential integrity fails at write time rather than surprising a reader. An aborted or interrupted run deletes its partial output, and the backstop if one ever survives is `dump_info`: the row is written last, so a database without one is a partial dump.

pyproject.toml

@@ -58,7 +58,12 @@ readme = { file = "README.md", content-type = "text/markdown" }
 requires-python = ">=3.12"
 
 [project.optional-dependencies]
-dev = ["allpairspy", "cloudpickle", "cryptography", "debugpy", "httpx", "hypothesis", "mongomock-motor", "moto[ecs,s3]", "pytest-asyncio", "pytest-mock", "ruff"]
+dev = ["allpairspy", "cloudpickle", "cryptography", "debugpy", "duckdb", "httpx", "hypothesis", "mongomock-motor", "moto[ecs,s3]", "pytest-asyncio", "pytest-mock", "ruff"]
+# The DuckDB dump (issue #123) is an offline batch tool most deployments
+# never run, so its only dependency stays behind an extra rather than
+# growing the API image:
+# ``pip install "cfdb[duckdb] @ git+https://github.com/abdenlab/cfdb.git"``.
+duckdb = ["duckdb"]
 
 [tool.uv.sources]
 # Pinned to a sha, not a branch: clodius considers ``tiles_v2`` unstable
@@ -101,6 +106,12 @@ markers = [
     "integration: slow or tool-dependent tests (e.g. samtools, htslib, real wool WorkerPool)",
 ]
 
+# Ship the dump schema DDL inside the wheel — src-layout auto-discovery
+# includes only ``*.py`` without this, and ``cfdb.dump`` reads
+# ``schema.sql`` via importlib.resources at run time.
+[tool.setuptools.package-data]
+"cfdb.dump" = ["*.sql"]
+
 [tool.setuptools_scm]
 local_scheme = "dirty-tag"
 version_scheme = "release-branch-semver"

src/cfdb/cli.py

@@ -4,6 +4,8 @@
 import requests
 from pymongo import MongoClient
 
+from cfdb.dcc_registry import get_all_dcc_names
+
 __client__ = None
 
 logging.basicConfig(level=logging.INFO)
@@ -23,7 +25,7 @@ def debug(ctx, param, value):
 
     import debugpy
 
-    debugpy.listen(5678)
+    debugpy.listen(value)
     logging.debug("Waiting for debugger to attach...")
     debugpy.wait_for_client()
     logging.debug("Debugger attached")
@@ -110,5 +112,154 @@ def sync(dcc_names: tuple[str, ...], api_url: str, api_key: str):
         raise SystemExit(1)
 
 
+@cli.group("dump")
+def dump():
+    """Export cfdb data to offline analytic formats."""
+
+
+@dump.command("duckdb")
+@click.argument("output", type=click.Path(dir_okay=False, writable=True))
+@click.option(
+    "--database-url",
+    default="mongodb://127.0.0.1:27017",
+    envvar="DATABASE_URL",
+    help="MongoDB connection string",
+    show_default=True,
+)
+@click.option(
+    "--database-name",
+    default="cfdb",
+    envvar="DATABASE_NAME",
+    help="MongoDB database holding the materialized files collection",
+    show_default=True,
+)
+@click.option(
+    "--submission",
+    "-s",
+    "submissions",
+    multiple=True,
+    type=click.Choice(get_all_dcc_names(), case_sensitive=False),
+    help="Restrict the dump to these consortia. Repeatable; omit for all.",
+)
+@click.option(
+    "--to-parquet",
+    "to_parquet",
+    type=click.Path(file_okay=False),
+    default=None,
+    help="Also export every table as Parquet into this directory.",
+)
+@click.option(
+    "--force",
+    is_flag=True,
+    help="Overwrite OUTPUT if it already exists.",
+)
+@click.option(
+    "--debug",
+    "-d",
+    callback=debug,
+    expose_value=False,
+    help="Run with debugger listening on the specified port.",
+    is_eager=True,
+    type=int,
+)
+def dump_duckdb(
+    output: str,
+    database_url: str,
+    database_name: str,
+    submissions: tuple[str, ...],
+    to_parquet: str | None,
+    force: bool,
+):
+    """
+    Dump the materialized files collection to a relational DuckDB database.
+
+    Streams every document in ``files`` (optionally restricted by
+    --submission) and writes OUTPUT, a single .duckdb file holding the
+    flattened schema defined in src/cfdb/dump/schema.sql. Requires the
+    ``duckdb`` extra:
+    pip install "cfdb[duckdb] @ git+https://github.com/abdenlab/cfdb.git".
+
+    Examples:
+
+        cfdb dump duckdb cfdb.duckdb
+
+        cfdb dump duckdb cfdb.duckdb -s 4dn -s encode --to-parquet ./parquet
+    """
+    try:
+        import duckdb
+    except ImportError:
+        click.echo(
+            "Error: the DuckDB dump requires the 'duckdb' extra: "
+            'pip install "cfdb[duckdb] @ git+https://github.com/abdenlab/cfdb.git"',
+            err=True,
+        )
+        raise SystemExit(1)
+
+    from pathlib import Path
+
+    from cfdb.db import mongo_client_kwargs
+    from cfdb.dump import DumpError, dump_files
+
+    output_path = Path(output)
+    if output_path.exists():
+        if not force:
+            click.echo(
+                f"Error: {output_path} already exists (pass --force to overwrite)",
+                err=True,
+            )
+            raise SystemExit(1)
+        output_path.unlink()
+
+    normalized = [s.lower() for s in submissions]
+    query = {"submission": {"$in": normalized}} if normalized else {}
+    client = MongoClient(database_url, **mongo_client_kwargs())
+    cursor = client[database_name]["files"].find(query, batch_size=1000)
+
+    con = duckdb.connect(str(output_path))
+    completed = False
+    try:
+        report = dump_files(
+            cursor,
+            con,
+            source_database=database_name,
+            submissions=normalized or None,
+        )
+        completed = True
+    except DumpError as e:
+        click.echo(f"Error: {e}", err=True)
+        raise SystemExit(1)
+    finally:
+        client.close()
+        if not completed:
+            # A partial dump is worse than no dump: a finally block (rather
+            # than except) runs this cleanup on every exit path, including
+            # KeyboardInterrupt.
+            con.close()
+            output_path.unlink(missing_ok=True)
+
+    # From here the dump is complete and self-describing; a failure in the
+    # secondary parquet export must not destroy it.
+    export_error: Exception | None = None
+    if to_parquet:
+        parquet_dir = str(Path(to_parquet)).replace("'", "''")
+        try:
+            con.execute(f"EXPORT DATABASE '{parquet_dir}' (FORMAT PARQUET)")
+        except Exception as e:
+            export_error = e
+    con.close()
+
+    click.echo(report.summary())
+    click.echo(f"Wrote {output_path}")
+    if export_error is not None:
+        click.echo(
+            f"Error: Parquet export to {to_parquet} failed ({export_error}); "
+            f"{output_path} is complete and was kept",
+            err=True,
+        )
+        raise SystemExit(1)
+    if to_parquet:
+        click.echo(f"Exported Parquet to {to_parquet}")
+
+
 if __name__ == "__main__":
     cli()

src/cfdb/dump/__init__.py

@@ -0,0 +1,14 @@
+"""Relational DuckDB dump of the materialized ``files`` collection.
+
+See :mod:`cfdb.dump.loader` for the flattening semantics and
+``schema.sql`` for the schema of record (issue #123).
+"""
+
+from cfdb.dump.loader import (
+    SCHEMA_VERSION,
+    DumpError,
+    DumpReport,
+    dump_files,
+)
+
+__all__ = ["SCHEMA_VERSION", "DumpError", "DumpReport", "dump_files"]