Add a command to convert datasets with aneurysm masks

.github/workflows/ci.yml

@@ -26,6 +26,24 @@ jobs:
       - name: Test
         run: uv run pytest
 
+  wheel:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v10.2.0
+      - name: Build and install the wheel outside the repository
+        run: |
+          uv build --wheel -o /tmp/dist
+          uv venv /tmp/venv
+          VIRTUAL_ENV=/tmp/venv uv pip install /tmp/dist/*.whl --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match
+      - name: Run the commands from another folder
+        working-directory: /tmp
+        run: |
+          mkdir -p data/P0001 && echo '{}' > data/P0001/config.json
+          /tmp/venv/bin/deepaneseg-train --help
+          /tmp/venv/bin/deepaneseg-preprocess data_dir=data
+          test -f data/0Work/split_pats.json
+
   docker:
     runs-on: ubuntu-latest
     steps:
@@ -35,4 +53,4 @@ jobs:
       - name: Smoke test
         run: |
           docker run --rm deepaneseg:cpu python -c "import torch, deepaneseg.utils; print(torch.__version__)"
-          docker run --rm deepaneseg:cpu python scripts/train.py --help
+          docker run --rm deepaneseg:cpu deepaneseg-train --help

Dockerfile

@@ -21,4 +21,4 @@ RUN uv sync --locked --no-install-project --no-dev --extra ${TORCH} && rm -rf /r
 COPY . .
 RUN uv sync --locked --no-dev --extra ${TORCH} && rm -rf /root/.cache/uv
 
-CMD ["python", "scripts/train.py", "--help"]
+CMD ["deepaneseg-train", "--help"]

README.md

@@ -25,9 +25,9 @@ deepaneseg/
   training/   dataset, losses, metrics, trainer
   inference/  patch-wise prediction and ADAM-style evaluation
   experimental/  post-paper models (vessel-coupled U-Nets) and their trainer
-configs/      Hydra configuration (see Configuration)
-scripts/      pipeline steps (see Usage)
-  experimental/  analysis scripts from the PhD experiments
+  cli/        pipeline commands (see Usage)
+  configs/    Hydra configuration (see Configuration)
+scripts/experimental/  analysis scripts from the PhD experiments
 ```
 
 The data is expected in the following layout:
@@ -48,56 +48,66 @@ Data_dir/
 ```
 
 # Configuration
-Settings are [Hydra](https://hydra.cc) configs in `configs/`: one file per script (`train.yaml`, `predict.yaml`,
-`preprocess.yaml`, `remove_skull.yaml`, `extract_points.yaml`) composed from groups (`data`, `model`, `optimizer`,
+Settings are [Hydra](https://hydra.cc) configs in `deepaneseg/configs/`: one file per command (`train.yaml`,
+`predict.yaml`, `evaluate.yaml`, `preprocess.yaml`, `remove_skull.yaml`, `extract_points.yaml`) composed from groups (`data`, `model`, `optimizer`,
 `scheduler`, `augmentation`). Any value can be overridden on the command line:
 ```
-uv run python scripts/train.py data_dir=Data_dir name=my_training batch_size=8 optimizer.lr=3e-4 model.f_maps=32
-uv run python scripts/train.py --cfg job --resolve data_dir=Data_dir name=my_training   # print the config
+uv run deepaneseg-train data_dir=Data_dir name=my_training batch_size=8 optimizer.lr=3e-4 model.f_maps=32
+uv run deepaneseg-train --cfg job --resolve data_dir=Data_dir name=my_training   # print the config
 ```
 Runs are reproducible: `seed` (0 by default) fixes the patient split and the training, and each training writes
 `run_info.json` with the git commit and library versions next to its checkpoints. For identical GPU runs, also set
 `deterministic=true` (slower).
 
 To add an alternative (for example another scheduler), add a file to the group, such as
-`configs/scheduler/step.yaml` with a `_target_` class, and select it with `scheduler=step`.
+`deepaneseg/configs/scheduler/step.yaml` with a `_target_` class, and select it with `scheduler=step`.
 
 # Usage
-Run every command from the repository root.
+Run the commands with `uv run` from the repository folder, or directly (`deepaneseg-train …`) from any folder once
+`.venv` is activated or the package is installed with pip.
+
+**0. Dataset conversion:** a dataset of scans with aneurysm label masks can be converted to the layout above. The
+defaults match the ADAM challenge (`<case>/orig/TOF.nii.gz`, `<case>/aneurysms.nii.gz`, where 2 marks treated
+aneurysms); other datasets only need their own `scan` and `mask` patterns. Each aneurysm becomes a two-point
+annotation, and `Data_dir/cases.csv` maps the patients to the original cases.
+```
+uv run deepaneseg-prepare source_dir=ADAM data_dir=Data_dir labels=[1]
+uv run deepaneseg-prepare source_dir=my_data data_dir=Data_dir scan='scans/{case}.nii.gz' mask='masks/{case}.nii.gz'
+```
 
 **1. Data preparation:** skull-strip the volumes, then select the negative patch centers.
 ```
-uv run python scripts/remove_skull.py data_dir=Data_dir
-uv run python scripts/extract_points.py data_dir=Data_dir
+uv run deepaneseg-remove-skull data_dir=Data_dir
+uv run deepaneseg-extract-points data_dir=Data_dir
 ```
 
 **2. Split:** assign the patients to training, validation and testing.
 ```
-uv run python scripts/preprocess_data.py data_dir=Data_dir          # split=[0.7,0.2,0.1] overwrite=true
+uv run deepaneseg-preprocess data_dir=Data_dir          # split=[0.7,0.2,0.1] overwrite=true
 ```
 
 **3. Training:** outputs go to `Data_dir/0Work/my_training`; running the same command again resumes training.
 The training and validation volumes are kept in memory as float32, about 4 bytes per voxel (150 MiB for a
 512×512×150 scan).
 ```
-uv run python scripts/train.py data_dir=Data_dir name=my_training
+uv run deepaneseg-train data_dir=Data_dir name=my_training
 ```
 
 **4. Testing:** predictions of the test patients are written to `Data_dir/0Work/my_training/prediction/test`: a
 probability map per patient, and its detections (center and radius in mm, size, probability) as CSV and as `.fcsv`
 markups for 3D Slicer.
 ```
-uv run python scripts/predict.py train_dir=Data_dir/0Work/my_training
+uv run deepaneseg-predict train_dir=Data_dir/0Work/my_training
 ```
 Any other scan, or folder of scans, can be predicted the same way (skull-stripped first, like the training data):
 ```
-uv run python scripts/predict.py train_dir=Data_dir/0Work/my_training input=scan.nii.gz output_dir=results
+uv run deepaneseg-predict train_dir=Data_dir/0Work/my_training input=scan.nii.gz output_dir=results
 ```
 
 **5. Evaluation:** detection metrics of the test predictions (ADAM challenge criteria: sensitivity and false
 positives per patient), written to `Data_dir/0Work/my_training/evaluation`.
 ```
-uv run python scripts/evaluate.py train_dir=Data_dir/0Work/my_training    # threshold=0.5 min_size=null
+uv run deepaneseg-evaluate train_dir=Data_dir/0Work/my_training    # threshold=0.5 min_size=null
 ```
 
 # Differences from the published code
@@ -111,7 +121,8 @@ uv sync --extra cu126    # NVIDIA GPU (CUDA 12.6)
 uv sync --extra cpu      # CPU only (also the build used on macOS)
 ```
 This creates `.venv` with the exact versions of `uv.lock`, including pytest and ruff. Commands then run with
-`uv run`, e.g. `uv run pytest`. Without uv, `pip install -e .` also works (PyTorch's default PyPI build).
+`uv run`, e.g. `uv run pytest`. Without uv, `pip install -e .` also works (PyTorch's default PyPI build), or, to only
+use the commands, `pip install git+https://github.com/youssefassis/DeepAneSeg`.
 
 # Docker
 Build the image with the GPU (default) or CPU PyTorch build:
@@ -122,9 +133,9 @@ docker build -t deepaneseg:cpu --build-arg TORCH=cpu .
 Mount the data folder and run any script; outputs are written under `Data_dir/0Work` on the host:
 ```
 docker run --rm --gpus all --shm-size=8g -v /path/to/Data_dir:/data deepaneseg \
-    python scripts/train.py data_dir=/data name=my_training
+    deepaneseg-train data_dir=/data name=my_training
 docker run --rm --gpus all --shm-size=8g -v /path/to/Data_dir:/data deepaneseg \
-    python scripts/predict.py train_dir=/data/0Work/my_training
+    deepaneseg-predict train_dir=/data/0Work/my_training
 ```
 GPU runs need the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/) on the
 host. `--shm-size` gives the data-loading workers enough shared memory, and `--user $(id -u):$(id -g)` creates the

scripts/evaluate.py → deepaneseg/cli/evaluate.py

@@ -1,7 +1,6 @@
-#!/usr/bin/env python3
-"""Score a training's predictions on its test patients; configuration in configs/evaluate.yaml.
+"""Score a training's predictions on its test patients; configuration in deepaneseg/configs/evaluate.yaml.
 
-Example: python scripts/evaluate.py train_dir=/data/0Work/exp1
+Example: deepaneseg-evaluate train_dir=/data/0Work/exp1
 A detection (connected component above the threshold) is a true positive when its center lies within an
 aneurysm's radius (ADAM challenge criteria). Writes per_patient.csv and summary.json to <train_dir>/evaluation.
 """

scripts/extract_points.py → deepaneseg/cli/extract_points.py

@@ -1,7 +1,6 @@
-#!/usr/bin/env python3
 """Select negative patch centers (on vessels and in the parenchyma) for every patient.
 
-Configuration in configs/extract_points.yaml. Example: python scripts/extract_points.py data_dir=/data
+Configuration in deepaneseg/configs/extract_points.yaml. Example: deepaneseg-extract-points data_dir=/data
 Writes <output> (CSV) and its 3D Slicer .fcsv counterpart in each patient folder.
 Requires the 'noskull volume' produced by remove_skull.py.
 """

scripts/predict.py → deepaneseg/cli/predict.py

@@ -1,9 +1,8 @@
-#!/usr/bin/env python3
-"""Predict with a trained model; configuration in configs/predict.yaml.
+"""Predict with a trained model; configuration in deepaneseg/configs/predict.yaml.
 
 Examples:
-    python scripts/predict.py train_dir=/data/0Work/exp1                      # test patients of the split
-    python scripts/predict.py train_dir=/data/0Work/exp1 input=scan.nii.gz    # any scan, or a folder of scans
+    deepaneseg-predict train_dir=/data/0Work/exp1                      # test patients of the split
+    deepaneseg-predict train_dir=/data/0Work/exp1 input=scan.nii.gz    # any scan, or a folder of scans
 Data and model settings come from the training's resolved configuration (<train_dir>/.hydra/config.yaml).
 For each case, writes the probability map <name>.nii.gz and the detections <name>_detections.csv (and .fcsv,
 for 3D Slicer).

deepaneseg/cli/prepare.py

@@ -0,0 +1,57 @@
+"""Convert a dataset of scans and aneurysm masks to the pipeline's layout; configuration in deepaneseg/configs/prepare.yaml.
+
+Example (ADAM challenge data): deepaneseg-prepare source_dir=/data/ADAM data_dir=/data/converted labels=[1]
+Writes one P???? folder per case (volume, F.csv with the aneurysms, config.json) and cases.csv, which maps the
+patients to the original case names.
+"""
+
+import json
+import os
+import shutil
+import sys
+
+import hydra
+import pandas as pd
+
+import deepaneseg.data.io as dio
+from deepaneseg.data.prepare import aneurysm_points, find_cases
+from deepaneseg.utils import get_logger
+
+
+@hydra.main(version_base="1.3", config_path="../configs", config_name="prepare")
+def main(cfg):
+    logger = get_logger("Prepare")
+    cases = find_cases(cfg.source_dir, cfg.scan)
+    if not cases:
+        sys.exit(f"No scan matches '{cfg.scan}' in {cfg.source_dir}")
+    masks = {case: os.path.join(cfg.source_dir, cfg.mask.replace("{case}", case)) for case in cases} if cfg.mask else {}
+    missing = sorted(case for case, path in masks.items() if not os.path.isfile(path))
+    if missing:
+        raise FileNotFoundError(f"No aneurysm mask '{cfg.mask}' for {', '.join(missing)}")
+    if dio.fetch_patient_dirs(cfg.data_dir) and not cfg.overwrite:
+        sys.exit(f"{cfg.data_dir} already contains patients; pass overwrite=true to replace them")
+
+    rows = []
+    for i, (case, scan) in enumerate(cases.items(), start=1):
+        patient = os.path.join(cfg.data_dir, f"P{i:04d}")
+        os.makedirs(patient, exist_ok=True)
+        volume = "volume.nii.gz" if scan.endswith(".gz") else "volume.nii"
+        shutil.copyfile(scan, os.path.join(patient, volume))
+        config = {"init volume": volume}
+        points = []
+        if case in masks:
+            mask, affine = dio.read_nii_from_file(masks[case])
+            points = aneurysm_points(mask, affine, labels=cfg.labels)
+            if len(points):
+                pd.DataFrame(points, columns=["x", "y", "z"]).to_csv(os.path.join(patient, "F.csv"), index=False)
+                config["pts aneurysm"] = "F.csv"
+        with open(os.path.join(patient, "config.json"), "w") as f:
+            json.dump(config, f, indent=2)
+        rows.append({"patient": f"P{i:04d}", "case": case, "scan": scan, "aneurysms": len(points) // 2})
+        logger.info(f"{case} -> P{i:04d}: {len(points) // 2} aneurysms")
+
+    pd.DataFrame(rows).to_csv(os.path.join(cfg.data_dir, "cases.csv"), index=False)
+
+
+if __name__ == "__main__":
+    main()

scripts/preprocess_data.py → deepaneseg/cli/preprocess.py

@@ -1,7 +1,6 @@
-#!/usr/bin/env python3
-"""Split the patients into training/validation/testing sets; configuration in configs/preprocess.yaml.
+"""Split the patients into training/validation/testing sets; configuration in deepaneseg/configs/preprocess.yaml.
 
-Example: python scripts/preprocess_data.py data_dir=/data split=[0.7,0.2,0.1]
+Example: deepaneseg-preprocess data_dir=/data split=[0.7,0.2,0.1]
 Writes the split to data.split_file (<data_dir>/0Work/split_pats.json by default).
 """
 

scripts/remove_skull.py → deepaneseg/cli/remove_skull.py

@@ -1,7 +1,6 @@
-#!/usr/bin/env python3
-"""Skull-strip every patient volume; configuration in configs/remove_skull.yaml.
+"""Skull-strip every patient volume; configuration in deepaneseg/configs/remove_skull.yaml.
 
-Example: python scripts/remove_skull.py data_dir=/data
+Example: deepaneseg-remove-skull data_dir=/data
 The result is saved next to each volume and registered as 'noskull volume' in the patient's config.json.
 """
 

scripts/train.py → deepaneseg/cli/train.py

@@ -1,7 +1,6 @@
-#!/usr/bin/env python3
-"""Train a model; configuration in configs/train.yaml.
+"""Train a model; configuration in deepaneseg/configs/train.yaml.
 
-Example: python scripts/train.py data_dir=/data name=exp1 batch_size=8
+Example: deepaneseg-train data_dir=/data name=exp1 batch_size=8
 Checkpoints, TensorBoard logs and the resolved configuration (.hydra/) are written to <data_dir>/0Work/<name>.
 Running the same command again resumes from the last checkpoint.
 """

deepaneseg/configs/__init__.py

@@ -0,0 +1 @@
+"""Hydra configuration of the deepaneseg commands (a package so that installed commands find it)."""

configs/data/default.yaml → deepaneseg/configs/data/default.yaml

@@ -1,9 +1,9 @@
 # Patient data and patch geometry
 data_dir: ${data_dir}                    # directory containing the patient folders (P????)
 work_dir: ${.data_dir}/0Work             # split file and trainings are written here
-split_file: ${.work_dir}/split_pats.json # written by preprocess_data.py
+split_file: ${.work_dir}/split_pats.json # written by deepaneseg-preprocess
 volume: init volume                      # key, in each patient's config.json, of the volume to train on
 normalize: Linear                        # Linear (rescale to [0, 1]), Normal (zero mean, unit variance) or null
-negative_patch_centers: points.csv       # written by extract_points.py in each patient folder
+negative_patch_centers: points.csv       # written by deepaneseg-extract-points in each patient folder
 patch_shape: [48, 48, 48]                # patch dimension, in voxels
 patch_size: [38, 38, 38]                 # patch size, in mm (resolution = patch_size / patch_shape)

configs/evaluate.yaml → deepaneseg/configs/evaluate.yaml

@@ -1,6 +1,6 @@
 # Detection metrics (ADAM challenge criteria) of a training's predictions on its test patients
-train_dir: ???                            # training directory written by train.py
-predictions: ${train_dir}/prediction/test # <patient>.nii.gz probability maps written by predict.py
+train_dir: ???                            # training directory written by deepaneseg-train
+predictions: ${train_dir}/prediction/test # <patient>.nii.gz probability maps written by deepaneseg-predict
 threshold: 0.5                            # probability above which a voxel belongs to a detection
 min_size: null                            # detections smaller than this, in voxels, are ignored; null keeps all
 

configs/predict.yaml → deepaneseg/configs/predict.yaml

@@ -1,10 +1,10 @@
 # Predicts with a trained model; data and model settings are read from the training's config
-train_dir: ???                    # training directory written by train.py
+train_dir: ???                    # training directory written by deepaneseg-train
 input: null                       # NIfTI scan or folder of scans; null: the test patients of the training's split
 output_dir: null                  # null: <train_dir>/prediction/test, or <train_dir>/prediction/scans for input
 checkpoint: last_checkpoint.pytorch
 volume: noskull volume            # key, in each patient's config.json, of the volume to predict on
-skull_strip: true                 # skull-strip input scans as remove_skull.py does, to match volume above
+skull_strip: true                 # skull-strip input scans as deepaneseg-remove-skull does, to match volume above
 threshold: 0.5                    # detections: connected components above this probability
 min_size: null                    # detections smaller than this, in voxels, are ignored; null keeps all
 margin: 8                         # voxels discarded at each patch border when stitching predictions

deepaneseg/configs/prepare.yaml

@@ -0,0 +1,11 @@
+# Converts a dataset of scans and aneurysm label masks (defaults: ADAM challenge layout) to the pipeline's layout
+source_dir: ???                    # dataset to convert
+data_dir: ???                      # output folder, one P???? folder per case
+scan: "{case}/orig/TOF.nii.gz"     # scan of each case, relative to source_dir; {case} matches one name
+mask: "{case}/aneurysms.nii.gz"    # aneurysm label mask of a case, relative to source_dir; null if there are none
+labels: null                       # mask values that are aneurysms, e.g. [1] for ADAM (2: treated); null: any non-zero
+overwrite: false                   # replace existing patients in data_dir
+
+hydra:
+  run:
+    dir: ${data_dir}/0Work/logs/${hydra.job.name}/${now:%Y-%m-%d_%H-%M-%S}

deepaneseg/data/prepare.py

@@ -0,0 +1,42 @@
+"""Conversion of a dataset of scans and aneurysm masks to the layout used by the pipeline."""
+
+import glob
+import os
+import re
+
+import numpy as np
+
+from deepaneseg.volume.selection import connected_components_to_spheres
+
+
+def find_cases(source_dir, scan_pattern):
+    """
+    Returns {case name: scan path}, sorted by case, for the files matching scan_pattern in source_dir.
+    scan_pattern is a path relative to source_dir where {case} stands for one folder or file name part,
+    e.g. "{case}/orig/TOF.nii.gz" (ADAM challenge) or "scans/{case}.nii.gz".
+    """
+    if scan_pattern.count("{case}") != 1:
+        raise ValueError(f"scan pattern must contain {{case}} exactly once, got '{scan_pattern}'")
+    before, after = scan_pattern.split("{case}")
+    regex = re.compile(re.escape(before) + r"([^/]+)" + re.escape(after) + "$")
+    cases = {}
+    for path in glob.glob(os.path.join(source_dir, scan_pattern.replace("{case}", "*"))):
+        match = regex.search(os.path.relpath(path, source_dir))
+        if match:
+            cases[match.group(1)] = path
+    return dict(sorted(cases.items()))
+
+
+def aneurysm_points(mask, affine, labels=None):
+    """
+    Annotation points of the aneurysms of a label mask: each connected component of the selected labels (any
+    non-zero value if labels is None) is approximated by a sphere, described by the two ends of a diameter along x,
+    as in the F.csv files. Returns a 2Nx3 array of points in mm.
+    """
+    selected = mask > 0 if labels is None else np.isin(mask, list(labels))
+    spheres = connected_components_to_spheres(selected, affine)
+    offset = np.zeros_like(spheres[:, :3])
+    offset[:, 0] = spheres[:, 3]
+    points = np.empty((2 * len(spheres), 3))
+    points[0::2], points[1::2] = spheres[:, :3] - offset, spheres[:, :3] + offset
+    return points