fturader

RF-DETR + SAM2 pipeline for FTU (Functional Tissue Unit) instance segmentation in histopathology images — H&E brightfield and multiplex fluorescence (CODEX / OME-TIFF) via a Beer-Lambert pseudo-H&E front end.

One call turns a tissue image into an (H, W) integer FTU label map — segment_ftu(image, channel_names, tissue_type). Install → drop in the weights → run.

New here? Read Quick start and run a getting-started notebook.
Deep Python API — loader decision tree, the low-level FTURader model, multiplex loaders, pseudo-H&E recipes, and the full support matrix: docs/python_api.md.
CLI reference: docs/cli.md · Pseudo-H&E internals: docs/multiplex_pseudo_he.md.

Requirements

Requirement	Version
Python	3.10
NVIDIA GPU + CUDA	≥ 11.8
torch	≥ 2.5.1
rfdetr	≥ 1.6.5 (validated on 1.8.0)
SAM-2	install from GitHub (pulled in automatically as `sam-2@git+…`)
supervision	≥ 0.26
tifffile	≥ 2024.1

The DINOv2 patch-size / positional-encoding warnings printed when RF-DETR loads a checkpoint are benign and version-independent — the FTU checkpoints were trained at patch_size=16 / resolution=1024 (not stock DINOv2's 14/518), so RF-DETR (any version) reports it is not loading stock backbone weights. The full RF-DETR weights load fine.

Installation

From source (editable):

pip install -e .[dev]      # [dev] adds pytest, jupyter; omit for runtime only

This pulls torch, rfdetr, supervision, tifffile, and installs SAM-2 from GitHub (sam-2@git+https://github.com/facebookresearch/sam2.git), so the install host needs network access to GitHub and a working C/CUDA toolchain for SAM-2's extensions. The optional [zarr] extra (pip install -e .[zarr]) adds lazy windowed reads for very large arrays.

Pre-trained weights

Running inference requires pre-trained model weights under weights_root (DEFAULT_WEIGHTS_ROOT = ~/.deepcell/models/ftusam_v0.1/, also the CLI's default --weights-root). The directory must contain per-organ RFDETRLarge_resolution_1024_<organ>.pth (kidney / largeintestine / lung / prostate / spleen) and a shared sam2.1_hiera_b+_epoch_300.pth. Two ways to get them, pick one:

1. Place local weights (no token). If you already hold the .pth files, drop them into ~/.deepcell/models/ftusam_v0.1/. segment_ftu and the CLI load from there by default — no network, no token.

2. Download from Deepcell (needs a token). Create an access token at users.deepcell.org, then set it as an environment variable in your shell (not in a notebook or source) and download:

export DEEPCELL_ACCESS_TOKEN=<your-token>
python -c "import fturader; fturader.download_model_weights()"   # → ~/.deepcell/models/ftusam_v0.1/

download_model_weights() fetches and unpacks the archive into ~/.deepcell/models/ftusam_v0.1/. Authentication goes through the DEEPCELL_ACCESS_TOKEN environment variable (_auth.py); a missing token raises a clear ValueError pointing back to users.deepcell.org.

Security: never commit a token or write it into a notebook cell / source file. Use a shell environment variable (or a secrets manager). The download path has not been smoke- tested here because no token was available; the local-weights path is the one exercised.

Smoke test (no GPU / no weights)

Verify the install can import and that the CLI parses, without a GPU or weights:

python -c "import fturader; from fturader.multiplex import SingleImageDataset, infer_multiplex; print('import OK')"
fturader --help
pytest -q -m unit          # fast, no GPU, no external checkpoints

A weight-free end-to-end dry run (channel classification only, no model) on any multichannel OME-TIFF:

fturader --input stitched.ome.tif --print-classification

Supported Organs (defaults)

Organ	min_mask_area (px²)	score_threshold
prostate	40	0.33
largeintestine	200	0.36
lung	1 000	0.25
kidney	5 000	0.25
spleen	10 000	0.38

Lung default raised in R3 (score_threshold 0.13 → 0.25; min_mask_area 1 000). The old 0.13 produced 400+ detections per crop with slow NMM merge; 0.25 gives cleaner, faster output (a 6000²-px crop @ 0.25 ≈ 219 FTUs / 41 s) with no hand-set threshold.

Tuning for fewer, cleaner detections: raise score_threshold and/or min_mask_area. CLI: --score-threshold / --min-mask-area (0 = organ default). Python: segment_ftu(..., score_threshold=…, min_mask_area=…) or any infer_wsi kwarg.

Quick start — `segment_ftu`

The shortest path to a result. Hand us a multichannel image as (C, H, W), the channel names, and the tissue type; get back an (H, W) integer FTU label map. This is the most general entry point — recipe="auto" chooses the pseudo-H&E channels per panel, and the model always runs the large-image tiling engine, so it works on the big mosaics (thousands of px on a side) that this pipeline targets.

import numpy as np
from fturader import segment_ftu

# A multiplex fluorescence mosaic: C channels, then height, width.
image = ...                          # np.ndarray, shape (C, H, W); C is unrestricted
channel_names = ["Hoechst1", "Cytokeratin", "Vimentin", "CollIV", ...]   # len == C

labels = segment_ftu(
    image,
    channel_names,
    tissue_type="largeintestine",    # one of SUPPORTED_ORGANS (selects the RF-DETR weights)
    recipe="auto",                   # auto-pick channels (recipes: docs/python_api.md)
    pixel_size_um=0.377,             # µm/px — pass it; detections are scaled by it
)
# labels: (H, W) int, 0 = background, 1..N = FTU instances

segment_ftu packages from_image → pseudo-H&E synthesis → tissue_type → weights → infer_wsi → rasterize labels. It needs pre-trained weights present under weights_root (see Pre-trained weights).

Useful options (full reference in src/fturader/api.py docstring):

Argument	Meaning
`mask=`	optional `(H, W)` boolean ROI; output labels are intersected with it (outside → `0`). `None` = whole image.
`recipe=`	`"auto"` (default) / `"auto:mean"` / `"auto:max"` / `"ck"` / an eosin marker list e.g. `["Cytokeratin", "Vimentin"]` / a `PseudoHEConfig`. `"v1"` / `"v2"` are deprecated aliases.
`he=`	`None` (default) synthesizes a named fluorescence panel; `True` passes an already-H&E `(H, W, 3)` image through unchanged. Channel count never decides H&E — `he` does.
`pixel_size_um=`	µm/px; `None` falls back to `0.4` with a synthesis warning.
`weights_root=`	weights directory; default `DEFAULT_WEIGHTS_ROOT` = `~/.deepcell/models/ftusam_v0.1`.
`return_report=True`	also return a machine-readable report dict `{"load": …, "synthesis": …, "infer": …}` (loader facts + structured synthesis config + inference plan with n_detections).
`**infer_kw`	forwarded to `infer_wsi` (e.g. `score_threshold`, `overlap_ratio`); the organ default `score_threshold` is used otherwise.

Client notes (read before running).

Working size: best at roughly 3000–10000 px on the long side. Far smaller crops give the tiler too little context; far larger only costs time.

auto can pick poorly on unusual panels. If the pseudo-H&E looks washed out, inspect describe() (via return_report=True) and specify channels manually — pass an eosin marker list, e.g. recipe=["Cytokeratin", "Vimentin"], or recipe="ck" for an epithelial organ that carries Cytokeratin.

Resource cost (U5–U9 profiling): VRAM ~1.4 GB; host RAM peak scales with area × channel count, roughly 16–35 GB for a ~10000²-px multi-channel mosaic; wall time scales with the number of detected FTUs.

For finer control over loading (separate tile files, in-memory arrays, block-wise focus) or for the infer_image / infer_wsi model objects directly, see docs/python_api.md.

Transparency: report & profiling

Three ways to see what a run will (or did) do — pick by what you want:

want	call	shape
human-readable summary of what synthesis will do	`dataset.describe()`	concise string (one `load` line + one `synthesis` line); `describe(verbose=True)` adds the full channel list, normalization, Beer-Lambert and best-focus z
machine-readable transparency bundle	`segment_ftu(..., return_report=True)`	one dict `{"load": …, "synthesis": …, "infer": …}`
just the tiling/route plan (no model run beyond geometry)	`model.infer_plan(image_hw, pixel_size_um)`	the `infer` section standalone (no pixels read, no inference run)

dataset.describe() covers the load + synthesis stages only; the infer stage (tiling, route, threshold, detection count) needs the model, so it lives in return_report=True / infer_plan. (The structured dict behind describe() is the private _describe_synthesis, exposed programmatically as report["synthesis"].)

Profiling — where wall time goes (and the NMM bottleneck). Pass profile=True to record per-stage timings:

model.infer_wsi(rgb, pixel_size_um=0.377, profile=True)
prof = model.last_profile                               # dict of per-stage seconds
# or, via the one-call entry:
labels, report = segment_ftu(..., profile=True, return_report=True)
prof = report["infer"]["profile"]

The CLI writes the same block as profile into the bundle summary.json. Keys: scale_s, n_tiles, tile_forward_s (= RF-DETR + SAM2 tile inference), assemble_s (cross-tile concat), merge_s (the NMM/NMS dedup — timed directly by wrapping sv.Detections.with_nmm), slicer_overhead_s, rescale_s, total_s, n_detections.

Empirical conclusion (R3 runs): tile_forward_s (inference) stays bounded at ~2–19 s across samples, while merge_s (the NMM merge) scales with detection count and dominates wall time at high counts:

sample	n_detections	merge_s
HBM573	6	0.02 s
CODEX (large intestine)	~26	~1.0 s
lung 6000²-px @ 0.25	~219	~27 s
HBM288 full	247	99 s (≈80 % of a 124 s run)

8× more detections (26 → 209) cost ~46× the merge time, with inference roughly unchanged. This is why raising score_threshold / min_mask_area makes runs both cleaner and faster.

Notebooks

Notebooks come in two tiers — start minimal, then go comprehensive:

Tier	Python	CLI
Getting started (minimal — one call, copy-paste)	`notebooks/getting_started_python.ipynb` — a single `segment_ftu` call	`notebooks/getting_started_cli.ipynb` — one `fturader` command, with the 3 key params (`--pixel-size` / `--channel-names` / `--he`) explained up front
Comprehensive (full tour)	`notebooks/walkthrough_python.ipynb` — `segment_ftu`, H&E + CODEX + OME, recipe comparison, bundle I/O	`notebooks/walkthrough_cli.ipynb` — the CLI equivalents

Read a getting-started notebook first for the shortest working path, then move to the matching comprehensive guide for the full feature tour. The kidney example was removed from the notebooks (kidney remains a fully supported organ in the API and the defaults table). Deprecated notebooks are kept under notebooks/legacy/.

The getting-started notebooks run on a small bundled crop with no download. The walkthrough notebooks are best read as a reference tour — every cell runs on the full raw HuBMAP slides, so to actually re-execute them you must first download the datasets they list (HuBMAP Data Portal links are in each walkthrough).

CLI

The unified fturader command handles both H&E brightfield and multiplex fluorescence (CODEX / OME-TIFF) inputs through a single --input. It always runs infer_wsi internally (tiling kicks in for large inputs).

Deprecated aliases kept for backward compatibility: fturader-infer (H&E only) and fturader-multiplex (multiplex only) share the same logic as fturader and will be removed in a future release.

Full parameter reference: docs/cli.md.

Quick examples

# H&E brightfield (kidney)
fturader \
    --input  /path/to/image.ome.tiff \
    --he true \
    --organ  kidney \
    --output-dir ./out/ \
    --save-overlay

# Multiplex CODEX directory (large intestine; manual eosin channels, or recipe=ck)
fturader \
    --input  /path/to/processed/ \
    --channel-names /path/to/extras/channelnames.txt \
    --pixel-size 0.377 \
    --recipe Cytokeratin,Vimentin \
    --organ  largeintestine \
    --output-dir ./out/

# Already-stitched OME-TIFF (channel names + pixel size read from OME-XML)
fturader \
    --input  stitched.ome.tif \
    --recipe auto \
    --organ  largeintestine \
    --output-dir ./out/

# Dry run: print channel classification only (no GPU / no weights)
fturader --input stitched.ome.tif --print-classification

Output bundle

File	Description
`labels.tif`	`(H, W)` uint16/uint32 instance map (zlib-compressed); 0=background, 1..N; high-confidence on top
`detections.npz`	compact-RLE per-instance masks + xyxy / confidence / rf_score / sam_score
`summary.json`	organ, pixel_size_um, n_detections, overlap_strategy, checkpoint paths, pseudo_he_report (effective pseudo-H&E config), …
`overlay_full.jpg`	exp4-style turbo filled-mask overlay (`--save-overlay`)
`overlay_2000.jpg`	same, short-side-2000px JPEG (`--save-overlay`)

Reload in Python:

from fturader.io import load_detections
dets, meta = load_detections("./out/")   # sv.Detections + metadata dict

Results

Instance segmentation (COCO AP / F1@)

Segmentation quality (pixel-level Dice & IoU)

Authors

Developed by Qilin Li (Van Valen Lab, California Institute of Technology), with contributions from Ross Barnowski.

If you use this software, please cite it using the metadata in CITATION.cff.

Navigation Menu

Uh oh!

fturader

Requirements

Installation

Pre-trained weights

Smoke test (no GPU / no weights)

Supported Organs (defaults)

Quick start — `segment_ftu`

Transparency: report & profiling

Notebooks

CLI

Quick examples

Output bundle

Results

Authors

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

fturader

Requirements

Installation

Pre-trained weights

Smoke test (no GPU / no weights)

Supported Organs (defaults)

Quick start — segment_ftu

Transparency: report & profiling

Notebooks

CLI

Quick examples

Output bundle

Results

Authors

Quick start — `segment_ftu`