Add scButterfly method (Multiome GEX↔ATAC)

[benjaminfreyuu]

Summary

Adds scButterfly (Cao et al., Nat. Commun. 2024, doi:10.1038/s41467-024-47418-x) — a dual-VAE + adversarial translator for single-cell cross-modality prediction — as a train/predict method.

• Scope: Multiome only (GEX↔ATAC) via scButterfly's high-level Butterfly class. Direction is detected from uns["modality"], covering both swap (ATAC→GEX) and normal (GEX→ATAC) variants. CITE-seq/ADT is out of scope and raises a clear ValueError.

What's included

• src/methods/scbutterfly/ — wrapper + scbutterfly_train + scbutterfly_predict, plus shared helpers (butterfly_common.py, chrom_utils.py).
• Registered in src/workflows/run_benchmark/{main.nf,config.vsh.yaml}.
• scripts/create_datasets/test_resources.sh — Multiome-only fixture-generation step (mirrors novel/simple_mlp).

Design notes

• Custom base image. scButterfly 0.0.9 hard-pins a 2022 stack (scipy==1.9.3, scvi-tools==0.19.0, episcanpy==0.3.2) that won't build on the shared openproblems/base_* Python. The engine uses python:3.9 + torch cu117 wheels (CUDA bundled, so the gpu label still works), scButterfly --no-deps + the pinned stack, and openproblems from openproblems-bio/core (as the base images do).
• Real serialized model. Train writes scButterfly's state_dicts (<model>/model/*.pt) + metadata.pkl; predict reconstructs the architecture and loads them, so it generalises to unseen test_mod1.
• Predict needs the training data. scButterfly has no transform-only preprocessing API (data_preprocessing refits HVG/peak-filter/TF-IDF on its input), so predict rebuilds the paired object from the train data and re-runs the deterministic preprocessing before loading weights. input_train_mod1 is therefore wired into predict.
• Runtime shims (applied before importing scButterfly): translate the removed size_average loss kwarg; clamp BCE input to [0,1] (float32 sigmoid can drift >1.0); make .cuda() a no-op when no GPU is present.

Verification

Verified locally:

• viash ns test -s src/methods/scbutterfly — passes both components.
• Manual train→predict end-to-end on bmmc_multiome/swap and .../normal: output matches test_mod2 in shape and var order, no NaNs, correct method_id/dataset_id.
• Predict loads real saved weights and infers (no retraining).

Needs CI / infra validation:

• GPU execution (local runs used a CPU .cuda() shim; the native GPU path was not exercised).
• The nextflow run_wf wrapper and full run_benchmark run (blocked locally by nextflow/JDK version constraints + component build size). The wrapper follows the standard train/predict pattern.
• The method fixture must be synced to the S3 test-resource bucket (rerun test_resources.sh, which regenerates and syncs it).

🤖 Generated with Claude Code

[rcannood]

Ran this end-to-end on a de.NBI Tesla T4 (train + predict) to check the gpu label is doing something real. scButterfly comes out clean on every axis -- nothing to fix here.

Registration: correct. scbutterfly is in both dependencies: of run_benchmark/config.vsh.yaml and the methods list in main.nf. It's the only one of the three that gets this right (#18 misses the dependency, #19 misses both).

GPU: genuinely used, in both steps. Built each component as an executable, ran on a T4 with --gpus all --ipc=host, sampling nvidia-smi --query-compute-apps once a second:

scbutterfly_train    EXIT=0  16s   peak 18% util   431 MiB   CUDA proc 8/15    -> USES GPU
scbutterfly_predict  EXIT=0  35s   peak  8% util   401 MiB   CUDA proc 25/33   -> USES GPU

(short run -- --rna_pretrain_epoch 2 --atac_pretrain_epoch 2 --translator_epoch 2 -- to confirm the device path, not to converge.)

Output contract: the prediction validates against file_prediction.yaml, and uns["method_id"] is right:

prediction: shape=(427, 1500) dtype=float32 uns.method_id='scbutterfly'
CONTRACT: OK

Worth calling out that the suffix-stripping in scbutterfly_predict:

method_id = meta["name"]
if method_id.endswith(_suffix): ...

is doing real work. #18 and #19 both write meta["name"] straight through and end up publishing senkin_predict / babel_predict as the method id, because the metrics copy ad_pred.uns["method_id"] verbatim into the score. You avoided that.

I merged current main into a local copy of this branch: clean merge, viash ns build green.

One thing to expect at merge time: all three PRs append to the same methods list in run_benchmark/main.nf and to test_resources.sh, so whichever lands second and third will conflict there. Trivial to resolve -- keep all the entries.