fMRI preprocessing pipeline

This README documents the modernization of the Pygers Princeton Handbook for Reproducible Neuroimaging (https://brainhack-princeton.github.io/handbook/index.html) code/preprocessing pipeline for fMRI data on PNI’s cluster: new self-owned Singularity containers under /jukebox/pinsk/singularity-containers with updated software versions, a self-owned TemplateFlow cache, and a selectable defacing backend (pydeface or PyFaceWipe).

---

1. One-time setup

Software versions

Tool	Old	New
heudiconv	0.8.0	1.4.0 /jukebox/pinsk/singularity-containers/heudiconv/heudiconv-1.4.0.simg
MRIQC	0.15.1	24.0.2 /jukebox/pinsk/singularity-containers/mriqc/mriqc-24.0.2.simg
fMRIPrep	20.2.0	25.2.5 /jukebox/pinsk/singularity-containers/fmriprep/fmriprep-25.2.5.simg
pydeface	module 2.0.0	2.1.0 conda env /usr/people/mpinsk/miniconda3/envs/pydeface/bin/pydeface
pyfacewipe	— (new, optional)	0.1.1 (prebuilt wheel) conda env /usr/people/mpinsk/miniconda3/envs/pyfacewipe/bin/python — optional alternative defacer

Build the containers

Run on scotty.

export SINGULARITY_CACHEDIR=/jukebox/pinsk/singularity-containers/.singularity_cache
export SINGULARITY_TMPDIR=/jukebox/pinsk/singularity-containers/.singularity_tmp
mkdir -p "$SINGULARITY_CACHEDIR" "$SINGULARITY_TMPDIR"

mkdir -p /jukebox/pinsk/singularity-containers/{heudiconv,mriqc,fmriprep}

singularity build /jukebox/pinsk/singularity-containers/heudiconv/heudiconv-1.4.0.simg \
    docker://nipy/heudiconv:1.4.0
singularity build /jukebox/pinsk/singularity-containers/mriqc/mriqc-24.0.2.simg \
    docker://nipreps/mriqc:24.0.2
singularity build /jukebox/pinsk/singularity-containers/fmriprep/fmriprep-25.2.5.simg \
    docker://nipreps/fmriprep:25.2.5

After a clean build you can rm -rf the cache/tmp dirs.

Verify each with singularity run --cleanenv <image> --version.

Build the pydeface conda env

conda create --name pydeface python=3.11
conda activate pydeface
pip install --upgrade pip
pip install pydeface==2.1.0
pydeface --help          # verify
conda deactivate

Launcher lands at /usr/people/mpinsk/miniconda3/envs/pydeface/bin/pydeface (referenced by the deface scripts). pydeface calls FSL at runtime, provided by the scripts' module load fsl/....

Build the pyfacewipe conda env (optional — alternative defacer)

PyFaceWipe is an optional alternative to pydeface: it skull-strips with SynthStrip and then wipes facial voxels. It handles T1w, T2w, and most other contrasts. It is much faster than pydeface, but setup is more involved because the published package predates numpy 2 / nibabel 5 and needs a small set of source patches. **See PyFaceWipe_cluster_install.md.

Set up the TemplateFlow cache

The pipeline now uses its own TemplateFlow cache instead of the Hasson lab's:

mkdir -p /jukebox/pinsk/templateflow

Populate it with an explicit pre-fetch:

conda activate base
pip install templateflow
export TEMPLATEFLOW_HOME=/jukebox/pinsk/templateflow
python -c "from templateflow import api; [api.get(t) for t in ('MNI152NLin2009cAsym','MNI152NLin6Asym')]"

---

2. FreeSurfer note

• fMRIPrep requires the FreeSurfer license (it runs recon-all). Keep license.txt in code/preprocessing/ and the --fs-license-file flag.
• MRIQC does not need FreeSurfer or a license. It uses SynthStrip, which is bundled inside the container. No module load freesurfer, no license.
• The cluster's freesurfer/… module is not used by either containerized tool.

---

3. File-by-file changes from the original Pyger files

run_heudiconv.py

• Container path -> /jukebox/pinsk/singularity-containers/heudiconv/heudiconv-1.4.0.simg
• Multi-echo now emits proper echo-1/2/3 entities (1.4.0), not _e1/_e2/_e3
• Before re-running: clear bids/.heudiconv/sub-001 or add --overwrite

step2_preproc.sh (BIDS-validator cleanup after conversion)

Does six things, in order:

1. delete scout/localizer files (*scout*) — not valid BIDS.
2. delete duplicated/canceled runs (*dup*) — heudiconv's __dup- files.
3. delete empty *task-rest*_events.tsv/.json stubs (rest has no events).
4. set IntendedFor on all fieldmaps explicitly.
5. tidy sidecar inheritance: set TaskName: "rest" in the dataset-level task-*_bold.json, drop per-acquisition keys (ImageType, dcmmeta_reorient_transform) from them, and remove the duplicate TaskName from per-run sidecars — clears SIDECAR_FIELD_OVERRIDE and the TaskName TODO.
6. prune each *_scans.tsv to drop rows for the files deleted in 1-3 — otherwise the validator raises SCANS_FILENAME_NOT_MATCH_DATASET (an error).

Why step 4 sets ALL fmaps: heudiconv's populate_intended_for is unreliable — it always skips the multi-echo fmap (nipy/heudiconv#554).

Note: heudiconv writes BIDS files read-only; step2 flips them writable for the edits and restores read-only afterward.

globals.sh (defacing backend switch)

• Added DEFACE_TOOL (pydeface default, or pyfacewipe) and a deface_one helper function that dispatches to the chosen tool. Both defacing scripts now call deface_one "$img" instead of a hardcoded pydeface, so one variable flips the whole pipeline. Also defines the backend paths ($PYDEFACE, $PYFACEWIPE_PY, $PYFACEWIPE_CLI) and an optional PYFACEWIPE_GPU=1 (CPU by default). Override per-run without editing files — see §4.

pyfacewipe_cli.py (NEW — pydeface-compatible PyFaceWipe wrapper)

• PyFaceWipe is import-only (no command line); this shim gives it one and makes it write the same <img>_defaced.nii.gz output pydeface does, so the two are drop-in interchangeable. It absorbs PyFaceWipe's quirks — defaces into a temp dir so it can't overwrite the input (its output basename equals the source basename), sets COMPUTERNAME, and honors PYFACEWIPE_GPU. Lives in the scripts dir; invoked by deface_one with the pyfacewipe env's Python.

deface.sh (defaces raw BIDS anat)

• pydeface $img -> deface_one "$img" — routes through the $DEFACE_TOOL switch (pydeface or pyfacewipe) from globals.sh. Output naming and the mv *_defaced.nii.gz $defaced_dir/ step are unchanged, so downstream is unaffected.

deface_template.sh (defaces fMRIPrep-preprocessed anat)

• Rewritten to find every native-space desc-preproc_{T1w,T2w} under the subject's fmriprep folder, so it works whether fMRIPrep writes anat at the subject level (sub-XX/anat/, the multi-session default) or per session (sub-XX/ses-YY/anat/). No session argument needed; guards skip missing files.
• Calls deface_one "$img" (the $DEFACE_TOOL switch) instead of $PYDEFACE directly; still excludes space-* (standard-space) copies.

run_fmriprep.sh

• Container path -> /jukebox/pinsk/singularity-containers/fmriprep/fmriprep-25.2.5.simg

• TemplateFlow -> own cache: SINGULARITYENV_TEMPLATEFLOW_HOME=/templateflow + --bind /jukebox/pinsk/templateflow:/templateflow (replaces the hasson bind)

• Output dir must end in /fmriprep (/project/data/bids/derivatives/fmriprep): fMRIPrep 21+ uses BIDS-derivatives layout and writes subjects directly into the output dir — no fmriprep/ subfolder (unlike 20.2.0). Passing .../derivatives scatters sub-XX/ at the derivatives root (next to mriqc//deface/) and breaks deface_template.sh's path; pointing it at .../derivatives/fmriprep restores the expected derivatives/fmriprep/sub-XX layout.

• Flag rename (required in 25.x): --bold2t1w-dof 6 -> --bold2anat-dof 6

• Added --me-output-echos for the multi-echo (acq-me225mm) run. fMRIPrep does the optimal (T2*-weighted) combination and outputs the combined _bold by default; this flag ALSO writes the STC/HMC/SDC-corrected per-echo series (..._echo-1/2/3_desc-preproc_bold) that tedana needs. fMRIPrep does not run ME-ICA denoising — run tedana separately on those echoes for TE-dependent denoising.

• All other flags (--fs-license-file, --no-submm-recon, --output-spaces, --write-graph, --nthreads/--omp-nthreads) unchanged and still valid.

• NOTE: 25.2.5 results differ from 20.2.0 — reprocess all subjects on one version.

  ME workflow handoff: fMRIPrep (optimal combination + per-echo outputs) -> tedana (ME-ICA denoising) is a separate downstream step.

run_mriqc.sh (participant level)

• Container path -> /jukebox/pinsk/singularity-containers/mriqc/mriqc-24.0.2.simg
• Removed --correct-slice-timing (deleted in modern MRIQC — would crash)
• Added own TemplateFlow cache (env + bind)

run_mriqc_group.sh (group level)

• Container path + removed --correct-slice-timing + TemplateFlow bind (as above)
• Kept --modalities T1w bold

slurm_fmriprep.sh

• Bug fix: --array=001, 002, 003 (spaces are invalid to SLURM) -> --array=1-3 (printf "%03d" still pads back to 001)
• Otherwise unchanged — run_fmriprep.sh and deface_template.sh are per-subject steps and correctly belong inside the array.

slurm_mriqc.sh

• Same --array=1-3 fix
• Bug fix: removed the group-level call. It was running inside every array task, so group MRIQC fired once per subject in parallel, aggregating incomplete outputs and clobbering the same group files. Group must run once, after all participant jobs finish (see below).

slurm_mriqc_group.sh (NEW)

• A single (non-array) job that runs run_mriqc_group.sh once, submitted with a dependency on the participant array.

---

4. How to run

# 0A. Download/unzip the new_study_template.zip. This contains the folder structure you need to use.

# 0B. Download/place all preprocessing files in the code/preprocessing folder.

# 0C. Edit the paths in the globals.sh file, including the defacing backend paths.
#  (Remember to update YOUREMAIL@princeton.edu in al slum_*.sh scripts).

# 1. Convert DICOMs -> BIDS  (edit subject/session args as needed) and deface anatomicals.
#     (Remember to update container path in run_heudiconv.sh.)
./step1_preproc.sh 001 01 <dicom_session_name>

# 2. Clean up files and make them BIDS-compatible to pass bids-validator
#     (Remember to update Step 4: IntendedFor Fieldmaps section to explicitly set your fieldmaps).
./step2_preproc.sh 001

# 3. Run bids_validator:
#     (Remember to install BIDS Validator first (see Section 7 further down).
conda activate bids-validator
bids-validator data/bids

# 4A. MRIQC: single subject:
#      (Remember to update container path and template flow path in run_mriqc.sh)
./run_mriqc.sh 001

# 4B. MRIQC: group:
#      (Remember to update container path and template flow path in run_mriqc_group.sh)
sbatch slurm_mriqc_group.sh

# 5. fMRIPrep + per-subject deface (array).
#      (Remember to update container path and template flow path in run_fmriprep.sh.)
sbatch slurm_fmriprep.sh

Choosing the defacing tool. Both defacing steps use pydeface by default. To use PyFaceWipe instead, set DEFACE_TOOL=pyfacewipe in the environment — no file edits:

# step1 (raw anat)
DEFACE_TOOL=pyfacewipe ./step1_preproc.sh 001 01 <dicom_session_name>

# fMRIPrep array (post-fmriprep anat) — --export passes the variable into the job
sbatch --export=ALL,DEFACE_TOOL=pyfacewipe slurm_fmriprep.sh

To make PyFaceWipe the permanent default, change the one line in globals.sh to DEFACE_TOOL=${DEFACE_TOOL:-pyfacewipe}. (PyFaceWipe requires its patched conda env — see PyFaceWipe_cluster_install.md; pydeface needs nothing new.)

---

5. Notes

• In bids-validator, residual "recommended" warnings are expected: the validator should report zero errors; remaining warnings are optional metadata the scanner didn't record or that doesn't apply (PulseSequenceType, PartialFourierDirection, SpoilingType, Instructions, TaskDescription, CogPOID, HEDVersion). Leave them — they don't affect fMRIPrep/MRIQC. Optionally add GeneratedBy + SourceDatasets to dataset_description.json and expand /README for provenance.
• Defacing backend: pydeface (default) or pyfacewipe, selected with DEFACE_TOOL (see §4). pyfacewipe needs its patched conda env (PyFaceWipe_cluster_install.md); pydeface relies on the module load fsl the scripts already do. Whichever you use, eyeball one defaced anatomical in fsleyes (face removed, cerebellum intact) before trusting the batch.
• Update #SBATCH --mail-user=YOUREMAIL@princeton.edu in the SLURM scripts.

---

6. Fieldmaps

Modify step2_preproc.sh to set your fieldmap’s IntendedFor correctly. step2_preproc.sh sets each fieldmap's IntendedFor explicitly, because heudiconv's auto-population is unreliable with multi-echo EPI: it always skips the multi-echo fieldmap (nipy/heudiconv#554), and I’ve had it miss a single-echo pair as well. The pipeline therefore does not rely on it.

---

7. BIDS validation — expected clean state

The validator is BIDS Validator 3.x. Install once in its own env, then run it against the BIDS root:

conda create -n bids-validator -c conda-forge deno
conda activate bids-validator
DENO_INSTALL_ROOT=$CONDA_PREFIX deno install -ERWN -g -f -n bids-validator jsr:@bids/validator

bids-validator /jukebox/pinsk/cerebellum/pilot/data/bids

After running step1_preproc (convert) + step2_preproc.sh on every subject and filling in dataset_description.json and README, a clean run should have zero errors. The only warnings that should remain are optional "recommended metadata" that the scanner didn't record or that doesn't apply — do NOT chase these (none affect fMRIPrep or MRIQC):

• HEDVersion (dataset_description) — only relevant if you use HED tags.
• PulseSequenceType, PartialFourierDirection, SpoilingType — scanner fields dcm2niix didn't extract.
• Instructions, TaskDescription, CogPOID — task-paradigm fields; N/A for resting state. fieldmap