Make the Nextflow workflows actually run, and execute the raw-reanalysis path for real

[sr320]

Ran the CALLA2026_OSHV pilot on real hardware against real public FASTQ. It found sixteen defects, and the headline is uncomfortable: the reproducibility layer did not work. Three of the four workflows did not compile on current Nextflow, the fourth silently skipped two thirds of its processes while exiting 0, and aree harmonize --input — one of the six commands in the project brief — read a different file than the one you passed it.

All sixteen are fixed. The RNA-seq raw_reanalysis path now runs end to end.

Why a pilot rather than the full study

One contrast, 11 libraries, first 1M read pairs each (~1.5 GB), streamed from ENA and truncated. Purpose: make the workflow run, not produce biology.

This was the right call. A 226 GB download and a day of compute would have died on line 38 of main.nf, at def VALID_MODES = [...].

The run

38 processes green: FASTQC ×22 → fastp ×11 → Salmon index → Salmon quant ×11 → MultiQC → DESeq2 → standardize → manifest → report. ~7 min on an M4, native tools via -profile local.

Two results worth recording:

• 22,301 genes, every one with a real lfcSE, 22,276 with an unadjusted p-value. This is exactly what HESSER2024_VCOR cannot supply and the whole reason this study was chosen for raw reanalysis.
• Identifier resolution is 100.0% exact (4,000-identifier dry run) versus 87.2% for HESSER — confirming the reference-choice prediction recorded in the study YAML. All 33,068 GeneIDs in the reference GTF are in the crosswalk, none unmatched.

The output is not valid biology and has not been harmonized. First-N reads are not a random subsample, and 1M pairs under-powers a DE test. CALLA2026_OSHV still contributes zero evidence records.

The sixteen defects

Would not compile (Nextflow 26.04 strict DSL2): bare script-level statements in rnaseq and proteomics; workflow.onComplete at script level in three workflows; then params and workflow both unresolvable inside the relocated closure, giving an NPE after every otherwise-successful run.

Ran but broken: proteomics RENDER_REPORT copied the template onto itself (cp rejects identical paths); metabolomics set qc_json_for_manifest = Channel.empty().collect(), which emits nothing rather than an empty list — so EMIT_MANIFEST never fired and RENDER_REPORT, depending on it, never fired either. Exit 0, completed=1, no manifest, no report, no error. That is the most dangerous item on the list; every other defect announces itself.

DESeq2 module could not accept a real study: condition levels hardcoded to c("control","treatment") — an all-NA factor for any study not named that way, i.e. every real study; quant_subdir mandatory; tx2gene header assumed; ignoreTxVersion strips versions from quant.sf but not from tx2gene, so any NCBI GTF stops matching; and a bare $ before a quote mangled by Groovy interpolation, corrupting the R regex into sub("\\..*DIFFERENTIAL_EXPRESSION_DESEQ2, "".

The Nextflow → AREE handoff, never actually connected: comparison resolved by filename only, so workflow output matched nothing; --input was ignored — it selected a comparison and then harmonized the registry's results_file instead of the file given. Existing tests missed this because they always passed the declared path, so reading the wrong file produced identical output. It also made raw reanalysis impossible by construction, since a raw-mode study has results_file: null. Plus manifest writing crashed on any path outside the repo, which is where workflow output lives.

Also: SAMPLE_QC declared multiqc_data but MultiQC writes multiqc_report_data, failing the task as a missing output after MultiQC exited 0 and wrote everything.

Environment findings

• Quarto fails on exFAT. RENDER_REPORT dies in Quarto's cleanup with the work dir on an exFAT volume; identical run succeeds on APFS. Publishing Salmon directories to exFAT also fails. Keep -work-dir and --outdir on APFS.
• A read glob must not begin with a character class. [SD]*_{1,2}.fastq.gz, used to skip macOS AppleDouble sidecars, silently breaks fromFilePairs key extraction and yields empty sample ids.
• No container needed on Apple Silicon. brew install nextflow fastp fastqc salmon + BiocManager + pip install multiqc is sufficient, and avoids x86 emulation.

Docs

implementation_status.md, roadmap.md, first_raw_reanalysis.md and all four workflow READMEs are corrected. The status table previously claimed "DSL2 wiring correct; DAG executes in -stub-run mode" — that had become false, and the page now says so explicitly rather than quietly editing it out.

Every page keeps the qualifications prominent: subsampled, one of six comparisons, no container ever pulled, nothing harmonized, and random-effects pooling on real data still unexercised.

Verification

149 Python tests, ruff clean, docs render, and all four workflows run processed_results_harmonization end to end — which none of them did before this branch.

Also removes ~80 Nextflow run artifacts that git add -A swept into the first commit, and gitignores them.

🤖 Generated with Claude Code