Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Lab 5: Scalable analysis workflow

PUBH 6854, Fall 2026. Abdulmateen Oyediran.

This repo turns four separate analysis steps into one Snakemake workflow. It cleans messy_samples.csv (the Lab 3 data), builds a one-row-per-patient feature table from cohort.db (the Lab 4 TCGA lung adenocarcinoma database), stacks the two into one combined table, and writes a short summary report with a figure. Every script, query and rule here was written new for this lab. Only the two raw data files came from earlier labs.

The workflow

Snakemake builds this graph from the inputs and outputs each rule declares. It was generated with snakemake --rulegraph mermaid-js and is saved in evidence/rulegraph.mmd.

flowchart TB
    id0[all]
    id1[combine_tables]
    id2[clean_samples]
    id3[build_feature_table]
    id4[make_report]
    style id0 fill:#D95757,stroke-width:2px,color:#333333
    style id1 fill:#57D98B,stroke-width:2px,color:#333333
    style id2 fill:#8BD957,stroke-width:2px,color:#333333
    style id3 fill:#D9BF57,stroke-width:2px,color:#333333
    style id4 fill:#57BFD9,stroke-width:2px,color:#333333
    id4 --> id0
    id1 --> id0
    id2 --> id1
    id3 --> id1
    id1 --> id4
Loading
Rule Declared inputs Output
clean_samples data/raw/messy_samples.csv, scripts/clean_samples.py, and reference_date from config.yaml results/samples_clean.tsv
build_feature_table data/raw/cohort.db, sql/patient_features.sql, scripts/build_feature_table.py results/patient_features.tsv
combine_tables results/samples_clean.tsv, results/patient_features.tsv, scripts/combine_tables.py results/combined_table.tsv
make_report results/combined_table.tsv, scripts/make_report.py, and reference_date from config.yaml results/report.md, results/age_by_source.png

There are no hidden dependencies:

  • Each script gets every file path from its rule's command in the Snakefile, so it can't read a file the workflow doesn't know about.
  • Each script is also listed as an input, so editing the code re-runs that step.
  • The one setting, the reference date, is a declared params, so changing it re-runs the steps that use it.

Each rule also writes a log to logs/ with that step's checks and counts.

Setup

conda env create -f environment.yml
conda activate pubh6854-lab5

You can use mamba in place of conda. Tested on Windows 11 with Python 3.12 and Snakemake 9.27.1.

How to run

Run every command from the repository folder.

Full run from a clean state:

snakemake --delete-all-output
snakemake

The first command deletes every file the workflow produces. The second rebuilds all of them.

Partial re-run: change any input, then:

snakemake -n
snakemake

-n is a dry run. It lists which rules will re-run and why, without running anything. Only the rules downstream of the change re-run.

Notes:

  • profiles/default/config.yaml sets cores: 1 and nolock: true, so a plain snakemake works. Snakemake picks this file up automatically.
  • nolock is there because on my Windows machine Snakemake couldn't delete its own lock folder at the end of each run (PermissionError on .snakemake\locks), even after I moved the project out of OneDrive. The lock only matters when two runs share a folder at once.
  • Snakemake 9 compares file contents, not just timestamps. Re-saving a file without changing it does not trigger a re-run.

Evidence

File What it shows
evidence/rulegraph.mmd The rule graph above, as Snakemake generated it
evidence/01_delete_all_output.txt snakemake --delete-all-output removing every output: the clean state
evidence/02_full_run_from_clean_state.txt The full run from that state: all four rules ran (5 of 5 steps, counting all)
evidence/03_rerun_nothing_changed.txt Running again with nothing changed: Nothing to be done
evidence/04_partial_rerun_dry_run.txt The dry-run preview after the change below
evidence/05_partial_rerun.txt The partial re-run itself (4 of 4 steps)

The partial re-run. I added one line to sql/patient_features.sql (patients.vital_status) so the TCGA feature table also records whether each patient is living or deceased. Then:

Rule Re-ran? Reason Snakemake gave
build_feature_table Yes Updated input files: sql/patient_features.sql
combine_tables Yes Input files updated by another job: results/patient_features.tsv
make_report Yes Input files updated by another job: results/combined_table.tsv
clean_samples No Nothing it depends on changed

The change was entirely in the TCGA branch, so the messy-samples branch was left alone. Skipping work that doesn't need redoing is the point of a workflow manager.

Outputs

File Rows What it is
results/samples_clean.tsv 60 Cleaned messy samples, one row per sample
results/patient_features.tsv 566 TCGA feature table, one row per patient
results/combined_table.tsv 626 The combined analytic table (60 + 566)
results/report.md The summary report, with the figure results/age_by_source.png

Columns of the combined table:

Column Meaning
source messy_samples or tcga_luad
record_id sample_id (like S0001) or TCGA patient_id
sex Female, Male, or NA (not recorded)
age_years Whole years. Messy samples: age on the reference date (2026-01-01). TCGA: age at diagnosis.
site Messy samples: enrollment site (Site A, B or C). TCGA: the institution that sent the tissue.

Missing values are written as NA in every output file.

Data decisions

Cleaning messy_samples.csv

  • Sample IDs: two styles (S0001, s-0003) become S plus 4 digits.
  • Patient name: dropped. Initial plus surname isn't unique (T. Johnson appears 3 times), so it can't identify a person, and the analysis doesn't need it.
  • Sex: 9 spellings become Male (22) and Female (20). unknown, U and blank become missing (18).
  • Site: 7 spellings (siteB, site_a, SITE-A, Site C, and others) become Site A (23), Site B (20) and Site C (17).
  • Date of birth: 4 formats (MM/DD/YYYY, DD-Mon-YYYY, MM.DD.YY, YYYY-MM-DD). Each is matched with a regex and read with its exact layout. Two-digit years up to 26 are read as 20xx and the rest as 19xx. Four-digit dates in the same file show participants born as late as 2015, which supports this. It affects two rows: S0017 (2015) and S0033 (2012). Python's built-in two-digit-year rule would have turned 53 into 2053.
  • Age: whole years on a fixed reference date, 2026-01-01, set in config.yaml. A fixed date means the same input always gives the same ages, whatever day the workflow runs.
  • Glucose units: the 13 rows labelled mmol/L hold values in the same range as the mg/dL rows (the 12 that were plain numbers run from 103 to 217). Real mmol/L readings would be about 3 to 15, so the label is wrong, not the number. I kept every value as mg/dL and flagged those rows in glucose_label_was_mmol instead of converting them.
  • Glucose asterisks: two readings had a trailing asterisk (209.0*, 241.2*). I kept the number, and glucose_had_asterisk records which rows had one. The file doesn't say what the asterisk means.
  • Notes: turned into a true/false redraw_requested column (4 rows).
  • The script stops with an error instead of guessing if it finds a spelling, date format or unit it doesn't recognize, a duplicate ID, or an age outside 0 to 110.

TCGA feature table

  • One row per patient. The script checks that the query returns exactly as many rows as the patients table (566), with no patient twice.
  • The gene columns count protein-altering mutations (every type except Silent) in TP53, KRAS, EGFR, STK11 and KEAP1. Tumors with no mutation calls at all get 0, as in Lab 4.
  • Sex, age, stage and vital status are copied as recorded. Missing values stay NA (sex 52, age 71, stage 54).

Combining

  • The two sources share no patient ID, so I stacked them on the columns they share instead of joining them, and added a source column. No row is linked to a row in the other source.
  • TCGA sex is lowercase in the database, so it is changed to Female/Male to match. TCGA age at diagnosis is rounded down to whole years.
  • Age and site mean different things in the two sources (see the column table above). The report says so too.

Repository layout

Snakefile                      the workflow: four rules plus rule all
config.yaml                    settings: the reference date for ages
profiles/default/config.yaml   default command-line options (cores: 1, nolock: true)
environment.yml                conda environment
data/raw/                      course inputs, unchanged: messy_samples.csv, cohort.db
sql/patient_features.sql       the query for the TCGA feature table
scripts/                       one script per rule
results/                       everything the workflow writes
logs/                          one log per rule, with that step's checks and counts
evidence/                      console output from the clean, no-change and partial runs
exploration/                   scripts I used to look at the raw data first (not part of the workflow)

AI use

See AI_USAGE.md.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages