PUBH 6854, Fall 2026. Abdulmateen Oyediran.
This repo turns four separate analysis steps into one Snakemake workflow. It cleans
messy_samples.csv (the Lab 3 data), builds a one-row-per-patient feature table from
cohort.db (the Lab 4 TCGA lung adenocarcinoma database), stacks the two into one combined
table, and writes a short summary report with a figure. Every script, query and rule here was
written new for this lab. Only the two raw data files came from earlier labs.
Snakemake builds this graph from the inputs and outputs each rule declares. It was generated
with snakemake --rulegraph mermaid-js and is saved in evidence/rulegraph.mmd.
flowchart TB
id0[all]
id1[combine_tables]
id2[clean_samples]
id3[build_feature_table]
id4[make_report]
style id0 fill:#D95757,stroke-width:2px,color:#333333
style id1 fill:#57D98B,stroke-width:2px,color:#333333
style id2 fill:#8BD957,stroke-width:2px,color:#333333
style id3 fill:#D9BF57,stroke-width:2px,color:#333333
style id4 fill:#57BFD9,stroke-width:2px,color:#333333
id4 --> id0
id1 --> id0
id2 --> id1
id3 --> id1
id1 --> id4
| Rule | Declared inputs | Output |
|---|---|---|
clean_samples |
data/raw/messy_samples.csv, scripts/clean_samples.py, and reference_date from config.yaml |
results/samples_clean.tsv |
build_feature_table |
data/raw/cohort.db, sql/patient_features.sql, scripts/build_feature_table.py |
results/patient_features.tsv |
combine_tables |
results/samples_clean.tsv, results/patient_features.tsv, scripts/combine_tables.py |
results/combined_table.tsv |
make_report |
results/combined_table.tsv, scripts/make_report.py, and reference_date from config.yaml |
results/report.md, results/age_by_source.png |
There are no hidden dependencies:
- Each script gets every file path from its rule's command in the
Snakefile, so it can't read a file the workflow doesn't know about. - Each script is also listed as an input, so editing the code re-runs that step.
- The one setting, the reference date, is a declared
params, so changing it re-runs the steps that use it.
Each rule also writes a log to logs/ with that step's checks and counts.
conda env create -f environment.yml
conda activate pubh6854-lab5
You can use mamba in place of conda. Tested on Windows 11 with Python 3.12 and Snakemake 9.27.1.
Run every command from the repository folder.
Full run from a clean state:
snakemake --delete-all-output
snakemake
The first command deletes every file the workflow produces. The second rebuilds all of them.
Partial re-run: change any input, then:
snakemake -n
snakemake
-n is a dry run. It lists which rules will re-run and why, without running anything. Only the
rules downstream of the change re-run.
Notes:
profiles/default/config.yamlsetscores: 1andnolock: true, so a plainsnakemakeworks. Snakemake picks this file up automatically.nolockis there because on my Windows machine Snakemake couldn't delete its own lock folder at the end of each run (PermissionErroron.snakemake\locks), even after I moved the project out of OneDrive. The lock only matters when two runs share a folder at once.- Snakemake 9 compares file contents, not just timestamps. Re-saving a file without changing it does not trigger a re-run.
| File | What it shows |
|---|---|
evidence/rulegraph.mmd |
The rule graph above, as Snakemake generated it |
evidence/01_delete_all_output.txt |
snakemake --delete-all-output removing every output: the clean state |
evidence/02_full_run_from_clean_state.txt |
The full run from that state: all four rules ran (5 of 5 steps, counting all) |
evidence/03_rerun_nothing_changed.txt |
Running again with nothing changed: Nothing to be done |
evidence/04_partial_rerun_dry_run.txt |
The dry-run preview after the change below |
evidence/05_partial_rerun.txt |
The partial re-run itself (4 of 4 steps) |
The partial re-run. I added one line to sql/patient_features.sql (patients.vital_status)
so the TCGA feature table also records whether each patient is living or deceased. Then:
| Rule | Re-ran? | Reason Snakemake gave |
|---|---|---|
build_feature_table |
Yes | Updated input files: sql/patient_features.sql |
combine_tables |
Yes | Input files updated by another job: results/patient_features.tsv |
make_report |
Yes | Input files updated by another job: results/combined_table.tsv |
clean_samples |
No | Nothing it depends on changed |
The change was entirely in the TCGA branch, so the messy-samples branch was left alone. Skipping work that doesn't need redoing is the point of a workflow manager.
| File | Rows | What it is |
|---|---|---|
results/samples_clean.tsv |
60 | Cleaned messy samples, one row per sample |
results/patient_features.tsv |
566 | TCGA feature table, one row per patient |
results/combined_table.tsv |
626 | The combined analytic table (60 + 566) |
results/report.md |
The summary report, with the figure results/age_by_source.png |
Columns of the combined table:
| Column | Meaning |
|---|---|
source |
messy_samples or tcga_luad |
record_id |
sample_id (like S0001) or TCGA patient_id |
sex |
Female, Male, or NA (not recorded) |
age_years |
Whole years. Messy samples: age on the reference date (2026-01-01). TCGA: age at diagnosis. |
site |
Messy samples: enrollment site (Site A, B or C). TCGA: the institution that sent the tissue. |
Missing values are written as NA in every output file.
- Sample IDs: two styles (
S0001,s-0003) becomeSplus 4 digits. - Patient name: dropped. Initial plus surname isn't unique (
T. Johnsonappears 3 times), so it can't identify a person, and the analysis doesn't need it. - Sex: 9 spellings become
Male(22) andFemale(20).unknown,Uand blank become missing (18). - Site: 7 spellings (
siteB,site_a,SITE-A,Site C, and others) becomeSite A(23),Site B(20) andSite C(17). - Date of birth: 4 formats (MM/DD/YYYY, DD-Mon-YYYY, MM.DD.YY, YYYY-MM-DD). Each is
matched with a regex and read with its exact layout. Two-digit years up to 26 are read as
20xx and the rest as 19xx. Four-digit dates in the same file show participants born as late
as 2015, which supports this. It affects two rows: S0017 (2015) and S0033 (2012). Python's
built-in two-digit-year rule would have turned
53into 2053. - Age: whole years on a fixed reference date, 2026-01-01, set in
config.yaml. A fixed date means the same input always gives the same ages, whatever day the workflow runs. - Glucose units: the 13 rows labelled mmol/L hold values in the same range as the mg/dL
rows (the 12 that were plain numbers run from 103 to 217). Real mmol/L readings would be
about 3 to 15, so the label is wrong, not the number. I kept every value as mg/dL and
flagged those rows in
glucose_label_was_mmolinstead of converting them. - Glucose asterisks: two readings had a trailing asterisk (
209.0*,241.2*). I kept the number, andglucose_had_asteriskrecords which rows had one. The file doesn't say what the asterisk means. - Notes: turned into a true/false
redraw_requestedcolumn (4 rows). - The script stops with an error instead of guessing if it finds a spelling, date format or unit it doesn't recognize, a duplicate ID, or an age outside 0 to 110.
- One row per patient. The script checks that the query returns exactly as many rows as the
patientstable (566), with no patient twice. - The gene columns count protein-altering mutations (every type except Silent) in TP53, KRAS, EGFR, STK11 and KEAP1. Tumors with no mutation calls at all get 0, as in Lab 4.
- Sex, age, stage and vital status are copied as recorded. Missing values stay
NA(sex 52, age 71, stage 54).
- The two sources share no patient ID, so I stacked them on the columns they share instead
of joining them, and added a
sourcecolumn. No row is linked to a row in the other source. - TCGA sex is lowercase in the database, so it is changed to
Female/Maleto match. TCGA age at diagnosis is rounded down to whole years. - Age and site mean different things in the two sources (see the column table above). The report says so too.
Snakefile the workflow: four rules plus rule all
config.yaml settings: the reference date for ages
profiles/default/config.yaml default command-line options (cores: 1, nolock: true)
environment.yml conda environment
data/raw/ course inputs, unchanged: messy_samples.csv, cohort.db
sql/patient_features.sql the query for the TCGA feature table
scripts/ one script per rule
results/ everything the workflow writes
logs/ one log per rule, with that step's checks and counts
evidence/ console output from the clean, no-change and partial runs
exploration/ scripts I used to look at the raw data first (not part of the workflow)
See AI_USAGE.md.