Skip to content

Plan paired PGS robustness studies and compare policies across seeds - #4

Draft
DylanCosto wants to merge 7 commits into
feature/robustness-experimentsfrom
feature/robustness-sweeps
Draft

DylanCosto wants to merge 7 commits into
feature/robustness-experimentsfrom
feature/robustness-sweeps

Conversation

@DylanCosto

@DylanCosto DylanCosto commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Repeated missing-data experiments require hand-written scenarios and separate inspection of each result. Add robustness-plan to expand authored policies across paired rates and seeds, and robustness-summarize to compare completed studies in HTML, JSON and TSV.

The planner retains controls and pairs deterministic masks across policies. The summarizer verifies experiment identity, complete rate/seed grids and output fingerprints before aggregating score errors, correlations, cutoff transitions and frozen-reference shifts. Differences are calculated within matching seeds first. Undefined metrics and participant denominators remain explicit. Models, baselines, cutoffs and reference groups are never pooled. These are descriptive sensitivity summaries, not confidence intervals or phenotype validation.

Include an independently checked offline study (examples/robustness/run.py --study), a public eight-person pilot, and the completed 128-target/64-reference PGS000018 study. The larger study tests four omission rates and five seeds across 42 scenarios. All 128 exact baselines and source-input fingerprints match the earlier source-verified genome-wide run. The checker independently recalculates descriptive metrics from native exact sums and participant reference rows; it does not reconstruct the new masks from source genotypes. Reproduction instructions and checked records accompany the release notes.

Repeated scenarios share frequency tables by resolved path and reuse random draws within each locus. The pilot's peak RSS fell from 2,060.48 MiB to 501.14 MiB after frequency sharing, with identical numerical outputs. Draw caching also preserves all numerical outputs; its observed elapsed time changed from 45.81 to 35.25 seconds. These are shared-host observations, not controlled speedup claims. The 128-person study used the frozen binary before these optimizations.

Validation

  • Stable Rust and Rust 1.88: all 179 tests passed, one optional local-data test ignored. Formatting and all-target Clippy passed.
  • Independent summary checker: four hand-calculated and corruption tests passed. Twelve additional synthetic runs checked both mask units, repeatability, one/four threads and full contribution attribution.
  • Public pilot: 112 participant/scenario rows, 98 score-metric checks and 150 distribution checks passed.
  • Larger public study: 5,376 participant/scenario rows, 294 score-metric checks, 84 score-distribution checks and 216 cutoff/reference-distribution checks passed.
  • Final Linux archive: 51 payload hashes and 49 local Markdown links checked; the extracted offline study passed 104 participant/scenario rows, 182 score-metric checks and 300 distribution checks. A second package build was byte-identical.
  • Hosted CI passed on final revision a6e6bc4: Linux/macOS tests, minimum Rust version, extracted-package examples, performance smoke and robustness smoke (run 38505852369).

Stacked on #3 (feature/robustness-experiments). The original candidate is preserved; an updated Linux candidate includes this study and its checks. No release tag or publication is created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant