Repository navigation
Plan paired PGS robustness studies and compare policies across seeds - #4
Draft
DylanCosto wants to merge 7 commits into
Draft
DylanCosto wants to merge 7 commits into
DylanCosto wants to merge 7 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repeated missing-data experiments require hand-written scenarios and separate inspection of each result. Add
robustness-planto expand authored policies across paired rates and seeds, androbustness-summarizeto compare completed studies in HTML, JSON and TSV.The planner retains controls and pairs deterministic masks across policies. The summarizer verifies experiment identity, complete rate/seed grids and output fingerprints before aggregating score errors, correlations, cutoff transitions and frozen-reference shifts. Differences are calculated within matching seeds first. Undefined metrics and participant denominators remain explicit. Models, baselines, cutoffs and reference groups are never pooled. These are descriptive sensitivity summaries, not confidence intervals or phenotype validation.
Include an independently checked offline study (
examples/robustness/run.py --study), a public eight-person pilot, and the completed 128-target/64-reference PGS000018 study. The larger study tests four omission rates and five seeds across 42 scenarios. All 128 exact baselines and source-input fingerprints match the earlier source-verified genome-wide run. The checker independently recalculates descriptive metrics from native exact sums and participant reference rows; it does not reconstruct the new masks from source genotypes. Reproduction instructions and checked records accompany the release notes.Repeated scenarios share frequency tables by resolved path and reuse random draws within each locus. The pilot's peak RSS fell from 2,060.48 MiB to 501.14 MiB after frequency sharing, with identical numerical outputs. Draw caching also preserves all numerical outputs; its observed elapsed time changed from 45.81 to 35.25 seconds. These are shared-host observations, not controlled speedup claims. The 128-person study used the frozen binary before these optimizations.
Validation
a6e6bc4: Linux/macOS tests, minimum Rust version, extracted-package examples, performance smoke and robustness smoke (run 38505852369).Stacked on #3 (
feature/robustness-experiments). The original candidate is preserved; an updated Linux candidate includes this study and its checks. No release tag or publication is created.