Skip to content

Fix unequal-fold PPI weighting - #13

Merged
Akram Zaytar (Akramz) merged 1 commit into
mainfrom
fix-unequal-folds
Nov 2, 2026
Merged

Akram Zaytar (Akramz) merged 1 commit into
mainfrom
fix-unequal-folds

Conversation

@calebrob6

Copy link
Copy Markdown
Member

Closes #11.

PPI now normalizes weights within each correction fold and averages the fold mean vectors equally, matching deployment imputation. The delta covariance sums the within-fold covariances and divides by the squared number of models. IWCV uses the same fold weighting, and effective sample size follows those normalized weights.

Bootstrap resampling preserves every correction fold. It samples rows or nested clusters within folds; clusters that span folds remain intact, and draws that omit a correction fold are redrawn. This keeps each replicate's imputation and correction on the same fixed-model target.

Direct cross-fit calls to ppi_estimate and ppi_macro_f1 with multiple deployment models now require fold_l. audit supplies it automatically. Single-model calls retain their existing behavior.

The issue's reproduction now returns accuracy 0.50 with a 95% interval of [0.448349, 0.551651], rather than 0.38 with an interval that excludes the truth.

The regression cases cover accuracy and pooled macro-F1 with unequal fold sizes, fold-specific inclusion weights, IWCV, delta variance, and row, nested-cluster and cross-fold-cluster bootstrap intervals. All 18 tests pass on Python 3.10 and 3.13 with pinned dependencies. Ruff lint passes, and a grouped audit with real logistic trainers produces identical bootstrap results for n_jobs=1 and n_jobs=2.

The README revision is separate in #12. This PR changes only the estimator, pipeline and regression tests.

@Akramz
Akram Zaytar (Akramz) merged commit 002157b into main Nov 2, 2026
5 checks passed
@Akramz
Akram Zaytar (Akramz) deleted the fix-unequal-folds branch November 2, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unequal correction folds bias PPI deployment estimates

2 participants