Skip to content

ENH: Optional sequential reproducibility control for repeated cross-fitting - #1

Closed
soodoku wants to merge 2 commits into
mainfrom
enh/reproducibility-control
Closed

soodoku wants to merge 2 commits into
mainfrom
enh/reproducibility-control

Conversation

@soodoku

@soodoku soodoku commented Nov 2, 2026

Copy link
Copy Markdown

DoubleML currently asks users to choose a fixed repetition count. This proposal adds a scalar mean diagnostic over existing per-repetition coefficients and an opt-in sequential driver for a single-treatment, nonclustered PLR model.

The driver fits a private one-repetition model on independently shuffled partitions, checks after burn-in, never refits earlier repetitions, and returns either target_reached or budget_exhausted. Its separate mean result carries no sampling SE or p-value; the supplied model and its median-based inference are preserved. Fold-specific tuning, custom scores, clustered data, and multiple treatments are explicitly excluded.

The normal stopping rule follows Ritzwoller–Romano. Its error-rate interpretation is asymptotic. In 1,000 independent pairs on fixed synthetic data, tolerance 0.02 used 29.953 repetitions on average and had 6.0% exceedance (95% Monte Carlo interval 4.69%–7.65%); tolerance 0.01 used 122.261 repetitions and had 6.2% exceedance (4.87%–7.87%). Neither exhausted the budget. These results do not establish exact 5% calibration.

Validation completed:

  • 39 feature tests; 100% statement coverage of the new module.
  • 774 utility/PLR/default regression tests.
  • Ruff, pinned Black across project source, configured mypy, and documentation lint.
  • Source distribution and wheel build; new exports verified in the wheel.
  • API documentation builds; 21 warnings concern existing dataset references and inaccessible online inventories.

Full pytest run: 11,588 passed, 66 skipped, 2 failed. Both failures are existing LPLR Optuna tuning assertions (nuisance_space-ml_a0-random and nuisance_space-ml_a0-tpe, line 93 of test_lplr_tune_ml_models.py). Both also fail in a clean snapshot of upstream 469daf260d4ea3520b8265acf17a9d29e800c5e0 using the same environment; no LPLR code is changed here. The full run used four pytest workers and unchanged upstream downloads for its two remote fixtures through a temporary local test adapter. The final six additional feature cases were checked separately; all 39 feature cases pass.

This is a draft review PR in the finite-sample fork, before upstream submission. The benchmark source, raw results, and limitations are included under benchmarks/.

This pull request includes code written with the assistance of AI.
The code has not yet been reviewed by a human.

@soodoku

soodoku commented Nov 2, 2026

Copy link
Copy Markdown
Author

Superseded by the ready-for-review upstream PR: DoubleML#429 . The feature branch is retained.

@soodoku soodoku closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant