Skip to content

Validate LPLR tuning against its cross-validation objective - #446

Open
soodoku wants to merge 1 commit into
DoubleML:mainfrom
finite-sample:fix/lplr-tuning-tests
Open

soodoku wants to merge 1 commit into
DoubleML:mainfrom
finite-sample:fix/lplr-tuning-tests

Conversation

@soodoku

@soodoku soodoku commented Nov 2, 2026 •

Copy link
Copy Markdown

The LPLR Optuna tests compare post-tuning two-fold DoubleML predictions with their untuned counterparts, although tuning optimizes a separate five-fold CV objective. They also compare classifier RMSE even though the default classifier tuning objective is log loss. Improvement in these different evaluations is not guaranteed.

On current main, four cases with an explicitly supplied ml_a fail. For example, the two-fold RMSE worsens from 0.6979 to 0.7056, while the actual five-fold tuning RMSE improves from 0.7212 to 0.6600. Independently recomputing the selected model's score with scikit-learn agrees with the reported Optuna score.

This test-only correction:

  • Uses the same explicit five-fold splits for baseline, tuning, and independent score verification, with the appropriate data subset and scorer for each nuisance model.
  • Creates a fresh seeded random/TPE sampler per case, preventing shared sampler state from making results depend on earlier tests.
  • Checks that selected parameters are configured for every nuisance model and that the subsequent causal fit is finite.
  • Independently fits the selected ml_a estimator on the public inner splits and compares its averaged predictions with DoubleML's stored predictions, verifying that fitting actually uses the chosen parameters.

No estimator or tuning production code changes.

Reproduce with pytest doubleml/plm/tests/test_lplr_tune_ml_models.py -q: 4 failed, 4 passed before; 8 passed after. Formatting and lint checks pass. Independent automated review found no blocking issues. Complete-suite validation of the combined branch in #445 passed 11,613 cases with 66 intentional skips and four worker-process crashes; all four affected cases passed on serial retry. All eight LPLR tuning cases passed in that full run.

This arose while validating #445, which also includes this test-fix commit so it can be validated with the correction in place. The fix can be reviewed and merged independently first.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant