Skip to content

fix: leave untuned learners' parameters unchanged in tune_ml_models - #447

Open
vrjdev wants to merge 1 commit into
DoubleML:mainfrom
vrjdev:fix-tune-ml-models-untuned-learners
Open

vrjdev wants to merge 1 commit into
DoubleML:mainfrom
vrjdev:fix-tune-ml-models-untuned-learners

Conversation

@vrjdev

@vrjdev vrjdev commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Description

With a partial ml_param_space, DoubleML.tune_ml_models installed parameters even for learners it did not tune. For those, _dml_tune_optuna returns get_params(deep=True) of a clone fitted on the full data, and tune_ml_models set these as fold-specific nuisance parameters.

  • For a Pipeline: the parameters contain that clone's fitted step objects, so all folds shared them. Every fold was then predicted by the last fold's fit, i.e. in-sample.
  • For any learner: parameters set earlier with set_ml_nuisance_params were overwritten with the constructor values.

Changes:

  • doubleml/double_ml.py: tune_ml_models only sets the parameters of learners that were actually tuned (tuned_result.tuned); untuned learners keep their current parameters. This matches the scalar API (DoubleMLScalar.tune_ml_models), which only applies results with result.tuned.
    • Tuned learners are handled as before.
    • The returned tuning results are unchanged.
    • Skipping tuned_result is None doesn't change behaviour: set_ml_nuisance_params(..., None) already kept the existing parameters.
  • doubleml/tests/test_dml_tune_optuna.py: two new tests, both of which fail before the fix.
    • A partially tuned PLR with an untuned Pipeline ml_l gives the same predictions, coefficient and standard error as the same model with the tuned ml_m parameters set directly, on the same splits.
    • Parameters set earlier for the untuned learner are kept.

Reference to Issues or PRs

Fixes #444

Comments

This changes what params shows after a partial tune_ml_models call. An untuned learner now keeps its previous entry (None, or the values set earlier) instead of a full copy of its constructor parameters. The fitted model only differs when the bug was hit.

PR Checklist

  • The title of the pull request summarizes the changes made.
  • The PR contains a detailed description of all changes and additions.
  • References to related issues or PRs are added.
  • The code passes all (unit) tests.
  • Enhancements or new feature are equipped with unit tests.
  • The changes adhere to the PEP8 standards.

With a partial ml_param_space, DoubleML.tune_ml_models also set parameters
for learners it did not tune: get_params(deep=True) of a clone fitted on
the full data. For a Pipeline these contain the fitted step objects, so all
folds shared them and every fold was predicted by the last fold's fit
(in-sample). For any learner, parameters set before with
set_ml_nuisance_params were overwritten by the constructor values. Only set
the parameters of tuned learners, as DoubleMLScalar.tune_ml_models does.

Check that a partially tuned PLR with an untuned Pipeline learner matches
the same model with the tuned parameters set directly, and that parameters
set before tuning are kept for untuned learners.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: tune_ml_models sets parameters for learners it did not tune: untuned Pipelines leak across folds and preset parameters are overwritten

1 participant