Repository navigation
Conversation
With a partial ml_param_space, DoubleML.tune_ml_models also set parameters for learners it did not tune: get_params(deep=True) of a clone fitted on the full data. For a Pipeline these contain the fitted step objects, so all folds shared them and every fold was predicted by the last fold's fit (in-sample). For any learner, parameters set before with set_ml_nuisance_params were overwritten by the constructor values. Only set the parameters of tuned learners, as DoubleMLScalar.tune_ml_models does. Check that a partially tuned PLR with an untuned Pipeline learner matches the same model with the tuned parameters set directly, and that parameters set before tuning are kept for untuned learners.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
With a partial
ml_param_space,DoubleML.tune_ml_modelsinstalled parameters even for learners it did not tune. For those,_dml_tune_optunareturnsget_params(deep=True)of a clone fitted on the full data, andtune_ml_modelsset these as fold-specific nuisance parameters.Pipeline: the parameters contain that clone's fitted step objects, so all folds shared them. Every fold was then predicted by the last fold's fit, i.e. in-sample.set_ml_nuisance_paramswere overwritten with the constructor values.Changes:
doubleml/double_ml.py:tune_ml_modelsonly sets the parameters of learners that were actually tuned (tuned_result.tuned); untuned learners keep their current parameters. This matches the scalar API (DoubleMLScalar.tune_ml_models), which only applies results withresult.tuned.tuned_result is Nonedoesn't change behaviour:set_ml_nuisance_params(..., None)already kept the existing parameters.doubleml/tests/test_dml_tune_optuna.py: two new tests, both of which fail before the fix.Pipelineml_lgives the same predictions, coefficient and standard error as the same model with the tunedml_mparameters set directly, on the same splits.Reference to Issues or PRs
Fixes #444
Comments
This changes what
paramsshows after a partialtune_ml_modelscall. An untuned learner now keeps its previous entry (None, or the values set earlier) instead of a full copy of its constructor parameters. The fitted model only differs when the bug was hit.PR Checklist