[vrjdev]

Description

With normalize_ipw=True, DoubleMLSSM rescaled the inverse probability weights by sum(d) / sum(d * s / (m * pi)), and by the analogue for the controls. That factor is roughly P(D = d) instead of about one. It shrank the bias-correction term of the doubly robust score, so estimates were biased whenever the outcome model was poor, even with correct propensity models.

• doubleml/irm/ssm.py: normalize with len(y) / sum(...), so that the weights average to one in each treatment arm. This matches the reference implementation causalweight::treatselDML (nrow(...) * ... / sum(...)). The change applies to both scores, missing-at-random and nonignorable.

• doubleml/irm/tests/_utils_ssm_manual.py: the same fix in the manual implementation, which had the same formula. test_ssm.py now checks the correct normalization.

• New doubleml/irm/tests/test_ssm_normalize_ipw.py, modelled on test_iivm_normalize_ipw.py from Fix IIVM normalize_ipw: normalize with the instrument z #413. It checks two things:

  • The normalized score equals the score with Hájek-normalized weights.
  • The normalized weights average to one in both treatment arms.

  It fails before the fix.

Reference to Issues or PRs

Fixes #426

Comments

Monte Carlo check: 60 replications, n = 4,000, correctly specified logistic models for m and pi, a constant outcome model (DummyRegressor), true ATE = 1.

	mean	bias	sd
normalize_ipw=False	0.981	−0.019	0.268
normalize_ipw=True, before	1.811	+0.811	0.119
normalize_ipw=True, after	0.993	−0.008	0.211

The R package has the same formula in R/double_ml_ssm.R (lines 640-641).

PR Checklist

• The title of the pull request summarizes the changes made.
• The PR contains a detailed description of all changes and additions.
• References to related issues or PRs are added.
• The code passes all (unit) tests.
• Enhancements or new feature are equipped with unit tests.
• The changes adhere to the PEP8 standards.

[SvenKlaassen]

Thanks @vrjdev,

I pushed a small follow-up. The idea is that normalize_ipw should do the same thing in SSM as in IRM: divide the inverse probability weights by their sample mean in each treatment arm, so they average to one. Writing it with np.mean makes that easier to see, and it avoids the n vs. sum(d) mix-up that caused the original bug.

Changes:

• ssm.py: _score_elements now computes the weights d·s/(m·π) and (1−d)·s/((1−m)·π) once. If normalize_ipw=True, they are divided by their mean, and both cases share the same score formula.
• test_ssm_normalize_ipw.py: I removed the assertion that recomputed the score by hand, since after the refactor it was the same expression as the implementation. Two tests remain:
  • a new test that sets s = 1 and π = 1 and checks that the SSM score matches the normalized IRM ATE score from DoubleMLIRM._score_elements (which uses _normalize_ipw), so SSM is checked against code it doesn't share
  • the existing checks that, with random s and π, the normalized weights average exactly one in each arm.

Both new tests fail on the old formula. All SSM tests pass.

Does this look fine to you?

[vrjdev]

Thanks @SvenKlaassen , looks good to me. Writing it with np.mean is clearer, and the comparison with the IRM score is a better independent check than recomputing the formula.