Skip to content

Add score_upgrade_prospects to train and score the leadership-upgrade model - #245

Merged
shivamlalakiya merged 3 commits into
mainfrom
e6-score-upgrade-prospects
Sep 25, 2026
Merged

shivamlalakiya merged 3 commits into
mainfrom
e6-score-upgrade-prospects

Conversation

@shivamlalakiya

Copy link
Copy Markdown
Contributor

Summary

  • Adds philanthropy.models.score_upgrade_prospects(gifts, *, activities=None, donors=None, threshold=1000.0, band=(100.0, 999.0), fiscal_year_start=7, as_of=None, random_state=None) -> (scores, report), the fit-and-score entry point over build_upgrade_snapshots: it trains a MajorGiftClassifier on every fully-resolved historical fiscal year, validates it on a walk-forward FiscalYearGroupedSplitter fold, and scores today's band-qualifying donors (cut at as_of, never at a future fiscal-year end) with a model refit on all history.
  • scores carries affinity_score (0-100), rank, decile, a per-donor top_reasons heuristic (z-score within the scored population weighted by global permutation importance, since there's no per-instance explainer in this package and SHAP stays out of scope for this series), and suggested_ask left NaN — there's no ask-amount label in this data, and training one on the realized FY T+1 amount would train on the very quantity the upgrade label is derived from.
  • report carries training-row counts, a low-data warning under ~500 rows, the activities_to_features id-match warning (captured via warnings.catch_warnings, not recomputed), and a top-N upgrade-rate lift over the naive "highest FY total" rule.
  • Placement: philanthropy.models, not philanthropy.ingest where build_upgrade_snapshots lives. Unlike that function (raw tables in, donor-level table out, no estimator), this one's work is mostly a model-selection split, a classifier fit, and a permutation-importance call — model-domain, not ingest-domain. It still calls build_upgrade_snapshots directly for the historical half rather than re-deriving its target/leakage logic.
  • build_upgrade_snapshots is refactored with no behaviour change: its per-year feature logic is split into two private helpers (_prepare_gifts, _snapshot_features_for_year) so the new function can build the same features for an unlabelled "current" row without duplicating that logic. All existing build_upgrade_snapshots tests pass unchanged.
  • CLI: philanthropy train --task upgrade reads a raw gift export via --source (reusing each source's own header-canonicalisation function rather than re-deriving it) and calls score_upgrade_prospects directly, writing a scored CSV to --out and printing the report; new --threshold, --band, --fiscal-year-start, --activity and --donors flags feed it. philanthropy features gained repeated --activity TYPE=PATH and --as-of flags to fold activities_to_features columns into the feature table the same way.
  • A pre-existing test_sklearn_compliance.py check assumed every philanthropy.models.__all__ entry was a class; adding a plain function exposed that assumption (issubclass() on a non-class raises TypeError), so it now skips non-class entries the same way it already skips non-estimator classes.

Judgment calls

  • Leakage guard: as_of (default: the latest gift date in the data) cuts the whole gift log once, up front, before either the historical training years or the current row are built, so appending a future-dated row can never move either — see the two explicit leakage tests in tests/test_score_upgrade_prospects.py.
  • Validation split: uses drop_repeat_donors=False (plain fiscal-year groups), because the upgrade label is time-varying per fiscal year for the same donor, not a static per-donor label — the case that splitter's own docstring says to keep True for.
  • Small-data handling: fewer than ~500 training rows warns and proceeds (flagged in the report); a walk-forward split needs at least 2 distinct historical fiscal years, and below that it warns and falls back to an in-sample fit for scores and feature-importance weights; zero historical rows is a hard ValueError.
  • Suggested ask: deliberately left NaN rather than forcing AskAmountRecommender onto a training signal this data doesn't have.

Test plan

  • python -c "from philanthropy.models import score_upgrade_prospects"
  • make ci (lint, mypy, doctests, full suite, coverage floor) — green
  • make riskcov — green, 98% over the risk tier (floor 93%)
  • New tests: tests/test_score_upgrade_prospects.py (leakage, band/threshold, low-data and activity-match-rate warnings, error paths) and CLI additions in tests/test_cli.py, including the required Python-vs-CLI identical-scores acceptance test (test_python_and_cli_upgrade_paths_produce_identical_scores)

… model

score_upgrade_prospects (philanthropy.models) is the fit-and-score entry
point over build_upgrade_snapshots: it trains a MajorGiftClassifier on
every fully-resolved historical fiscal year, validates it with a
walk-forward FiscalYearGroupedSplitter fold, and scores today's
band-qualifying donors with a model refit on all history. The scored
output carries an affinity score, rank, decile, a per-donor top-reasons
heuristic built from global permutation importance, and a suggested-ask
column left NaN (no ask-amount label exists yet to train one honestly).
The report includes training-row counts, a low-data warning under ~500
rows, the activities_to_features id-match warning, and a top-N
upgrade-rate lift over the naive "highest FY total" rule.

build_upgrade_snapshots itself is refactored (no behaviour change) to
split its per-year feature logic into two private helpers so the new
function can build the same features for an unlabelled "current" row
without duplicating the target/leakage logic.

Wired into the CLI: `philanthropy train --task upgrade` reads a raw gift
export via --source and calls score_upgrade_prospects directly, writing
a scored CSV and printing the report; `philanthropy features` gained
repeated --activity TYPE=PATH and --as-of flags to fold engagement data
into the feature table the same way.
test_python_and_cli_upgrade_paths_produce_identical_scores failed on
some pandas versions because the two code paths' donor_id index ended
up as StringDtype with a different na_value sentinel (nan vs pd.NA)
even though the actual index values already matched (checked
separately on the line above). assert_series_equal was enforcing that
internal dtype flavor by default; pass check_index_type=False so the
test asserts what it's actually meant to: identical scores, not
identical extension-dtype metadata.
@shivamlalakiya
shivamlalakiya merged commit 884d1f9 into main Sep 25, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant