Add score_upgrade_prospects to train and score the leadership-upgrade model - #245
Merged
Merged
Conversation
… model score_upgrade_prospects (philanthropy.models) is the fit-and-score entry point over build_upgrade_snapshots: it trains a MajorGiftClassifier on every fully-resolved historical fiscal year, validates it with a walk-forward FiscalYearGroupedSplitter fold, and scores today's band-qualifying donors with a model refit on all history. The scored output carries an affinity score, rank, decile, a per-donor top-reasons heuristic built from global permutation importance, and a suggested-ask column left NaN (no ask-amount label exists yet to train one honestly). The report includes training-row counts, a low-data warning under ~500 rows, the activities_to_features id-match warning, and a top-N upgrade-rate lift over the naive "highest FY total" rule. build_upgrade_snapshots itself is refactored (no behaviour change) to split its per-year feature logic into two private helpers so the new function can build the same features for an unlabelled "current" row without duplicating the target/leakage logic. Wired into the CLI: `philanthropy train --task upgrade` reads a raw gift export via --source and calls score_upgrade_prospects directly, writing a scored CSV and printing the report; `philanthropy features` gained repeated --activity TYPE=PATH and --as-of flags to fold engagement data into the feature table the same way.
test_python_and_cli_upgrade_paths_produce_identical_scores failed on some pandas versions because the two code paths' donor_id index ended up as StringDtype with a different na_value sentinel (nan vs pd.NA) even though the actual index values already matched (checked separately on the line above). assert_series_equal was enforcing that internal dtype flavor by default; pass check_index_type=False so the test asserts what it's actually meant to: identical scores, not identical extension-dtype metadata.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
philanthropy.models.score_upgrade_prospects(gifts, *, activities=None, donors=None, threshold=1000.0, band=(100.0, 999.0), fiscal_year_start=7, as_of=None, random_state=None) -> (scores, report), the fit-and-score entry point overbuild_upgrade_snapshots: it trains aMajorGiftClassifieron every fully-resolved historical fiscal year, validates it on a walk-forwardFiscalYearGroupedSplitterfold, and scores today's band-qualifying donors (cut atas_of, never at a future fiscal-year end) with a model refit on all history.scorescarriesaffinity_score(0-100),rank,decile, a per-donortop_reasonsheuristic (z-score within the scored population weighted by global permutation importance, since there's no per-instance explainer in this package and SHAP stays out of scope for this series), andsuggested_askleftNaN— there's no ask-amount label in this data, and training one on the realized FY T+1 amount would train on the very quantity the upgrade label is derived from.reportcarries training-row counts, a low-data warning under ~500 rows, theactivities_to_featuresid-match warning (captured viawarnings.catch_warnings, not recomputed), and a top-N upgrade-rate lift over the naive "highest FY total" rule.philanthropy.models, notphilanthropy.ingestwherebuild_upgrade_snapshotslives. Unlike that function (raw tables in, donor-level table out, no estimator), this one's work is mostly a model-selection split, a classifier fit, and a permutation-importance call — model-domain, not ingest-domain. It still callsbuild_upgrade_snapshotsdirectly for the historical half rather than re-deriving its target/leakage logic.build_upgrade_snapshotsis refactored with no behaviour change: its per-year feature logic is split into two private helpers (_prepare_gifts,_snapshot_features_for_year) so the new function can build the same features for an unlabelled "current" row without duplicating that logic. All existingbuild_upgrade_snapshotstests pass unchanged.philanthropy train --task upgradereads a raw gift export via--source(reusing each source's own header-canonicalisation function rather than re-deriving it) and callsscore_upgrade_prospectsdirectly, writing a scored CSV to--outand printing the report; new--threshold,--band,--fiscal-year-start,--activityand--donorsflags feed it.philanthropy featuresgained repeated--activity TYPE=PATHand--as-offlags to foldactivities_to_featurescolumns into the feature table the same way.test_sklearn_compliance.pycheck assumed everyphilanthropy.models.__all__entry was a class; adding a plain function exposed that assumption (issubclass()on a non-class raisesTypeError), so it now skips non-class entries the same way it already skips non-estimator classes.Judgment calls
as_of(default: the latest gift date in the data) cuts the whole gift log once, up front, before either the historical training years or the current row are built, so appending a future-dated row can never move either — see the two explicit leakage tests intests/test_score_upgrade_prospects.py.drop_repeat_donors=False(plain fiscal-yeargroups), because the upgrade label is time-varying per fiscal year for the same donor, not a static per-donor label — the case that splitter's own docstring says to keepTruefor.ValueError.NaNrather than forcingAskAmountRecommenderonto a training signal this data doesn't have.Test plan
python -c "from philanthropy.models import score_upgrade_prospects"make ci(lint, mypy, doctests, full suite, coverage floor) — greenmake riskcov— green, 98% over the risk tier (floor 93%)tests/test_score_upgrade_prospects.py(leakage, band/threshold, low-data and activity-match-rate warnings, error paths) and CLI additions intests/test_cli.py, including the required Python-vs-CLI identical-scores acceptance test (test_python_and_cli_upgrade_paths_produce_identical_scores)