Add build_upgrade_snapshots for leadership upgrade-model training tables - #242
Merged
Merged
Conversation
Fundraising teams running an upgrade model (will a mid-level donor move into leadership giving next year?) need a training table with one row per donor per fiscal year, built without letting next year's giving leak into this year's features. build_upgrade_snapshots(gifts, *, fiscal_years, threshold=1000, band=(100, 999), fiscal_year_start=7, activities=None, donors=None) does that: for each fiscal year T, it takes every donor whose FY T total lands in the upgrade band (and stays below threshold even if the band is wider), computes gift-derived features (prior-year totals, trend, largest gift, gift count, consecutive years given, months since last gift) from data through the end of T, optionally joins activities_to_features and static donor attributes, and sets target to whether the donor crossed threshold in FY T+1. Lives in philanthropy.ingest: it is the same shape as the other bridges there (raw/normalised tables in, one donor-level feature table out) and calls activities_to_features directly. preprocessing was the other candidate, but every class there is an sklearn TransformerMixin, and this function changes row count and manufactures a label column, which a transform() cannot do; model_selection's declared scope is cross-validation, not building the table a splitter consumes. Fiscal-year boundaries reuse FiscalYearTransformer rather than reimplementing the month arithmetic. The output's fiscal_year and donor-id columns feed FiscalYearGroupedSplitter directly, covered by an end-to-end test. Tested with 37 cases: band inclusion/exclusion (including the threshold carve-out on a wider band), target reading only FY T+1, feature values staying put when FY T+1+ rows are appended, idempotency, the activities/donors joins, and the splitter integration. make ci and make riskcov both pass with the new module at 100% coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fundraising teams running an upgrade model, will a mid-level donor move into leadership giving next year, need a training table with one row per donor per fiscal year, built so next year's giving can't leak into this year's features.
philanthropy.ingest.build_upgrade_snapshots(gifts, *, fiscal_years, threshold=1000, band=(100, 999), fiscal_year_start=7, activities=None, donors=None)does that.For each fiscal year T in
fiscal_years, the population is every donor whose FY T total giving falls insideband(inclusive) and stays belowthresholdeven ifbandis wider than that; a donor already giving at leadership level in T is not an upgrade candidate. Every feature is built from gift (and, if given, activity) rows dated at or before the end of FY T: prior-year totals (T-1, T-2), trend, largest single gift, gift count, consecutive years given, and months since the last gift.activities_to_featuresis called withas_ofset to the end of FY T when an activity log is supplied, and static donor attributes are joined in as given.targetis 1 if the donor's FY T+1 total reachesthreshold, computed from FY T+1 gift data only, including 0 for a donor who gave nothing at all that year. The output'sfiscal_yearand donor-id columns feedFiscalYearGroupedSplitterdirectly (covered by an end-to-end test).Why
philanthropy.ingestIt's the same shape as every other bridge already there: raw/normalised tables in, one donor-level feature table out, and it calls
activities_to_featuresdirectly rather than duplicating its logic.philanthropy.preprocessingwas the other candidate, but every class there is an sklearnTransformerMixin, and this function changes row count and manufactures a label column, neither of which atransform()can do.philanthropy.model_selectionhouses the splitter this feeds, but its declared scope is cross-validation, not building the table a splitter consumes.Fiscal-year boundaries reuse
FiscalYearTransformerrather than reimplementing the month arithmetic, matching itsfiscal_year_startconvention.Note on branch base
This was originally planned to stack on
e2-activities-to-features(PR #232) since it depends onactivities_to_features, which wasn't merged yet when this work started. Both #232 and PR #233 merged tomainwhile this was in progress, so this PR is based directly on currentmaininstead; no stacking needed.Test plan
python -c "from philanthropy.ingest import build_upgrade_snapshots; print('OK')"tests/test_upgrade_snapshots.py(37 tests): band inclusion/exclusion at both bounds, the threshold carve-out on a wider band,fiscal_yearset to T not T+1, multi-year stacking, target reading FY T+1 only (including "no gift at all" -> 0, and a T+2 gift never affecting it), feature values unchanged by the size or presence of FY T+1+ rows, idempotency on repeat calls, gift-derived feature correctness (totals, trend, largest gift, gift count, consecutive years given, months since last gift), theactivitiesanddonorsjoins (including the activity cutoff matching the end of FY T and a missing donor getting NaN rather than a dropped row), validation errors, and an end-to-endFiscalYearGroupedSplitterintegration test.make ci(flake8, mypy, doctest, full suite, coverage floor): 2177 passed, 8 skipped, overall coverage 98.62% against the 92% floor;_upgrade_snapshots.pyitself at 100%.make riskcov(risk-tier floor overpreprocessing/,models/,ingest/,cli.py,utils/_persistence.py): 98% against the 93% floor.sh scripts/install_hooks.shpre-push hook ran the full suite on push: 2177 passed, 8 skipped.Changelog / docs
## [Unreleased]->### AddedinCHANGELOG.md.build_upgrade_snapshotsto the Tier 2 (Beta) table indocs/reference/index.md: the gift-derived feature set is a minimal starting recipe and likely to be refined, same reasoning as the sibling ingest bridges.