diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 72c3d66..eb648fb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,6 +47,14 @@ jobs: # One leg is enough: these exercise the public API against the working # tree, not the OS. --no-cov because pytest-cov's fail_under gate is for # philanthropy/ itself, not notebook cells. + # Notebook 04 fetches the KDD Cup 1998 archive (~35 MB) from UCI. Cache + # it so a slow or unavailable mirror does not fail every push; the key + # is the loader, which pins the file's checksum. + - name: Cache KDD Cup 1998 archive + uses: actions/cache@v6 + with: + path: ~/philanthropy_data + key: kdd98-${{ hashFiles('philanthropy/datasets/_kdd98.py') }} - name: Execute example notebooks run: python -m pytest --nbmake examples/notebooks -q --no-cov diff --git a/CHANGELOG.md b/CHANGELOG.md index 560b560..b2b686b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,14 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) ## [Unreleased] ### Added +- `examples/notebooks/04_kdd98_end_to_end.ipynb` and the tutorial page + "End to End on Real Donor Data": the whole library path on the 95,412 real + donors of KDD Cup 1998, from a wide export to a cleaned gift log, a Raiser's + Edge export with a pledge row, as-of RFM features, a lapse-model leakage + backtest, a response model with permutation importance, a gift-size model + with a calibrated interval and ask ladder, the mailing decision against the + $0.68 piece cost, a disparity check, and a saved model bundle. It runs in CI + with the other notebooks, and the downloaded archive is cached there. - Added test coverage in `tests/test_encounter_timezone.py` guarding that `EncounterRecencyTransformer` raises a `KeyError` naming the invalid timezone and does not emit the misleading "Already tz-aware" error. Closes #204. diff --git a/docs/tutorials/end_to_end_on_real_donor_data.md b/docs/tutorials/end_to_end_on_real_donor_data.md new file mode 100644 index 0000000..644f8f5 --- /dev/null +++ b/docs/tutorials/end_to_end_on_real_donor_data.md @@ -0,0 +1,55 @@ +--- +description: "The whole PhilanthroPy path on 95,412 real donors from KDD Cup 1998: ingest, as-of features, leakage, response and gift-size models, the mailing decision in dollars, and a disparity check." +--- + +# End to End on Real Donor Data + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/PhilanthroPy-Project/PhilanthroPy/blob/main/examples/notebooks/04_kdd98_end_to_end.ipynb) + +The other tutorials use synthetic data. This one runs on a real donor file: +**KDD Cup 1998**, 95,412 donors with the gift history of 22 earlier mailings and +the outcome of one more. The notebook is +[`examples/notebooks/04_kdd98_end_to_end.ipynb`](https://github.com/PhilanthroPy-Project/PhilanthroPy/blob/main/examples/notebooks/04_kdd98_end_to_end.ipynb); +open it in Colab with the badge above, or run it locally in about a minute. +The first run downloads the dataset (about 35 MB) once to `~/philanthropy_data`. + +## What it covers + +| Step | What you do | PhilanthroPy pieces | +|---|---|---| +| 1. Size up the file | Concentration, response rate, average gift | `gift_concentration_gini`, `top_donor_share` | +| 2. Clean gift log | Reshape a wide export into one row per gift, clean it, add fiscal years; retention and lifetime value | `CRMCleaner`, `FiscalYearTransformer`, `donor_retention_rate`, `donor_lifetime_value`, `plot_retention_waterfall` | +| 3. CRM export | Read the gifts back as a Raiser's Edge export, with a pledge row that a naive sum would double-count | `read_raisers_edge_gifts`, `raisers_edge_gifts_to_features` | +| 4. As-of features | Recency, frequency, monetary value and tenure as of the mailing date | `RFMTransformer(as_of=...)` | +| 5. Leakage | Backtest a lapse model with features built as of each period against the same features built over the whole file | `LapsePredictor`, `FiscalYearGroupedSplitter` | +| 6. Response model | Who will respond, with a leakage-safe wealth-screen imputer, and why they score high | `WealthScreeningImputer`, `MajorGiftClassifier`, `plot_affinity_distribution`, `donor_feature_importance` | +| 7. Gift size | Predicted gift with a calibrated 90% interval, and an ask ladder | `AskAmountRecommender`, `GiftIntervalCalibrator`, `interval_report` | +| 8. Mailing decision | Mail when expected gift beats the $0.68 cost, compared against mailing everyone | `fundraising_roi`, `cost_per_dollar_raised` | +| 9. Disparity check | Selection rate by recorded gender, four-fifths rule | `selection_rate_by_group`, `disparate_impact_ratio` | +| 10. Save | Store the fitted pipeline with its features and versions | `save_model`, `load_model` | + +## What the notebook finds + +From one run (`random_state=0`, 30% of donors held out): + +- **Leakage inflates the backtest.** On a 20,000-donor sample, a walk-forward + backtest of the lapse model reads ROC-AUC 0.717 with as-of features and 0.801 + when the same totals are built over the whole file. On the held-out final + mailing the whole-history backtest overpromises by 0.218, against 0.170 for + the as-of one. The full experiment on all 95,412 donors is in + [Real-data replication](../explanation/real_data_replication.md). +- **Response model:** ROC-AUC 0.613 on held-out donors. The strongest signals + are last gift, largest gift and number of gifts. +- **Gift-size interval:** certified 90.1% coverage; 88.7% observed on the test + responders. +- **Mailing decision:** model-targeted mailing sends 18,109 pieces instead of + 28,624 and nets $3,877 instead of $3,035 on the held-out donors. +- **Disparity:** selection rates of 0.62 (F) and 0.66 (M), a ratio of 0.94, + above the usual 0.8 flag. + +The Raiser's Edge reader in step 3 is newer than the 0.7.1 release; until the +next release, install from GitHub (the first cell shows how). + +!!! note "Dataset terms" + Under the KDD Cup 1998 terms, teaching material must not name the + organisation that supplied the data. Cite it only as "KDD Cup 1998". diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 2ab12e3..eef5dec 100755 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -5,6 +5,7 @@ Tutorials teach PhilanthroPy one step at a time. Each lesson is learning-oriente * [Building Your First Model](building_your_first_model.md) * [Avoiding Temporal Data Leakage](avoiding_temporal_data_leakage.md) * [Building a Grateful Patient Pipeline](building_a_grateful_patient_pipeline.md) +* [End to End on Real Donor Data](end_to_end_on_real_donor_data.md) ## Which estimator do I need? diff --git a/examples/notebooks/04_kdd98_end_to_end.ipynb b/examples/notebooks/04_kdd98_end_to_end.ipynb new file mode 100644 index 0000000..cece2b0 --- /dev/null +++ b/examples/notebooks/04_kdd98_end_to_end.ipynb @@ -0,0 +1,712 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7583a29c", + "metadata": {}, + "source": [ + "# 04: End to end on a real donor file (KDD Cup 1998)\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/PhilanthroPy-Project/PhilanthroPy/blob/main/examples/notebooks/04_kdd98_end_to_end.ipynb)\n", + "\n", + "Notebooks 01 to 03 run on synthetic data. This one runs on **95,412 real donors**:\n", + "the KDD Cup 1998 direct-mail file, with each donor's gift history across 22\n", + "earlier mailings and the outcome of one more mailing (`TARGET_B` = gave,\n", + "`TARGET_D` = how much).\n", + "\n", + "It walks the whole path a development shop would take:\n", + "\n", + "1. Size up the donor file (concentration, retention, lifetime value)\n", + "2. Turn a wide export into a gift log, clean it, and add fiscal years\n", + "3. Read the same gifts as a Raiser's Edge export, pledges and all\n", + "4. Build features **as of** the decision date\n", + "5. See what leakage does to a lapse model's backtest\n", + "6. Score who will respond, and explain the score\n", + "7. Predict gift size with an honest interval, and build an ask ladder\n", + "8. Decide who to mail, in dollars\n", + "9. Check the mailing list for group disparity\n", + "10. Save the model for next year\n", + "\n", + "The first run downloads the dataset (about 35 MB) to `~/philanthropy_data`\n", + "once; later runs read the cached copy. Nothing about your own data is sent\n", + "anywhere.\n", + "\n", + "> **Dataset terms.** Under the KDD Cup 1998 terms, teaching material must not\n", + "> name the organisation that supplied the data; it is cited here only as\n", + "> \"KDD Cup 1998\"." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3a09df81", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:56:35.771741Z", + "iopub.status.busy": "2026-09-24T22:56:35.771600Z", + "iopub.status.idle": "2026-09-24T22:56:39.774395Z", + "shell.execute_reply": "2026-09-24T22:56:39.773938Z" + } + }, + "outputs": [], + "source": [ + "# In Colab, install the library first (the Raiser's Edge reader needs 0.8+ or main):\n", + "# !pip install \"git+https://github.com/PhilanthroPy-Project/PhilanthroPy.git\"\n", + "import tempfile\n", + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "import pandas as pd\n", + "from sklearn.metrics import roc_auc_score\n", + "from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split\n", + "from sklearn.pipeline import make_pipeline\n", + "\n", + "from philanthropy.datasets import fetch_kdd98_donors\n", + "from philanthropy.ingest import raisers_edge_gifts_to_features, read_raisers_edge_gifts\n", + "from philanthropy.inspection import donor_feature_importance\n", + "from philanthropy.metrics import (\n", + " cost_per_dollar_raised,\n", + " disparate_impact_ratio,\n", + " donor_lifetime_value,\n", + " donor_retention_rate,\n", + " fundraising_roi,\n", + " gift_concentration_gini,\n", + " interval_report,\n", + " selection_rate_by_group,\n", + " top_donor_share,\n", + ")\n", + "from philanthropy.model_selection import FiscalYearGroupedSplitter\n", + "from philanthropy.models import (\n", + " AskAmountRecommender,\n", + " GiftIntervalCalibrator,\n", + " LapsePredictor,\n", + " MajorGiftClassifier,\n", + ")\n", + "from philanthropy.preprocessing import (\n", + " CRMCleaner,\n", + " FiscalYearTransformer,\n", + " RFMTransformer,\n", + " WealthScreeningImputer,\n", + ")\n", + "from philanthropy.utils import load_model, save_model\n", + "from philanthropy.visualisation import plot_affinity_distribution, plot_retention_waterfall\n", + "\n", + "RANDOM_STATE = 0\n", + "donors = fetch_kdd98_donors()\n", + "donors.shape" + ] + }, + { + "cell_type": "markdown", + "id": "8d030e92", + "metadata": {}, + "source": [ + "## 1. Size up the donor file\n", + "\n", + "Before any model: how concentrated is giving, and what does a donor look like?\n", + "`RAMNTALL` is each donor's lifetime giving up to the mailing being predicted." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "141db23e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:56:39.775794Z", + "iopub.status.busy": "2026-09-24T22:56:39.775685Z", + "iopub.status.idle": "2026-09-24T22:56:39.784250Z", + "shell.execute_reply": "2026-09-24T22:56:39.783835Z" + } + }, + "outputs": [], + "source": [ + "lifetime = donors[\"RAMNTALL\"]\n", + "print(f\"Donors: {len(donors):,}\")\n", + "print(f\"Lifetime giving, total: ${lifetime.sum():,.0f}\")\n", + "print(f\"Gini of lifetime giving: {gift_concentration_gini(lifetime):.3f}\")\n", + "print(f\"Share from the top 10% donors: {top_donor_share(lifetime, top_fraction=0.10):.1%}\")\n", + "print(f\"Responded to the next mailing: {donors['TARGET_B'].mean():.2%}\")\n", + "print(f\"Average gift when they did: ${donors.loc[donors['TARGET_D'] > 0, 'TARGET_D'].mean():.2f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "5421bf89", + "metadata": {}, + "source": [ + "## 2. From a wide export to a clean gift log\n", + "\n", + "The file is wide: one row per donor, with `RDATE_3`..`RDATE_24` (gift date,\n", + "encoded `YYMM`) and `RAMNT_3`..`RAMNT_24` (amount) for each earlier mailing.\n", + "Most CRM exports you meet are long instead: one row per gift. Reshape, then run\n", + "the two cleaning steps every analysis starts with." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "94f3e224", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:56:39.785428Z", + "iopub.status.busy": "2026-09-24T22:56:39.785348Z", + "iopub.status.idle": "2026-09-24T22:56:40.768663Z", + "shell.execute_reply": "2026-09-24T22:56:40.768256Z" + } + }, + "outputs": [], + "source": [ + "parts = []\n", + "for i in range(3, 25):\n", + " promo = donors[[\"CONTROLN\", f\"RDATE_{i}\", f\"RAMNT_{i}\"]].dropna()\n", + " promo.columns = [\"donor_id\", \"yymm\", \"gift_amount\"]\n", + " parts.append(promo)\n", + "gifts = pd.concat(parts, ignore_index=True)\n", + "gifts[\"gift_date\"] = pd.to_datetime(\n", + " \"19\" + gifts[\"yymm\"].astype(int).astype(str).str.zfill(4), format=\"%Y%m\"\n", + ")\n", + "gifts = gifts[[\"donor_id\", \"gift_date\", \"gift_amount\"]]\n", + "\n", + "gifts = (CRMCleaner(date_col=\"gift_date\", amount_col=\"gift_amount\")\n", + " .set_output(transform=\"pandas\").fit_transform(gifts)\n", + " .astype({\"donor_id\": int}))\n", + "fiscal = (FiscalYearTransformer(date_col=\"gift_date\", fiscal_year_start=7)\n", + " .set_output(transform=\"pandas\").fit_transform(gifts).astype(int))\n", + "gifts = pd.concat([gifts, fiscal], axis=1)\n", + "print(f\"{len(gifts):,} gifts from {gifts['donor_id'].nunique():,} donors, \"\n", + " f\"{gifts['gift_date'].min():%b %Y} to {gifts['gift_date'].max():%b %Y}\")\n", + "gifts.head()" + ] + }, + { + "cell_type": "markdown", + "id": "0f7e5ec9", + "metadata": {}, + "source": [ + "**Retention and lifetime value.** Compare the donors who gave in fiscal 1995\n", + "(July 1994 to June 1995) with those who gave in fiscal 1996. Read the number\n", + "with care: this file only contains people who gave between June 1995 and June\n", + "1996, so retention here runs higher than on a full donor file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "be98cae8", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:56:40.769831Z", + "iopub.status.busy": "2026-09-24T22:56:40.769757Z", + "iopub.status.idle": "2026-09-24T22:56:41.087156Z", + "shell.execute_reply": "2026-09-24T22:56:41.086758Z" + } + }, + "outputs": [], + "source": [ + "by_fy = gifts.groupby(\"fiscal_year\")[\"donor_id\"].apply(set)\n", + "prior, current = by_fy[1995], by_fy[1996]\n", + "earlier = set().union(*[by_fy[fy] for fy in by_fy.index if fy < 1995])\n", + "\n", + "retention = donor_retention_rate(current, prior)\n", + "lapsed = len(prior - current)\n", + "recovered = len((current - prior) & earlier)\n", + "acquired = len(current - prior - earlier)\n", + "print(f\"Retention FY1995 to FY1996: {retention:.1%}\")\n", + "\n", + "avg_annual_gift = gifts[gifts[\"fiscal_year\"] == 1996].groupby(\"donor_id\")[\"gift_amount\"].sum().mean()\n", + "ltv = donor_lifetime_value(avg_annual_gift, lifespan_years=5, retention_rate=retention)\n", + "print(f\"Average annual giving per FY1996 donor: ${avg_annual_gift:.2f}\")\n", + "print(f\"5-year lifetime value at that retention: ${ltv:.2f}\")\n", + "\n", + "plot_retention_waterfall(len(prior), acquired, lapsed, recovered);" + ] + }, + { + "cell_type": "markdown", + "id": "c564d9a9", + "metadata": {}, + "source": [ + "## 3. The same gifts as a Raiser's Edge export\n", + "\n", + "Real shops hand over a CRM export, not a tidy table. Write the gift log out\n", + "under Raiser's Edge's own column labels, add a pledge row the way Raiser's Edge\n", + "records one (a commitment row separate from the payment against it), and read\n", + "it back. The reader drops commitment rows so pledged dollars are not counted\n", + "twice." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "90eb5cbe", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:56:41.088396Z", + "iopub.status.busy": "2026-09-24T22:56:41.088279Z", + "iopub.status.idle": "2026-09-24T22:56:42.055148Z", + "shell.execute_reply": "2026-09-24T22:56:42.054363Z" + } + }, + "outputs": [], + "source": [ + "tmp = Path(tempfile.mkdtemp())\n", + "re_export = gifts.rename(columns={\n", + " \"donor_id\": \"Constituent ID\", \"gift_date\": \"Gift Date\", \"gift_amount\": \"Gift Amount\",\n", + "})[[\"Constituent ID\", \"Gift Date\", \"Gift Amount\"]].assign(**{\"Gift Type\": \"Cash\"})\n", + "pledge = pd.DataFrame({\n", + " \"Constituent ID\": [95515], \"Gift Date\": [\"1995-01-01\"],\n", + " \"Gift Amount\": [5000.0], \"Gift Type\": [\"Pledge\"],\n", + "})\n", + "pd.concat([re_export, pledge]).to_csv(tmp / \"re_gifts.csv\", index=False)\n", + "\n", + "raw = read_raisers_edge_gifts(tmp / \"re_gifts.csv\")\n", + "re_features = raisers_edge_gifts_to_features(raw, reference_date=\"1997-06-01\")\n", + "naive = raisers_edge_gifts_to_features(raw, reference_date=\"1997-06-01\", exclude_gift_types=None)\n", + "print(f\"Donor 95515 total, pledge filtered: ${re_features.loc['95515', 'total_gift_amount']:,.0f}\")\n", + "print(f\"Donor 95515 total, naive sum: ${naive.loc['95515', 'total_gift_amount']:,.0f}\")\n", + "re_features[[\"total_gift_amount\", \"gift_count\", \"largest_gift_amount\", \"recency_days\", \"years_active\"]].head()" + ] + }, + { + "cell_type": "markdown", + "id": "aa9d8169", + "metadata": {}, + "source": [ + "## 4. Features as of the decision date\n", + "\n", + "The mailing being predicted went out in **June 1997**. A handful of gifts in\n", + "the log are dated after that, late responses to earlier mailings. Anything\n", + "dated after the decision could not have been known when the list was pulled,\n", + "so `RFMTransformer(as_of=...)` drops it before rolling up recency, frequency\n", + "and monetary value." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f59873e2", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:56:42.056218Z", + "iopub.status.busy": "2026-09-24T22:56:42.056137Z", + "iopub.status.idle": "2026-09-24T22:56:42.106165Z", + "shell.execute_reply": "2026-09-24T22:56:42.105773Z" + } + }, + "outputs": [], + "source": [ + "AS_OF = \"1997-06-01\"\n", + "late = gifts[gifts[\"gift_date\"] >= AS_OF]\n", + "print(f\"Gifts dated on or after {AS_OF}: {len(late)} from {late['donor_id'].nunique()} donors\")\n", + "\n", + "rfm = (RFMTransformer(as_of=AS_OF, include_tenure=True)\n", + " .fit_transform(gifts[[\"donor_id\", \"gift_date\", \"gift_amount\"]])\n", + " .set_index(\"donor_id\"))\n", + "rfm.head()" + ] + }, + { + "cell_type": "markdown", + "id": "22027d0c", + "metadata": {}, + "source": [ + "## 5. What leakage does to a backtest\n", + "\n", + "This is the failure the library exists to prevent. Reshape the history into a\n", + "donor-period panel (one row per donor per mailing), label each row \"did this\n", + "donor lapse at the next mailing\", and fit `LapsePredictor` two ways:\n", + "\n", + "- **as of** each period: totals built only from mailings up to that period\n", + "- **whole history**: the same totals built over the entire file, including\n", + " mailings after the period being predicted\n", + "\n", + "Both are scored with walk-forward `FiscalYearGroupedSplitter`, which never\n", + "trains on the future. A random 20,000-donor sample keeps the run to about a\n", + "minute; the full result on all 95,412 donors is in the\n", + "[real-data replication](https://philanthropy-project.github.io/PhilanthroPy/explanation/real_data_replication/)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c4bc828e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:56:42.107321Z", + "iopub.status.busy": "2026-09-24T22:56:42.107248Z", + "iopub.status.idle": "2026-09-24T22:57:02.265360Z", + "shell.execute_reply": "2026-09-24T22:57:02.264767Z" + } + }, + "outputs": [], + "source": [ + "sample = donors.sample(20_000, random_state=RANDOM_STATE)\n", + "ramnt = sample[[f\"RAMNT_{i}\" for i in range(24, 2, -1)]].fillna(0.0).to_numpy() # oldest first\n", + "gave = ramnt > 0\n", + "n_periods = ramnt.shape[1]\n", + "\n", + "rows_as_of, rows_whole = [], []\n", + "cum_total, cum_n = np.zeros(len(sample)), np.zeros(len(sample))\n", + "for p in range(n_periods):\n", + " cum_total, cum_n = cum_total + ramnt[:, p], cum_n + gave[:, p]\n", + " next_gave = gave[:, p + 1] if p < n_periods - 1 else sample[\"TARGET_B\"].to_numpy() == 1\n", + " common = dict(period=p, recent=ramnt[:, p], lapsed=(~next_gave).astype(int))\n", + " rows_as_of.append(pd.DataFrame(dict(total=cum_total, n=cum_n, **common)))\n", + " rows_whole.append(pd.DataFrame(dict(total=ramnt.sum(1), n=gave.sum(1), **common)))\n", + "as_of_panel = pd.concat(rows_as_of, ignore_index=True)\n", + "whole_panel = pd.concat(rows_whole, ignore_index=True)\n", + "\n", + "\n", + "def backtest(panel, cv):\n", + " panel = panel[panel[\"period\"] < panel[\"period\"].max()] # hold out the final period\n", + " X, y = panel[[\"total\", \"n\", \"recent\"]].to_numpy(), panel[\"lapsed\"].to_numpy()\n", + " model = LapsePredictor(n_estimators=50, max_depth=10, random_state=RANDOM_STATE)\n", + " groups = panel[\"period\"].to_numpy() if isinstance(cv, FiscalYearGroupedSplitter) else None\n", + " return cross_val_score(model, X, y, cv=cv, groups=groups, scoring=\"roc_auc\").mean()\n", + "\n", + "\n", + "walk = FiscalYearGroupedSplitter(n_splits=3, drop_repeat_donors=False)\n", + "honest = backtest(as_of_panel, walk)\n", + "leaky = backtest(whole_panel, walk)\n", + "random_cv = backtest(as_of_panel, StratifiedKFold(3, shuffle=True, random_state=RANDOM_STATE))\n", + "print(f\"Walk-forward, as-of features: ROC-AUC {honest:.3f}\")\n", + "print(f\"Walk-forward, whole-history features: ROC-AUC {leaky:.3f} (inflated by {leaky - honest:+.3f})\")\n", + "print(f\"Random K-fold, as-of features: ROC-AUC {random_cv:.3f} (split choice moves it {random_cv - honest:+.3f})\")" + ] + }, + { + "cell_type": "markdown", + "id": "870ebca4", + "metadata": {}, + "source": [ + "The whole-history model looks better in the backtest because its totals\n", + "include gifts from after the period being predicted: it has seen part of the\n", + "answer. The real test is the final period, which neither model trained on.\n", + "Fit each on every earlier period and score that one. Both backtests\n", + "overpromise here, because the final period is the separate June 1997 mailing\n", + "rather than one more period of the same history; what matters is that the\n", + "whole-history backtest overpromises by more, and that extra is the part that\n", + "came from seeing the answer. A backtest is only useful if it tells you what\n", + "the model will do next, and the leaky one tells you the least." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dd13e0f6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:57:02.267376Z", + "iopub.status.busy": "2026-09-24T22:57:02.267265Z", + "iopub.status.idle": "2026-09-24T22:57:07.512460Z", + "shell.execute_reply": "2026-09-24T22:57:07.511921Z" + } + }, + "outputs": [], + "source": [ + "def held_out(panel):\n", + " last = panel[\"period\"].max()\n", + " train, latest = panel[panel[\"period\"] < last], panel[panel[\"period\"] == last]\n", + " model = LapsePredictor(n_estimators=50, max_depth=10, random_state=RANDOM_STATE)\n", + " model.fit(train[[\"total\", \"n\", \"recent\"]].to_numpy(), train[\"lapsed\"].to_numpy())\n", + " score = model.predict_lapse_score(latest[[\"total\", \"n\", \"recent\"]].to_numpy())\n", + " return roc_auc_score(latest[\"lapsed\"], score), score\n", + "\n", + "\n", + "honest_future, lapse_score = held_out(as_of_panel)\n", + "leaky_future, _ = held_out(whole_panel)\n", + "print(f\"As-of: backtest promised {honest:.3f}, final period delivered {honest_future:.3f} \"\n", + " f\"(overpromised by {honest - honest_future:.3f})\")\n", + "print(f\"Whole-history: backtest promised {leaky:.3f}, final period delivered {leaky_future:.3f} \"\n", + " f\"(overpromised by {leaky - leaky_future:.3f})\")\n", + "pd.Series(lapse_score, name=\"lapse score (0-100)\").describe().round(1)" + ] + }, + { + "cell_type": "markdown", + "id": "08464d8d", + "metadata": {}, + "source": [ + "## 6. Who will respond to the next mailing?\n", + "\n", + "One row per donor: demographics, the wealth screen (`WEALTH1`, `WEALTH2`,\n", + "`INCOME`, each missing for a quarter to half of donors), the file's own giving\n", + "summary, and the as-of RFM features from section 4. `WealthScreeningImputer`\n", + "learns its fill values from the training rows only, and `MajorGiftClassifier`\n", + "handles the remaining gaps natively.\n", + "\n", + "This target is a single point in time (every donor is scored on the same\n", + "mailing), so a stratified random split is the right split here, unlike the\n", + "panel above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25c1454d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:57:07.514121Z", + "iopub.status.busy": "2026-09-24T22:57:07.514036Z", + "iopub.status.idle": "2026-09-24T22:57:09.759391Z", + "shell.execute_reply": "2026-09-24T22:57:09.758807Z" + } + }, + "outputs": [], + "source": [ + "base_cols = [\"AGE\", \"INCOME\", \"WEALTH1\", \"WEALTH2\", \"NUMCHLD\", \"RAMNTALL\", \"NGIFTALL\",\n", + " \"LASTGIFT\", \"AVGGIFT\", \"MAXRAMNT\", \"MINRAMNT\", \"TIMELAG\"]\n", + "X = (donors.set_index(\"CONTROLN\")[base_cols]\n", + " .assign(HOMEOWNER=lambda d: (donors.set_index(\"CONTROLN\")[\"HOMEOWNR\"] == \"H\").astype(int))\n", + " .join(rfm.add_prefix(\"rfm_\")))\n", + "y_resp = donors.set_index(\"CONTROLN\").loc[X.index, \"TARGET_B\"]\n", + "y_amount = donors.set_index(\"CONTROLN\").loc[X.index, \"TARGET_D\"]\n", + "gender = donors.set_index(\"CONTROLN\").loc[X.index, \"GENDER\"]\n", + "\n", + "X_train, X_test, yb_train, yb_test, yd_train, yd_test, g_train, g_test = train_test_split(\n", + " X, y_resp, y_amount, gender, test_size=0.3, stratify=y_resp, random_state=RANDOM_STATE\n", + ")\n", + "\n", + "response_model = make_pipeline(\n", + " WealthScreeningImputer(wealth_cols=[\"WEALTH1\", \"WEALTH2\", \"INCOME\"]),\n", + " MajorGiftClassifier(random_state=RANDOM_STATE),\n", + ")\n", + "response_model.fit(X_train, yb_train)\n", + "p_respond = response_model.predict_proba(X_test)[:, 1]\n", + "affinity = response_model[-1].predict_affinity_score(response_model[:-1].transform(X_test))\n", + "print(f\"Test ROC-AUC: {roc_auc_score(yb_test, p_respond):.3f}\")\n", + "plot_affinity_distribution(affinity, labels=yb_test.to_numpy());" + ] + }, + { + "cell_type": "markdown", + "id": "a876eea6", + "metadata": {}, + "source": [ + "**Why does a donor score high?** Permutation importance: shuffle one column at\n", + "a time on held-out donors and measure how much the score degrades." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "09cc849b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:57:09.760994Z", + "iopub.status.busy": "2026-09-24T22:57:09.760870Z", + "iopub.status.idle": "2026-09-24T22:57:10.867328Z", + "shell.execute_reply": "2026-09-24T22:57:10.866623Z" + } + }, + "outputs": [], + "source": [ + "imp_rows = X_test.sample(8_000, random_state=RANDOM_STATE).index\n", + "importance = donor_feature_importance(\n", + " response_model, X_test.loc[imp_rows], yb_test.loc[imp_rows],\n", + " feature_names=list(X.columns), n_repeats=3, random_state=RANDOM_STATE, scoring=\"roc_auc\",\n", + ")\n", + "importance.head(10)" + ] + }, + { + "cell_type": "markdown", + "id": "83f75c4e", + "metadata": {}, + "source": [ + "## 7. How much will they give, with an honest range?\n", + "\n", + "Among donors who responded, predict gift size with `AskAmountRecommender`,\n", + "then wrap it in `GiftIntervalCalibrator`, which calibrates a 90% interval on\n", + "responders the model never saw. `interval_report` checks it on the test set." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a490e3a5", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:57:10.868965Z", + "iopub.status.busy": "2026-09-24T22:57:10.868849Z", + "iopub.status.idle": "2026-09-24T22:57:11.858393Z", + "shell.execute_reply": "2026-09-24T22:57:11.857717Z" + } + }, + "outputs": [], + "source": [ + "responders = yd_train > 0\n", + "X_resp, y_resp_amt = X_train[responders], yd_train[responders]\n", + "X_fit, X_cal, y_fit, y_cal = train_test_split(X_resp, y_resp_amt, test_size=0.3, random_state=RANDOM_STATE)\n", + "\n", + "amount_model = AskAmountRecommender(random_state=RANDOM_STATE).fit(X_fit, y_fit)\n", + "interval_model = GiftIntervalCalibrator(amount_model, alpha=0.10, score=\"log\").fit(X_cal, y_cal)\n", + "\n", + "test_resp = yd_test > 0\n", + "interval = interval_model.predict_gift_interval(X_test[test_resp])\n", + "print(f\"Certified coverage: {interval.attained_level[0]:.1%} (asked for {interval.requested_level:.0%})\")\n", + "interval_report(yd_test[test_resp], interval.lower, interval.upper, alpha=0.10)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1765f20a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:57:11.859937Z", + "iopub.status.busy": "2026-09-24T22:57:11.859849Z", + "iopub.status.idle": "2026-09-24T22:57:11.873136Z", + "shell.execute_reply": "2026-09-24T22:57:11.872684Z" + } + }, + "outputs": [], + "source": [ + "ladder = amount_model.ask_ladder(X_test[test_resp].head(5), multipliers=(1.0, 1.5, 2.5))\n", + "pd.DataFrame(ladder, columns=[\"ask\", \"target\", \"stretch\"],\n", + " index=X_test[test_resp].head(5).index).round(2).assign(\n", + " actual_gift=yd_test[test_resp].head(5))" + ] + }, + { + "cell_type": "markdown", + "id": "e11dde87", + "metadata": {}, + "source": [ + "## 8. Who should we mail? Decide in dollars\n", + "\n", + "Each piece cost **$0.68** to mail. Mail a donor when the expected gift,\n", + "P(respond) × predicted amount, beats that cost, and compare against mailing\n", + "everyone on the held-out 30%." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cef02168", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:57:11.874603Z", + "iopub.status.busy": "2026-09-24T22:57:11.874481Z", + "iopub.status.idle": "2026-09-24T22:57:11.911299Z", + "shell.execute_reply": "2026-09-24T22:57:11.910927Z" + } + }, + "outputs": [], + "source": [ + "COST = 0.68\n", + "expected_gift = p_respond * amount_model.predict(X_test)\n", + "mail = expected_gift > COST\n", + "\n", + "\n", + "def campaign(mask):\n", + " raised, cost = yd_test[mask].sum(), COST * mask.sum()\n", + " return raised, cost\n", + "\n", + "\n", + "for name, mask in [(\"Mail everyone\", np.ones(len(X_test), bool)), (\"Model-targeted\", mail)]:\n", + " raised, cost = campaign(mask)\n", + " print(f\"{name:15s} pieces {mask.sum():6,d} raised ${raised:9,.0f} cost ${cost:8,.0f} \"\n", + " f\"net ${raised - cost:8,.0f} ROI {fundraising_roi(total_raised=raised, total_fundraising_expense=cost):5.2f} \"\n", + " f\"cost per $ {cost_per_dollar_raised(total_fundraising_expense=cost, total_raised=raised):.2f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "1ebcedd3", + "metadata": {}, + "source": [ + "## 9. Does the list treat groups evenly?\n", + "\n", + "Selection rate by recorded gender, and the four-fifths-rule ratio (below 0.8\n", + "is the usual flag). Gender is not a model input here, but proxies can carry it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "58319199", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:57:11.912973Z", + "iopub.status.busy": "2026-09-24T22:57:11.912874Z", + "iopub.status.idle": "2026-09-24T22:57:11.926765Z", + "shell.execute_reply": "2026-09-24T22:57:11.926280Z" + } + }, + "outputs": [], + "source": [ + "mf = g_test.isin([\"M\", \"F\"]).to_numpy()\n", + "print(selection_rate_by_group(mail[mf].astype(int), g_test[mf]))\n", + "print(f\"Disparate impact ratio: {disparate_impact_ratio(mail[mf].astype(int), g_test[mf]):.3f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "54dbfd5e", + "metadata": {}, + "source": [ + "## 10. Save the model for next year\n", + "\n", + "`save_model` stores the fitted pipeline with its feature list, target and\n", + "library versions; `load_model` warns if you reload it under different versions." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "22b12838", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-24T22:57:11.928194Z", + "iopub.status.busy": "2026-09-24T22:57:11.928119Z", + "iopub.status.idle": "2026-09-24T22:57:11.976625Z", + "shell.execute_reply": "2026-09-24T22:57:11.976011Z" + } + }, + "outputs": [], + "source": [ + "bundle_path = save_model(response_model, tmp / \"response_model.joblib\",\n", + " features=list(X.columns), target=\"TARGET_B\")\n", + "bundle = load_model(bundle_path)\n", + "reloaded = bundle[\"model\"]\n", + "assert np.allclose(reloaded.predict_proba(X_test)[:, 1], p_respond)\n", + "sorted(bundle)" + ] + }, + { + "cell_type": "markdown", + "id": "252261b6", + "metadata": {}, + "source": [ + "## Where to go next\n", + "\n", + "- [Avoiding temporal data leakage](https://philanthropy-project.github.io/PhilanthroPy/tutorials/avoiding_temporal_data_leakage/): the as-of idea step by step\n", + "- [Real-data replication](https://philanthropy-project.github.io/PhilanthroPy/explanation/real_data_replication/): the full leakage experiment on all 95,412 donors\n", + "- [Which estimator do I need?](https://philanthropy-project.github.io/PhilanthroPy/tutorials/): a decision table from question to class" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/mkdocs.yml b/mkdocs.yml index 962d8e5..dff65ec 100755 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -109,6 +109,7 @@ nav: - tutorials/building_your_first_model.md - tutorials/avoiding_temporal_data_leakage.md - tutorials/building_a_grateful_patient_pipeline.md + - tutorials/end_to_end_on_real_donor_data.md - How-To Guides: - how-to/index.md - how-to/use_the_cli.md