docs: end-to-end notebook on the KDD Cup 1998 donor file

.github/workflows/ci.yml

@@ -47,6 +47,14 @@ jobs:
       # One leg is enough: these exercise the public API against the working
       # tree, not the OS. --no-cov because pytest-cov's fail_under gate is for
       # philanthropy/ itself, not notebook cells.
+      # Notebook 04 fetches the KDD Cup 1998 archive (~35 MB) from UCI. Cache
+      # it so a slow or unavailable mirror does not fail every push; the key
+      # is the loader, which pins the file's checksum.
+      - name: Cache KDD Cup 1998 archive
+        uses: actions/cache@v6
+        with:
+          path: ~/philanthropy_data
+          key: kdd98-${{ hashFiles('philanthropy/datasets/_kdd98.py') }}
       - name: Execute example notebooks
         run: python -m pytest --nbmake examples/notebooks -q --no-cov
 

CHANGELOG.md

@@ -6,6 +6,14 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
 ## [Unreleased]
 
 ### Added
+- `examples/notebooks/04_kdd98_end_to_end.ipynb` and the tutorial page
+  "End to End on Real Donor Data": the whole library path on the 95,412 real
+  donors of KDD Cup 1998, from a wide export to a cleaned gift log, a Raiser's
+  Edge export with a pledge row, as-of RFM features, a lapse-model leakage
+  backtest, a response model with permutation importance, a gift-size model
+  with a calibrated interval and ask ladder, the mailing decision against the
+  $0.68 piece cost, a disparity check, and a saved model bundle. It runs in CI
+  with the other notebooks, and the downloaded archive is cached there.
 - Added test coverage in `tests/test_encounter_timezone.py` guarding that
   `EncounterRecencyTransformer` raises a `KeyError` naming the invalid timezone
   and does not emit the misleading "Already tz-aware" error. Closes #204.

docs/tutorials/end_to_end_on_real_donor_data.md

@@ -0,0 +1,55 @@
+---
+description: "The whole PhilanthroPy path on 95,412 real donors from KDD Cup 1998: ingest, as-of features, leakage, response and gift-size models, the mailing decision in dollars, and a disparity check."
+---
+
+# End to End on Real Donor Data
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/PhilanthroPy-Project/PhilanthroPy/blob/main/examples/notebooks/04_kdd98_end_to_end.ipynb)
+
+The other tutorials use synthetic data. This one runs on a real donor file:
+**KDD Cup 1998**, 95,412 donors with the gift history of 22 earlier mailings and
+the outcome of one more. The notebook is
+[`examples/notebooks/04_kdd98_end_to_end.ipynb`](https://github.com/PhilanthroPy-Project/PhilanthroPy/blob/main/examples/notebooks/04_kdd98_end_to_end.ipynb);
+open it in Colab with the badge above, or run it locally in about a minute.
+The first run downloads the dataset (about 35 MB) once to `~/philanthropy_data`.
+
+## What it covers
+
+| Step | What you do | PhilanthroPy pieces |
+|---|---|---|
+| 1. Size up the file | Concentration, response rate, average gift | `gift_concentration_gini`, `top_donor_share` |
+| 2. Clean gift log | Reshape a wide export into one row per gift, clean it, add fiscal years; retention and lifetime value | `CRMCleaner`, `FiscalYearTransformer`, `donor_retention_rate`, `donor_lifetime_value`, `plot_retention_waterfall` |
+| 3. CRM export | Read the gifts back as a Raiser's Edge export, with a pledge row that a naive sum would double-count | `read_raisers_edge_gifts`, `raisers_edge_gifts_to_features` |
+| 4. As-of features | Recency, frequency, monetary value and tenure as of the mailing date | `RFMTransformer(as_of=...)` |
+| 5. Leakage | Backtest a lapse model with features built as of each period against the same features built over the whole file | `LapsePredictor`, `FiscalYearGroupedSplitter` |
+| 6. Response model | Who will respond, with a leakage-safe wealth-screen imputer, and why they score high | `WealthScreeningImputer`, `MajorGiftClassifier`, `plot_affinity_distribution`, `donor_feature_importance` |
+| 7. Gift size | Predicted gift with a calibrated 90% interval, and an ask ladder | `AskAmountRecommender`, `GiftIntervalCalibrator`, `interval_report` |
+| 8. Mailing decision | Mail when expected gift beats the $0.68 cost, compared against mailing everyone | `fundraising_roi`, `cost_per_dollar_raised` |
+| 9. Disparity check | Selection rate by recorded gender, four-fifths rule | `selection_rate_by_group`, `disparate_impact_ratio` |
+| 10. Save | Store the fitted pipeline with its features and versions | `save_model`, `load_model` |
+
+## What the notebook finds
+
+From one run (`random_state=0`, 30% of donors held out):
+
+- **Leakage inflates the backtest.** On a 20,000-donor sample, a walk-forward
+  backtest of the lapse model reads ROC-AUC 0.717 with as-of features and 0.801
+  when the same totals are built over the whole file. On the held-out final
+  mailing the whole-history backtest overpromises by 0.218, against 0.170 for
+  the as-of one. The full experiment on all 95,412 donors is in
+  [Real-data replication](../explanation/real_data_replication.md).
+- **Response model:** ROC-AUC 0.613 on held-out donors. The strongest signals
+  are last gift, largest gift and number of gifts.
+- **Gift-size interval:** certified 90.1% coverage; 88.7% observed on the test
+  responders.
+- **Mailing decision:** model-targeted mailing sends 18,109 pieces instead of
+  28,624 and nets $3,877 instead of $3,035 on the held-out donors.
+- **Disparity:** selection rates of 0.62 (F) and 0.66 (M), a ratio of 0.94,
+  above the usual 0.8 flag.
+
+The Raiser's Edge reader in step 3 is newer than the 0.7.1 release; until the
+next release, install from GitHub (the first cell shows how).
+
+!!! note "Dataset terms"
+    Under the KDD Cup 1998 terms, teaching material must not name the
+    organisation that supplied the data. Cite it only as "KDD Cup 1998".

docs/tutorials/index.md

@@ -5,6 +5,7 @@ Tutorials teach PhilanthroPy one step at a time. Each lesson is learning-oriente
 * [Building Your First Model](building_your_first_model.md)
 * [Avoiding Temporal Data Leakage](avoiding_temporal_data_leakage.md)
 * [Building a Grateful Patient Pipeline](building_a_grateful_patient_pipeline.md)
+* [End to End on Real Donor Data](end_to_end_on_real_donor_data.md)
 
 ## Which estimator do I need?