Retention Radar

Ranks a SaaS customer base by 30-day churn risk and turns the model output into a weekly, revenue-weighted list of which accounts the retention team should call first.

Live demo | Data structure spec

---

Summary

• Problem: a retention team can only work a fixed number of accounts per week, so the question is not "who will churn" but "which accounts are worth the slot".
• Approach: a calibrated XGBoost model on a 7-table weekly account panel (146 features), scored into risk bands and a queue ranked by expected revenue at risk.
• Result: ROC-AUC 0.82, 6.3x lift on the top 50 accounts, and expected-value ranking protects 32% more monthly revenue than ranking by probability alone.
• Live: hosted demo dashboard, plus a one-command local pipeline (schema, synthetic data, model, scores, dashboard).

---

Why this matters

• Churn is where recurring-revenue businesses quietly lose money, and the retention budget is always smaller than the at-risk base.
• The output is an action, not a score: a ranked worklist with the reason each account is flagged and the revenue it represents.
• The modelling choices here (calibration, expected-value ranking, honest uncertainty) are the difference between a dashboard people trust and one they ignore.

---

Results

Evaluated with 5-fold out-of-fold predictions over 21,727 labelled account-weeks, churn base rate 5.4%.

Metric	Value
ROC-AUC	0.824
PR-AUC	0.201 (base rate 0.054)
Brier score	0.046
Calibration error (ECE)	0.009
Precision @ top 50	0.34, a 6.3x lift
Precision @ top 25	0.40, a 7.4x lift

• Precision@K is the headline, not accuracy. At a 5% base rate, predicting "no churn" for everyone is 95% accurate and useless.
• Expected-value ranking beats probability ranking. Ordering the same 50 retention slots by risk x monthly revenue raises expected revenue covered from $13,945 to $18,365 per month (+31.7%) and swaps 22 of the 50 accounts.
• Calibration is load-bearing. The class-weighted model over-predicts churn by 2.5x raw; isotonic calibration cuts the Brier score 42% (0.080 to 0.046) and aligns predicted with observed rates, with no loss of ranking quality.
• Per-segment metrics name the segments where the model fails (accounts with zero activity, and "surprise churn" among healthy-looking accounts) rather than hiding them behind an average.

---

Key decisions

• Grouped split by account, out-of-fold evaluation. Weekly snapshots of one account are near-duplicates, so a random row split leaks an account's future into its own training rows. 5-fold GroupKFold gives every row one out-of-fold prediction; the shipped model is refit on all data afterward.
• Explicit leakage list, not inferred. Forbidden columns (churn_date, cancellation flags, downstream outcome fields) are transcribed from the spec, so adding a schema column forces a deliberate keep-or-drop decision.
• Labels are right-censored. A snapshot in the panel's final 30 days has no observable outcome, so its label is NULL, not 0.
• Queue ranked by expected value. A retention slot on a small account that is certain to leave protects less revenue than one on a large account that is merely likely to. Both orderings are stored so the difference stays auditable.
• Risk bands are multiples of the base rate (MEDIUM at 2x, HIGH at 4x, CRITICAL at 8x), because fixed thresholds tuned against uncalibrated scores made the top band unreachable once probabilities were corrected.
• Reproducible down to the IDs. Account UUIDs come from the seeded RNG, because IDs decide fold assignment and uuid4 moved Precision@50 by more than 3x between runs at a fixed seed.

---

How it works

7 source tables  ->  churn_prediction_wide (SQL view)  ->  numeric feature matrix
   ->  XGBoost + isotonic calibration  ->  risk bands + priority queue  ->  dashboard

• Data: a synthetic panel generator stands in for the upstream systems (usage, billing, support, revenue). Swapping in real data means replacing one module; nothing downstream changes.
• Model: XGBoost when available, scikit-learn HistGradientBoostingClassifier otherwise. Both handle missing values natively, so missingness stays informative.
• Serving: score writes per-account probabilities, bands, and a ranked queue; the Streamlit dashboard filters the portfolio and exports the worklist.

---

Tests

67 tests, about 25 seconds, run against their own temporary database and mutation-checked. Reintroducing the uuid4 bug, the probability-ranking bug, an unfiltered-scoring bug, a leakage-exclusion removal, or a pass-through calibrator each makes the suite fail. They assert label correctness, leakage exclusion, scoring order, and calibration quality.

---

Run locally

pip install -r requirements.txt
python -m churn_app.cli all          # schema, data, model, scores (~20s)
streamlit run churn_app/dashboard/app.py

make all, make dashboard, and make test wrap the same steps.

---

Scope

The upstream production systems in the source spec do not exist here, so the app runs on synthetic data with the same statistical structure (clonal account behaviour, 5% class imbalance, censored labels). Schema, feature assembly, leakage policy, training, evaluation, and scoring are production code and move to real sources by replacing the data-loading module.

Objective 1 (weekly scoring and prioritisation) is implemented end to end. Time-to-churn survival models, per-cell calibration of the upstream score, and cause prediction are scoped in the repo but not built.

---

Author

Deepak Kushwaha, Data Scientist, MSc at University of Naples Federico II GitHub | LinkedIn