Add a Salesforce NPSP Opportunity export reader

CHANGELOG.md

@@ -10,6 +10,13 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
   user-supplied export's headers to canonical names and raises one
   `ValueError` listing every still-missing required column, for callers
   building a column-mapping UI over an arbitrary CRM export.
+- `philanthropy.ingest.read_npsp_opportunities` and
+  `npsp_opportunities_to_features`: a Salesforce Nonprofit Success Pack (NPSP)
+  Opportunity export bridge, alongside the existing CiviCRM and Raiser's Edge
+  ones. Drops `Pledged` Recurring Donation instalment rows by default
+  (`DEFAULT_EXCLUDED_STAGES`) so an instalment isn't counted both as the
+  pledge and again once it closes `Won`. Wired into the CLI as
+  `philanthropy features --source npsp`.
 - `philanthropy.ingest.activities_to_features(activities, *, as_of,
   donors=None)`: aggregates a long, multi-source activity log (event
   attendance, volunteer shifts, email clicks, ...) into per-donor,

docs/how-to/use_the_cli.md

@@ -17,7 +17,7 @@ Four subcommands: `features`, `train`, `score`, `validate`.
 philanthropy features --source raisers_edge --data gifts.csv --out features.csv
 ```
 
-`--source` accepts `raisers_edge` (Blackbaud Raiser's Edge and RE NXT) or `civicrm`. `--data` takes a single CSV or a directory of them, walked recursively, which is the shape of a folder of monthly exports. Omit `--out` and the CSV goes to stdout.
+`--source` accepts `raisers_edge` (Blackbaud Raiser's Edge and RE NXT), `npsp` (Salesforce Nonprofit Success Pack), or `civicrm`. `--data` takes a single CSV or a directory of them, walked recursively, which is the shape of a folder of monthly exports. Omit `--out` and the CSV goes to stdout.
 
 The output has one row per donor and these columns:
 
@@ -27,7 +27,7 @@ Header spelling is normalised for you, so a desktop Export (`Constituent ID`, `G
 
 !!! warning "A pledge is not a payment, and `features` knows the difference"
 
-    In Raiser's Edge a pledge and the money paid against it are **separate gift records**, and a recurring gift row is a template rather than a sum ever received. Adding up the amount column double-counts every committed dollar. `features` drops the commitment rows (`Pledge`, `Matching Gift Pledge`, `Recurring Gift`) and the ledger corrections, and keeps the payments (`Pay-Cash`, `PledgePayment`, `RecurringGiftPayment`, ...). Export the **Gift Type** field or it cannot do this, and it will warn you. The excluded set is the `exclude_gift_types` parameter of `philanthropy.ingest.raisers_edge_gifts_to_features` if your site spells its types differently. For CiviCRM the equivalent traps are test-mode rows and non-`Completed` contributions, and they are dropped the same way.
+    In Raiser's Edge a pledge and the money paid against it are **separate gift records**, and a recurring gift row is a template rather than a sum ever received. Adding up the amount column double-counts every committed dollar. `features` drops the commitment rows (`Pledge`, `Matching Gift Pledge`, `Recurring Gift`) and the ledger corrections, and keeps the payments (`Pay-Cash`, `PledgePayment`, `RecurringGiftPayment`, ...). Export the **Gift Type** field or it cannot do this, and it will warn you. The excluded set is the `exclude_gift_types` parameter of `philanthropy.ingest.raisers_edge_gifts_to_features` if your site spells its types differently. NPSP writes the same split through Opportunity stage instead of a separate record: a Recurring Donation instalment is created `Pledged` and only moved to `Closed Won` (or a site's own `Posted`) once received, and depending on the org's instalment settings both rows can exist for the same money. `features` drops `Pledged` rows for `npsp`; override with the `exclude_stages` parameter of `philanthropy.ingest.npsp_opportunities_to_features` if your org's stages differ. For CiviCRM the equivalent traps are test-mode rows and non-`Completed` contributions, and they are dropped the same way.
 
 !!! note "`features` does not invent a label"
 

docs/reference/index.md

@@ -55,6 +55,7 @@ Everything reachable from `philanthropy.__all__` is listed below. A symbol not l
 | `constituent_events_to_features`, `read_constituent_events` | `ingest` | Tracks the UniSchema `ConstituentEvent` schema, which is versioned upstream. |
 | `civicrm_contributions_to_features`, `read_civicrm_contributions` | `ingest` | Tracks CiviCRM's contribution export labels and APIv4 field names, which move with the CRM. |
 | `raisers_edge_gifts_to_features`, `read_raisers_edge_gifts`, `DEFAULT_EXCLUDED_GIFT_TYPES` | `ingest` | Tracks Raiser's Edge export labels and the RE NXT gift-type vocabulary; the excluded-type default will grow as real exports arrive. |
+| `npsp_opportunities_to_features`, `read_npsp_opportunities`, `DEFAULT_EXCLUDED_STAGES` | `ingest` | Tracks NPSP Opportunity export labels and a per-org-configurable stage vocabulary; the excluded-stage default will grow as real exports arrive. |
 | `map_columns` | `ingest` | The one-error-per-missing-column message shape may still change. |
 | `activities_to_features` | `ingest` | The activity-type feature set (`_count_12m`, `_distinct`, ...) may grow as more source types are onboarded. |
 | `plot_affinity_distribution`, `plot_retention_waterfall` | `visualisation` | Chart composition is presentation, not contract. |

philanthropy/cli.py

@@ -33,7 +33,7 @@
 
 # Gift-export readers `features` can front. Each name maps to a
 # (reader, aggregator) pair in _cmd_features.
-_FEATURE_SOURCES = ("civicrm", "raisers_edge")
+_FEATURE_SOURCES = ("civicrm", "raisers_edge", "npsp")
 
 # The donor-level columns `features` emits, in order. Named here so
 # `philanthropy features --help` answers "what do I pass to --features?"
@@ -203,6 +203,9 @@ def _cmd_features(args: argparse.Namespace) -> None:
     if args.source == "raisers_edge":
         read = ingest.read_raisers_edge_gifts
         to_features = ingest.raisers_edge_gifts_to_features
+    elif args.source == "npsp":
+        read = ingest.read_npsp_opportunities
+        to_features = ingest.npsp_opportunities_to_features
     else:
         read = ingest.read_civicrm_contributions
         to_features = ingest.civicrm_contributions_to_features
@@ -248,9 +251,10 @@ def _build_parser() -> argparse.ArgumentParser:
             "the models consume, then feed that to `train` and `score`. "
             "Emitted columns, in order: " + _FEATURE_COLUMNS + ". Commitment "
             "rows (pledges, recurring gift templates) are dropped for "
-            "raisers_edge, and test-mode and non-Completed rows for civicrm, "
-            "so a committed dollar is not counted twice. No label is produced: "
-            "`train --target` needs a column you define yourself."
+            "raisers_edge, Pledged instalment rows for npsp, and test-mode "
+            "and non-Completed rows for civicrm, so a committed dollar is "
+            "not counted twice. No label is produced: `train --target` "
+            "needs a column you define yourself."
         ),
     )
     features.add_argument(

philanthropy/ingest/__init__.py

@@ -17,6 +17,11 @@
 (pledges, matching gift pledges, recurring gift templates) first so a pledged
 dollar is not counted both as the promise and as the payments against it.
 
+NPSP: ``read_npsp_opportunities`` loads a Salesforce Nonprofit Success Pack
+Opportunity export CSV; ``npsp_opportunities_to_features`` aggregates it,
+dropping ``Pledged`` instalment rows first so a Recurring Donation instalment
+is not counted both in its ``Pledged`` stage and again once it closes ``Won``.
+
 ``map_columns`` renames a user-supplied export's headers to the canonical
 names a bridge above expects, raising one error listing every column still
 missing after the rename.
@@ -37,6 +42,11 @@
     read_constituent_events,
 )
 from ._map_columns import map_columns
+from ._npsp import (
+    DEFAULT_EXCLUDED_STAGES,
+    npsp_opportunities_to_features,
+    read_npsp_opportunities,
+)
 from ._raisers_edge import (
     DEFAULT_EXCLUDED_GIFT_TYPES,
     raisers_edge_gifts_to_features,
@@ -45,12 +55,15 @@
 
 __all__ = [
     "DEFAULT_EXCLUDED_GIFT_TYPES",
+    "DEFAULT_EXCLUDED_STAGES",
     "activities_to_features",
     "civicrm_contributions_to_features",
     "constituent_events_to_features",
     "map_columns",
+    "npsp_opportunities_to_features",
     "raisers_edge_gifts_to_features",
     "read_civicrm_contributions",
     "read_constituent_events",
+    "read_npsp_opportunities",
     "read_raisers_edge_gifts",
 ]