Skip to content

Add a Salesforce NPSP Opportunity export reader - #233

Merged
shivamlalakiya merged 2 commits into
mainfrom
e3-npsp-reader
Sep 25, 2026
Merged

shivamlalakiya merged 2 commits into
mainfrom
e3-npsp-reader

Conversation

@shivamlalakiya

Copy link
Copy Markdown
Contributor

Summary

Adds a Salesforce Nonprofit Success Pack (NPSP) Opportunity export reader alongside the existing CiviCRM and Raiser's Edge bridges, and wires it into the CLI as philanthropy features --source npsp.

  • philanthropy.ingest.read_npsp_opportunities(path): reads an Opportunity export CSV, or a directory of them, delegating to the shared CiviCRM reader.
  • philanthropy.ingest.npsp_opportunities_to_features(opportunities, *, reference_date=None, exclude_stages=DEFAULT_EXCLUDED_STAGES): rolls the export up into the donor-level feature table, dropping Pledged rows by default before handing the rest to civicrm_contributions_to_features.
  • philanthropy.ingest.DEFAULT_EXCLUDED_STAGES = ("Pledged",).

Why the Pledged filter exists: NPSP's Recurring Donations feature creates one Opportunity per instalment. The upcoming instalment is created in the Pledged stage and only moves to a closed/won stage (Closed Won, or a site's own Posted) once the gift is actually received. Depending on an org's "Installment Opportunity Auto-Creation" setting, the Pledged row and the Closed Won row recording the same instalment's receipt can both exist with the same amount and close date, so summing every Opportunity's Amount naively counts that instalment twice - the same commitment-vs-payment trap the Raiser's Edge bridge already guards against, expressed through Opportunity stage instead of a separate gift record.

Header aliases accept a Salesforce report export's column labels (Account Name/Account ID, Close Date, Amount, Stage, Record Type), the raw Opportunity API field names (AccountId, CloseDate, StageName, RecordType.Name), and NPSP's own Data Import template headers (Donation Date, Donation Amount, Donation Stage, Donation Record Type Name), plus Primary Contact for the Contact-keyed donor case.

NPSP vocabulary was checked against Salesforce's own public documentation rather than assumed:

I deliberately did not model the npe01__OppPayment__c Payment child object - this bridge reads an Opportunity-level export, matching the PR's scope, and I didn't want to assert Payment field names I couldn't verify against a primary source.

Files changed

  • philanthropy/ingest/_npsp.py (new)
  • philanthropy/ingest/__init__.py (export the three new names)
  • philanthropy/cli.py (--source npsp)
  • tests/test_npsp.py (new, same depth as tests/test_raisers_edge.py)
  • tests/test_cli.py (two CLI integration tests for the new source)
  • docs/how-to/use_the_cli.md, docs/reference/index.md (Tier 2/Beta row)
  • CHANGELOG.md

Test plan

  • python -c "from philanthropy.ingest import DEFAULT_EXCLUDED_STAGES, npsp_opportunities_to_features, read_npsp_opportunities" before writing tests, per CONTRIBUTING.md's order.
  • make ci: 2117 passed, 8 skipped; coverage 98.52% (floor 92%); philanthropy/ingest/_npsp.py at 100%.
  • make riskcov: risk-tier coverage 98% (floor 93%).
Required test coverage of 92.0% reached. Total coverage: 98.52%
2117 passed, 8 skipped, 195 warnings in 114.93s (0:01:54)
==> All CI checks passed locally.
==> Checking risk-tier coverage floor (93%)...
TOTAL   2163  22  548  26  98%

Mirrors the existing Raiser's Edge bridge: read_npsp_opportunities loads
an Opportunity export CSV (or a folder of them) and
npsp_opportunities_to_features rolls it up into the donor-level feature
table used by train/score, delegating the actual aggregation to the
shared CiviCRM roll-up.

NPSP's Recurring Donations feature creates one Opportunity per
instalment, starting it in the Pledged stage and only moving it to a
closed/won stage (Closed Won, or a site's own Posted) once the money is
actually received. Depending on an org's instalment settings, the
Pledged row and the Closed Won row recording its receipt can both exist
for the same amount and close date, so summing every row's Amount
double-counts that instalment. Pledged rows are dropped by default
(DEFAULT_EXCLUDED_STAGES), same as Raiser's Edge drops its Pledge and
Recurring Gift rows.

Header aliases accept a Salesforce report export's column labels
(Account Name, Close Date, Amount, Stage), the raw Opportunity API field
names (AccountId, CloseDate, StageName), and NPSP's own Data Import
template headers (Donation Date, Donation Amount, Donation Stage).

Wired into the CLI as `philanthropy features --source npsp`. Added
tests/test_npsp.py at the same depth as tests/test_raisers_edge.py plus
two CLI integration tests, a CHANGELOG entry, a paragraph in
docs/how-to/use_the_cli.md, and a Tier 2 (Beta) row in
docs/reference/index.md.
@shivamlalakiya
shivamlalakiya enabled auto-merge (squash) September 25, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant