Add activities_to_features for multi-source engagement data - #232
Merged
Merged
Conversation
A no-code upload flow lets a user hand over as many activity files as they have (event attendance, volunteer shifts, email clicks), each tagged with a type and mapped onto a shared long-table shape. This adds the aggregator every one of those types runs through: per activity type found in the data, count_12m, count_36m, days_since_last and distinct, plus hours_12m / amount_12m when those columns are present. as_of is required and is the leakage boundary; rows dated after it are dropped before anything is counted, and a type or donor with no qualifying rows gets zero columns or zero values rather than inventing a value from future data. Generalises the aggregation already used for the UniSchema constituent event bridge, so a new activity type never needs new model code.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The no-code upload flow (map columns) needs a way to fold arbitrarily many activity files (event attendance, volunteer shifts, email clicks, ...) into per-donor features, without new model code every time a source is added.
philanthropy.ingest.activities_to_features(activities, *, as_of, donors=None)takes a long table (contact_id,activity_date,activity_type, optionalamount/hours/name) and, for every activity type actually present in the data after theas_ofcutoff, produces<type>_count_12m,<type>_count_36m,<type>_days_since_last,<type>_distinct, plus<type>_hours_12m/<type>_amount_12mwhen those columns exist. A donor with no rows of a present type gets zeros; a type with zero rows anywhere in the (cutoff) data contributes no columns at all, rather than a column of zeros for every donor.as_ofis required, not optional, and is the leakage boundary: everything is filtered toactivity_date <= as_ofbefore any count, window, or recency figure is computed, so a row dated after the cutoff cannot change the output.donorsis optional and used only to report the match rate (the share of distinct activitycontact_ids also found indonors), warning when it's under 80%, which is usually a sign the activity export is keyed on a different id (email vs. an internal CRM id) than the file it's being joined against. It does not add rows for donors absent from the activity log.This generalises the aggregation pattern already used for the UniSchema constituent-event bridge (
philanthropy/ingest/_constituent_events.py) to an open-ended, data-discovered set of activity types instead of a fixed handful.Test plan
python -c "from philanthropy.ingest import activities_to_features; print('OK')"tests/test_activities_to_features.py(23 tests): windowed counts, days-since-last, zero-vs-absent columns, amount/hours columns, multiple types, leakage (future rows ignored, idempotent on repeat), match-rate warning (DataFrame/Series/plain-iterable donors, low vs. high match rate), donors doesn't inject extra rows, missing required column, empty input, every row filtered by cutoff, blank type / unparseable date dropped, DataFrame input, sort ordermake ci(flake8, mypy, doctest, full suite, 92% coverage floor) - all green,_activities.pyat 100% coveragemake riskcov(risk-tier floor overpreprocessing/,models/,ingest/,cli.py) - green