📮 Complaint Router NLP

Read a real consumer-complaint narrative → route it to the right team → attach the insight (sentiment + theme).

An end-to-end, notebook-driven NLP + ML pipeline over the U.S. Consumer Financial Protection Bureau complaint database — from raw API pull to a two-stage router with an error analysis and an ops-facing insight dashboard.

Why this project

Complaint triage in a financial institution is a routing problem: a free-text grievance comes in, and someone (or something) has to send it to Mortgage Servicing, or Card Services, or the credit-bureau disputes desk — fast, and correctly, because a misrouted complaint is a regulatory clock still ticking in the wrong inbox.

The modelling is deliberately not the interesting part. The interesting part is the data: real people writing under stress, with typos (morgage, forclosure), run-on sentences, and CFPB's own XXXX / XX/XX/XXXX PII redactions baked into the text. That is what production text actually looks like, and it is why this repo leans on per-lane recall, honest error analysis, and a frank section on the accuracy ceiling rather than a single leaderboard number.

Everything is built on the CFPB Consumer Complaint Database via the free, public CCDB5 API — no key, 13.8M+ complaints back to 2011, updated daily.

Highlights

Two-stage hierarchical router — a coarse lane classifier, plus a department sub-router that fires only inside the Loans lane (mirrors how real triage escalates).
Cursor-paginated ingest — search_after chaining against the CCDB5 API, with a documented note on the frm-offset trap that silently returns page 1 forever.
Full classical NLP preprocessing — cleaning → NLTK tokenization → stop-word removal → POS-aware WordNet lemmatization, plus near-duplicate collapse to stop train/test leakage.
Word ⊕ character n-gram TF-IDF — char_wb 3–5-grams absorb the corpus's constant misspellings that break a pure word model.
Model bake-off — C-tuned Linear SVM vs. Logistic Regression (OvR) vs. ComplementNB vs. Random Forest (LSA-reduced), scored on a held-out split; the winner is persisted as prod.
Sentiment (VADER) and topic modelling (LDA, 10 topics) as first-class insight outputs — surfaced in a per-lane dashboard, deliberately not used as classifier features.
route_complaint() — a single call: raw text → lane → (department) → confidence + sentiment + dominant theme.
Reproducible — fixed RANDOM_SEED = 42 everywhere; every stage caches its artifacts so downstream notebooks don't recompute.

Feature coverage

Everything the one-liner and description promise is implemented — here is exactly where:

Capability	Notebook · cell	Status
Read a real consumer-complaint narrative	`00` fetch → `route_complaint()` in `05` takes raw text	✅
Text preprocessing — cleaning · tokenization	`01` · Cleaning → tokenization → stop-word removal → lemmatization	✅
POS-aware lemmatization	`01` · same cell (WordNet + `pos_tag`), cached to `lemma_cache.parquet`	✅
Near-duplicate removal (anti-leakage)	`01` · Deduplicate & drop near-empty narratives	✅
Feature extraction — TF-IDF	`02` · TF-IDF vectorization (word 1–2-gram ⊕ `char_wb` 3–5-gram union)	✅
EDA (class balance, length, redaction density, word clouds)	`02` · lane-balance / length / redaction / word-cloud cells	✅
Sentiment analysis	`03` · whole notebook — VADER `compound`/`pos`/`neu`/`neg` per complaint	✅
Topic modeling — LDA	`04` · whole notebook — 10-topic LDA, top terms, topic×lane heatmap	✅
Classification — SVM and Random Forest	`05` · Stage 1 bake-off (also Logistic Regression + ComplementNB)	✅
Hyper-parameter tuning	`05` · tune the Linear SVM's C on the validation split	✅
Two-stage routing (lane → department)	`05` · Stage 1 + Stage 2 – sub-router inside the `Loans` lane	✅
Detailed metrics (accuracy · precision · recall · F1 · ROC-AUC · PR-AUC)	`05` · Detailed metrics + ROC & Precision–Recall curves cells	✅
Confusion matrix + per-class recall	`05` · Confusion matrix / Per-lane recall	✅
Error analysis (real misrouted narratives)	`05` · Error analysis – the 3 weakest lanes	✅
Held-out test evaluation + end-to-end score	`05` · End-to-end evaluation on the held-out test set + Scoreboard	✅
`route_complaint(text)` — text → lane → dept → confidence + sentiment + theme	`05` · Routing demo	✅
Ops-facing insight dashboard	`05` · Per-lane insight dashboard (volume · recall · sentiment · theme)	✅
Saved figures + metric CSVs	`05` · `reports/figures/.png`, `reports/.csv`	✅
Raw ingest from the live CFPB API	`00` · cursor-paginated CCDB5 pull	✅

Not built yet (see Roadmap): route_complaint() is a notebook function / loadable .joblib, not a deployed service or pip-installable package; there is no CLI, no calibrated probability, and no DistilBERT baseline.

Architecture

                     ┌──────────────────────────────┐
   raw complaint ───▶│  clean · collapse redactions │
      narrative      └──────────────┬───────────────┘
                                    │
                        word ⊕ char TF-IDF (fitted, 140k dims)
                                    │
                     ┌──────────────▼───────────────┐
       STAGE 1       │   Linear SVM  (C = 0.1)       │
    lane classifier  │   4 classes · acc 0.899       │
                     └──────────────┬───────────────┘
                                    │
              ┌─────────────────────┼───────────────────────────┐
              │                     │                           │
        lane = "Loans"        lane ∈ {Mortgage,           (done — lane is
              │                Reporting & Collections,   the routing target)
              │                Banking & Payments}
              ▼
   ┌────────────────────┐
   │   STAGE 2           │   Student Loan Servicing · Auto Finance
   │  Loans sub-router   │   Short-Term Lending · Consumer Lending
   │  Linear SVM, 4-way  │
   └────────────────────┘

  side channels (insight, not features):
    VADER sentiment  ──▶ per-lane "how angry is this queue" signal
    LDA topic weights ──▶ dominant theme label per complaint

Lanes (stage 1) — the exact set is controlled by two merge switches in 01:

`MERGE_REPORTING_COLLECTIONS`	`MERGE_BANKING_PAYMENTS`	Lanes	Stage-1 accuracy
off	off	7 · Mortgage, Credit Reporting, Debt Collection, Cards, Deposits, Money Movement, Loans	0.82
on	off	6 · …reporting + collections merged	0.836
on	on (default)	4 · Mortgage, Loans, `Reporting & Collections`, `Banking & Payments`	0.899

product → lane / department is a first-match-wins keyword config (LANE_RULES / DEPARTMENT_RULES in 01), not a hard-coded product list — CFPB has renamed its product taxonomy several times and a keyword router tolerates that, the way a real config has to.

Both merges collapse label clusters that are genuinely ambiguous, not just hard for the model: a collection account is the thing people dispute on their report, and a Cash App dispute / a debit-card problem on a checking account / a prepaid-card issue get filed interchangeably. The 4-lane scheme is the one that clears the ≥ 0.90 target.

UML diagrams

Rendered by GitHub's built-in Mermaid support. The class view is a conceptual model — the notebook implements it as the route_complaint() function plus module-level globals rather than literal classes.

Component / class diagram — inference time

classDiagram
    direction LR

    class ComplaintRouter {
        +FeatureUnion vectorizer
        +LinearSVC laneModel
        +LinearSVC loanSubRouter
        +route(text) RoutingResult
    }
    class TextCleaner {
        +Pattern redactionRe
        +clean(text) str
    }
    class InsightEngine {
        +SentimentIntensityAnalyzer vader
        +LatentDirichletAllocation lda
        +CountVectorizer ldaCountVec
        +Lemmatizer lemmatizer
        +annotate(text) Insight
    }
    class RoutingResult {
        +str lane
        +str department
        +float confidence
        +Insight insight
    }
    class Insight {
        +str sentiment
        +float sentimentScore
        +int dominantTopic
        +str themeTerms
    }

    ComplaintRouter ..> TextCleaner : uses
    ComplaintRouter ..> InsightEngine : attaches
    ComplaintRouter --> RoutingResult : produces
    RoutingResult *-- Insight
    InsightEngine --> Insight : builds

Sequence diagram — a `route_complaint()` call

sequenceDiagram
    autonumber
    actor Caller
    participant R as route_complaint()
    participant C as TextCleaner
    participant V as TF-IDF FeatureUnion
    participant S1 as Stage-1 SVM (lane)
    participant S2 as Stage-2 SVM (Loans)
    participant I as VADER + LDA

    Caller->>R: route(narrative)
    R->>C: collapse XXXX redactions, squeeze whitespace
    C-->>R: clean text
    R->>V: transform([clean])
    V-->>R: sparse vector (1 x ~140k)
    R->>S1: decision_function → softmax over OvR margins
    S1-->>R: lane, confidence
    alt lane == "Loans"
        R->>S2: decision_function → softmax
        S2-->>R: department, confidence
    else any other lane
        Note over R: department = lane
    end
    R->>I: polarity_scores(text) · lda.transform(lemmatize(text))
    I-->>R: sentiment label + score, dominant theme
    R-->>Caller: {lane, department, confidence, sentiment, theme}

Activity diagram — the training pipeline (`00` → `05`)

flowchart TD
    A(["CFPB CCDB5 API"]) -->|"00 · cursor-paginated"| B["data/raw/ per-product CSVs"]
    B -->|"01 · clean · POS-lemmatize · label · near-dup dedupe · split"| C["data/processed/ train · val · test .csv"]

    C -->|"02 · fit word + char TF-IDF on train"| D["tfidf_vectorizer.joblib<br/>tfidf_*.npz"]
    C -->|"03 · VADER on narrative_clean"| E["sentiment_*.csv"]
    C -->|"04 · LDA-10 on lemmas"| F["lda_model.joblib<br/>doctopic_*.npy"]

    D --> G{"05 · bake-off:<br/>SVM · LogReg · CNB · RF"}
    G -->|"pick best macro-F1"| H["complaint_router.joblib<br/>Linear SVM, C=0.1"]
    C -->|"Loans rows only"| I["05 · stage-2 sub-router"]
    D --> I
    I --> J["complaint_router_loans.joblib"]

    G -.->|insight| K[["per-lane dashboard"]]
    E -.->|insight| K
    F -.->|insight| K

    H --> L(["route_complaint()"])
    J --> L

State diagram — a complaint's routing lifecycle

stateDiagram-v2
    [*] --> Received
    Received --> Cleaned : collapse redactions, squeeze whitespace
    Cleaned --> Vectorized : word ⊕ char TF-IDF
    Vectorized --> LaneAssigned : stage-1 SVM (6-way)

    LaneAssigned --> DepartmentAssigned : lane == Loans
    LaneAssigned --> Routed : lane != Loans

    DepartmentAssigned --> Routed : stage-2 SVM (4-way)
    Routed --> Annotated : attach VADER sentiment + LDA theme
    Annotated --> [*]

    note right of LaneAssigned
        confidence = softmax over
        one-vs-rest decision-function margins
        (ordering signal, not a probability)
    end note

Entity–relationship diagram — data & artifacts

erDiagram
    RAW_COMPLAINT  ||--|| PROCESSED_ROW : "cleaned & labelled by 01"
    PROCESSED_ROW  ||--|| SENTIMENT_ROW : "row-aligned (03)"
    PROCESSED_ROW  ||--|| DOCTOPIC_ROW  : "row-aligned (04)"
    PROCESSED_ROW  }o--|| LANE          : "stage-1 target"
    PROCESSED_ROW  }o--o| DEPARTMENT    : "stage-2 target (Loans only)"
    LANE           ||--o{ DEPARTMENT    : "Loans lane expands to 4"

    RAW_COMPLAINT {
        int    complaint_id PK
        string product
        string sub_product
        string issue
        string company
        string narrative
    }
    PROCESSED_ROW {
        int    complaint_id PK
        string lane FK
        string department FK
        string narrative_clean
        string lemmas
        int    word_count
        int    redaction_count
    }
    SENTIMENT_ROW {
        float  sent_compound
        float  sent_pos
        float  sent_neu
        float  sent_neg
        string sentiment
    }
    DOCTOPIC_ROW {
        float  topic_0_to_9
        int    dominant_topic
    }
    LANE {
        string name PK
    }
    DEPARTMENT {
        string name PK
    }

Pipeline

00  fetch        CCDB5 API ──cursor-paginated (search_after)──▶ data/raw/<product>.csv
01  preprocess   clean · tokenize · stop-word · POS-aware lemmatize · near-dup dedupe
                 · product→lane+department labels · stratified 70/15/15 split
                                                    ──▶ data/processed/{train,val,test}.csv
02  features     EDA by lane (balance · length · redaction density · word clouds)
                 TF-IDF: word 1–2 gram (80k) ⊕ char_wb 3–5 gram (60k), fit on train
                                                    ──▶ models/tfidf_vectorizer.joblib
                                                    ──▶ data/processed/tfidf_{split}.npz
03  sentiment    VADER compound/pos/neu/neg per complaint (on narrative_clean)
                                                    ──▶ data/processed/sentiment_{split}.csv
04  topics       LatentDirichletAllocation(10) over the lemma corpus
                 top terms per topic · topic×lane heatmap · dominant topic per complaint
                                                    ──▶ models/lda_model.joblib (+ count vec)
                                                    ──▶ data/processed/doctopic_{split}.npy
05  classify     bake-off: Linear SVM (C-tuned) · LogReg (OvR) · ComplementNB · Random Forest
                 → acc · precision · recall · F1 · ROC-AUC for every model
                 → best model: detailed per-class table (P·R·F1·ROC-AUC·PR-AUC·support)
                              + ROC and Precision-Recall curve panels
                 confusion matrix · per-lane recall · misrouted-example error analysis
                 stage-2 Loans sub-router (same detailed table + curves)
                 held-out test: detailed table + curves · consolidated scoreboard
                 route_complaint() demo · per-lane insight dashboard
                                                    ──▶ models/complaint_router*.joblib

Notebooks 02–05 read the artifacts written by earlier stages, so 01 must run first.

Repository layout

complaint router nlp/
├── notebooks/
│   ├── 00_fetch_data.ipynb          # CFPB API  → data/raw/*.csv
│   ├── 01_preprocessing.ipynb       # clean · lemmatize · label · split
│   ├── 02_features_eda.ipynb        # EDA + word⊕char TF-IDF
│   ├── 03_sentiment.ipynb           # VADER sentiment
│   ├── 04_topic_modeling_lda.ipynb  # LDA (10 topics)
│   └── 05_classification.ipynb      # bake-off · 2-stage router · demo · dashboard
├── data/
│   ├── raw/                         # per-product CFPB pulls        (git-ignored, ~140 MB)
│   └── processed/                   # splits + cached features      (git-ignored, ~490 MB)
├── models/                          # fitted estimators             (*.joblib git-ignored)
│   └── distilbert_checkpoints/      # reserved for the transformer stretch goal
├── reports/
│   └── figures/                     # PNGs saved by 05 (bake-off, ROC/PR, confusion, recall)
├── LICENSE.md
├── CONTRIBUTING.md
├── requirements.txt
└── README.md

data/ and models/*.joblib are git-ignored — all regenerable by running the notebooks. reports/ (figures + metric CSVs written by 05) is kept so plots can be embedded here. Source (notebooks/, requirements.txt, docs) is tracked.

Quickstart

Prerequisites: Python 3.12, ~2–3 GB free RAM for the modelling notebooks, an internet connection for the first data pull.

# 1. environment
python -m venv .venv
.venv\Scripts\activate           # Windows
# source .venv/bin/activate      # macOS / Linux
pip install -r requirements.txt

# 2. NLTK corpora (one-off)
python -m nltk.downloader punkt punkt_tab stopwords wordnet omw-1.4 \
    averaged_perceptron_tagger averaged_perceptron_tagger_eng vader_lexicon

# 3. register the venv as a Jupyter kernel  (the notebooks expect this exact name)
python -m ipykernel install --user --name complaint-router-venv --display-name complaint-router-venv

# 4. launch
jupyter notebook            # or: jupyter lab

Then open notebooks/00_fetch_data.ipynb and Run All, top to bottom, 00 → 05. Select the complaint-router-venv kernel in each notebook (the default python3 kernel will not have nltk / wordcloud).

Headless (CI / batch)

for nb in notebooks/0*.ipynb; do
  jupyter nbconvert --to notebook --execute --inplace \
    --ExecutePreprocessor.timeout=3600 \
    --ExecutePreprocessor.kernel_name=complaint-router-venv "$nb"
done

On a memory-constrained machine, run them one at a time rather than in a single loop (see Troubleshooting).

Running the pipeline

#	Notebook	~Runtime*	Re-run when…	Produces
00	`00_fetch_data`	secs (cache) / ~10–15 min (first pull)	once, ever — or to refresh the raw data	`data/raw/<product>.csv` — ~105k complaints, `MAX_PER_CLASS = 6000`/product
01	`01_preprocessing`	~15–20 min first time,secs on re-run (lemma cache)	you change labels/lanes/split — e.g. the merge switches	`data/processed/{train,val,test}.csv`, `lemma_cache.parquet`
02	`02_features_eda`	~3–5 min	you change TF-IDF params or`USE_CHAR_NGRAMS`	`models/tfidf_vectorizer.joblib`, `data/processed/tfidf_{split}.npz`
03	`03_sentiment`	~2 min	after`01`	`data/processed/sentiment_{split}.csv`
04	`04_topic_modeling_lda`	~5–10 min	after`01`, or to change `N_TOPICS`	`models/lda_model.joblib`, `models/lda_count_vectorizer.joblib`, `data/processed/doctopic_{split}.npy`, `lda_topic_terms.csv`
05	`05_classification`	~10–15 min	you change the`C` grid, models, or upstream artifacts	`models/complaint_router.joblib`, `reports/figures/.png`, `reports/*.csv` + all plots/curves inline

* Wall-clock on a mid-range laptop CPU with the venv kernel.

00 never re-fetches unless you ask. With a valid data/raw/ cache and the default FORCE_REFETCH = False, the notebook makes zero API calls — it just verifies the cache. You can also skip 00 entirely once the cache exists.

01's lemmatization is cached. The POS-tagged lemma pass (the one slow step) writes data/processed/lemma_cache.parquet keyed by complaint_id. Lemmas depend only on the narrative text, never on the lane / cap / merge settings, so re-running 01 after a config change only processes rows the cache has never seen — a pure re-run is seconds. Delete the parquet to force a full rebuild.

Using the trained router

The notebooks define route_complaint() inline. To use the persisted model outside Jupyter, load the three artifacts and reproduce the ~20 lines of inference:

import re
import numpy as np
import joblib

vectorizer = joblib.load("models/tfidf_vectorizer.joblib")   # word ⊕ char FeatureUnion
lane_svm   = joblib.load("models/complaint_router.joblib")    # stage 1 (tuned Linear SVM)
loan_svm   = joblib.load("models/complaint_router_loans.joblib")  # stage 2

_REDACT = re.compile(r"X{2,}", re.IGNORECASE)

def _clean(text: str) -> str:
    text = _REDACT.sub(" __REDACTED__ ", str(text))
    return re.sub(r"\s+", " ", text).strip()

def _softmax_margin(model, X):
    m = np.atleast_1d(model.decision_function(X)[0])
    e = np.exp(m - m.max())
    return e / e.sum()

def route(text: str) -> dict:
    X = vectorizer.transform([_clean(text)])
    p = _softmax_margin(lane_svm, X)
    lane = lane_svm.classes_[int(p.argmax())]
    out = {"lane": lane, "confidence": round(float(p.max()), 3), "department": lane}
    if lane == "Loans":
        q = _softmax_margin(loan_svm, X)
        out["department"] = loan_svm.classes_[int(q.argmax())]
        out["confidence"] = round(float(q.max()), 3)
    return out

route("the dealership added gap insurance to my auto loan without telling me and my payment is higher")
# {'lane': 'Loans', 'department': 'Auto Finance', 'confidence': 0.79}

Note on confidence: LinearSVC does not emit calibrated probabilities. The value above is a softmax over one-vs-rest decision-function margins — a useful ordering signal for review queues, not a true probability. Wrap the estimator in CalibratedClassifierCV if you need real probabilities.

Data

Source

CFPB Consumer Complaint Database, via the CCDB5 search API (GET https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/). Only complaints with a consumer narrative (has_narrative=true) are pulled — the free-text field is the whole point. 00_fetch_data discovers the current product categories from an aggregation query, then pulls a capped, stratified sample per category with search_after cursor pagination, caching each product to its own CSV.

⚠️ The frm pagination trap. The API's frm offset is a no-op past page 1 on its own — every request just returns page 1 again unless paired with the search_after cursor from the previous response's _meta.break_points. A naive frm += size loop silently produces thousands of duplicate rows. 00 does this correctly and asserts the resulting duplicate-narrative ratio is < 50%.

`data/processed/{train,val,test}.csv` schema

Column	Type	Notes
`complaint_id`	int	CFPB primary key
`product`, `sub_product`, `issue`	str	CFPB taxonomy (source of the labels)
`date_received`, `state`, `company`	str	metadata, not used by the model
`lane`	str	stage-1 target — 1 of 6
`department`	str	stage-2 target — fine team (only used within `Loans`)
`narrative`	str	original complaint text
`narrative_clean`	str	light normalization only — redaction runs collapsed to`__REDACTED__` / `__REDACTED_DATE__`, whitespace collapsed; case & punctuation kept. Feeds TF-IDF and VADER.
`lemmas`	str	fully processed — lowercase, non-alpha stripped, stop-words removed, POS-aware WordNet lemmas, space-joined. FeedsLDA only.
`word_count`, `char_count`, `redaction_count`	int	derived features for EDA

Intermediate artifacts

File	Written by	Read by	Contents
`data/processed/{split}.csv`	01	02–05	labelled + preprocessed splits
`data/processed/lemma_cache.parquet`	01	01	`complaint_id → lemmas` cache so re-runs skip the slow POS-lemmatization
`models/tfidf_vectorizer.joblib`	02	05	fitted word ⊕ char`FeatureUnion`
`data/processed/tfidf_{split}.npz`	02	05	sparse TF-IDF matrices (`float32`, ~140k cols)
`data/processed/sentiment_{split}.csv`	03	05	VADER scores,row-aligned with the split CSV
`models/lda_model.joblib`, `models/lda_count_vectorizer.joblib`	04	05	LDA + its`CountVectorizer`
`data/processed/doctopic_{split}.npy`	04	05	dense doc–topic weight matrices (N × 10)
`data/processed/lda_topic_terms.csv`	04	05	top 12 terms per topic
`models/complaint_router.joblib`	05	you	prod model — the bake-off winner (currently the tuned Linear SVM, stage 1)
`models/complaint_router_loans.joblib`	05	you	stage-2`Loans` sub-router
`models/complaint_router_{svm,logreg,rf}.joblib`	05	—	individual bake-off contestants, for inspection

Configuration knobs

Knob	Notebook	Default	Effect
`MAX_PER_CLASS`	`00`	`6000`	rows pulled per CFPB`product` (API hard-caps at 10k)
`FORCE_REFETCH`	`00`	`False`	`True` re-pulls from the API; `False` reuses a valid `data/raw/` cache (zero network)
`MAX_PER_LANE`	`01`	`8000`	down-sample cap per lane, appliedbefore lemmatization
`MERGE_REPORTING_COLLECTIONS`	`01`	`True`	fold`Credit Reporting` + `Debt Collection` into one lane
`MERGE_BANKING_PAYMENTS`	`01`	`True`	fold`Cards` + `Deposits` + `Money Movement` into `Banking & Payments` (→ 4-lane scheme)
`MIN_NARRATIVE_CHARS`	`01`	`20`	drop near-empty narratives
`USE_CHAR_NGRAMS`	`02`	`True`	add the`char_wb` 3–5-gram block to the TF-IDF union (~2× peak memory)
`N_TOPICS`	`04`	`10`	LDA topic count
`C` grid	`05`	`[0.02 … 1.0]`	Linear SVM regularization search (chosen on val macro-F1)
`RANDOM_SEED`	all	`42`	reproducibility

Results

Default config: 4 lanes (MERGE_REPORTING_COLLECTIONS + MERGE_BANKING_PAYMENTS both on) — Mortgage, Loans, Reporting & Collections, Banking & Payments. Features: TF-IDF word 1–2-grams (80k) ⊕ char_wb 3–5-grams (60k), fit on train. C grid-searched on the validation split → best C = 0.1.

Stage-1 bake-off — validation

Model	Accuracy	Precision (macro)	Recall (macro)	F1 (macro)	F1 (weighted)	ROC-AUC (macro)	ROC-AUC (weighted)	PR-AUC (macro)
TF-IDF + Linear SVM	0.8993	0.9032	0.9006	0.9018	0.8994	0.9777	0.9770	0.9491
TF-IDF + Logistic Regression (OvR)	0.8964	0.9006	0.8977	0.8990	0.8965	0.9771	0.9764	0.9504
TF-IDF + ComplementNB	0.8317	0.8395	0.8363	0.8335	0.8292	0.9572	0.9554	0.9082
TF-IDF (LSA-160) + Random Forest	0.8301	0.8397	0.8312	0.8347	0.8308	0.9591	0.9578	0.9093

Linear SVM goes to prod — accuracy 0.899, macro-F1 0.902, macro ROC-AUC 0.978. Logistic Regression is within noise; the tree / NB baselines trail by ~7 points. All ROC-AUC and PR-AUC figures are one-vs-rest.

Held-out test — stage 1 & the full pipeline

The model generalises: validation 0.899 → test 0.894 (a 0.5-point drop), macro ROC-AUC actually a hair higher on test (0.9788).

Stage	Split	Accuracy	Precision (macro)	Recall (macro)	F1 (macro)	ROC-AUC (macro)	n
Stage 1 · Linear SVM	validation	0.8993	0.9032	0.9006	0.9018	0.9777	4,498
Stage 1 · Linear SVM	test	0.8944	0.8985	0.8961	0.8971	0.9788	4,498
Stage 2 · Loans sub-router	validation	0.8415	0.8490	0.8190	0.8322	0.9607	1,199
End-to-end (stage 1 → 2)	test	0.8604	—	—	0.8193	—	4,498

Per-lane, stage 1 (test):

Lane	Precision	Recall	F1	ROC-AUC	Support
Mortgage	0.954	0.921	0.937	0.991	900
Banking & Payments	0.889	0.926	0.907	0.986	1,200
Reporting & Collections	0.876	0.867	0.872	0.969	1,198
Loans	0.875	0.870	0.873	0.970	1,200

No lane collapsed — every recall is ≥ 0.87.

Stage 2 — Loans sub-router (validation)

A 4-way Linear SVM over Loans complaints only. Student Loan Servicing and Short-Term Lending are clean; Consumer Lending (a catch-all) is the weak spot.

Department	Precision	Recall	F1	ROC-AUC	Support
Student Loan Servicing	0.987	0.929	0.957	0.996	254
Short-Term Lending	0.817	0.894	0.854	0.944	519
Auto Finance	0.883	0.763	0.819	0.965	207
Consumer Lending	0.709	0.690	0.699	0.937	219
macro avg	0.849	0.819	0.832	0.961	1,199

Why the 4-lane merge clears 0.90

At 6 lanes, stage 1 tops out at 0.836: the Cards / Deposits / Money Movement cluster is not cleanly separable — a Cash App dispute, a debit-card problem on a checking account, a prepaid-card issue all get filed by consumers under whichever CFPB product felt right, so the labels themselves disagree, not just the model. Folding that cluster into one Banking & Payments lane removes the boundary the classifier can't win and lifts stage-1 accuracy to 0.899 — while the ranking metric (ROC-AUC) barely moves (0.967 → 0.978), because the model had already learned the signal; only the hard decision was ambiguous.

Remaining lever if you need the 6-lane granularity back at ≥ 0.90: a DistilBERT / RoBERTa fine-tune (~0.88–0.91 on 6 lanes), at the cost of hours of CPU / a GPU and ≥ 4 GB RAM. Checkpoint dir reserved at models/distilbert_checkpoints/.

Progression

Setup	Stage-1 accuracy
11 fine departments · word-only · lemma features	0.77
7 coarse lanes · word-only TF-IDF	0.82
6 lanes (reporting + collections merged) · word ⊕ char n-grams · tuned`C`	0.836
4 lanes (+ banking + payments merged) · same features	0.899

Insight dashboard (`05`)

Per-lane volume, stage-1 recall, mean VADER sentiment, and dominant LDA theme. Payments-type narratives skew angriest (mean compound ≈ −0.3 to −0.4); LDA themes line up with the lanes without ever seeing the label — mortgage → escrow / foreclosure / servicer; reporting & collections → dispute / FCRA / inaccurate; banking & payments → check / fee / transfer / overdraft.

Design decisions (developer FAQ)

Why TF-IDF on narrative_clean and not the lemmas? Lemmatization plus non-alpha stripping throws away signal a linear classifier leans on — $2,700, APR, co-signer, chapter 7, and the density of redaction tokens (which varies by product). Lemmatization rarely helps linear text models and usually hurts. The lemmas column exists only to keep LDA topics readable.

Why character n-grams? The corpus is full of misspellings — morgage, forclosure, reposession, refinanace. A pure word model treats each as an OOV token. char_wb 3–5-grams match the shared subword skeleton and are robust to it. Worth ~1–2 points here; the cost is roughly doubled peak memory during fit, hence the USE_CHAR_NGRAMS switch.

Why two stages instead of one flat classifier over all departments? The coarse lanes are separable (0.90 at 4 lanes, 0.84 at 6); the fine departments are not, from a single flat model — an 11-way classifier scored 0.77 and its macro-F1 was dragged down by tiny, ambiguous classes. Hierarchy lets stage 1 be accurate and lets stage 2 specialize on a homogeneous subset (Loans), and it matches how a real triage desk escalates.

Why Linear SVM over Random Forest (the brief asked for both)? Text features are high-dimensional and sparse — exactly where linear models shine and where tree ensembles struggle. Random Forest needs dense, low-dimensional input, so its TF-IDF is first LSA-reduced to 160 components (only ~19% variance retained). It trails the SVM by ~7 points and is kept purely as a documented contrast.

Why aren't sentiment and LDA topics used as features? They were tried — sent_compound plus 10 topic weights stacked onto the TF-IDF matrix. Eleven dense columns beside 140k sparse dimensions moved validation accuracy by < 0.1 point, and the topic weights are collinear with the labels in a way that would not generalize. They earn their place as insight outputs, not inputs.

Why report per-lane recall so prominently? A router that silently never predicts one lane can still post high overall accuracy on an imbalanced set. Per-lane recall is the actual SLA — "what fraction of Mortgage complaints reach the Mortgage desk" — so it is the headline chart in 05, sorted worst-first.

Why a keyword product → lane map instead of a lookup table? CFPB has restructured its product taxonomy repeatedly since 2011 (e.g. Credit reporting → Credit reporting, credit repair services, or other personal consumer reports). A first-match-wins keyword config tolerates renames and reads like a real router config that has to survive upstream churn.

Why cap per lane / per product? Raw category volumes span three orders of magnitude (credit reporting has ~1.7M narratives, Other financial service has ~290). Uncapped, one lane dominates and the classifier learns the prior instead of the language. Capping + class_weight="balanced" keeps every lane learnable.

Model card


Task	Multiclass text classification — 4 routing lanes by default (stage 1; configurable to 6 or 7 via the merge switches) + conditional 4-way department classification within`Loans` (stage 2).
Input	A single consumer-complaint narrative, English free text.
Output	`lane`, optional `department`, an uncalibrated confidence, plus attached VADER sentiment and dominant LDA theme.
Training data	tens of thousands of CFPB complaints (2011–2026) that have a consumer narrative, capped at`MAX_PER_LANE` per lane, stratified by lane; exact and near-duplicate narratives removed.
Metrics	Stage-1 (4-lane): validation acc 0.899 / macro-F1 0.902 / macro ROC-AUC 0.978; held-out test acc 0.894 / macro-F1 0.897 / macro ROC-AUC 0.979. Loans sub-router (val) acc 0.842. End-to-end (test) acc 0.860. Full breakdown: `05` outputs + `reports/`.
Known limitations	(1) At 6+ lanes, label noise in the`Cards` / `Deposits` / `Money Movement` cluster caps stage-1 accuracy near 0.84 — the default 4-lane merge sidesteps this by design, at the cost of top-level granularity. (2) English only. (3) Trained on complaints that have narratives — a self-selected sub-population. (4) The department taxonomy is a plausible simulation, not a real CFPB org chart. (5) Confidence is a margin softmax, not a probability.
Intended use	Triage assistance and complaint analytics — surfacing a suggested lane and a batch-level view.Not for automated decisions that affect a consumer without human review.
Out of scope	Legal/eligibility determinations; PII extraction; anything treating the`confidence` score as a calibrated probability.

Troubleshooting

Symptom	Cause & fix
`ModuleNotFoundError: No module named 'nltk'` when a cell runs	Wrong kernel. Select`complaint-router-venv` (Kernel ▸ Change Kernel). The global `python3` kernel does not have the project deps.
Kernel dies at theimport cell / a notebook run is killed with no traceback	Out of RAM. The modelling notebooks need ~2–3 GB free; a kernel needs ~0.4–1 GB just to import`pandas`/`sklearn`. Close other memory-heavy apps and retry. The notebooks are already written lean (`float32` matrices, `usecols` on CSV reads, capped `n_jobs`, `compress=3` dumps).
Runningall six notebooks in one shell loop gets killed ~30 min in	Some environments cap long-lived background jobs. Run notebooksone at a time; `01` alone is ~15–20 min.
`04` / LDA is very slow	Reduce`N_TOPICS`, or lower `MAX_PER_LANE` in `01` and re-run from there.
`route_complaint()` returns low confidence (0.2–0.5)	Expected.`LinearSVC` margins are not probabilities; the softmax is a rough proxy. Use `CalibratedClassifierCV` for real probabilities.
I need the fine department for anon-`Loans` lane	Not modelled — only`Loans` is sub-routed. Add entries to `DEPARTMENT_RULES` and a sibling sub-router in `05`.
`00` produces mostly duplicate rows	The`search_after` cursor chain broke for a category. Inspect one raw API response; check `_meta.break_points`. The built-in assertion should have caught this.
First`00` run is slow / rate-limited	Normal — it is a live API pull of ~105k records with a polite`time.sleep(0.15)` between pages. It runs once; the default `FORCE_REFETCH = False` skips the network on every later run.
`01` re-runs the ~15-min lemmatization every time	It shouldn't — check`data/processed/lemma_cache.parquet` exists and is being read (the cell prints `lemma cache: N hit · M to compute`). A big `M` after only a lane/cap change means the cache was deleted or `complaint_id` changed. `pyarrow` must be installed.

Roadmap

4-lane merge (Banking & Payments) as a config flag → clears the 0.90 target.
DistilBERT fine-tune path (models/distilbert_checkpoints/ reserved) with a SVM-baseline comparison table.
Calibrated probabilities — wrap the prod estimator in CalibratedClassifierCV and expose a real p(lane).
Sub-routers for Cards / Deposits / Reporting & Collections so every lane has a fine department.
Package route_complaint() as an importable module + a small CLI (python -m complaint_router "<text>").
Date-windowed fetch to break the API's 10k-per-product frm ceiling and pull a larger, time-balanced sample.
pyLDAvis interactive topic view in 04.
CI — nbconvert --execute smoke test on a tiny fixture sample.

Contributing

Contributions are welcome — see CONTRIBUTING.md for the full guide. The essentials:

The notebooks are the source of truth; a clean "Restart & Run All" on the complaint-router-venv kernel must pass, 00 → 05.
Respect the artifact contract — if you change what a notebook writes, update every downstream reader and the Intermediate artifacts / Configuration knobs tables.
Preserve RANDOM_SEED = 42 and the stratified 70/15/15 split; never tune on the test set.
Re-run touched notebooks so committed outputs match the code. data/ and models/*.joblib stay git-ignored — never commit regenerable artifacts.
One concern per PR; include before/after numbers for any metric change.

Acknowledgements & data attribution

Complaint data © Consumer Financial Protection Bureau, retrieved via the public Consumer Complaint Database API. This project is not affiliated with, sponsored by, or endorsed by the CFPB. The internal "department" taxonomy is a simulation for demonstration and does not represent any real organization.

Built with scikit-learn, NLTK (incl. the VADER sentiment lexicon), pandas, matplotlib / seaborn, and wordcloud.

License

The complaint data it is trained on comes from the CFPB Consumer Complaint Database (a U.S. Government work, public domain in the U.S.) via the public CCDB5 API, and is not covered by the MIT license — see LICENSE.md § Scope and the Acknowledgements above. Third-party libraries in requirements.txt are under their own licenses.

Built an end-to-end NLP complaint-routing system (Python, scikit-learn, NLTK): text preprocessing, POS-aware lemmatization, TF-IDF word + char n-grams, VADER sentiment and LDA topic modeling feeding a two-stage Linear SVM router that classifies ~100K CFPB consumer complaints into routing lanes at 90% accuracy (0.90 macro-F1, 0.98 ROC-AUC) — ~7 points ahead of Random Forest and Naïve Bayes baselines.

Maintained by
Shaikh Rumman Fardeen
GitHub: @srummanf
rummanfardeen4567@gmail.com

Project Links
Source Code
MIT License | Contributing

Name		Name	Last commit message	Last commit date
models		models
notebooks		notebooks
reports		reports
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE.md		LICENSE.md
README.md		README.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

📮 Complaint Router NLP

Table of contents

Why this project

Highlights

Feature coverage

Architecture

UML diagrams

Component / class diagram — inference time

Sequence diagram — a route_complaint() call

Activity diagram — the training pipeline (00 → 05)

State diagram — a complaint's routing lifecycle

Entity–relationship diagram — data & artifacts

Pipeline

Repository layout

Quickstart

Headless (CI / batch)

Running the pipeline

Using the trained router

Data

Source

data/processed/{train,val,test}.csv schema

Intermediate artifacts

Configuration knobs

Results

Stage-1 bake-off — validation

Held-out test — stage 1 & the full pipeline

Stage 2 — Loans sub-router (validation)

Why the 4-lane merge clears 0.90

Progression

Insight dashboard (05)

Design decisions (developer FAQ)

Model card

Troubleshooting

Roadmap

Contributing

Acknowledgements & data attribution

License

About

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Sequence diagram — a `route_complaint()` call

Activity diagram — the training pipeline (`00` → `05`)

`data/processed/{train,val,test}.csv` schema

Insight dashboard (`05`)