feat(pii_remover): default NER to IndicBERTv2 Naamapadam model

[AkhileshNegi]

Summary

pii_remover runs Presidio with spaCy en_core_web_lg for NER. That model is English-only, so for Indian NGO traffic PERSON/LOCATION were effectively never detected in Hindi, Marathi, Kannada or Telugu text, and Hinglish was over-redacted (mera naam Rajesh Kumar hai → <PERSON>). Regex recognizers (phone, email, PAN, Aadhaar…) were fine since they are script-agnostic.

This PR switches the default NER to neuronsbyisshu/indicbert-hi-ner-naamapadam (IndicBERTv2 fine-tuned on Hindi Naamapadam, CC0, ~0.3B params, CPU-only) via the existing transformers engine path. spaCy en_core_web_sm stays for tokenization. nlp_engine_type="spacy" still works as a fallback. No new dependency, no API key, public HF repo.

Changes:

• pii_remover.py: DEFAULT_TRANSFORMERS_MODEL → IndicBERT model, nlp_engine_type default → "transformers".
• Dockerfile: pre-download the new model instead of dslim/bert-base-NER-uncased.
• Test updated for the new default; docs (pii-remover.md, DEVELOPER_GUIDE.md, ARCHITECTURE.md) updated.

Evaluation

Row-by-row comparison sheet: https://docs.google.com/spreadsheets/d/1843aBnpVtFJu4SJDNFx3x6w-L445bp7-ucm955WwNQU/edit?usp=sharing (filter column J ≠ tie to see only rows where the models differ).

Both runs use the same harness metrics as app/evaluation/pii/run.py, with all 15 entity types enabled so NER is actually exercised.

Dataset 1: pii_multilang_3300.csv (3300 rows, 550 templates × 6 langs; names/locations in native script for hi/mr/kn/te)

Sentence-level F1 (did the validator fire on a PII row):

lang	spaCy en_core_web_lg	IndicBERT
all	0.89 (72 FP, 536 FN)	0.92 (13 FP, 433 FN)
en	0.93	0.92
hinglish	0.95 (38 FP)	0.92 (0 FP)
hi	0.87	0.92
mr	0.86	0.92
kn	0.87	0.92
te	0.86	0.91

Value recall for NER-dependent types (spaCy → IndicBERT):

	en	hinglish	hi	mr	kn	te
NAME	0.75 → 0.66	0.73 → 0.57	0.07 → 0.75	0.11 → 0.75	0.11 → 0.61	0.09 → 0.58
LOCATION	0.93 → 0.97	0.89 → 0.96	0.05 → 1.00	0.05 → 0.99	0.07 → 0.99	0.26 → 1.00
NRP	0.65 → 0.00	0.58 → 0.05	~0 → ~0	~0 → ~0	~0 → ~0	~0 → ~0

Row-level wins (ties excluded): IndicBERT 722, spaCy 167. spaCy's 72 false positives on clean rows are mostly Hinglish junk tagged PERSON (bhai, kitni hai); 11 of IndicBERT's 13 are real names the dataset marks as non-PII (Bill Gates, John Wick).

Dataset 2: pii_detection_testing_dataset.csv (419 rows, 300 en / 119 hi)

split	spaCy	IndicBERT
all acc / F1	0.84 / 0.86 (17 FP)	0.89 / 0.90 (0 FP)
en acc / F1	0.86 / 0.89	0.86 / 0.88
hi acc / F1	0.78 / 0.78	0.97 / 0.97

Hindi PERSON recall 0.17 → 1.00, LOCATION 0.00 → 1.00.

Regex entities

Identical across both models on every dataset (PHONE/EMAIL/URL/IP/PAN/VOTER 1.00, CREDIT_CARD 0.94, PASSPORT 0.80, VEHICLE_REG 0.70, AADHAAR 0.2–0.4, IBAN 0.23, MEDICAL_LICENSE 0.15). The low ones are Presidio pattern / synthetic-data mismatches (invalid Verhoeff, spaced vehicle plates) and are out of scope here.

Cost

	spaCy	IndicBERT
latency mean / p95	6 ms / 8 ms	20 ms / 30 ms
model load	0.7 s	5–13 s
disk	~600 MB	~1.1 GB

Checklist

• Ran the validator end-to-end via the evaluation harness on both datasets.
• Tests updated and passing (app/tests/validators/test_pii_remover.py, 17 passed).

Notes

• Trade-offs: NRP is not a label in this model, so it will not be detected (spaCy only got it for English anyway). Lone Latin-script first names inside Indic text (डॉ Aiswarya) are the weakest case (~0.45 recall).
• DEFAULT_ENTITY_TYPES still excludes PERSON/LOCATION/NRP; this PR only changes which model answers when they are requested. Turning them on by default is a separate decision.
• Follow-up candidate: ai4bharat/IndicNER (trained on all 11 Indic languages, MIT) once gated HF access is granted; same plug-in point, should lift kn/te name recall.
• Dockerfile still ships en_core_web_lg so the spaCy fallback keeps working; can be dropped later to save ~600 MB.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one): 1

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: ProjectTech4DevAI/kaapi-guardrails/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: 44f03ec5-5365-4159-89db-6fe39fa160a4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[AkhileshNegi]

Comparison on the published 419-sample evaluation set

This is the same dataset behind the numbers on the docs page (Privacy Protection). The current docs figures (acc 0.84 / P 0.92 / R 0.80 / F1 0.86) reproduce exactly with spaCy en_core_web_lg and all entity types enabled. Same harness, same rows, IndicBERT alongside:

	spaCy en_core_web_lg (docs today)	IndicBERT
Accuracy	0.84	0.89
Precision	0.92	1.00
Recall	0.80	0.82
F1	0.86	0.90
False positives (of 163 clean rows)	17	0
English F1 (300 rows)	0.89	0.88
Hindi F1 (119 rows)	0.78	0.97
Hindi PERSON recall	0.17 (1/6, 12 FP)	1.00 (6/6)
Hindi LOCATION recall	0.00	1.00
Latency p95	16 ms	41 ms

Regex entities (phone, email, IP, PAN, Aadhaar, passport, vehicle, voter) are byte-identical between the two runs.

Notes on the dataset itself, affecting both models equally:

• 4/5 Hindi Aadhaar values fail the Verhoeff checksum, so Presidio rejects them by design.
• Voter IDs are in DL/12/345/678901 form; Presidio's IN_VOTER pattern expects ABC1234567.
• The 4 IndicBERT "false positives" in Hindi are डॉ. मिश्रा, डॉ. खान etc., real names the dataset left unlabelled.

If this merges, the docs page should be updated to acc 0.89 / P 1.00 / R 0.82 / F1 0.90, response time ~0.04 s.