An NLP dataset quality auditor. Run one command, get a reproducible report covering five quality checks and an overall score. This pip installable python library is built so a dataset that scores well is something you can actually cite as evidence of quality, not just a number.
pip install inference-audit-pk
Every stage of an NLP pipeline from publishing a dataset, choosing one for training , depends on quality that's usually checked with a one-off script written from scratch each time, producing results that aren't comparable across projects or papers. inference-audit replaces that with a standard, versioned, reproducible check anyone can run and cite.
Every Python example in this README is wrapped in if __name__ == "__main__":. This is required on Windows, not a style choice: language_contamination uses a process pool internally for performance, and Windows has no fork(), so it re-imports your entry script to create each worker process. Without the guard, that re-import re-triggers your top-level code, causing a runtime error. If you're calling Auditor.audit() from inside a function (e.g. in a larger application) rather than a bare top-level script, the guard is not needed there — it only matters for code sitting directly at module level in the file you run.
The simplest case — audits a dataset using the five checks with default settings. annotation_consistency will be skipped (not scored zero) since no confidence column is provided.
from inference_audit import Auditor
if __name__ == "__main__":
auditor = Auditor()
report = auditor.audit("dataset.csv", label_col="emotion", text_col="text")
report.save("audit_report.html") # human-readable, self-contained HTML
report.to_json("audit_report.json") # machine-readable, for CI pipelinesinference-audit run dataset.csv \
--label-col emotion \
--text-col text \
--output audit_report.htmlPowerShell:
inference-audit run dataset.csv `
--label-col emotion `
--text-col text `
--output audit_report.htmlIf your dataset includes an annotation confidence or agreement score per row, pass it in to enable the annotation_consistency check — otherwise this check is silently skipped rather than scored.
from inference_audit import Auditor
if __name__ == "__main__":
auditor = Auditor()
report = auditor.audit(
"dataset.csv", label_col="emotion", text_col="text",
conf_col="confidence",
)
report.save("audit_report.html")inference-audit run dataset.csv \
--label-col emotion \
--text-col text \
--conf-col confidence \
--output audit_report.htmlPowerShell:
inference-audit run dataset.csv `
--label-col emotion `
--text-col text `
--conf-col confidence `
--output audit_report.htmllanguage_contamination defaults to flagging English/Hindi as contamination risks. If your dataset has a different realistic contamination risk (e.g. an English dataset where French or Spanish leakage is the actual concern), override the default rather than relying on it.
from inference_audit import Auditor
if __name__ == "__main__":
auditor = Auditor()
report = auditor.audit(
"dataset.csv", label_col="label", text_col="text",
concern_languages=("fr", "es"),
)
report.save("audit_report.html")inference-audit run dataset.csv \
--label-col label \
--text-col text \
--concern-languages fr,es \
--output audit_report.htmlPowerShell:
inference-audit run dataset.csv `
--label-col label `
--text-col text `
--concern-languages fr,es `
--output audit_report.htmlUse --fail-below <score> to make the command exit with a non-zero status if the dataset doesn't meet a quality bar — this is what makes it usable as a CI gate on a pull request.
inference-audit run dataset.csv --label-col emotion --text-col text --fail-below 70Label distribution — flags severe class imbalance. A dataset where one label dominates and another barely appears is harder to train on fairly and easier to get misleading results from.
Near duplicates — flags verbatim and near-verbatim repeated samples using MinHash/LSH similarity, including cases where the same text appears under two different labels (a common annotation error).
Language contamination — flags text confidently detected in a language your dataset isn't supposed to contain. Checks against a configurable list of "concern languages" (default: English and Hindi) rather than trying to guess the dataset's "true" language, since off-the-shelf language detection has no reliable profile for every language — Roman Urdu being one notable example. Override concern_languages/--concern-languages for datasets with a different realistic contamination risk.
Missing values — flags null, whitespace-only, and suspiciously short text samples that likely don't carry real content.
Annotation consistency — flags low-confidence or inconsistent annotations, when a confidence/agreement column is available. Skipped (not scored zero) if no such column is provided.
Each check returns a score (0–100), an optional warning, and detailed metrics. An overall score is computed as a weighted average — label distribution and near-duplicates count for more, reflecting which issues most often affect real published datasets.
| Flag | Required | Description |
|---|---|---|
path |
Yes | Dataset file (.csv, .json, or .parquet) |
--label-col |
Yes | Name of the label column |
--text-col |
Yes | Name of the text column |
--output |
No | Output path, .html or .json (default: audit_report.html) |
--conf-col |
No | Confidence/agreement column, enables the annotation consistency check |
--concern-languages |
No | Comma-separated ISO 639-1 codes to flag if detected (default: en,hi) |
--fail-below |
No | Exit with code 1 if the overall score is below this value (0–100) |
The CLI does not require the if __name__ == "__main__": guard — that's already handled internally.
- Language detection has no real signal for some languages. The underlying detection library has no language profile for some languages (Roman Urdu being a notable example), so
language_contaminationdoesn't attempt to identify "the dataset's language" — it only flags confident detections of specific languages on a configurable concern list. Text in an undetectable language can still occasionally be misdetected as a concern-list language (measured false-positive rate: roughly 10–12% on real Roman Urdu test data) — this is a bounded, documented limitation, not a bug. - Performance on very large datasets. Recent optimizations (deduplication before comparison, parallelized language detection) significantly improved runtime on large datasets, but
near_duplicatesandlanguage_contaminationremain the two most compute-intensive checks. On datasets over 100K rows, a full audit can still take up to several minutes (up to ~5 minutes observed on a 100K+ row real-world dataset) depending on text diversity and duplication rate. If you're auditing a very large corpus, expect these two checks to dominate total runtime. --concern-languagesdefaults to English/Hindi. This is a reasonable general-purpose default, particularly for code-switched or Roman-script corpora where English/Hindi leakage is a common risk — but it is not universally correct. If you're auditing a dataset with a different realistic contamination risk, override this flag rather than relying on the default.- On Windows, the Python API requires an
if __name__ == "__main__":guard when called from a top-level script, due to howlanguage_contamination's process pool is created withoutfork()support. See the note above the Quickstart section. The CLI is unaffected.
Running against a real 134K-row Roman Urdu emotion corpus:
Overall score: 74
label_distribution: 61 (severe class imbalance detected)
near_duplicates: 78.6
language_contamination: 71.1
missing_values: 88
annotation_consistency: skipped (no confidence column provided)
See sample_report.html and sample_report.json in this repository for full rendered examples, including the score chart.
- Python 3.9+
- pandas, jinja2, matplotlib, datasketch, langdetect, typer (installed automatically)
git clone https://github.com/Inference-LAB/inference-audit.git
cd inference-audit
python -m venv venv
source venv/bin/activate # or venv\Scripts\Activate.ps1 on Windows
pip install -e .
pytest --cov=inference_auditMIT — see LICENSE.
Built as part of the INFERENCE Lab Engineering Fellowship, Cohort 01 — Project C.