Final Evalution

.github/workflows/ci.yml

@@ -1,20 +1,46 @@
 name: CI
- 
+
 on:
   push:
-    branches: [ main ]
+    branches:
+      - main
   pull_request:
-    branches: [ main ]
-
+    branches:
+      - main
+
 jobs:
   test:
     runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
+
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v4
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
         with:
-          python-version: "3.10"
-      # - name: Install dependencies
-      #   run: pip install -e ".[dev]"
-      # - name: Run tests
-      #   run: pytest --cov=<package_name>
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+          pip install pytest pytest-cov
+
+      - name: Run tests with coverage (90% required)
+        run: |
+          pytest --cov=inference_audit \
+                 --cov-report=xml:coverage_report.xml \
+                 --cov-report=term-missing \
+                 --cov-fail-under=90
+
+      - name: Upload coverage report artifact
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: coverage-report-py${{ matrix.python-version }}
+          path: coverage_report.xml

.gitignore

@@ -3,3 +3,10 @@ __pycache__/
 .vscode/
 
 venv/
+
+htmlcov/
+coverage_report.xml
+.coverage
+
+*.egg-info/
+build/

README.md

@@ -1,18 +1,20 @@
 # inference-audit
 
-An NLP dataset quality auditor. Run one command, get a reproducible report covering five quality checks and an overall score — built so a dataset that scores well is something you can actually cite as evidence of quality, not just a number.
+An NLP dataset quality auditor. Run one command, get a reproducible report covering five quality checks and an overall score.This pip installable python library is built so a dataset that scores well is something you can actually cite as evidence of quality, not just a number.
 
 ```
 pip install inference-audit-pk
 ```
 
 ## Why this exists
 
-Every stage of an NLP pipeline — publishing a dataset, choosing one for training — depends on quality that's usually checked with a one-off script written from scratch each time, producing results that aren't comparable across projects or papers. `inference-audit` replaces that with a standard, versioned, reproducible check anyone can run and cite.
+Every stage of an NLP pipeline from publishing a dataset, choosing one for training , depends on quality that's usually checked with a one-off script written from scratch each time, producing results that aren't comparable across projects or papers. `inference-audit` replaces that with a standard, versioned, reproducible check anyone can run and cite.
 
 ## Quickstart
 
-### Python
+### Default usage
+
+The simplest case — audits a dataset using the five checks with default settings. `annotation_consistency` will be skipped (not scored zero) since no confidence column is provided.
 
 ```python
 from inference_audit import Auditor
@@ -24,12 +26,37 @@ report.save("audit_report.html")     # human-readable, self-contained HTML
 report.to_json("audit_report.json")  # machine-readable, for CI pipelines
 ```
 
-### CLI
+```bash
+inference-audit run dataset.csv \
+  --label-col emotion \
+  --text-col  text \
+  --output    audit_report.html
+```
+
+PowerShell:
+```powershell
+inference-audit run dataset.csv `
+  --label-col emotion `
+  --text-col  text `
+  --output    audit_report.html
+```
+
+### With a confidence column
+
+If your dataset includes an annotation confidence or agreement score per row, pass it in to enable the `annotation_consistency` check — otherwise this check is silently skipped rather than scored.
+
+```python
+report = auditor.audit(
+    "dataset.csv", label_col="emotion", text_col="text",
+    conf_col="confidence",
+)
+```
 
 ```bash
 inference-audit run dataset.csv \
   --label-col emotion \
   --text-col  text \
+  --conf-col  confidence \
   --output    audit_report.html
 ```
 
@@ -38,10 +65,41 @@ PowerShell:
 inference-audit run dataset.csv `
   --label-col emotion `
   --text-col  text `
+  --conf-col  confidence `
   --output    audit_report.html
 ```
 
-Use `--fail-below <score>` to make the command exit with a non-zero status if the dataset doesn't meet a quality bar — this is what makes it usable as a CI gate on a pull request:
+### With a custom concern-languages list
+
+`language_contamination` defaults to flagging English/Hindi as contamination risks. If your dataset has a different realistic contamination risk (e.g. an English dataset where French or Spanish leakage is the actual concern), override the default rather than relying on it.
+
+```python
+report = auditor.audit(
+    "dataset.csv", label_col="label", text_col="text",
+    concern_languages=("fr", "es"),
+)
+```
+
+```bash
+inference-audit run dataset.csv \
+  --label-col label \
+  --text-col  text \
+  --concern-languages fr,es \
+  --output    audit_report.html
+```
+
+PowerShell:
+```powershell
+inference-audit run dataset.csv `
+  --label-col label `
+  --text-col  text `
+  --concern-languages fr,es `
+  --output    audit_report.html
+```
+
+### CI quality gate
+
+Use `--fail-below <score>` to make the command exit with a non-zero status if the dataset doesn't meet a quality bar — this is what makes it usable as a CI gate on a pull request.
 
 ```bash
 inference-audit run dataset.csv --label-col emotion --text-col text --fail-below 70
@@ -53,7 +111,7 @@ inference-audit run dataset.csv --label-col emotion --text-col text --fail-below
 
 **Near duplicates** — flags verbatim and near-verbatim repeated samples using MinHash/LSH similarity, including cases where the same text appears under two different labels (a common annotation error).
 
-**Language contamination** — flags text confidently detected in a language your dataset isn't supposed to contain. Checks against a configurable list of "concern languages" (default: English and Hindi, reflecting realistic contamination risk for Roman Urdu corpora) rather than trying to guess the dataset's "true" language, since off-the-shelf language detection has no reliable profile for Roman Urdu.
+**Language contamination** — flags text confidently detected in a language your dataset isn't supposed to contain. Checks against a configurable list of "concern languages" (default: English and Hindi) rather than trying to guess the dataset's "true" language, since off-the-shelf language detection has no reliable profile for every language — Roman Urdu being one notable example. Override `concern_languages`/`--concern-languages` for datasets with a different realistic contamination risk.
 
 **Missing values** — flags null, whitespace-only, and suspiciously short text samples that likely don't carry real content.
 
@@ -75,9 +133,9 @@ Each check returns a score (0–100), an optional warning, and detailed metrics.
 
 ## Known Limitations
 
-- **Language detection has no real signal for Roman Urdu.** The underlying detection library has no language profile for it, so `language_contamination` doesn't attempt to identify "the dataset's language" — it only flags confident detections of specific languages on a configurable concern list. Genuine Roman Urdu text can still occasionally be misdetected as a concern-list language (measured false-positive rate: roughly 10–12% on real test data) — this is a bounded, documented limitation, not a bug.
-- **Near-duplicate detection performance** on very large datasets (100K+ rows) can approach the tool's own time budget; if you're auditing a very large corpus, expect this to be the slowest of the five checks.
-- **`--concern-languages` defaults to English/Hindi**, chosen for the tool's primary use case (Roman Urdu corpora). If you're auditing a dataset with different realistic contamination risks, override this flag rather than relying on the default.
+- **Language detection has no real signal for some languages.** The underlying detection library has no language profile for some languages (Roman Urdu being a notable example), so `language_contamination` doesn't attempt to identify "the dataset's language" — it only flags confident detections of specific languages on a configurable concern list. Text in an undetectable language can still occasionally be misdetected as a concern-list language (measured false-positive rate: roughly 10–12% on real Roman Urdu test data) — this is a bounded, documented limitation, not a bug.
+- **Performance on very large datasets.** Recent optimizations (deduplication before comparison, parallelized language detection) significantly improved runtime on large datasets, but `near_duplicates` and `language_contamination` remain the two most compute-intensive checks. On datasets over 100K rows, a full audit can still take up to several minutes (up to ~5 minutes observed on a 100K+ row real-world dataset) depending on text diversity and duplication rate. If you're auditing a very large corpus, expect these two checks to dominate total runtime.
+- **`--concern-languages` defaults to English/Hindi.** This is a reasonable general-purpose default, particularly for code-switched or Roman-script corpora where English/Hindi leakage is a common risk — but it is not universally correct. If you're auditing a dataset with a different realistic contamination risk, override this flag rather than relying on the default.
 
 ## Example Output
 
@@ -93,7 +151,7 @@ missing_values:           88
 annotation_consistency:   skipped (no confidence column provided)
 ```
 
-See `report.html` in this repository for a full rendered example, including the score chart.
+See `sample_report.html` and `sample_report.json` in this repository for full rendered examples, including the score chart.
 
 ## Requirements
 
@@ -119,5 +177,5 @@ MIT — see [LICENSE](LICENSE).
 
 Built as part of the Inference Lab Engineering Fellowship, Cohort 01 — Project C.
 
-- Khadija Faisal (Lead Engineer)
-- Muhammad Shoaib Altaf (Research & Implementation Engineer)
+- **Khadija Faisal** (Lead Engineer) — [GitHub](https://github.com/khadijja1) · [LinkedIn](https://www.linkedin.com/in/khadijjafaisal)
+- **Muhammad Shoaib Altaf** (Research & Implementation Engineer) — [GitHub](https://github.com/Shoaib-Altaf) · [LinkedIn](https://www.linkedin.com/in/muhammad-shoaib-altaf-6ab3a8326)