Skip to content

Feat/shoaib annotation consistency - #10

Merged
Khubaib8281 merged 3 commits into
mainfrom
feat/shoaib-annotation-consistency
Sep 9, 2026
Merged

Khubaib8281 merged 3 commits into
mainfrom
feat/shoaib-annotation-consistency

Conversation

@Shoaib-Altaf

Copy link
Copy Markdown
Collaborator

What does this PR do?

Implements check_annotation_consistency — Week 4 deliverable, flags samples with
annotation confidence below a configurable threshold.

Which module(s) does it touch?

inference_audit/checks/annotation_consistency.py (new)
tests/test_annotation_consistency.py (new)

How did you test it?

Ran: pytest tests/test_annotation_consistency.py -v

============================= test session starts ==============================
collected 10 items

test_high_confidence_scores_high PASSED
test_low_confidence_scores_in_expected_range PASSED
test_no_confidence_column_provided_is_graceful_skip PASSED
test_missing_column_returns_none_score PASSED
test_empty_dataframe_returns_none_score PASSED
test_invalid_threshold_above_one_returns_none PASSED
test_invalid_threshold_negative_returns_none PASSED
test_null_confidence_values_reported_explicitly PASSED
test_all_null_confidence_returns_none PASSED
test_non_numeric_values_returns_none PASSED

============================== 10 passed in 0.47s ==============================

Also ran against both real fixtures: high_confidence.csv scored 100.0 (expected 90-100), low_confidence.csv scored 45.4 (expected 30-50) — both landed cleanly inside the Role Guide's expected ranges, no discrepancy to flag on this check.

Edge cases handled:

  • conf_col=None (not provided) -> graceful skip, CheckResult(score=None, ...) — this is the expected normal case for datasets without confidence data, not an error
  • Missing column -> CheckResult(score=None, ...)
  • Empty dataframe -> CheckResult(score=None, ...)
  • confidence_threshold outside [0, 1] -> rejected with a clear explanation, since a threshold outside that range is meaningless for confidence values
  • Null values in the confidence column -> computed and reported explicitly in details and the warning (excluded from the rate calculation denominator, never silently dropped without a trace) — applying the same discipline the director's review
    required for check_label_distribution's null-label handling
  • All confidence values null -> CheckResult(score=None, ...), nothing to assess
  • Non-numeric values in the confidence column (e.g. strings) -> rejected with a clear explanation rather than silently miscompared against a numeric threshold — applying the same discipline the director's review required for check_missing_values' non-string handling

Checklist:

  • Tests added or updated
  • Docstrings on all new public functions
  • No unused imports
  • README updated if public API changed (N/A — no public API change)
  • Lead Engineer has reviewed and approved before submitting to director

@Khubaib8281
Khubaib8281 merged commit 40db8f0 into main Sep 9, 2026
1 check passed
@Khubaib8281
Khubaib8281 deleted the feat/shoaib-annotation-consistency branch September 9, 2026 03:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants