[robertotru]

Summary

This pull request addresses evaluation defects and observability limitations in the VerifyCraft review quality benchmark identified during full-split live DeepSeek evaluations.

1. Root Cause Analysis & Scoring Accuracy

• Defect in dev_01_req_violation_auth_expiry: Ground truth previously enforced rigid literal substring matching ("expiration"). Valid review findings expressing semantic equivalents (e.g. "Expired tokens are accepted as valid" citing token.expires_at) were misclassified simultaneously as a false negative and an ungrounded false positive. Resolved by introducing versioned aliases ("expired", "expires_at", "expiry") and evidence_anchors ("expires_at", "authenticate") linked to a stable defect_id.
• Adjudication for dev_09 KeyError Claim: In dev_09, get_lead_user_id(users: list[dict]) accesses primary['id']. Flagging this as a defect without input schema evidence is an unsupported caller assumption. Rather than silently dropping or ignoring it, a versioned AdjudicationCatalog was established (benchmarks/data/review_eval/adjudications.json), classifying it auditably as an unsupported_claim (retained as a false positive).
• Matching Invariants: Implemented 1-to-1 bipartite matching so that duplicate findings targeting a single defect cannot inflate True Positive yields or satisfy multiple distinct defects simultaneously.
• Requirement Status Normalization: Fixed status vocabulary mapping in src/verifycraft/review/engine.py and benchmark scoring to normalize valid provider statuses ("satisfied") into the canonical appears_satisfied enum.

2. Offline Rescoring & Auditability (--rescore)

• Added --rescore <PATH> CLI option to evaluate past live run reports deterministically and hermetically without live provider API calls.
• Prints decision deltas showing exactly what changed, why, and the impact on release gate policy.

3. CLI Observability & Stream Separation

• Replaced monolithic, non-interactive execution with streaming progress logs written directly to sys.stderr and flushed immediately.
• Added --quiet (-q) flag to suppress progress logs.
• When --json is supplied, machine-readable JSON is emitted cleanly to stdout while progress and diagnostics remain isolated on stderr.
• Added fails-closed credential validation that exits with code 1 and actionable remediation instructions when required keys are missing.

4. Verification & Invariants

• Unit tests added in tests/unit/test_benchmark_accuracy.py (11 new tests).
• All 336 project tests passing (pytest), strict type checking passing (mypy), and formatting/linting passing (ruff).
• No modifications were made to bypass tests or weaken quality gate thresholds (85% precision, 90% recall, 0 invalid locations).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud