Four small, runnable primitives for evaluating LLM systems honestly. Each file is self-contained and prints its own result. No dependencies except pytest.
python skill4_parse.py
Turns model text into validated data or fails loudly. Three checks: structure, type/range, and whether a claimed quotation appears verbatim in the source.
Expected output: two PASS lines, then PE-03 (wrong type), PE-04 (out of range), PE-05 (fabricated quote). The last case is the point - valid JSON, valid score, rejected because the model quoted "100 percent" from a source saying 99.2.
python skill5_trail.py
Appends run records chained by hash, then tampers with row 1 and re-verifies.
Expected: intact: (True, -1) then after tamper: (False, 1).
A hash stored beside its payload proves nothing - an attacker recomputes it. Chaining each row to the previous one makes tampering detectable and locates it.
python skill6_wilson.py
Expected output includes:
9/10 = 90% 95% CI [0.596, 0.982] width 0.386
900/1000 = 90% 95% CI [0.880, 0.917] width 0.037
5/5 = 100% 95% CI [0.566, 1.000] width 0.434
Same headline percentage, very different amounts of evidence. 5/5 cannot rule out a 43% failure rate. A confidence interval describes the coverage of the procedure across repeated samples, not the probability of this one interval.
pytest -q
Expected: 2 passed. Then break it on purpose:
sed -i 's/"score": -1, "reason"/"score": 4, "reason"/' mysystem.py
pytest -q # 2 failed
sed -i 's/"score": 4, "reason"/"score": -1, "reason"/' mysystem.py
A suite that has never been watched to fail is not coverage. Green on degenerate data is indistinguishable from green on correct code.
Note the two tests fail differently by design: one is an aggregate rate gate, one pins a specific behaviour (abstention). Aggregate gates alone can absorb a broken behaviour without crossing the threshold.
Written by Rehan Masood.