Skip to content

fix(eval): refuse contaminated independent L0 measurements - #210

Merged
scbrown merged 1 commit into
mainfrom
gennaro/95-measurement-contract-20261023
Oct 23, 2026
Merged

scbrown merged 1 commit into
mainfrom
gennaro/95-measurement-contract-20261023

Conversation

@scbrown

@scbrown scbrown commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Independent L0 cells can currently accept unexplained silence or reducing-ledger contamination as a skip, and hook errors can carry numeric zero recall. Add a versioned opt-in outcome contract to the sealed-reuse driver: contamination, unsupported/contradictory output and unexplained silence are errors with null efficacy metrics, triggering the existing quarantine. Valid headers and explicit operational/gate skips retain their meanings, including rounded equal score/gate displays.

Legacy scoring and all original rows remain unchanged. The reconciliation accounts for 213 injected, 34 header errors, 23 dedup-worded skips, 42 operational skips and 18 gate skips. Historical stdout is not in the scalar projection, so the exact cause of the 34 header errors remains unknown. Do not decode arbitrary output into invented zero-quality results. Reuse the already delivered state isolation/path qualification rather than duplicating it.

This delivers the next source/fixture step on #95, not a fresh campaign. The proposal names the remaining gates: current-native-runtime enrollment (sealed reuse currently admits only exact 0.28.0 bytes), measured aggregate storage and acceptance of the unchanged six-hour/4-CPU/6-GiB/no-swap/no-GPU execution envelope. Its proposed 32-GiB total storage ceiling is not accepted or proven sufficient. No indexing, model/provider call, GPU run, resource increase or resumption of old outputs occurs here; #96/#97 stay gated.

Validation: full just test-eval 623 passed; 24 focused contract controls include real CLI → hook fixture → classifier → actual cell quarantine, with fake indexing. Three inverse changes fail the intended driver controls (3, 3 and 1 failures, no instrument errors). Candidate and inverse HOME guards: 157 paths/five resources unchanged. New-test Ruff, diff check and file-size check pass. Initial inverse fixture missed the scorer package and is retained as invalid; corrected inverse copies run the intended controls. No Rust source changed or native build exercised.

Demo, actual source fixture transcript at 20a9ecf: for identical silent or ledger-skip input, legacy reports skipped, while independent-hook-outcomes/v1 reports error; a standard-header positive remains injected in both. Reproduce with cd eval && python3 -m pytest tests/test_l0_outcome_contract.py -q; the driver controls also assert null error metrics and real quarantine. This is a text/source demonstration, not installed behavior or new retrieval quality.

Rollback: revert this instrument change, preserve old and separately named future outputs. Current normal classification b5bd74f6: T1/floor1/score1.39, review policy advice review_required=false, current neutral76579334. Normal CI/hold/preflight admission still applies; Tim retains landing custody. This does not accept the proposed campaign envelope. Ref aegis-8rga / bobbin#95.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant