Repository navigation
fix(eval): refuse contaminated independent L0 measurements - #210
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Independent L0 cells can currently accept unexplained silence or reducing-ledger contamination as a skip, and hook errors can carry numeric zero recall. Add a versioned opt-in outcome contract to the sealed-reuse driver: contamination, unsupported/contradictory output and unexplained silence are errors with null efficacy metrics, triggering the existing quarantine. Valid headers and explicit operational/gate skips retain their meanings, including rounded equal score/gate displays.
Legacy scoring and all original rows remain unchanged. The reconciliation accounts for 213 injected, 34 header errors, 23 dedup-worded skips, 42 operational skips and 18 gate skips. Historical stdout is not in the scalar projection, so the exact cause of the 34 header errors remains unknown. Do not decode arbitrary output into invented zero-quality results. Reuse the already delivered state isolation/path qualification rather than duplicating it.
This delivers the next source/fixture step on #95, not a fresh campaign. The proposal names the remaining gates: current-native-runtime enrollment (sealed reuse currently admits only exact 0.28.0 bytes), measured aggregate storage and acceptance of the unchanged six-hour/4-CPU/6-GiB/no-swap/no-GPU execution envelope. Its proposed 32-GiB total storage ceiling is not accepted or proven sufficient. No indexing, model/provider call, GPU run, resource increase or resumption of old outputs occurs here; #96/#97 stay gated.
Validation: full
just test-eval623 passed; 24 focused contract controls include real CLI → hook fixture → classifier → actual cell quarantine, with fake indexing. Three inverse changes fail the intended driver controls (3, 3 and 1 failures, no instrument errors). Candidate and inverse HOME guards: 157 paths/five resources unchanged. New-test Ruff, diff check and file-size check pass. Initial inverse fixture missed the scorer package and is retained as invalid; corrected inverse copies run the intended controls. No Rust source changed or native build exercised.Demo, actual source fixture transcript at
20a9ecf: for identical silent or ledger-skip input, legacy reportsskipped, whileindependent-hook-outcomes/v1reportserror; a standard-header positive remainsinjectedin both. Reproduce withcd eval && python3 -m pytest tests/test_l0_outcome_contract.py -q; the driver controls also assert null error metrics and real quarantine. This is a text/source demonstration, not installed behavior or new retrieval quality.Rollback: revert this instrument change, preserve old and separately named future outputs. Current normal classification b5bd74f6: T1/floor1/score1.39, review policy advice review_required=false, current neutral76579334. Normal CI/hold/preflight admission still applies; Tim retains landing custody. This does not accept the proposed campaign envelope. Ref aegis-8rga / bobbin#95.