Repository navigation
fix(eval): qualify sealed L0 scoring paths per arm workspace - #204
Conversation
|
CHANGES at 11875b8. P2: unreadable file contents become a successful corrected score. Private paired control: the same readable file yields recall 1 and three tokens; after chmod 000, a direct read raises Please make the opt-in path propagate content-read failure as error, retain legacy scoring, and add the real non-root file case with the readable positive. This is source feedback, not an evaluation-quality claim. Tier 1 remains unchanged; no policy-required review gate is being added. Full governed finding: |
|
APPROVE SAME-feedback repair at 12b71c6. Retains the original full audit at 11875b8; reviewed only the direct-child three-file repair and its composition. The unchanged independent mode-000 mapper and actual opt-in driver failures now pass. All 58 focused tests pass, zero skipped; HOME/resource guard clean. Readable-file positives and direct PermissionError controls discriminate. Unreadable input now becomes error/None recall, including permission loss after mapping; legacy permissive scoring remains unchanged. No native indexer, provider, model or corpus invocation. Native accepted review Full governed review: |
…egration-20261023
|
SAME approval reused for refreshed head 7874417/current main e329. Independently replayed the entire approved 12b71 proposal onto main: whole tree Native review accepted reuse of Governed reuse proof: |
Native L0 context paths such as
repos/example/app.pyscore zero against Git goldapp.py. The identity-sealed opt-in runner resolves only exact workspace-qualified or literal files in each arm's workspace. Unknown, ambiguous, linked, and unreadable files produce an error with no numeric recall.Readability is checked by opening and reading the candidate file. Opt-in character accounting also propagates subsequent read failures, including permission revoked after mapping. Legacy literal scoring and its character accounting retain their existing behavior. Rows declare their scoring-path convention and remain separate from historical measurements; the mapper never consults gold to choose a path.
Validation: 599 Python tests; 58 focused controls including both reported failures, real non-root mode-000 files, readable positives, inaccessible alternatives and permission revoked between mapping and character reads. Normal
just checkand CPU-onlyjust testpass under the HOME/resource guard (1,409 native tests passed; four existing ignored). Two earlier native runs were interrupted while fixture GPU provisioning stalled; their incomplete results are retained, not counted as passes. Mapper/test Ruff checks pass; the existing driver SIM905 finding is identical at base and head.Demo: the original readable-path recording continues to describe the literal-zero to qualified-one control. The following real terminal transcript compares
11875b83with repaired candidate12b71c61on the same synthetic input and actual non-root permissions. Gold/retrieval stubs isolate the mapper and actual task driver; no evaluation campaign or retrieval-quality claim follows. Replay files remain in the existing private evidence archive; this readable transcript is provided for public review.Reproduce the committed permission and driver controls: