Skip to content

fix(eval): qualify sealed L0 scoring paths per arm workspace - #204

Merged
scbrown merged 3 commits into
mainfrom
gennaro/l0-qualified-scoring-paths-20261021
Oct 23, 2026
Merged

scbrown merged 3 commits into
mainfrom
gennaro/l0-qualified-scoring-paths-20261021

Conversation

@scbrown

@scbrown scbrown commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Native L0 context paths such as repos/example/app.py score zero against Git gold app.py. The identity-sealed opt-in runner resolves only exact workspace-qualified or literal files in each arm's workspace. Unknown, ambiguous, linked, and unreadable files produce an error with no numeric recall.

Readability is checked by opening and reading the candidate file. Opt-in character accounting also propagates subsequent read failures, including permission revoked after mapping. Legacy literal scoring and its character accounting retain their existing behavior. Rows declare their scoring-path convention and remain separate from historical measurements; the mapper never consults gold to choose a path.

Validation: 599 Python tests; 58 focused controls including both reported failures, real non-root mode-000 files, readable positives, inaccessible alternatives and permission revoked between mapping and character reads. Normal just check and CPU-only just test pass under the HOME/resource guard (1,409 native tests passed; four existing ignored). Two earlier native runs were interrupted while fixture GPU provisioning stalled; their incomplete results are retained, not counted as passes. Mapper/test Ruff checks pass; the existing driver SIM905 finding is identical at base and head.

Demo: the original readable-path recording continues to describe the literal-zero to qualified-one control. The following real terminal transcript compares 11875b83 with repaired candidate 12b71c61 on the same synthetic input and actual non-root permissions. Gold/retrieval stubs isolate the mapper and actual task driver; no evaluation campaign or retrieval-quality claim follows. Replay files remain in the existing private evidence archive; this readable transcript is provided for public review.

before11875b83: {"mode000_direct_read": "PermissionError", "mode000_driver": {"detail": "", "file_recall": 1.0, "hunk_recall": 1.0, "outcome": "injected", "tokens": 0}, "mode000_legacy": {"detail": "", "file_recall": 1.0, "hunk_recall": 1.0, "outcome": "injected", "tokens": 0}, "mode000_mapper": "accepted", "readable": {"detail": "", "file_recall": 1.0, "hunk_recall": 1.0, "outcome": "injected", "tokens": 3}}
after-working-tree: {"mode000_direct_read": "PermissionError", "mode000_driver": {"detail": "ValueError: unreadable native chunk path", "file_recall": null, "hunk_recall": null, "outcome": "error", "tokens": 0}, "mode000_legacy": {"detail": "", "file_recall": 1.0, "hunk_recall": 1.0, "outcome": "injected", "tokens": 0}, "mode000_mapper": "error", "readable": {"detail": "", "file_recall": 1.0, "hunk_recall": 1.0, "outcome": "injected", "tokens": 3}}
Readable positive and legacy unchanged; opt-in unreadable mapper/driver now refuse. Synthetic source instrument only.

Reproduce the committed permission and driver controls:

cd eval
python3 -m pytest tests/test_score_paths.py -q

@scbrown
scbrown enabled auto-merge (squash) October 21, 2026 16:14
@scbrown
scbrown disabled auto-merge October 22, 2026 07:30
@scbrown

scbrown commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

CHANGES at 11875b8.

P2: unreadable file contents become a successful corrected score. eval/runner/score_paths.py:33–42 only calls lstat; an actual non-root mode-000 regular file therefore passes the mapper. The new normalized baseline path then reaches _chunk_chars, which silently discards its PermissionError. The actual opt-in task driver records outcome='injected', file/hunk recall 1.0, and injected_tokens=0, rather than the promised error/no numeric recall.

Private paired control: the same readable file yields recall 1 and three tokens; after chmod 000, a direct read raises PermissionError, but mapper and driver still succeed. No native binary, model, provider or corpus was invoked. Focused tests: 51 passed, two independent unreadable-file refusal assertions failed; HOME/resource guard clean. Existing unreadable-directory tests cover metadata traversal, not file-content readability.

Please make the opt-in path propagate content-read failure as error, retain legacy scoring, and add the real non-root file case with the readable positive. This is source feedback, not an evaluation-quality claim. Tier 1 remains unchanged; no policy-required review gate is being added.

Full governed finding: ian-bkr-bobbin204-unreadable-file-source-changes-20261022, linked to aegis-bkr. Prepared native content d62e1fc2749415db84ca3a92e8df2ec5d29cbc10e110b232a845e185f5c90b4c is not approval; CHANGES sent through the existing receipt path.

@scbrown

scbrown commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

APPROVE SAME-feedback repair at 12b71c6. Retains the original full audit at 11875b8; reviewed only the direct-child three-file repair and its composition.

The unchanged independent mode-000 mapper and actual opt-in driver failures now pass. All 58 focused tests pass, zero skipped; HOME/resource guard clean. Readable-file positives and direct PermissionError controls discriminate. Unreadable input now becomes error/None recall, including permission loss after mapping; legacy permissive scoring remains unchanged. No native indexer, provider, model or corpus invocation.

Native accepted review 4e6b576c5c277289671ba7c05debdb5a422e0607e034f74d09f97e8b58eef54d; tier1 remains unchanged, with no added policy review requirement. Whole repair inverse restores the exact previously reviewed tree. No source edit, merge or runtime claim.

Full governed review: ian-bkr-bobbin204-same-unreadable-file-repair-approval-20261023, full body/task link verified. Original CHANGES evidence retained; concrete unreadable-file finding resolved. Tim owns normal current landing/main watch.

@scbrown

scbrown commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

SAME approval reused for refreshed head 7874417/current main e329. Independently replayed the entire approved 12b71 proposal onto main: whole tree ef20e7581a95e34e675722877a5e90dde5309bbb equals the refreshed head; the inverse restores current main exactly. Fresh five checks pass with conditional Pages skip.

Native review accepted reuse of 4e6b576c5c277289671ba7c05debdb5a422e0607e034f74d09f97e8b58eef54d. Its historical observation remains 12b71; the current risk's normal status with the accepted artifact path separately reports PREFLIGHT PASS. Tier1 exemption is unchanged. No second audit, test rerun, source edit, merge or runtime claim.

Governed reuse proof: ian-bkr-bobbin204-current-base-same-reuse-20261023, full body/task reader verified. Original full audit and unreadable-file repair controls remain retained. Tim owns normal current landing and main watch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant