Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/demos/l0-score-paths/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Exact L0 scoring paths

A synthetic header in the measured frozen native format names
`repos/fixture/app.py`; Git gold names `app.py`. The same two-line source scores
file recall 0 under legacy literal comparison and 1 through the opt-in exact
workspace mapping. A foreign-only path refuses, and original literal scoring
remains unchanged. This demonstrates the instrument, not retrieval quality.

Run `python3 docs/demos/l0-score-paths/probe.py` from the repository root.
The source base is `9624689e5eca15f198910201fd89ebf3afd1af9e`; the candidate
adds the mapper and opt-in driver wiring. Exact candidate identity is in the PR.
No native executable, model, provider, graph or campaign is invoked.

[Real terminal capture](output.log) and [timing](timing.log) can be downloaded
and replayed with:

```sh
scriptreplay --log-timing timing.log --log-out output.log --maxdelay 0.1
```

The private paired control also applies this mapper to retained actual native
hook chunks. Its outcome does not change or reinterpret the earlier partial run.
7 changes: 7 additions & 0 deletions docs/demos/l0-score-paths/output.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Script started on 2026-10-20 23:26:32-04:00 [COMMAND="stty -onlcr; python3 docs/demos/l0-score-paths/probe.py" <not executed on terminal>]
Before (legacy literal paths): file_recall=0.0
After (opt-in exact workspace mapping): file_recall=1.0
Foreign-only path: REFUSED (unknown or ambiguous native chunk path)
Original literal scoring unchanged; synthetic instrument only, no retrieval/campaign

Script done on 2026-10-20 23:26:32-04:00 [COMMAND_EXIT_CODE="0"]
29 changes: 29 additions & 0 deletions docs/demos/l0-score-paths/probe.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
"""Synthetic transport-path instrument demo; no retrieval, model or campaign."""
import sys
import tempfile
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parents[3] / 'eval'))
from runner.l0 import parse_injected_chunks, score_chunks
from runner.score_paths import workspace_chunks

with tempfile.TemporaryDirectory() as directory:
workspace = Path(directory) / 'fixture'
workspace.mkdir()
(workspace / 'app.py').write_text('def authenticate(token):\n return bool(token)\n')
native = parse_injected_chunks('--- repos/fixture/app.py:1-2 authenticate ---')
gold = {'app.py': [(1, 2)]}
before = score_chunks(native, gold)['file_recall']
after = score_chunks(workspace_chunks(workspace, native), gold)['file_recall']
assert before == 0 and after == 1
print(f'Before (legacy literal paths): file_recall={before}')
print(f'After (opt-in exact workspace mapping): file_recall={after}')
try:
workspace_chunks(workspace, [('repos/foreign/app.py', 1, 2)])
except ValueError as exc:
print(f'Foreign-only path: REFUSED ({exc})')
else:
raise AssertionError('foreign path accepted')
assert native == [('repos/fixture/app.py', 1, 2)]
assert score_chunks(native, gold)['file_recall'] == before
print('Original literal scoring unchanged; synthetic instrument only, no retrieval/campaign')
3 changes: 3 additions & 0 deletions docs/demos/l0-score-paths/timing.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
0.036814 103
0.000012 68
0.000012 85
30 changes: 24 additions & 6 deletions docs/eval/l0-bounded-index-reuse.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,28 @@ Rollback: revert the source commit, retain all immutable prior/new results, and
do not reinterpret a partial run as complete. No production indexing, release,
resource increase, timer or service change is part of this source delivery.

A separate scoring caveat remains: the retained native hook emits repository-
A paired scoring probe found that the retained native hook emits repository-
qualified paths such as `repos/fixture/app.py`, while the existing scorer matches
paths literally against Git-relative gold paths. This source change preserves
that scorer contract and the original scores; the private hook-equivalence
controls compare native chunks directly, not claimed recall improvements. Path
normalization and any resulting study interpretation require explicit paired
scorer controls before full-cohort quality acceptance.
paths literally against Git-relative gold paths. Legacy scoring and original
scores stay unchanged; the private hook-equivalence controls compare native
chunks directly, not claimed recall improvements. The opt-in correction below
is paired against those same chunks before any full-cohort interpretation.

The opt-in scorer now resolves that path caveat with exact workspace ownership:
only a literal existing file or the exact current `repos/<workspace-name>/`
prefix can match. Exactly one nonlinked candidate is required. Ambiguity,
unknown/foreign-only paths and traversal refuse as an error. No gold-dependent
matching is used; normalization is applied before baseline character counting,
while actual hook character accounting is preserved. The manifest records
`workspace-qualified-or-literal/v1`; legacy scoring and old rows stay unchanged.
The paired private instrument probe checks the same retained actual native chunks,
and driver controls verify recall 0 before versus 1 with the opt-in mapping.
This repairs interpretation of paths, not a measured improvement in retrieval.

Path qualification uses the workspace that produced the current arm’s chunks.
The no-coupling copy is checked independently: an original-workspace file cannot
satisfy a missing derived-workspace file, even when both directories share a name.

An unreadable candidate is an explicit error, not evidence of absence. Strict
metadata reads refuse instead of treating an inaccessible alternative as a
unique path match.
3 changes: 2 additions & 1 deletion eval/runner/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -1843,6 +1843,7 @@ def l0(tasks_dir, task_ids, arms, budgets, out_path, workdir, index_timeout, reu
cells = planned_cells(tasks, arm_list, budget_list)
manifest["planned_cells"] = cells
manifest["index_reuse"] = "sealed-no-coupling/v1" if reuse_model_dir else "legacy"
manifest["scoring_paths"] = "workspace-qualified-or-literal/v1" if reuse_model_dir else "literal"
if reuse_limits is not None:
manifest["resource_limits"] = reuse_limits
manifest["wall_supervisor_required_seconds"] = 21600
Expand Down Expand Up @@ -1924,7 +1925,7 @@ def index(ws: Path, overrides=None):
target, Path(bobbin), reuse_model_dir, overrides, journal,
lambda a=arm, b=budget: L0.run_task(
task, ws, [a], [b], bobbin, env, index,
nocoupling_workspace=fork))
nocoupling_workspace=fork, normalize_paths=True))
except Exception as exc:
# Preserve the explicit failure and all prior cells;
# remaining cells stay missing. No reset/resend follows.
Expand Down
15 changes: 12 additions & 3 deletions eval/runner/l0.py
Original file line number Diff line number Diff line change
Expand Up @@ -668,6 +668,7 @@ def run_task(
index_workspace: Callable[[Path, dict[str, str] | None], None],
*,
nocoupling_workspace: Path | None = None,
normalize_paths: bool = False,
) -> list[ArmScore]:
"""Score every (arm, budget) for one task on an already-checked-out workspace.

Expand Down Expand Up @@ -696,8 +697,10 @@ def run_task(
continue
try:
chars = 0
scoring_workspace = workspace
if arm in ABLATIONS:
ws = nocoupling_ws if arm == "-coupling" else workspace
scoring_workspace = ws
outcome, detail, chunks, chars = run_hook_arm(
ws, prompt, arm, budget, env, bobbin
)
Expand All @@ -715,7 +718,11 @@ def run_task(
outcome, detail = (
("injected", "") if chunks else ("skipped", "baseline found nothing")
)
chars = _chunk_chars(workspace, chunks)
if normalize_paths:
from runner.score_paths import workspace_chunks
chunks = workspace_chunks(scoring_workspace, chunks)
if arm not in ABLATIONS:
chars = _chunk_chars(workspace, chunks, strict=normalize_paths)
except Exception as exc: # noqa: BLE001 -- recorded, never swallowed
base.outcome, base.detail = "error", f"{type(exc).__name__}: {exc}"[:300]
results.append(base)
Expand All @@ -735,12 +742,14 @@ def run_task(
return results


def _chunk_chars(workspace: Path, chunks: list[Chunk]) -> int:
def _chunk_chars(workspace: Path, chunks: list[Chunk], *, strict: bool = False) -> int:
total = 0
for path, s, e in chunks:
try:
lines = (workspace / path).read_text(errors="replace").splitlines()
except OSError:
except OSError as exc:
if strict:
raise ValueError("unreadable native chunk path") from exc
continue
total += sum(len(x) + 1 for x in lines[s - 1 : e])
return total
Expand Down
51 changes: 51 additions & 0 deletions eval/runner/score_paths.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
"""Resolve frozen native context paths without guessing another repository.

Only the opt-in, identity-sealed L0 runner uses this mapping. Legacy rows and
literal scoring remain unchanged. An ambiguous or unknown path is an error,
never a zero-recall observation.
"""
import stat
from pathlib import Path, PurePosixPath


def workspace_chunks(workspace: Path, chunks: list[tuple[str, int, int]]) -> list[tuple[str, int, int]]:
"""Map literal or exact repos/<workspace-name>/ paths to existing files."""
root = workspace.resolve(strict=True)
prefix = f"repos/{workspace.name}/"
result = []
for path, start, end in chunks:
if (not isinstance(path, str) or not path or "\\" in path or "\0" in path
or PurePosixPath(path).is_absolute() or ".." in PurePosixPath(path).parts
or PurePosixPath(path).as_posix() != path):
raise ValueError("unsafe native chunk path")
names = {path}
if path.startswith(prefix):
names.add(path[len(prefix):])
candidates = set()
for name in names:
if not name:
continue
target = root
metadata = None
for part in PurePosixPath(name).parts:
target = target / part
try:
metadata = target.lstat()
except (FileNotFoundError, NotADirectoryError):
metadata = None
break
except OSError as exc:
raise ValueError("unreadable native chunk path") from exc
if stat.S_ISLNK(metadata.st_mode):
raise ValueError("linked native chunk path")
if metadata is not None and stat.S_ISREG(metadata.st_mode):
try:
with target.open("rb") as contents:
contents.read(1)
except OSError as exc:
raise ValueError("unreadable native chunk path") from exc
candidates.add(name)
if len(candidates) != 1:
raise ValueError("unknown or ambiguous native chunk path")
result.append((candidates.pop(), start, end))
return result
Loading
Loading