Skip to content

fix(evals): resolve tool paths from BASE cwd - #77

Open
svozza wants to merge 1 commit into
mainfrom
fix/eval-base-cwd-grading
Open

svozza wants to merge 1 commit into
mainfrom
fix/eval-base-cwd-grading

Conversation

@svozza

@svozza svozza commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • grade relative read-only tool paths against the BASE cwd configured by cc_loop
  • treat pathless Grep and Glob requests as searches of that fixed cwd
  • keep rejecting explicit PR-head paths, sibling-prefix paths, and traversal outside BASE
  • add a deterministic regression for PR refactor: use direct standard JSON schemas #76 attempt 1's pathless Grep followed by an absolute BASE Read

Why

PR #76 run 32378471020 attempt 1 found the line-12 slice_dictionary defect, searched from the real BASE cwd, read the pinned parameters/ssm.py caller, and submitted the correct SSM impact. The eval nevertheless failed because its Grep request omitted an explicit BASE path.

That is stricter than production. cc_loop fixes the CLI cwd to BASE and exposes no tool that can change it, so pathless Grep/Glob and relative paths have a deterministic location. The existing grader discarded that known execution context and produced a false behavioral failure.

This changes the grader, not expect.json: the scenario still requires investigation of both slice_dictionary and parameters/ssm.py under BASE.

Verification

  • uv lock --check
  • uv sync --frozen --all-groups --no-install-project --check
  • uv run --frozen --group test python -m pytest tests/test_run_evals.py -p no:cacheprovider -q (380 passed, 2 skipped)
  • uv run --frozen --group test python -m pytest tests/ -p no:cacheprovider -q (2301 passed, 2 skipped)
  • uv run --frozen --group typecheck ty check --python .venv/bin/python

A direct branch workflow dispatch was attempted as run 32382298342, but the eval job was rejected before starting because the runtime environment intentionally allows only main. The PR-triggered pull_request_target workflow is the supported credentialed path for executing PR-head eval code.

@svozza
svozza deployed to ai-pr-review-runtime August 20, 2026 14:48 — with GitHub Actions Active
@svozza

svozza commented Aug 20, 2026

Copy link
Copy Markdown
Owner Author

Final verification on PR head af5746c:

  • Quality check: test_verifier passed; typecheck passed
  • Protected eval run 32382365886: 42/42 review scenarios passed, including caller_impact_needs_investigation; 0 behavioral failures; 0 invalid samples
  • Plan evals: 6/6 passed

The attempted direct branch dispatch (32382298342) produced no sample because the main-only runtime environment rejected the job before steps started; the PR-triggered pull_request_target run is the supported credentialed execution path and completed successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant