Skip to content

Preserve shared scoring and native answer admission after extraction - #2

Open
Maisiqi wants to merge 1 commit into
lexmount:mainfrom
Maisiqi:codex/pr51-scoring-port
Open

Maisiqi wants to merge 1 commit into
lexmount:mainfrom
Maisiqi:codex/pr51-scoring-port

Conversation

@Maisiqi

@Maisiqi Maisiqi commented Oct 16, 2026

Copy link
Copy Markdown

Change

Web Fetch answers can fail acceptance after harmless HTML/Markdown conversion, while metadata or a nearby table row can incorrectly supply a missing answer. This change ports Browser Eval PR51's shared scoring fixes into the benchmark package after Browser Eval's evaluator extraction.

  • Match body and structured answers through the shared parsers, preserve table row ownership and native link titles, and distinguish declared exclusions from diagnostic hints.
  • Make page identity an explicit task requirement. Preserve historical controls and add separate executable cases for revised policies.
  • Validate declared answer fields, files, title and final URL against retained native bytes. The scorer uses the package's existing delivered-file acceptance; the execution harness retains evidence persistence and provider adapters.
  • Bind the projection validator into the scoring source identity and exercise the same controls through standalone scoring.

Validation

make check passes all four task-suite definition checks, 1559 tests and 81 subtests. Controls accept a visible title paired with its own row's link and reject borrowed, hidden or mismatched titles; distinguish diagnostic text from an explicit task ban; and reject answer fields or result URLs that differ from retained native bytes.

The four task-suite definitions and frozen input hashes are unchanged. The synthetic scoring-control registry is SHA-256 159adaa11e5272adfe51555fddeba33283c5e5809a29a1d30bfc5ad911bcbe70. No measured outputs or provider calls are included. The existing Browser Eval PR51 will consume this pinned package revision and retain its saved report inputs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant