Skip to content

Extract Web Fetch benchmark code into pinned LexBench package - #54

Merged
lanyue-llk merged 5 commits into
mainfrom
codex/extract-webfetch-benchmark
Oct 16, 2026
Merged

lanyue-llk merged 5 commits into
mainfrom
codex/extract-webfetch-benchmark

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Self-authored Web Fetch tasks and scoring were mixed with Browser Eval's execution code. Move their ownership to LexBench Web Fetch, and make Browser Eval import its pinned package directly. LexBench can validate inputs and score saved responses or delivered files without Browser Eval.

Remove the local task model, scorer, scenario compiler, task-authoring rules and frozen definitions. Move their scenario tests with them. Browser Eval retains adapters, execution policy, reference/evidence lifecycle, experiment profiles and reporting. Frozen inputs, certificates and measured results retain their original bytes and bindings.

Pin the dependency, lockfile and compact input binding to LexBench main commit 6d8d72637bb96f4938b9a2f7c46d0c7dffd74945, merged through lexbench-webfetch#1. uv sync --frozen fetches the Git source, builds a wheel and installs the Python package. A sibling checkout is not required.

Historical input paths resolve through the installed owner, including equivalent paths and directory links. Keep the existing checkout-input CI guard effective after the manifests move: discover historical manifests from the compact binding, validate their installed hashes, and check the original checkout directories alongside local candidate manifests. Accidental dataset, source and reference copies remain forbidden.

Refresh the matching English/Chinese READMEs, repository About text, architecture and contribution guidance. LexBench includes licensing, citation/provenance, security/contribution guidance, CI and source/wheel packaging; third-party captures and measured evidence remain external.

Validation

  • Final remote CI passes: the complete make ci gate, all four wheel installation variants, real Chromium reference capture and Python 3.12–3.14 base compatibility.
  • Related reference-input, dependency, runtime and portable CLI tests: 119 tests and 6 subtests pass against the installed main dependency.
  • The actual CI entry accepts clean controls and rejects all 15 input-copy scenarios across five historical directories. The new cases failed before the fix. Local candidate checks and mismatched binding-hash rejection remain effective.
  • LexBench's main tree matches the independently tested package: 1,203 tests and 76 subtests pass in source and isolated-wheel environments without Browser Eval. All 894 active scoring controls preserve their prior output except the relocated implementation fingerprint.
  • Independent follow-up review confirms the latest P2 is fixed and no material findings remain on 5dd76e4e. It independently repeats the public CI scenarios and verifies the installed main commit, with 116 related tests and 23 subtests passing. The corresponding review discussion is resolved.

@lanyue-llk lanyue-llk changed the title Separate LexBench Web Fetch from the Browser Eval harness Extract Web Fetch benchmark code into pinned LexBench package Oct 16, 2026
Comment thread harness/web_fetch/reference_inputs.py Outdated
@lanyue-llk

Copy link
Copy Markdown
Contributor Author

本轮独立审查完成,固定 head 为 318dff021b3064a04248b3e7ecc2a40ab86c3d54,base 为 637bd5171b296af6431b7a570f3d8a48ec5c671f。

发现 1 个本 PR 的 P2 回退:历史输入路径别名绕过绑定和禁止修改检查。已用临时独立 checkout 实际写入并读出未绑定的本地数据;相同路径在拆分前会被拒绝修改。另有独立包 CLI 的输入缺口,已在 lexbench-webfetch#1 单独定位。

现场验证范围:

  • 在本会话目录创建独立 checkout,读取准确 base/head 和完整 diff,检查任务模型、评分、文件验收、编译、历史输入映射、源码指纹、安装边界和调用迁移;未使用原开发工作区。
  • 完整 Python 测试:5,391 通过、2,322 子测试通过、82 跳过、0 失败。另跑的输入/评分/CLI/并发边界测试为 123 通过、68 子测试通过。
  • 894 个有效评分场景分别执行拆分前 base 与当前独立包,完整指标和任务判定除源码指纹外全部相同。22 份迁移的冻结定义文件与 Browser Eval base 字节完全一致。
  • 从源码构建 wheel,在源码目录外分别验证基础包、WebMainBench、开源适配器和参考审计四种安装;四组冒烟测试均通过,覆盖评分、证据组装、修订、包内资源、命令行及本地采集等入口。
  • 报告 UI、加载器、Browser Use UI、原生适配器测试、命名检查及两组静态检查通过。

测试结论不覆盖上述跳过用例,也不包含真实供应商调用或完整历史采集材料的重新认证。发现的问题通过额外场景复现,现有测试通过不代表这两个输入边界已经正确。除已定位问题外,本轮没有发现其他需要修改的实质问题。没有提交、推送或合并代码;本轮不跟进后续新 head。

Comment thread harness/web_fetch/reference_inputs.py Outdated
@lanyue-llk

Copy link
Copy Markdown
Contributor Author

第二轮独立复审结论:上轮历史输入问题部分修复,P2 仍未关闭。

已从 GitHub 核对并固定审查 head 4c05e03f1ddbf67b2d385b1f65c33eb80796d964。普通 evaluation/risk-control/../dataset.json 现在正确使用包内绑定,拒绝修改,也不会读取本地 shadow 数据;输入文件符号链接和直接链接到 checkout 的场景通过现有边界测试。

但目录链接与其后面的 .. 组合仍可绕过同一保护。已实际构造 linked-checkout -> real/checkout,比较真实路径与 linked-checkout/../checkout/evaluation/dataset.json:两者解析到同一历史输入,后者却通过禁止修改检查,并由 read_dataset_bytes() 读出未绑定的本地 shadow 数据。根因是 os.path.abspath() 按文本折叠 ..,与文件系统解析目录链接后的实际位置不同。详细复现与修复要求。这是上轮同一 P2 的遗漏场景,不能据此关闭原问题。

本轮现场测试:

  • 路径绑定、参考输入、可移植 CLI、CLI 选项、quickstart、评分和并发场景:150 项通过、68 个子测试通过。
  • 独立复测上述目录链接场景,确认同一历史路径仍可读入 shadow 数据。
  • 核对安装依赖的实际 Git commit、pyproject.toml、锁文件和 revision 绑定,均为 522ede3e54409be5ca40f4496eaf4ceaa3cb411d。
  • 独立包源码及从源归档构建的 wheel 各通过 1,203 项测试、76 个子测试;上轮 CLI 元数据问题已修复。

除这个尚未解决的路径问题,本轮没有发现其他实质问题。未重复第一轮已通过且未受本次修改影响的完整慢检查,未进行真实供应商评测。只使用本会话独立 checkout,没有修改、提交、推送或合并代码。

@lanyue-llk

Copy link
Copy Markdown
Contributor Author

第三轮独立复审完成:上轮遗留的 P2 已修复,本轮没有发现新的实质问题。

已从 GitHub 核对并固定审查 head 7ea42b812438a6e38894070b126b8177cace21ff。重新构造上轮的 linked-checkout -> real/checkout 和 linked-checkout/../checkout/evaluation/dataset.json,现在它与真实 checkout 路径使用相同的包内绑定,拒绝修改,也不会读出本地 shadow 数据。第二轮遗留问题可以关闭。

修复覆盖了共同原因:先解析实际父目录,正确处理目录链接后面的 ..;保留文件叶节点,继续拒绝输入文件符号链接;保留原路径的绑定检查,避免将输入子目录链接到外部后绕过 checkout 已声明的冻结输入。这几项集中在同一个解析入口,各自对应实际场景,未见需要额外抽象或删减的实现。

本轮独立验证:

  • 参考输入、依赖边界、可移植 CLI、CLI 选项、quickstart、评分、并发、参考修订输出及安全场景:178 项测试和 91 个子测试通过。
  • 22 份已绑定定义分别通过真实目录、checkout 链接、多级链接及链接后接 .. 等六条路径检查,共 132 种别名,全部映射到同一包内定义。
  • 五类历史数据集共 30 种路径,均拒绝修改和读取未绑定的本地 shadow 数据;60 种有效或悬空输入文件链接均被拒绝;普通未绑定候选数据集仍可正常读写。
  • 四类输入子目录链接到外部 shadow 数据的场景,均保持冻结绑定和禁止修改保护。

关联 HTTP 测试首次被沙箱禁止监听本机地址;在允许本机测试服务的环境重跑同一组场景后,全部通过。没有修改产品代码,也没有因环境错误跳过这些场景。

LexBench 仍为 522ede3e54409be5ca40f4496eaf4ceaa3cb411d,第二轮对 CLI 元数据修复的结论继续有效。本轮未无故重复第一轮完整慢检查,也未进行真实供应商评测。只使用本会话独立 checkout,没有提交、推送或合并代码。两条原审查问题均已解决,剩余实质问题为 0。

@lanyue-llk lanyue-llk left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 7ea42b812438a6e38894070b126b8177cace21ff against base 637bd5171b296af6431b7a570f3d8a48ec5c671f. One P2 regression remains, detailed inline: relocating the manifests disables the existing checkout-input CI guard for the historical suites. Fix this before merging.

The ownership split is appropriate: LexBench owns task definitions and deterministic acceptance; Browser Eval retains execution, evidence and reporting. Direct imports and one pinned implementation provide a sufficient shared boundary. The guard needs a small extension to its existing discovery logic, with no additional abstraction or compatibility layer.

Independent validation: 22 relocated definitions are byte-identical to base; all 894 active scoring controls preserve their complete metrics and verdicts except the implementation fingerprint. The previous path fix passes 132 definition aliases, 30 dataset aliases and 60 file-link scenarios. Four isolated wheel installation variants, 25 actual-Chrome reference-capture tests, static checks, report UI, native adapters and the fixture CLI pass. Across the full test sweep and two macOS alias cases rerun under their required system temp root, 5,406 tests and 2,322 subtests passed; 82 environment/historical-input cases skipped. No product code was changed, pushed or merged.

Comment thread evaluation/lexbench-webfetch.json
@lanyue-llk

Copy link
Copy Markdown
Contributor Author

本轮独立复审通过,审查 head 为 5dd76e4eea442531292cceb3ce3d4e6551746b9a。迁出 manifest 后遗漏 checkout 输入检查的 P2已修复,可以关闭。本轮没有发现新的实质问题,剩余实质问题为 0。

根因确实是发现范围缩小:本地 manifest 移走后,原来的扫描无法再找到历史冻结目录。当前修改从现有 compact binding 补充五个历史 manifest 的逻辑路径,继续通过安装包校验绑定哈希,并检查原 checkout 目录;本地候选 manifest 的扫描也保留。修改集中在原检查器,没有复制输入规则或另建一套验收流程。检查器实现 · 对应回归场景

现场验证结果:

  • 参考输入、依赖边界、路径别名、参考修订输出及安全场景:116 项测试和 23 个子测试通过。
  • 将修复前 7ea42b81 与当前 head 分别导出到本会话的独立源码快照,直接运行公开 CI 入口 scripts/check_track_naming.py。对五个历史目录分别误放 dataset.json、sources/、list-references/,共 15 种场景:修复前全部退出 0;当前全部拒绝,错误为 duplicated in checkout。两版干净对照和当前最终清理后的对照均退出 0。
  • 在同一当前快照加入普通本地候选 manifest:没有误放输入时通过,加入候选的 sources/ 后拒绝。将绑定中的 manifest 哈希改错,也会报 Installed benchmark definition differs from checkout binding。这两项均通过实际 CI 入口验证。
  • pyproject.toml、uv.lock、compact revision 和实际安装依赖均指向 LexBench main 提交 6d8d72637bb96f4938b9a2f7c46d0c7dffd74945。已从 Git 获取该 main,并确认其 tree 为 0794fa7c967fd01d494c67dd9e5798ed241889c3,与先前独立测试过的 522ede3e 完全一致。正式 main 提交

本轮只在独立 checkout 和临时源码快照中复现,没有访问原开发工作区,没有修改产品代码、提交、推送或合并。未重复先前已通过且未受本次修改影响的完整慢检查,也未进行真实供应商评测。

@lanyue-llk
lanyue-llk merged commit 158f355 into main Oct 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant