Repository navigation
Correct evaluator and canonical replay contracts - #3
Merged
Merged
Conversation
# Conflicts: # src/webarena_verified/core/evaluation/data_types/base.py # src/webarena_verified/core/evaluation/data_types/markdown_string.py # tests/api/test_corrected_task_contracts.py # tests/dataset/test_known_contract_corrections.py
Author
独立防作弊审查结论已按“是否降低门槛、修改答案或为特定浏览器放行”重新审查最终 diff,并加入对抗用例。结论:最终 head 审查中确实发现并修复了三类会产生假阳性的共享问题:
另外纠正了 task 760 把 Allentown 机场当作市区起点的问题;这是评测数据错误修正,不是放宽条件。Browser Eval 的旧报告仍绑定旧评测版本,不会把旧轨迹结果冒充为新合同结果。 验证:目标测试 45 passed;核心、数据完整性与 API 2355 passed / 3 skipped;Ruff、ty、diff check 通过。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
目的
修正所有执行器共用的网络评测语义和已确认错误的 canonical 任务合同。修改以 lexmount fork 的 lex-main 为基线,已合并。
整体设计
改动范围
正反控制覆盖尾随错误请求、动态字段变化、数组类型、路线反向和空 HAR。修复适用于所有浏览器;浏览器名称、轨迹和部署地址由 browser-eval 维护。
评测报告与版本
812 题全量轨迹重放报告 · 固定报告版本。
报告包含各方法的官方成绩、任务合同、逐题证据及已声明修正。报告使用自身冻结的评测器版本,具体版本由 browser-eval 的 evaluator-transition.json 和 dataset.json 绑定;本 PR 的最新合并提交与历史测量分开记录。
验证和依赖
3f8e55934ea21fb32599e86242b89988902f3a0f。测试与自动 review 配置包含仓库测试工作流及 review 接入;此 PR 未提供 GitHub check 结果,以上数字来自已有本地验证。#1、#2 已在 lex-main;已关闭 #4 的修复已收敛到本 PR。轨迹执行及报告维护见 browser-eval #32。