Repository navigation
修复 WebArena 轨迹重放评测并发布五方法对比与独立解读 - #23
Conversation
|
Claude finished @lanyue-llk's task in 5m 39s —— View job PR #23 代码审查
Review 结论有条件通过 核心目标（历史分不变、视口缺陷证据、诊断附录、hash 绑定）实现正确。发现两处需在合并前或发布前确认的问题，另有一处术语不一致建议修复。 问题及建议[P1] 新增的"修复日志"导航页（ 建议：将 [P1] PR body 明确声明："The existing report profile points to the COM deployment; the available CN credential lists no sites, so the target must be resolved before publishing." 建议：在合并前记录目标解决方案，或在 [P2] 本 PR 将部分英文标签从 "Qualified input/trajectory" 改为 "Fixed input/trajectory"，但下列位置仍保留旧术语，造成同一报告内术语不一致（中文侧"标准轨迹"无变化，问题仅在英文侧）：
建议：统一到 "Fixed arguments" / "Fixed trajectory and official evaluation passed" / "Fixed action arguments" / "Fixed actions"，或回退为全部保留 "Qualified"，避免同一报告术语混用。 其余已核验：原始 90 题 verdict 不变（ |
|
Claude finished @lanyue-llk's task in 4m 14s —— View job PR Review
FindingsNo critical bugs or security vulnerabilities were found. Several minor issues are noted below. 1. Inconsistent symlink check for fix-log source path — if source.is_symlink() or not source.is_file() or _sha256(source) != binding["source_sha256"]:This only checks whether or path.is_symlink()
or any(parent.is_symlink() for parent in path.parents)An attacker who can create a symlink one level above the source path could redirect the read. The SHA-256 check limits the blast radius but doesn't eliminate the inconsistency. 2. raw = Path(str(ref.get("path") or "")).expanduser()
if any(path.is_symlink() for path in [raw, *raw.parents]) or not raw.is_file():
raise ValueError("missing or unsafe current-run retrieval evidence")Unlike 3. LOCAL_PATH_RE = re.compile(r"/(?:Users|home|private|tmp|var)/[^\s\"']+")
4. observation = next(
item for item in observations if item["browser"] == audit["browser"]
)If no observation matches No issues were found with:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35202694197. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 3m 5s —— View job PR #23 代码审查 / Code Review
先前审查问题的跟进 / Follow-up on prior review findingsThree of the five issues from the previous automated reviews are already fixed in this PR's code:
Two findings remain unaddressed: [Bug] raw = Path(str(ref.get("path") or "")).expanduser()
if any(path.is_symlink() for path in [raw, *raw.parents]) or not raw.is_file():
[Bug] Lines 490, 520, 523, 761, and 1450 still use No other bugs or security issues found. The new publish idempotency logic, |
|
Claude finished @lanyue-llk's task in 3m 29s —— View job PR #23 代码审查
Review 结论有条件通过 核心目标（历史分不变、视口缺陷证据、诊断附录、hash 绑定）实现正确。发现一处需在合并前修复的 P1 缺口，另有一处 P2 术语不一致建议修复。前次审查指出的三个问题（ 问题及建议[P1] 本 PR 新增了
建议：将 [P2] 本 PR 将部分英文标签从
中文侧"标准轨迹"无变化，问题仅在英文侧。 建议：将上述位置统一为 残余风险（不阻塞合并）
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35203503207. Please check Claude's inline feedback and summary comment for details.
PR #23 代码审查
|
|
Claude finished @lanyue-llk's task in 1m 42s —— View job PR #23 Code Review
Two bugs remain unfixed from prior reviews. All other previously flagged issues are confirmed fixed in the current code. [Bug] raw = Path(str(ref.get("path") or "")).expanduser()
if any(path.is_symlink() for path in [raw, *raw.parents]) or not raw.is_file():
[Bug] closure = Path(str(attestation.get("diagnostic_closure") or ""))
checks["diagnostic_closure"] = (
closure.is_file()
and not closure.is_symlink()
and sha256(closure) == source_closure_sha
)This checks Previously flagged issues now confirmed fixed:
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35204322710. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 2m 32s —— View job PR #23 Code Review
All previously flagged issues are confirmed fixed in the current code. No new bugs found. Confirmed fixed:
The new idempotency machinery ( |
|
Claude finished @lanyue-llk's task in 3m 25s —— View job PR #23 代码审查
Review 结论通过 本次 push 修复了前序所有 P1 问题。核心目标（历史分不变、视口缺陷证据、诊断附录、hash 绑定）实现正确。 问题及建议未发现需要阻塞合并的问题。 残余风险（不阻塞合并）
前次审查问题确认均已修复
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35204708938. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 56s —— View job PR #23 Code Review
No bugs or security issues found. The latest commit (
All previously confirmed fixes remain in place. No new code paths introduce path traversal, shell injection, or integrity gaps. |
|
Claude finished @lanyue-llk's task in 4m 47s —— View job PR #23 代码审查
Review 结论通过 本轮 push 修复了前序所有标注问题，核心目标（历史分不变、视口缺陷证据、诊断附录、hash 绑定）实现正确，无需阻塞合并的新缺陷。 问题及建议未发现需要阻塞合并的问题。 已独立核验的全部修复
残余风险（不阻塞合并） CN 部署凭据尚无绑定站点，PR description 已明确声明，发布前需先解决目标绑定；COM 目标（ |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35205358856. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 38s —— View job PR #23 Code Review
No bugs or security issues found. The new commits since the previous review cycle ( All previously confirmed fixes remain in place:
|
PR #23 代码审查
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35208353988. Please check Claude's inline feedback and summary comment for details.
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35559694902. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 54s —— View job PR #23 Code Review
No bugs or security issues found in the one new commit ( Confirmed safe:
All previously confirmed fixes remain in place (last full verification at the
|
|
Claude finished @lanyue-llk's task in 1m 10s —— View job PR #23 代码审查
Review 结论通过 自上一轮审查（ 问题及建议未发现需要阻塞合并的问题。 前次审查 P0/P1 修复确认仍在位
残余风险（不阻塞合并）
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35560875508. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 3m 35s —— View job PR #23 Code Review
No bugs or security issues found in the one new commit ( Confirmed safe in new code:
All previously confirmed fixes remain in place:
|
|
Claude finished @lanyue-llk's task in 6m 11s —— View job PR #23 代码审查
Review 结论不通过（结构性问题） 安全与正确性问题（前序 P0/P1）均已修复，不重复。本轮回应用户内联评论提出的四处问题，独立核验后归纳为以下结构性缺陷： 问题及建议[P1]
建议：将图片从 git 删除；构建时从 artifact-site 的已知资源 URL 下载，或存储在 [P1]
建议：厘清 [P1] 新增专项脚本（本 PR 新增或仅针对当前报告）：
此模式意味着下一轮报告将再产生同等规模的专项脚本。 建议：
[P2]
建议：后续重构时将 关于"动吗"/"从吗"上述 P1 项中唯有 33MB PNG 入库 是本 PR 新引入且可立即操作的问题（ 建议分步处理：
如需直接在此 PR 执行第1项（删除PNG + 修改加载逻辑 + 更新 AGENTS.md），请确认后我可以操作。 |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35691585620. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 45s —— View job PR #23 Code Review
No bugs or security issues found in the one new commit ( Confirmed safe in new code:
All previously confirmed fixes remain in place:
|
|
Claude finished @lanyue-llk's task in 7m 56s —— View job PR #23 代码审查
Review 结论不通过 最新提交 问题及建议[P1]
if any((root / route).exists() for route in ("diagnostics.html", "fixes.html", "data/diagnostic-closure.json")):
raise ValueError("three-method report contains a separate diagnostic appendix")同时从 已发布报告（PR body 链接：https://artifact-site.lexmount.cn/v/coY1pMVPbD6Z8cVLaUi_ZbpSnalfKpxY）是在包含上述代码时构建的，包含 用当前代码重建报告会产生不同的页面集合；若对现有发布包运行当前 建议：在合并前选择其一：
[P1] 此目录由本 PR 新增， 建议：从 git 删除 [P1]
建议：明确将
前序已确认修复（未被本次提交破坏）
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35695618830. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 6m 30s —— View job PR #23 代码审查
Review 结论通过 最新提交 问题及建议未发现需要阻塞合并的问题。 本轮提交核验（
|
| 问题 | 状态 |
|---|---|
| 33 MB PNG 入 git | ✅ 已修复：22 个 PNG 全部从 research/trajectory-replay-illustrations/ 删除，改用 research/trajectory-replay-content.json + copy_bound_assets(~/.browser-eval/report-assets/trajectory-replay/) |
research/trajectory-replay-interpretation.json（861 行）入 git |
✅ 已修复：从 git 删除，同样通过 content manifest 引用 |
trajectories-corrected.json vs trajectories.json 两文件歧义 |
✅ 已修复：README 明确说明两者分工（corrected = 当前权威数据集，original = 资格验证与修正溯源前体） |
revisions/ 快照副本歧义 |
✅ 已修复：README 明确说明为"不可变历史合同输入，非报告输出副本" |
audit_replay_cart_assertion.py 硬编码值 |
✅ 已修复：全部改为 --dataset、--candidate、--product 参数化 |
build_task67_retrieval_contract.py（120 行）专项脚本 |
✅ 已删除 |
revise_original36_for_corrected_reference.py（168 行）专项脚本 |
✅ 已删除 |
run_remote_* 三脚本代码重复 |
✅ 部分修复：新增 harness/trajectory_replay/remote_execution.py 共享模块，run_remote_three_method_cohort.py 和 run_remote_candidate_cohort_diagnostic.py 已迁移；run_remote_mutation_diagnostic.py 因执行模型不同暂未迁移 |
| AGENTS.md 无内容外置规则 | ✅ 已修复：新增"report-specific prose/illustrations 不入库，改用 compact 绑定"和"删除废弃生成器"规则 |
新增模块核验（无安全问题）
harness/reporting/assets.py:copy_bound_assets：路径验证（is_absolute(),.., symlink）+ SHA-256 完整性 + 排他写入（'xb'），设计正确harness/reporting/assets.py:check_external_asset_boundary：CI 守门测试test_repository_has_no_copies_of_bound_external_content已接入，git ls-files research/trajectory-replay-illustrations/确认为空，测试通过harness/trajectory_replay/remote_execution.py：subprocess全部使用列表参数，无shell=True；profile 目录参数安全注入
残余风险（不阻塞合并）
scripts/check_webarena_replay_report.py:499拒绝含diagnostics.html/fixes.html的报告，而当前已发布版本（ver_e40a44044966438192bea4cb1dc96e56,state: "verified"）是用旧代码构建的，包含这些页面。该检查是针对未来新构建的前向门控；已发布报告的验证记录在发布时已完成，历史结论不受新检查影响。如需重新发布须用当前代码重建（无诊断附录页）。run_remote_mutation_diagnostic.py（621 行）仍使用独立 browser launch/execution 实现，未迁移至remote_execution.py；mutation 测试执行模型与标准重放有差异，可后续独立重构。interpretation.py:24-25路径无is_absolute()+ 父目录 symlink 检查（此前审查已标注为可接受，SHA-256 完整性绑定在位）。
|
Claude finished @lanyue-llk's task in 4m 18s —— View job PR #23 Code Review
No bugs or security issues found in Confirmed safe in new code:
Confirmed fixed from prior review cycles:
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35697198559. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 3m 17s —— View job PR #23 代码审查
Review 结论通过 三个新提交均为清理性变更，无新缺陷，前序所有 P0/P1 问题确认不受影响。 本轮提交核验
移除
合并提交，将已合并入 main 的 PR #25（WCXB Moli 对照报告）变更引入本分支。轨迹重放侧仅 AGENTS.md 新增一行，将 WCXB builds 使用
删除 保留的 问题及建议未发现需要阻塞合并的问题。 残余风险（不阻塞合并，已在前次审查记录）
前序 P0/P1 修复确认未被本轮破坏
|
|
Claude finished @lanyue-llk's task in 2m 29s —— View job PR #23 Code Review
No bugs or security issues found in the four new commits ( Confirmed safe in new code:
All previously confirmed fixes remain in place (last full verification at
|
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35709005103. Please check Claude's inline feedback and summary comment for details.
结果与范围
同一批 90 题，保留原三组结果，追加两组 1.1.9 方法；共 450 条结果。
每组开始重置一次站点，按冻结顺序执行，每题使用新浏览器上下文。旧结果不被新方法覆盖。不同日期和负载下的耗时不用于声称源码提速。
根因、合同与证据
报告与独立解读
/v/分享码/results.html下导致的 404，保留分享权限、失败类别与方法筛选；原图下载也使用可访问的资源地址。已有验证