Repository navigation
Complete WebArena-Verified Chromium and Moli trajectory replay - #32
Conversation
|
Claude finished @lanyue-llk's task in 4m 30s —— View job PR Review
Reviewed the changed harness files, scripts, and evaluator fixes. No critical bugs found. Two issues worth addressing: Bug: access_levels = evidence.get("access_levels")
if not isinstance(access_levels, list) or not access_levels:
raise ValueError(...)
# passes silently if every element is None
if any(level is not None and level >= required_access_level for level in access_levels):
raise ValueError("observed_permission_denied found sufficient access")If Suggested fix: require that all elements are non- Note: elif action == "evaluate":
script = call.get("script")
...
value = action_page.evaluate(script)The No other bugs or security issues found. The harness-level guards (symlink checks, SHA-256 evidence pinning, schema validation in |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35865866351. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 3m 55s —— View job PR Review
Bug: This PR adds line 307 to validate that non- if any(level is not None and (not isinstance(level, int) or level < 0) for level in access_levels):
raise ValueError("observed_permission_denied access levels must be non-negative integers or null")But the original bug flagged in the previous review remains: an all- if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")No other bugs or security issues found in the new code. The |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35868669290. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 53s —— View job PR Review
Bug: This was flagged in the previous two review rounds. The partial fix added in the last push (line 307) validates that non- access_levels = evidence.get("access_levels")
if not isinstance(access_levels, list) or not access_levels:
raise ValueError(...)
if any(level is not None and (not isinstance(level, int) or level < 0) for level in access_levels):
raise ValueError(...) # ← added last push, correct
# ← MISSING: no check that at least one element is a concrete int
if any(level is not None and level >= required_access_level for level in access_levels):
raise ValueError("observed_permission_denied found sufficient access")
# [None, None] reaches here and emits PERMISSION_DENIED_ERRORAn if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")No other bugs or security issues found in the new code. The new |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35871430764. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 1s —— View job PR Review
Bug: This bug has been flagged in each of the three prior review rounds. The current code: access_levels = evidence.get("access_levels")
if not isinstance(access_levels, list) or not access_levels:
raise ValueError(...)
if any(level is not None and (not isinstance(level, int) or level < 0) for level in access_levels):
raise ValueError(...) # added in round 2 — validates non-None elements
# ← STILL MISSING: no check that at least one element is a concrete int
if any(level is not None and level >= required_access_level for level in access_levels):
raise ValueError("observed_permission_denied found sufficient access")
# [None, None] reaches here and emits PERMISSION_DENIED_ERRORAn if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")No other bugs or security issues found in the new code. The new |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35876884858. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 3s —— View job PR Review
Bug: The fix has not been applied. Current code: access_levels = evidence.get("access_levels")
if not isinstance(access_levels, list) or not access_levels:
raise ValueError(...)
if any(level is not None and (not isinstance(level, int) or level < 0) for level in access_levels):
raise ValueError(...) # validates non-None elements — correct
# ← MISSING: no check that at least one element is a concrete int
if any(level is not None and level >= required_access_level for level in access_levels):
raise ValueError("observed_permission_denied found sufficient access")
# [None, None] reaches line 315 and emits PERMISSION_DENIED_ERROR
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")No other bugs or security issues found in the new code. The new |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35878081317. Please check Claude's inline feedback and summary comment for details.
PR Review
|
|
Claude finished @lanyue-llk's task in 1m 34s —— View job PR Review
Bug fixed: This bug was flagged in each of the five prior review rounds. The fix has been applied locally (commit The change is two lines: # response_builder.py, after line 308
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")Plus one new parametrized test case in '{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',Please apply this one-liner manually. Without it, No other bugs or security issues found in the new code on this push. |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35887630361. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 29s —— View job PR Review
Bug: This bug has been flagged in every prior review round. The current code is unchanged from round 5: access_levels = evidence.get("access_levels")
if not isinstance(access_levels, list) or not access_levels:
raise ValueError(...)
if any(level is not None and (not isinstance(level, int) or level < 0) for level in access_levels):
raise ValueError(...) # validates non-None elements — correct
# MISSING: no check that at least one element is a concrete int
if any(level is not None and level >= required_access_level for level in access_levels):
raise ValueError("observed_permission_denied found sufficient access")
# [None, None] reaches line 315 and emits PERMISSION_DENIED_ERROR
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")No other bugs or security issues found in the new code. The last two commits only modify |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35887976551. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 29s —— View job PR Review
Bug fixed locally (push blocked): This bug has been flagged in every prior review. The fix was applied in commit The change is two lines — one in # response_builder.py, after line 308 (after the non-negative int check)
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")# test_response_builder.py — new parametrized case in test_response_builder_permission_denied_fails_closed
'{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',Please apply these two lines manually. Without them, No other bugs or security issues found in the new code on this push. |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35888548456. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 39s —— View job PR Review
Bug: This bug has been flagged in every prior review round. The current code (after the non-negative int check at line 308) still has no guard that at least one element is a concrete integer. The fix was applied locally as commit Please apply these two changes manually:
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")
'{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',No other bugs or security issues found in the new code. The two new commits ( |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35891852257. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 31s —— View job PR Review
Bug fixed locally (push blocked): This bug has been flagged in every prior review round. The fix was committed locally as Please apply these two changes manually:
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")
'{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',Without these two lines, No other bugs or security issues found in the new code on this push. |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35895412026. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 10s —— View job PR Review
Bug fixed locally (push still blocked): This bug has been flagged in every prior review round. The fix has been committed locally as Please apply these two changes manually:
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")
'{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',Without these changes, No other bugs or security issues found in the new code on this push. |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35897801287. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 28s —— View job PR Review
Bug fixed locally (push still blocked): This bug has been flagged in every prior review. The fix has been committed locally as Please apply these two changes manually:
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")
'{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',Without these two lines, No other bugs or security issues found in the new code on this push. The new |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35898685995. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 31s —— View job PR Review
Bug fixed locally (push still blocked): This bug has been flagged in every prior review round. The fix has been committed locally as Please apply these two changes manually:
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")
'{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',Without these changes, No other bugs or security issues found in the new code on this push. The |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35899764397. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 40s —— View job PR Review
Bug fixed locally (push still blocked): This bug has been flagged in every prior review. The fix has been committed locally as Please apply these two changes manually:
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")
'{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',Without these changes, No other bugs or security issues found in the new code. The |
There was a problem hiding this comment.
Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/35951130012. Please check Claude's inline feedback and summary comment for details.
|
Claude finished @lanyue-llk's task in 1m 41s —— View job PR Review
Bug fixed locally (push still blocked): This bug has been flagged in every prior review. The fix was committed locally as Please apply these two changes manually:
if not any(isinstance(level, int) for level in access_levels):
raise ValueError("observed_permission_denied requires at least one concrete access level")
'{"scope_complete":true,"allowed":false,"access_levels":[null,null],"required_access_level":40}',Without these changes, No other bugs or security issues found in the new code. The |
e2c376d to
8aac1d3
Compare
|
Claude encountered an error after 2s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 1s —— View job I'll analyze this and get back to you. |
本轮架构审查与修复已把分支真正 rebase 到最新
独立复核后没有采纳 验证：相关 pytest 61 passed；正式发布 gate 通过并返回当前公开报告 URL。按要求不等待 CI。 |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
Review closeout on current headRechecked the historical review comments against current head
Regression coverage exercises header injection, a blocked observation call, and the abnormal hosted-task cleanup path. The two affected test modules pass in full: 89 tests. Ruff and The repeated suggestion to require an integer in |
|
Claude finished @lanyue-llk's task in 8m 4s —— View job 代码审查结论审查范围：上次审查节点 结论：通过，可合并。 已确认修复（历史 P0/P1）
新提交发现[P2]
|
Follow-up review closeoutThe remaining P2 was valid and is fixed in Regression coverage exercises one Chinese participant and two English participants. |
|
Claude finished @lanyue-llk's task in 7m 8s —— View job Review 结论通过 所有历史追踪的 P0/P1 缺陷均已确认修复，新增代码无阻塞合并的问题。 问题及建议[P2] 同一段落内既用
当参与方法数不为 6 时，同一句话会出现"N 种方法……六种方法"的矛盾表述。 建议：将硬编码的"六种方法"替换为 历史 P0/P1 修复确认
新增关键路径验证
|
1 participant
目的
提供 WebArena-Verified 全量轨迹重放：浏览器共用轨迹、Playwright 执行、官方验收和报告管线。812 题全量集与原有 90 题采样集保持独立，复用同一套基础设施。
整体设计
仓库边界
Kitesurf 通过远程 CDP 接入共享 Playwright backend；访问 Rainbond 私网站点时使用受 allowlist 限制的请求桥。整任务 watchdog 覆盖连接、登录、动作、证据收尾、验收和关闭。执行账本保留物理调用，通过后的 case 不为重复确认而重跑。
评测报告
五款浏览器、六种方法的 812 题轨迹重放 · 固定报告版本。
当前报告包含 4,872 条逻辑结果、6,554 次物理执行。所有方法均通过 Playwright 重放，不调用模型。CPU、内存和网络资源没有采集；耗时来自已有执行记录。
报告的 Moli 成绩对应已冻结的历史二进制与修正账本，不代表 10 月 8 日重基后的提交重新执行了 812 题。旧四浏览器报告的分母和执行次数已由当前报告替代。
发布与验证
ver_c0f0693e9c08401bb4d2b88393234e55。