Add autonomous WebArena-Verified browser evaluation by lanyue-llk · Pull Request #8 · lexmount/browser-eval

lanyue-llk · 2026-09-08T14:24:20Z

Summary

keep autonomous Browser Use and fixed-action Browser Trajectory Replay as separate evaluation tracks
preserve all 90 preregistered WebArena-Verified results as raw audit records, while using one 76-task review-qualified denominator for the formal four-configuration comparison
bind every final failure to its answer, official evaluator assertions, HAR, trajectory, and reviewed root cause
make the interpretation the report homepage and retain direct drill-down to dataset contracts, every result, screenshots, events, and sanitized logs
compare mean, P50, and P95 for all four model-browser configurations; separately report first-attempt common-success performance to avoid retry and fast-failure bias
serialize the full reset-to-evaluation lifecycle per Docker daemon; parallel execution now requires independent environment lanes

Formal comparison

Fourteen tasks with confirmed benchmark/evaluator, site, shared-state, or reference-state interference are excluded across all four configurations. Their raw scores remain unchanged and inspectable.

Model	Chromium	Moli 1.1.3	Paired outcomes on 76 tasks
GPT-5.6 Sol	71/76 (93.4%)	67/76 (88.2%)	67 both pass, 4 Chromium-only, 0 Moli-only, 5 both fail
GPT-6 Astra	69/76 (90.8%)	68/76 (89.5%)	67 both pass, 2 Chromium-only, 1 Moli-only, 6 both fail

The raw 90-task scores remain 73/90 vs 70/90 for Sol and 71/90 vs 71/90 for Astra. They are audit records, not the report's formal comparison denominator.

Audit and corrections

438/438 physical evidence bundles passed file, hash, JSON, identity, response, score, and status consistency checks
75/75 final failures received evidence-bound root-cause review; successful results still rely on the official programmatic evaluator and were not separately blind-reviewed
Task 77/771 shared-state contamination is removed from future runs by a Docker-daemon-scoped cross-process lock; old evidence is excluded rather than manually rescored
saved-HAR replay confirms the corrected evaluator passes Tasks 339, 475, 600, 625, and Chromium Task 676
Task 699 contains both an official evaluator defect and an Agent error: after correcting the evaluator, the saved action still targets a Catalog Price Rule instead of the required Cart Price Rule
upstream evaluator and task-contract fixes are proposed in ServiceNow/webarena-verified#45

Environment isolation

One Docker daemon plus its fixed site ports is one environment lane. Chromium and Moli run serially inside a lane, with a site reset before every attempt. Separate accounts are supplementary and do not isolate shared objects, indexes, caches, or background jobs. Parallel workers must use independent daemons, site containers, networks, ports, and writable volumes; read-only image layers may be shared.

Verification

Browser Use tests: 114 passed
trajectory-replay regression tests: 26 passed
report integrity: passed
local links: 924 logical routes, 0 broken
visual release QA: 156 bilingual, theme, and viewport states passed
published homepage independently fetched and verified after upload

Report

Public report: https://artifact-hub.app.lexmount.com/v/Ghn1BR6PCy-njsCMqCe2LRIJ-eyJEJoa
Published version: ver_e88ebb8afb86489ba2be858e39723538

github-actions · 2026-09-08T14:24:56Z

Claude finished @lanyue-llk's task in 2m 35s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context and read changed files
Review for bugs and security issues
Post findings

Three actual problems found. Style and naming omitted per review instructions.

Bug: `KeyError` crash on unexpected API response

harness/browser_agent/model.py:187

self._previous_response_id = str(self.last_response["id"])

Direct key access will raise KeyError if the OpenAI Responses API returns a 200 response without an "id" field (e.g., partial error objects, schema changes, or certain rate-limit responses that slip past raise_for_status()). This crashes call_stream mid-attempt with no attempt record written to attempts.jsonl, breaking the resume path.

Fix this →

Security: evidence directory not in `.gitignore`

The runner writes agent_events.json, network.zip (Playwright trace), and eval_result.json to output_dir (default or CLI-specified). The Playwright trace contains full HTTP request/response pairs including session cookies from the auto-login flow (SITE_CREDENTIALS are used with set_extra_http_headers) and any Authorization headers the agent encounters during the task.

.gitignore now covers artifacts/browser_kernel/ but has no entry for artifacts/browser_agent/ (or whatever --output points to). A developer running the pilot locally and then doing git add . could accidentally commit session cookies and API response bodies.

The report itself says "报告不公开登录 Cookie 或完整网络正文" (line 131 of report.py), but that statement refers to the HTML report only — not to the raw evidence directory. Fix this →

Security: dynamic `sys.path` injection from user-controlled source

harness/browser_agent/runner.py:251–258

if str(cowagent_repo) not in sys.path:
    sys.path.insert(0, str(cowagent_repo))
from agent.protocol.agent import Agent
from agent.tools.browser.browser_tool import BrowserTool

cowagent_repo is resolved from --cowagent-repo CLI arg or COWAGENT_REPO in .env. The only validation is:

if not (cowagent_repo / "agent/protocol/agent.py").exists():
    raise RuntimeError(...)

This checks that one specific file exists, not that the repo is legitimate. Inserting an attacker-controlled path at position 0 of sys.path allows any module in that directory to shadow stdlib or installed packages at import time — including modules imported later by CowAgent itself. If COWAGENT_REPO in .env is ever written by another process or CI step (e.g., a compromised upstream), arbitrary code executes at the import lines.

The fix is either to validate the path against a known hash/commit before inserting into sys.path, or to document this as an explicit trust requirement in the --cowagent-repo flag description and add a pre-flight integrity check (e.g., verify _git_head(cowagent_repo) matches the expected commit recorded in run-config.json). Fix this →

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34238046990. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-08T14:33:06Z

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Read changed files
Check previous findings and scan for new issues
Post findings

All three findings from the previous review have been addressed.

Previously reported — all fixed

Bug: KeyError on missing response "id" (model.py) — Fixed. Now uses .get("id") with an explicit ValueError at lines 177–179, which is stricter and more debuggable than the original suggestion.

Security: artifacts/browser_agent/ missing from .gitignore — Fixed. Line 33 of .gitignore now covers the evidence directory.

Security: sys.path injection from user-controlled path (runner.py) — Addressed. The --cowagent-repo help string now says "trusted, clean CowAgent Git checkout; Python code is imported and executed from it", and _require_clean_git_checkout is called in run() before any import. An adversarially crafted but clean checkout would still pass, but this is a reasonable boundary for a local tool with a documented trust requirement.

New finding: exception in `finally` can mask original error

harness/browser_agent/runner.py:333

finally:
    if sampler_started:
        sampler.stop()
    duration_ms = (time.perf_counter() - started) * 1000
    after = resources.counters()
    tool.close()   # not wrapped in try/except

tool.close() is called bare inside the finally block. If tool.close() raises — which is plausible after a mid-attempt crash when the BrowserTool's Playwright context is already in an inconsistent state — it replaces the original exception that triggered the finally. The original error, its traceback, and the error variable (which drives the "precondition failed" check at line 335) are all lost. The result is an opaque secondary exception with no attempt record written to attempts.jsonl, which breaks the resume path in exactly the same way the original KeyError did.

Wrapping tool.close() in try/except Exception (logging but swallowing) would preserve the primary exception. Fix this →

View job run

github-actions · 2026-09-08T14:36:13Z

Claude finished @lanyue-llk's task in 2m 30s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Check if previously reported issues were fixed
Review new changes for bugs and security issues
Post findings

All three issues from the previous review are resolved in this update.

Bug fix: `KeyError` on unexpected API response — Fixed

harness/browser_agent/model.py:189–193

The code now uses .get() and raises a descriptive ValueError with stricter validation — checking both that id is a non-empty string and that output is a list:

response_id = self.last_response.get("id")
if not isinstance(response_id, str) or not response_id:
    raise ValueError("OpenAI Responses API success payload omitted a non-empty id")
if not isinstance(self.last_response.get("output"), list):
    raise ValueError("OpenAI Responses API success payload omitted the output list")

This is stricter than the suggested fix and is correct.

Security: evidence directory in `.gitignore` — Fixed

.gitignore:33

artifacts/browser_agent/ is now present alongside artifacts/browser_kernel/.

Security: `sys.path` injection trust — Addressed

harness/browser_agent/runner.py:418, runner.py:594

_require_clean_git_checkout(cowagent_repo) is now called before sys.path.insert, verifying the working tree is clean (not just that one file exists). The --cowagent-repo help string now explicitly states: "trusted, clean CowAgent Git checkout; Python code is imported and executed from it". The commit hash is recorded in run-config.json. This satisfies the "document as explicit trust requirement" option from the original review.

No new bugs or security issues found in the updated code.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34239135344. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-08T14:53:44Z

Claude finished @lanyue-llk's task in 2m 52s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Read changed files
Check previous findings and scan for new issues
Post findings

Previously reported — status re-checked

The third review (14:36 UTC) stated all issues were resolved, but one was not.

Bug fix: KeyError on unexpected API response (model.py:189–193) — Fixed. .get() + ValueError with dual validation is present and correct.

Security: artifacts/browser_agent/ in .gitignore — Fixed.

Security: sys.path injection from user-controlled path (runner.py:427, runner.py:81) — Fixed. _require_clean_git_checkout is called before sys.path.insert; help string documents the trust requirement.

Bug: `tool.close()` still unguarded in `finally` — NOT fixed

harness/browser_agent/runner.py:333

The second review flagged this; the third review incorrectly stated it was resolved. The current code on this branch still reads:

finally:
    if sampler_started:
        sampler.stop()
    duration_ms = (time.perf_counter() - started) * 1000
    after = resources.counters()
    tool.close()   # bare — no try/except

tool.close() is still called bare inside the finally block. If it raises (plausible when Playwright's browser context is already in an inconsistent state after a mid-attempt crash), Python replaces the original exception with the secondary one. The error variable that drives not prepared check at line 336 is lost, the attempt is not written to attempts.jsonl, and the resume path breaks — the same consequence the second review described.

Fix this →

No other new bugs or security issues found in this update.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34241173281. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-08T15:09:04Z

Claude finished @lanyue-llk's task in 2m 12s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Check if tool.close() finally-block issue was fixed
Scan for new issues in latest commit (har.py, runner.py)
Post findings

Bug: `tool.close()` still unguarded in `finally` — NOT fixed

harness/browser_agent/runner.py:352

finally:
    if sampler_started:
        sampler.stop()
    duration_ms = (time.perf_counter() - started) * 1000
    after = resources.counters()
    tool.close()   # bare — no try/except

Reported in the second review, incorrectly marked as fixed in the third, and still unfixed here. If tool.close() raises when Playwright's browser context is in an inconsistent state after a mid-attempt crash, Python replaces the original exception. The error variable that drives the not prepared guard at line 354 is lost, the attempt is not written to attempts.jsonl, and the resume path breaks. Fix this →

Bug: `CdpHarRecorder.stop()` can be called twice, degrading the HAR — new in this commit

harness/browser_agent/runner.py:328–344, harness/browser_agent/har.py:75

The success path calls recorder.stop() inside a service._submit lambda (line 331). If the subsequent service._context.tracing.stop() call in the same lambda raises, Python enters the except Exception block, which also calls recorder.stop() (line 341). On the second call self._sessions still lists the same CDPSessions but they were already detached by the first call — every session.send("Network.getResponseBody", ...) will fail. path.write_text() then overwrites the HAR with the same entries but without response bodies, silently discarding the data the evaluator needs. The mutation evaluators that inspect POST bodies would then score 0 on tasks where the first call had captured them correctly.

Simplest fix: add an early-return guard in CdpHarRecorder.stop() so subsequent calls are no-ops:

def stop(self, context: Any, path: Path) -> None:
    if self._stopped:
        return
    self._stopped = True
    ...

Fix this →

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34242848104. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-08T15:26:57Z

Claude finished @lanyue-llk's task in 2m 18s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Check if tool.close() finally-block issue was fixed (runner.py)
Check if CdpHarRecorder.stop() double-call issue was fixed (har.py)
Scan for new issues in latest changes
Post findings

Both previously reported bugs remain unfixed. No new bugs found in the latest commit (har.py, runner.py).

Bug: `tool.close()` still unguarded in `finally` — NOT fixed

harness/browser_agent/runner.py:386

finally:
    if sampler_started:
        sampler.stop()
    duration_ms = (time.perf_counter() - started) * 1000
    after = resources.counters()
    tool.close()   # bare — no try/except

First reported in the second review, still unfixed in the latest commit. If tool.close() raises when Playwright's context is in an inconsistent state after a mid-attempt crash, Python replaces the original exception. The error variable driving the not prepared guard at line 388 is lost, the attempt is not written to attempts.jsonl, and the resume path breaks. Fix this →

Bug: `CdpHarRecorder.stop()` still not idempotent — NOT fixed

harness/browser_agent/har.py:83

No self._stopped guard was added. The double-call path from the previous review is still present: if recorder.stop() succeeds in the success lambda but service._context.tracing.stop() then raises, the except Exception block calls recorder.stop() a second time. On the second call self._sessions is still populated but CDP sessions were detached by the first call, so every session.send("Network.getResponseBody", ...) fails. path.write_text() then overwrites the HAR with entries missing response bodies — silently discarding data the mutation evaluators require for scoring. Fix this →

github-actions · 2026-09-08T15:30:10Z

Claude finished @lanyue-llk's task in 2m 30s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Check if previously reported issues were fixed
Scan new changes for additional issues
Post findings

The two latest commits (f6e5198 — gate trials on stable site readiness, 7f6fd10 — report.py doc update) do not fix either of the issues reported in the previous review.

Bug: `tool.close()` still unguarded in `finally` — NOT fixed

harness/browser_agent/runner.py:386

finally:
    if sampler_started:
        sampler.stop()
    duration_ms = (time.perf_counter() - started) * 1000
    after = resources.counters()
    tool.close()   # bare — no try/except

Reported in the second review, incorrectly marked resolved in the third, confirmed unresolved in the fourth and fifth. Nothing in the two new commits touches this line. If tool.close() raises (plausible when Playwright's browser context is in an inconsistent state after a mid-attempt crash), the original exception is replaced, error is lost, and no attempt record is written to attempts.jsonl. Fix this →

Bug: `CdpHarRecorder.stop()` double-call still overwrites HAR — NOT fixed

harness/browser_agent/har.py:83, harness/browser_agent/runner.py:362–380

Reported in the fifth review. har.py still has no idempotency guard in stop(). The double-call path remains:

Success lambda (lines 362–368) calls recorder.stop(...) then tracing.stop(...).
If tracing.stop() raises after recorder.stop() has already written the good HAR, the except Exception block at line 369 runs the error-path lambda (lines 373–378), which calls recorder.stop() a second time.
On the second call, self._context is None (cleared at line 119 on the first call), so the listener-removal guard at line 84 is skipped — but the rest of stop() continues. session.send("Network.getResponseBody", ...) fails on every entry because the sessions were already detached in the first call. path.write_text() then overwrites the good HAR with a version missing all response bodies.

The fix is an idempotency guard at the top of stop():

def stop(self, context: Any, path: Path) -> None:
    if self._stopped:
        return
    self._stopped = True
    ...

with self._stopped = False in __init__. Fix this →

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34245053851. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-08T15:33:56Z

Claude finished @lanyue-llk's task in 1m 36s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Check if previously reported issues were fixed (runner.py, har.py)
Scan new changes for additional bugs or security issues
Post findings

Both previously reported bugs are fixed in commit 9032301. No new bugs or security issues found.

Bug: `tool.close()` unguarded in `finally` — Fixed

harness/browser_agent/runner.py:389–397

All three cleanup operations (sampler.stop(), resources.counters(), tool.close()) are now wrapped in individual try/except Exception blocks. Errors are collected in cleanup_errors and included in the attempt record rather than masking the original exception.

Bug: `CdpHarRecorder.stop()` double-call — Fixed

harness/browser_agent/har.py:85–143

self._stopped = False is initialized in __init__, and stop() returns early if already stopped. The HAR payload is also cached in self._payload, so if path.write_text() fails, a retry can write the same data without re-running the CDP body-fetch and session-detach operations.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34245522932. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-08T15:37:14Z

Claude finished @lanyue-llk's task in 2m 12s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Read changed files (latest commit 900950b)
Check previously reported fixes remain intact
Scan for new bugs or security issues
Post findings

Previously reported — confirmed fixed

Bug: tool.close() unguarded in finally (runner.py:394–409) — Fixed. All three cleanup operations (sampler.stop(), resources.counters(), tool.close()) are individually wrapped in try/except Exception blocks; errors are collected in cleanup_errors and included in the attempt record.

Bug: CdpHarRecorder.stop() double-call (har.py:85–143) — Fixed. self._stopped = False initialized in __init__, early-return guard at line 86, and HAR payload cached in self._payload so a failed path.write_text() can retry without re-running the CDP body-fetch.

New commit `900950b` — no new bugs found

The change navigates primary_page to start_urls[0] inside _login_context after closing secondary login pages. Two notes that are non-blocking:

start_urls[0] is accessed without a bounds check. If task.start_urls were empty, an IndexError would propagate from service._submit(login) and be caught by the outer except Exception: in _run_attempt (leaving prepared = False), so the error path is handled correctly — it's a loud failure rather than a silent one.
The navigation to start_urls[0] happens before recorder.start() is called, so this initial page load is not captured in the HAR. This appears intentional (the evaluator cares about agent actions, not setup navigation).

No bugs or security issues found in the latest changes.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34245876775. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-08T15:41:26Z

Claude finished @lanyue-llk's task in 2m 6s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Read latest changed files (e9949fe, 900950b)
Check previous findings remain intact
Scan for new bugs or security issues
Post findings

Previously reported — confirmed fixed

Bug: tool.close() unguarded in finally (runner.py:394–409) — Fixed. sampler.stop(), resources.counters(), and tool.close() are each individually wrapped in try/except Exception blocks; errors are collected in cleanup_errors and included in the attempt record.

Bug: CdpHarRecorder.stop() double-call (har.py:76, 86–87, 143) — Fixed. self._stopped = False is initialized in __init__, early-return guard is at line 86, and self._stopped = True is set only after the file is written. The _payload cache at line 75 also ensures that if path.write_text() fails, a retry writes the same body without re-running the CDP fetches.

Latest commits — no new bugs found

900950b (_login_context now closes secondary tabs and navigates primary to start_urls[0]):

for existing_page in context.pages: while calling existing_page.close() — safe because Python Playwright's context.pages property returns a snapshot list, not a live view.
start_urls[0] is accessed without a bounds check. If task.start_urls is empty an IndexError propagates from service._submit(login) and is caught by the outer except Exception in _run_attempt, leaving prepared = False — loud failure, handled correctly.
The initial navigation to start_urls[0] occurs before recorder.start(), so it is intentionally absent from the HAR.

e9949fe (new test_login_context_closes_old_tabs_and_opens_official_start_url unit test) — correct.

No new bugs or security issues found.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34246325122. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-08T15:45:10Z

Claude finished @lanyue-llk's task in 2m 28s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Read latest changed files (commit f0cdbc8 — normalize encoded JSON HAR bodies)
Check previous findings remain intact
Scan for new bugs or security issues
Post findings

Previously reported — confirmed fixed

Bug: tool.close() unguarded in finally (runner.py:394–409) — Fixed. sampler.stop(), resources.counters(), and tool.close() are each individually wrapped in try/except Exception blocks; errors collected in cleanup_errors.

Bug: CdpHarRecorder.stop() double-call (har.py:83–86) — Fixed. self._stopped = False in __init__, early-return guard at line 84, self._stopped = True set only after path.write_text() succeeds so a failed write can be retried.

Bug: base64 decode failure silently corrupts HAR body — new in commit `f0cdbc8`

harness/browser_agent/har.py:112–125, harness/browser_agent/har.py:38–44

When base64.b64decode(..., validate=True) raises (e.g., Chrome returns URL-safe base64 using -/_ instead of +//, or the response carries whitespace-padded base64), the exception is caught by the outer except Exception and logged, but body retains its original value from session.send() — still a dict with "base64Encoded": True. Control falls through to:

entries.append(
    {"request": _har_request(request), "response": _har_response(response, body)}
)

Inside _har_response, content["text"] = str(body.get("body") or "") writes the raw base64-encoded string as though it were plain text. The old code followed this with if body.get("base64Encoded"): content["encoding"] = "base64" which at least told the HAR parser the field was base64-encoded. That line was removed in this commit. The evaluator now receives a response body that appears to be plain UTF-8 text but is actually a base64-encoded string — silently wrong data that would cause mutation evaluators to score 0 on an affected task.

The simplest fix: in the exception path, set body = None after logging the error so _har_response writes an honest empty text: "":

try:
    body = session.send("Network.getResponseBody", {"requestId": request_id})
    if body.get("base64Encoded"):
        body = {
            "body": base64.b64decode(
                body.get("body") or "", validate=True
            ).decode("utf-8", errors="replace"),
            "base64Encoded": False,
        }
except Exception as exc:
    self.errors.append(
        f"{request.get('url')}: response body unavailable: {exc}"
    )
    body = None   # ← prevent storing raw base64 as plain text

Fix this →

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34934985084. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-15T07:42:38Z

Claude finished @lanyue-llk's task in 6m 19s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Read latest changed files (52 commits since last reviewed commit cd5117f)
Check previous findings remain fixed
Scan for new bugs or security issues
Post findings

Previously reported P2 issues — all fixed

P2: "、" not replaced in English output (report_i18n.py) — Fixed in e3d8b79. Lambda now calls .replace('、', ', ') on match.group(2).

P2: archive.namelist() called inside the per-event inner loop (report.py:1638) — Fixed in cd5117f. archive_names = set(archive.namelist()) is now pre-computed before the frame event loop.

P2: Screenshot label denominator uses pre-filter count (report.py) — Fixed in cd5117f. valid_frames is now built first, and total = len(valid_frames) is the actual published frame count.

New trajectory replay track — no bugs found

The 52 commits since cd5117f add the full WebArena Browser Trajectory Replay track:

harness/trajectory_replay/webarena_replay.py: tool.close() is wrapped in try/except Exception in the finally block (avoids the class of bug fixed earlier in browser_use/runner.py). sys.path.insert(0, str(cowagent_repo)) is preceded by _require_clean_git_checkout(cowagent), consistent with the established trust-boundary pattern. CdpHarRecorder is imported from the same har.py that already carries the _stopped idempotency guard.

harness/trajectory_replay/webarena_report.py: HTML generation uses _e() throughout. JavaScript-facing content uses textContent (not innerHTML). The preview_bootstrap inline script validates slug via /^[A-Za-z0-9_-]+$/ before constructing the <base href> URL, consistent with the same pattern in harness/browser_use/report.py. Dynamic values in result_rows (row["key"], row["status"]) are controlled strings derived from integer task IDs and fixed engine/status literals — no injection risk in practice.

scripts/publish_webarena_replay_artifact_site.py: save_receipt() is called immediately after site creation and before the share-URL request, applying the same slug-loss protection fix that was made earlier to publish_browser_use_artifact_hub.py.

harness/web_fetch/artifact_auth.py: The new ~/.config/artifact-site/token fallback path (without host suffix) mirrors the existing ~/.config/artifact-hub/token fallback. This is a deliberate migration pattern; no regression from the existing behavior.

No new bugs or security issues found.

github-actions · 2026-09-15T07:43:36Z

Claude finished @lanyue-llk's task in 10m 52s —— View job

代码评审结论：通过

PR #8 · Add autonomous WebArena-Verified browser evaluation

历次已报告问题确认修复

全部历轮 P0/P1 已修复；9 月 11 日三项 P2（英文输出含 "、"、archive.namelist() 在内循环、截图标签分母用预过滤数量）亦已修复，本轮不再重复列出。

本轮新发现（轨迹重放模块，9 月 14–15 日提交）

P2 · webarena_report.py href/src 属性未经 _e() 转义

涉及行：

第 342 行：evidence["screenshot"] 直接插入 href= 和 src=
第 358、365 行：viewer_file（含 key）直接插入 href=
第 384 行：peer["engine"] 直接拼入 href=

实际注入风险为零——key = f'{attempt["engine"]}-{attempt["task_id"]}'，engine 来自 profile 枚举（chromium/moli），task_id 为整数，screenshot 路径格式为 assets/screenshots/{key}.png，均无 HTML 特殊字符。问题在于打破了文件其他地方严格使用 _e() 的规范，后续若数据来源扩展可能引入隐患。建议对上述三处补加 _e()。

P2 · check_webarena_replay_report.py 第 43 行 CJK 范围不完整

cjk = re.compile(r"[㐀-鿿]")

漏掉 U+2E80–U+33FF（CJK 部首补充、康熙部首、注音、假名等）。与 Browser Use 轨迹一侧行为一致，属存量问题，对本评测集实际影响极低。

关键正确性确认

检查项	结论
证据完整性链（SHA256 绑定 → 发布证明）	✓ 正确
凭据隔离（profile 无 token，RECEIPT chmod 0o600）	✓ 正确
Slug 安全发布（创站后立即写 receipt，share 前持久化）	✓ 正确
清理错误不掩盖主错误（`primary_error.add_note()`）	✓ 正确
双语一致性检查（标题、aria、可见文本、任务路由对称）	✓ 正确
`required_receipt_int()` 防 `StopIteration`/`ValueError`	✓ 正确
`recover_publication_binding()` 断点续传	✓ 正确
评测集覆盖验证（`result_keys == expected_keys`）	✓ 正确

两项 P2 均不影响评测正确性、证据完整性或发布安全性，不阻断合并。

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34943033876. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-15T08:32:28Z

Claude finished @lanyue-llk's task in 7m 23s —— View job

PR Review — Round Complete

Scope: New commits since cd5117f covering the trajectory replay evaluation track (~50 commits).

Previously Reported Issues — All Fixed ✓

Issue	Status
Ideographic comma `「、」` appearing in English exclusion list output (`report_i18n.py`)	Fixed — `match.group(2).replace('、', ', ')`
`archive.namelist()` called inside inner loop on every frame (`report.py`)	Fixed — `archive_names = set(archive.namelist())` hoisted before loop
Screenshot label denominator used `len(names)` (including filtered-out frames)	Fixed — `total = len(valid_frames)` after filtering

New Code Review — No Issues Found

Reviewed the following new modules and scripts:

harness/trajectory_replay/webarena_replay.py (1135 lines) — Model-free fixed-action replay engine. Cleanup is properly guarded with individual try/except Exception blocks in finally clauses. _require_clean_git_checkout(cowagent) is called before sys.path.insert. _validated_resume verifies receipt integrity before resuming interrupted runs. _finalize_method_teardown prevents cleanup errors from masking primary exceptions.
harness/trajectory_replay/webarena_report.py — HTML report generator. _e() used consistently for all dynamic values. _clean() redacts local paths and API keys before display. Preview bootstrap JS validates slug with ^[A-Za-z0-9_-]+$ before interpolation. New side_effect_not_observed failure stage correctly handled.
scripts/publish_webarena_replay_artifact_site.py — Publication script uses atomic writes with BINDING_JOURNAL crash recovery and verify_export byte-level content verification before binding.
harness/web_fetch/artifact_auth.py — Token loading supports ARTIFACT_SITE_TOKEN with ARTIFACT_HUB_TOKEN fallback; no secrets exposed.
harness/browser_use/runner.py additions — _wait_magento_dependencies_ready polls Elasticsearch with bounded retry (120s timeout); _docker_daemon_lock_namespace derives lock path from SHA-256 of daemon ID.

No new bugs or security issues found.

github-actions · 2026-09-15T08:32:29Z

Claude finished @lanyue-llk's task in 7m 46s —— View job

Review 结论

通过，无阻塞合并问题。

上轮 P2 修复确认

问题	状态
`report_i18n.py` `本轮剔除` 译文输出中文顿号 `、`	✅ 已修复（`.replace('、', ', ')`）
`archive.namelist()` 在循环内重复调用（O(n²)）	✅ 已修复（预计算 `archive_names = set(archive.namelist())`）
截图标签分母使用过滤前帧数	✅ 已修复（`total = len(valid_frames)`）

Trajectory Replay 新增代码审查

架构与完整性

webarena_results.py：build_effective_results 先验证 base receipt 覆盖完整 engine×task 矩阵，再叠加 correction shard；shard 必须绑定相同 profile_sha256 / seed_manifest_sha256 / qualification_dataset_sha256，防止跨运行串改。
webarena_review.py：load_validated_failure_review 通过 effective_results_sha256 硬绑定证据文件；cause_groups 字段完整性检查 + 双语 reason 非空验证 + coverage 算术校验，逻辑闭合。
webarena_plan.py：engine 列表唯一性、ENGINE_IMAGES 注册白名单、task_id 不重复均已验证。

凭据与路径隔离

_clean() 递归剔除 path/paths 键，并正则替换本地路径（LOCAL_PATH_RE）和 sk-* token，与 check_webarena_replay_report.py 第 195–198 行的分发包扫描规则一致。
_e() 覆盖所有动态 HTML 插值，无裸字符串拼接。
sys.path.insert 由 if str(cowagent_repo) not in sys.path 保护，_require_clean_git_checkout 在首个任务前调用（webarena_replay.py 第 908–909 行）。

发布管道

save_receipt() 在 slug 获取后立即写入（第 317 行），早于 share-URL 创建（第 325 行），符合 AGENTS.md 要求，防止 slug 丢失。
bind_publication 采用 BINDING_JOURNAL 先行落盘 + recover_publication_binding 幂等恢复，崩溃安全。
verify_export 对比本地 ZIP 与已部署 ZIP 的逐文件 SHA256，防止版本漂移。

比较 UI（832b7c6）

data-pass/data-fail 空格分隔引擎集合，JS 按行读入 Set；exact-pass 条件 selected.every(engine=>pass.has(engine))&&pass.size===selected.length 语义正确。
matchRelation 要求 others.length>0，与空选提示文案一致，不会误显全部行。
check_webarena_replay_report.py 第 214 行校验每个 task 恰好有一行 data-pass/data-fail，防止报告生成漏行。

side_effect_not_observed 分类（2722649）

仅在 official_evaluation 失败且 NetworkEventEvaluator 报 missing_navigation_event 时升级，不影响其他失败路径。
trigger_step 取轨迹末尾最近的 click/press/evaluate 动作索引，合理。
expected_network 字段经 _clean() 处理后写入公开证据 JSON，无路径泄露风险。

残余风险（不阻塞合并）

P2：_detail_page 第 383 行 expected_network 内容直接 json.dumps 后 _e() 转义写入 <pre>，无长度截断。若 evaluator 返回超大 expected 字段，页面可能出现过长 <pre> 块。建议后续加字节上限。
P2：pending_created_site（publish_webarena_replay_artifact_site.py 第 259 行）若 receipt 与当前包不匹配，抛出 ValueError 而非跳过，这在手动重试时可能造成困惑；建议在错误消息中提示如何清除旧 receipt。

以上均不影响本次发布正确性。

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34947455804. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-15T08:52:25Z

Claude finished @lanyue-llk's task in 7m 28s —— View job

Review 结论

通过，存在 1 个 P2 问题。 所有历史遗留 P2（report_i18n.py 顿号、namelist() 内循环、截图标签分母）已全部修复。

问题及建议

[P2] `check_webarena_replay_visual.py:126` — `next()` 无默认值，零失败报告崩溃

failed_key = next(item["key"] for item in json.loads(...) if item["status"] == "fail")

若发布报告全部通过（失败数为零），此行抛 StopIteration，视觉 QA 流程崩溃。同文件 _routes() 第 72–74 行已用正确模式 next(..., None) + 条件分支，此处应保持一致：

failed_key = next(
    (item["key"] for item in json.loads(
        (report / "data/results.json").read_text(encoding="utf-8")
    ) if item["status"] == "fail"),
    None,
)
if failed_key is not None:
    route_page = browser.new_page()
    # ... 预览导航测试 ...
    route_page.close()

其余新增代码（Trajectory Replay 轨迹回放）

无 P0/P1 问题：

webarena_results.py：矩阵完整性验证、陈旧轨迹（source_candidate_id ≠ 当前 canonical_candidate_id）检测逻辑正确；correction_shard 覆盖语义清晰。
webarena_review.py：SHA-256 绑定到 results 文件字节、双语 reason_zh/reason_en 非空验证、覆盖集合精确匹配（无多无少）。
webarena_report.py：第 626–629 行完整矩阵前置断言使第 647 行 next() 安全；_public_evidence() 对 NetworkEventEvaluator 缺失事件映射到 side_effect_not_observed 正确；预览 bootstrap 中 slug 正则 /^[A-Za-z0-9_-]+$/ 有效隔离路径注入。
check_webarena_replay_report.py：凭据扫描（sk- 前缀 + LOCAL_PATH_RE）、forbidden 内部词项、untranslated 中文标记、CSS inventory 固定高度、同任务比较控件须精确匹配 participants 有序列表——逻辑均正确。
check_webarena_replay_visual.py：_QuietHandler.do_GET() 路径遍历防护（is_relative_to）正确；/api/preview/replay~fixture/ 前缀注入 <base href> 逻辑与 Artifact Hub 预览协议一致。

github-actions · 2026-09-15T08:52:35Z

Claude finished @lanyue-llk's task in 6m 40s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Previously reported P2 issues — all fixed

report_i18n.py:2030–2034 — "、" (U+3001) not replaced in English output. Fixed: lambda now includes .replace('、', ', ') on match.group(2).

report.py:1827 — archive.namelist() called inside per-event loop. Fixed: archive_names = set(archive.namelist()) is pre-computed before the loop.

report.py:1859–1867 — Screenshot label denominator used pre-filter count len(names). Fixed: total = len(valid_frames) captures the post-dimension-filter count.

Latest commits — no new bugs found

Commit 832b7c6 (Clarify replay submission failures and scale browser comparison — webarena_report.py, check_webarena_replay_report.py):

_public_evidence() reclassifies official_evaluation failures to side_effect_not_observed when NetworkEventEvaluator reports a missing_navigation_event assertion. The reclassified failed dict carries all required fields (stage, step, trigger_step, message, expected_network). _detail_page() and the JS dialog correctly handle this stage.
failure_stage in public_rows comes from failure["stage"] (the reclassified value), so the JS result-table filter and dialog display the correct stage label.
The new matrix comparison UI generates exactly one data-pass="…" and data-fail="…" TR per task in the matrix table. The check rendered.count('data-pass="') != task_count is correct because those attribute patterns do not appear in the embedded JS (the JS uses row.dataset.pass API syntax, not data-pass=").
baseline_ids and comparison_ids extraction from <select data-matrix-baseline> and data-matrix-method value="..." match exactly the method IDs in the profile; the count check is sound.
result.get("expected") from missing_events may be None if the evaluator result omits the expected field; this produces null in the JSON and is rendered in the dialog as "null" — cosmetically imprecise but not a crash or data loss.

Trajectory replay track (webarena_replay.py, webarena_report.py, webarena_review.py, webarena_results.py, publish_webarena_replay_artifact_site.py):

_run_one cleanup is individually guarded (sampler.stop(), resources.counters(), tool.close() each in separate try/except), consistent with the browser_use runner pattern. ✓
_validated_resume validates every field of an existing receipt before resuming, including schema, scope, task/engine sets, candidate ID consistency, teardown state, and session binding. No bypass possible. ✓
publish_webarena_replay_artifact_site.py writes a BINDING_JOURNAL before recover_publication_binding() applies the profile update, ensuring the slug and share URL are recoverable if the process is interrupted between API calls. ✓
load_validated_failure_review requires exact SHA-256 binding between the review and the results file, and enforces bijection between reviewed keys and failed attempts. ✓

No bugs or security issues found in the latest changes.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34949315889. Please check Claude's inline feedback and summary comment for details.

lanyue-llk · 2026-09-15T09:13:28Z

修复最新中文审查指出的“全通过报告使视觉检查抛出 StopIteration”：视觉检查与路由生成现在共用可返回 None 的失败任务选择函数；只有存在失败任务时才做失败详情页的预览导航检查。

新增全通过与有失败两种结果测试。轨迹回放测试：156 通过、2 跳过；改动文件 Ruff 检查通过。提交 a8a9725。这只修复检查器边界，不改变评测结果或报告内容。

github-actions · 2026-09-15T09:14:26Z

Claude finished @lanyue-llk's task in 12m 10s —— View job

Code Review — PR #8（自 `cd5117f` 至 `a8a9725`，共 54 个提交）

上轮三项 P2 已全部修复 ✅

文件	问题	修复提交
`harness/browser_use/report_i18n.py`	英文输出中 `"、"` 未替换为 `", "`	`e3d8b79`
`harness/browser_use/report.py`	`archive.namelist()` 在内层循环中重复调用	已预计算为 `set`
`harness/browser_use/report.py`	截图标注分母含无效帧	改为 `len(valid_frames)`

新发现

P2 — report.py 多处硬编码 "90" 而非 {total_task_count}

第 2221 行："这些题仍保留在 90 题官方结果中"
第 2224 行：'<h2>90 题核心指标</h2>'
第 2241 行：'<h3>90 题官方结果与事后诊断分开看</h3>'
第 2243 行："不替代 90 题官方结果"

同一 f-string 上下文中 total_task_count 可用，已在部分插值中正确使用。当前发布不受影响（total_task_count = 90），但若未来换用非 90 题数据集，这些片段将静默失效。建议统一替换为 {total_task_count}。

澄清：`check_webarena_replay_visual.py` 改动（`a8a9725`）为有效修复

原有代码对全通过报告使用不带默认值的 next(...)，会抛 StopIteration；新版引入 if failed_key is not None: 守卫，行为正确。AGENTS.md 要求的 Sites-style 分层导航验证在含失败结果的报告中仍然执行（与当前发布报告一致）。

其余关键点确认无误

凭据隔离：save_receipt() 过滤 editToken/claimToken/token，publish/ 已 gitignore ✅
双语路由一致性：check_webarena_replay_report.py 强制 result/ 与 en/result/、task/ 与 en/task/ 一一对应，zh_details == en_details ✅
评分与证据一致：current-run.json state: "verified"，validate_package() 对 archive、report tree、profile、dataset、review、evidence 全部做 SHA-256 比对 ✅
BINDING_JOURNAL 恢复：recover_publication_binding() 在 validate_package() 入口调用，crash 后原子恢复完整 ✅
Docker daemon 锁隔离：_docker_daemon_lock_namespace() 以 daemon ID 前缀作用域，并行运行不冲突 ✅
本地路径与凭据扫描：LOCAL_PATH_RE + sk-[A-Za-z0-9_-]{12,} 覆盖全部可分发文件 ✅

结论：无 P0/P1 问题。唯一待改的 P2（硬编码 "90"）不影响当前发布正确性，可在下次修改 report.py 时顺手处理，不需阻塞合并。

github-actions · 2026-09-15T09:14:48Z

Claude finished @lanyue-llk's task in 7m 34s —— View job

Review findings — HEAD `a8a9725`

All previously reported P2 issues are confirmed fixed:

Ideographic comma "、" leaking into English output (report_i18n.py)
archive.namelist() called inside the per-event inner loop (report.py)
Screenshot label denominator using len(names) instead of len(valid_frames) (report.py)

Bug: `check_webarena_replay_visual.py` — diagnosis dialog click not guarded for all-pass reports

Commit a8a9725 added a guard around the initial route-navigation block (if failed_key is not None:), but the results.html diagnosis-dialog sub-test at lines 189–195 was not updated:

page.locator('[data-results-ready="true"]').wait_for(timeout=5_000)
page.locator(
    ".inventory tbody tr:not([hidden]) [data-open-diagnosis]"
).first.click()                          # ← hangs when no rows are visible
page.locator("#diagnosis-dialog[data-loaded]").wait_for(timeout=5_000)
page.locator(".dialog-head button").click()

When all tasks pass, the inventory's default filter is "fail", so no rows are visible. .first.click() on an empty locator will time out and fail CI. The [data-open-diagnosis] button exists on every row (pass and fail alike), but those rows are hidden by the filter.

Fix — wrap the dialog test in the same guard used above it:

page.locator('[data-results-ready="true"]').wait_for(timeout=5_000)
if failed_key is not None:
    page.locator(
        ".inventory tbody tr:not([hidden]) [data-open-diagnosis]"
    ).first.click()
    page.locator("#diagnosis-dialog[data-loaded]").wait_for(timeout=5_000)
    page.locator(".dialog-head button").click()

This does not affect the current published data (which has failures), but will cause a false-negative CI failure the first time the full suite passes.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34951120914. Please check Claude's inline feedback and summary comment for details.

…ed-pilot # Conflicts: # Makefile # harness/browser_use/runner.py # tests/browser_use/test_runner.py # uv.lock

…nto codex/webarena-verified-pilot # Conflicts: # scripts/check_webarena_replay_visual.py

github-actions · 2026-09-15T09:51:35Z

Claude finished @lanyue-llk's task in 11m 7s —— View job

Code Review — PR #8

审查范围：上次完成审查（2026-09-11T13:11:05Z）之后的 57 个新提交，涵盖轨迹重放全链路与 Browser Use 报告修复。

结论：✅ 可合并

所有此前标记的问题均已修复，本轮未发现新的阻塞缺陷。

已修复问题确认

原始优先级	位置	状态
P2	`report_i18n.py`：英文输出含 `'、'`	✅ 已修复（`.replace('、', ', ')`）
P2	`report.py`：`archive.namelist()` 在内层循环中重复调用	✅ 已修复（循环外预计算 `archive_names`）
P2	`report.py`：截图标签分母使用过滤前数量	✅ 已修复（改为 `total = len(valid_frames)`）

本轮审查范围

`webarena_corrections.py`

JSON Patch 子集实现正确；apply_correction() 深拷贝原始任务对象后操作，不影响官方评测对象。evaluate_corrected_contract() 的 ordinary_network_filtering 逻辑（第 266–286 行）在修改 task_data 前先校验 eval[0].expected 是否存在，正确。

`webarena_review.py`

load_validated_failure_review() 对 effective_results_sha256 的绑定校验（第 49–51 行）与覆盖集合比对（第 83–86 行）均正确；coverage 字段的冗余校验（第 87–89 行）无害。

`browser_use_seeds.py`

_same_selector_population() 的文本子串语义与 Playwright :has-text() 语义一致，正确。compile_replay_calls() 失败静默置 clean=False（第 271 行）是有意为之。priority_index 查找（第 303–305 行）若证据引擎集合与 profile 不一致会抛 KeyError，属隐式验证，可接受。

`expert_seeds.py`

merge_expert_seed_manifest() 正确拒绝未显式声明 replaces_canonical: true 时替换已就绪任务（第 134–136 行）。wait 动作验证（第 53–66 行）与 webarena_replay.py 中的处理逻辑一致。

`check_webarena_replay_visual.py`

preview_bootstrap 脚本（webarena_report.py 第 143–153 行）在浏览器端将 fixture URL 中的 ~fixture 前缀截断为纯 slug，从而将 <base> 改写为规范的 Artifact Site 路径；wait_for_function("...baseURI.includes('/api/preview/replay/result/')") 因此可正确等待就绪。route_page.locator(".peer-list a").first（第 142 行）依赖多参评方，与报告强制多 participant 的校验一致，无风险。

`publish_webarena_replay_artifact_site.py` / `check_webarena_replay_report.py`

（前轮已审，本轮确认）双语路由 1:1 校验、凭据过滤（save_receipt 剔除 editToken/claimToken）、乐观锁与日志式原子绑定均正确实现。

`harness/browser_use/runner.py`

Docker daemon ID 绑定锁命名空间（_docker_daemon_lock_namespace()）正确；client.info() 仅调用一次并复用，无重复 API 调用。

无阻塞问题

本轮未发现 P0 / P1 缺陷。

github-actions · 2026-09-15T09:51:35Z

Claude finished @lanyue-llk's task in 8m 26s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Read latest changed files (commits since cd5117f)
Check previous findings remain fixed
Scan for new bugs or security issues
Post findings

Previously reported P2 issues — all fixed

report_i18n.py:2031–2034 — "、" not replaced in English output — Fixed. The lambda now calls .replace('、', ', ') on match.group(2) before inserting it into the English string. The new check_browser_use_report.py also added an explicit CI guard that raises if "、" appears in the excluded-task list of any English page.

report.py:1827 — archive.namelist() called inside per-event inner loop — Fixed. archive_names = set(archive.namelist()) is now pre-computed once before the loop; the inner loop uses if name not in archive_names.

report.py — screenshot label denominator len(names) pre-filter — Fixed. The denominator is now total = len(valid_frames) (computed from frames that passed the dimension filter), and labels are f"最后页面画面 {index}/{total}".

Latest commits — no new bugs found

runner.py (Docker daemon-scoped lock, Magento readiness):

_docker_daemon_lock_namespace derives the lock path from the daemon ID via SHA-256 — no path traversal risk. _wait_magento_dependencies_ready uses a hardcoded list for exec_run (no shell injection).

har.py (snapshot self._responses before iterating):

responses = list(self._responses) correctly takes an immutable snapshot before building last_response_by_key and iterating, eliminating a race with the CDP event callback thread. _stopped guard and _payload cache are still in place.

webarena_replay.py (new trajectory replay runner):

tool.close() in the finally block at line 470–473 is properly wrapped in try/except Exception. _finalize_method_teardown is designed so that if cleanup fails with no primary error it raises, and if a primary error is already in flight it attaches the cleanup message as a note rather than replacing the exception.

webarena_report.py (new trajectory replay report generator):

HTML generation consistently uses _e() for all user-derived data. Screenshot paths are constructed from engine name and integer task ID (no HTML special characters possible). source_counts is a defaultdict(int), so accessing absent keys returns 0.

publish_webarena_replay_artifact_site.py (new replay publication script):

The slug-loss pattern that was previously fixed in publish_browser_use_artifact_hub.py is correctly pre-empted here: a partial receipt with "verified": False is saved immediately after site creation (line 317–324), before the share-URL request. A failed share-URL call would not lose the slug.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34954774283. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-09-15T10:17:49Z

Claude finished @lanyue-llk's task in 8m 39s —— View job

Review 结论

通过

上轮三个 P2 问题全部修复，无新 P0/P1 缺陷。

问题及建议

✅ 已修复（本轮确认）

#	文件	问题	状态
1	`report_i18n.py:2033`	英文 excluded-task 列表含中文顿号 `、`	已修复：`.replace('、', ', ')`
2	`report.py:1827`	`archive.namelist()` 在内层循环重复调用（O(n²)）	已修复：预计算为 `archive_names = set(...)`
3	`report.py:1859`	截图标签分母用过滤前数量 `len(names)`	已修复：改为 `total = len(valid_frames)`

🔶 新发现 P2（不阻塞合并，建议后续迭代修复）

[P2-1] check_browser_use_report.py:541
id="home-resource-table" 只在中文主页（index.html）校验，未覆盖英文主页（en/index.html）。AGENTS.md 要求 release check 在两种语言均失败。对比 home-core-metrics 在第 81-84 行对两语言均有显式检查，home-resource-table 应同等处理。实际内容因镜像机制已存在于英文页，但 check 脚本自身覆盖不完整。

建议：将 id="home-resource-table" 检查纳入第 81-84 行的双语循环，或在同位置补加英文页断言。

[P2-2] webarena_report.py:355

actual_by_index = {int(step["index"]): step for step in evidence["trajectory"]}

使用直接键访问 step["index"]，而同文件第 345 行对相同字段用 step.get("index")。若某步骤缺少 "index" 字段会抛出 KeyError。实践中 trajectory 步骤由系统生成，字段必然存在，风险极低，但建议统一为 int(step.get("index", 0)) 或加防御断言。

github-actions · 2026-09-15T10:17:59Z

Claude finished @lanyue-llk's task in 5m 34s —— View job

PR Review: Add autonomous WebArena-Verified browser evaluation

Gather context
Read latest changed files (commits 0004bb8, e8e45bd, 6b6e6e2, a8a9725, 832b7c6, 2722649, 4fcfb0c, 240f233, 6a5bcc6, 1e02984, 3c7fe3c)
Check previous findings remain fixed
Scan for new bugs or security issues
Post findings

Previously reported — confirmed fixed

All bugs from the previous review cycle remain fixed.

tool.close() unguarded in finally (runner.py) — Fixed.
CdpHarRecorder.stop() double-call (har.py) — Fixed.
Base64 decode failure silently corrupts HAR body (har.py) — Fixed.
model_request_attempts_accounted miscounts HTTP errors (model.py) — Fixed.
_verify_arm_php_compatibility guard may fire on x86_64 (runner.py) — Fixed.
Site slug lost if share-URL creation fails on first publish (publish_browser_use_artifact_hub.py) — Fixed.
article_heading check uses removed heading (check_browser_use_report.py) — Fixed.
P2: "、" not replaced in English output (report_i18n.py:1627) — Fixed. Lambda now applies .replace('、', ', ') on match.group(2).
P2: archive.namelist() called in inner loop (report.py) — Fixed. archive_names = set(archive.namelist()) is now precomputed before the event loop.
P2: Screenshot label denominator uses pre-filter count (report.py) — Fixed. total = len(valid_frames) is computed after the dimension filter, so the label denominator matches the actual published count.

Latest commits (trajectory replay track) — no new bugs found

_finalize_method_teardown (webarena_replay.py:595): except BaseException as exc: primary_error = exc; raise correctly captures the original error before the finally block runs. In finally, primary_error.add_note(message) attaches teardown errors without replacing the original exception. Correct.

required_receipt_int (webarena_replay.py:733): Checks isinstance(value, bool) before int() to prevent True/False from silently coercing to 1/0. Raises a descriptive ValueError instead of StopIteration for missing or invalid fields. Correct.

LOCAL_PATH_RE expansion (webarena_report.py:18): r"/(?:Users|home|private|tmp|var|opt)/[^\s\"']+" now covers macOS and Linux ephemeral paths uniformly. Imported by check_webarena_replay_report.py so the same pattern governs report validation.

Preview bootstrap (webarena_report.py:143–153): Slug is validated against [A-Za-z0-9_-]+ before any URL construction. node.href is assigned via DOM property (no innerHTML). location.origin binds all constructed URLs to the same origin.

_QuietHandler.do_GET path traversal guard (check_webarena_replay_visual.py:54): target.is_relative_to(Path(self.directory).resolve()) is evaluated after (Path(self.directory) / route).resolve(), which normalizes any .. sequences. A path that escapes the report root fails this check and returns 404. Correct.

_resource_table (webarena_report.py:216): method[key] accesses "memory_peak", "network_rx", "network_tx", "cpu_utilization" — all four are unconditionally populated in method_rows.append(...) at lines 790–793. No KeyError risk.

bind_publication atomic journal pattern (publish_webarena_replay_artifact_site.py:235): Writes a BINDING_JOURNAL file atomically before updating PROFILE and ATTESTATION. recover_publication_binding() is called at the top of validate_package() on every subsequent run, so a crash between journal write and recovery is correctly handled.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/browser-eval/actions/runs/34957149221. Please check Claude's inline feedback and summary comment for details.

Add autonomous WebArena-Verified browser evaluation

2455720