diff --git a/.agents/skills/benchmark-report/SKILL.md b/.agents/skills/benchmark-report/SKILL.md index 3df41057c..25aaee9a4 100644 --- a/.agents/skills/benchmark-report/SKILL.md +++ b/.agents/skills/benchmark-report/SKILL.md @@ -22,6 +22,8 @@ Read the repository's `AGENTS.md` first. Project rules override this general ski 9. Rerun only calls whose inputs, provider route/version, runtime contract, or source validity changed. Recompute downstream scores and reports without provider calls when only scoring, annotations, or presentation changed. Never retry merely to obtain a better result. A reference or verdict correction must identify the shared root cause and add executable positive/negative controls through the production scorer; bind those regression IDs in the revision receipt instead of patching stored verdicts. 10. Build the public report from the frozen evidence. Every page should answer a reader question. Show task and run identifiers, the exact expected-versus-actual evidence behind a verdict, denominators and unavailable metrics, method differences, limits, concise reproduction instructions, and a compact review history. 11. Release only after code tests, evidence integrity, report links, responsive/visual checks, secret scans, target-site checks, and deployed-output verification pass. Preserve earlier sites unless replacement was explicitly requested. +12. Present one authoritative logical result per method and task. The frozen protocol chooses the score-bearing execution; a renderer must never silently replace that rule. Keep every physical call in the evidence chain and aggregate all latency and cost. Do not make execution ordinal labels part of the public comparison unless retry behavior is the declared subject. +13. In this repository, a finished report release defaults to Artifact Site at `https://artifact-hub.app.lexmount.com`, following its current [`for-agents.md`](https://artifact-hub.app.lexmount.com/for-agents.md). Build and validate before the external write, bind the returned slug, share URL, and version in the current profile, use optimistic locking for later updates, and verify the unauthenticated share page. Use repository publish targets when available; do not reconstruct a target from prose or an earlier receipt. Export the whole deployed tree only when the specific report publisher requires it or when diagnosing a release failure. ## Stop conditions diff --git a/.agents/skills/benchmark-report/references/report-and-release.md b/.agents/skills/benchmark-report/references/report-and-release.md index 57e3da41f..b7cb6aba3 100644 --- a/.agents/skills/benchmark-report/references/report-and-release.md +++ b/.agents/skills/benchmark-report/references/report-and-release.md @@ -18,7 +18,21 @@ Keep environment and reproduction together and deduplicated. Show repository/ver For a multilingual report, generate every language from one semantic report model. Keep route inventory, section order, tables, values, filters and evidence links aligned one-to-one. Localize navigation and explanatory prose for the target language instead of translating mechanically; preserve task names, IDs, URLs, references, provider outputs and other measured evidence unless the benchmark explicitly evaluates translated content. A summary-only secondary language must be labeled as a summary, not presented as the full report. -Use a compact review page resembling release history: current reviewed coverage and unresolved count, then one card or row per review round with scope, findings, corrections, and remaining work. Do not publish agent work logs, giant execution manifests, or downloads merely because the evidence exists. +Use a compact review page resembling release history: one card or row per review round with scope, findings, corrections, status, and remaining work. Keep per-result review evidence on result routes or in the evidence store; do not turn the review history into another task-result inventory. + +Expose one logical result per method and task. A declared additional execution is an orchestration detail: preserve its immutable evidence, follow the score authority frozen before the run, and add every execution's latency and cost. Do not split the public result table by execution ordinal unless that distinction is the preregistered research question. + +Keep dataset inventory separate from measured results. Dataset pages explain the task, selection, reference and acceptance contract; result pages own method verdicts, timing, trajectories and run-derived solvability observations. Define participants as reproducible method stacks and render comparisons from a registry. When the participant set can grow beyond two, group each task's pass/fail participants and provide one baseline selector plus multi-select comparison participants and explicit outcome relations; optional pairwise aggregates may supplement this view. Do not hard-code a two-column comparison or keep adding one wide column per participant. + +Name report tracks by the procedure a reader must distinguish, such as autonomous use versus deterministic trajectory replay, rather than by an internal module layer. Once a repository selects canonical track names, migrate code, commands, report routes and checks together; do not keep compatibility aliases unless external callers are explicitly in scope. + +On result-detail pages, use the opening summary for the verdict, a concise failure locator, and short scalar facts. Put long evaluator messages, expected/actual payloads, traces, and logs in their dedicated evidence sections; never force diagnostic prose or raw JSON into a narrow KPI card. + +For self-hosted web environments, include a deployment and network topology, image identities, included/excluded sites and compatibility layers. Distinguish configured preparation from independently bound preparation receipts. Split environment preparation, browser-session setup, agent run, tool-call wall time, evidence finalization and evaluator time at collection; label historical mixed timers by their actual boundaries and never call wrapper/CDP wall time pure browser-core speed. + +Never publish user prompts, assistant reasoning, plans, internal notes, conversational back-and-forth, or tool-work narration. A formal report explains the benchmark and evidence; it is not the authoring notebook. + +Write every title and paragraph for the reader, not as a reply to author feedback. Convert corrections into neutral artifact facts; never mention who requested a change, quote review conversation, or use a rebuttal-style heading when a descriptive heading works. Review history may record an objective finding and resolution, while release history records only reader-visible effects. ## Claims and rankings @@ -26,6 +40,10 @@ Pair every ranking with its task family, denominator, acceptance meaning, a typi Prefer concise tables for exact comparisons and prose for interpretation. Remove repeated caveats after placing each limitation where it changes interpretation. Run a factual pass before a prose/humanization pass; never let editing change numbers or scope. +For a standard numeric comparison, show mean, P50, and P95 when all three are useful. P50 is the median, so never present “median” and P50 as separate statistics. Declare whether higher or lower is better and visually mark only a unique, directionally valid winner; leave ties, identifiers, call counts, and metrics without a universal direction neutral. Do not rank raw totals when unequal workloads make them non-comparable without also exposing the workload or normalized value. + +Do not add a second static inventory when existing filters expose the same result slice. Keep one searchable result table and use focused summaries only when they add a distinct analytical relationship. + ## Release gates Before publication: @@ -39,6 +57,10 @@ Before publication: 7. fetch the unauthenticated share URL and compare the deployed file tree or archive with the submitted package; 8. record a concise release entry with score/denominator effects and remaining limits. +The responsive/visual gate must cover every core route class, both languages, desktop and narrow widths, and light and dark themes. Check that short headers do not wrap one character per line, long cells receive deliberate width or scrolling, filters remain operable, code and evidence blocks have readable contrast, and text-heavy diagrams use scalable/vector rendering or retain a readable minimum width. A screenshot of only the home page is insufficient. + +Render release history like a report, not a heading plus a raw table: show the current version summary first, keep evidence/handling/impact details expandable, retain older versions in a compact history, and provide the structured record download. Each entry states what changed, why, the formal handling, and the score, denominator, latency, evidence, and limitation effects. + Credentials belong in the approved secret store, never HTML or Git. If a service requires an account, list the account and official acquisition location in the report. Do not include live keys. Publication authorization does not imply permission to overwrite another report. Bind the release target in the current profile and reject receipts for a different service or site. Return the public share link, stable site ID, PR/change link, verification result, and unresolved count. diff --git a/.env.example b/.env.example index 1352fb8f7..5bbd49344 100644 --- a/.env.example +++ b/.env.example @@ -19,12 +19,13 @@ OBSCURA_BINARY=obscura CHROMIUM_BINARY=/Applications/Google Chrome.app/Contents/MacOS/Google Chrome KITESURF_CDP_ENDPOINT=wss://kitesurf.cloudflare.app/devtools/browser -# Browser Use optional model-based diagnostics only (not Web Fetch scoring) +# Browser Trajectory Replay diagnostics and autonomous Browser Use evaluation OPENAI_API_KEY= -OPENAI_BASE_URL= +OPENAI_BASE_URL=https://api.openai.com/v1 -# Optional publishing override; default: ~/.config/artifact-hub/token -ARTIFACT_HUB_TOKEN= +# Optional Artifact Site publishing override; default: +# ~/.config/artifact-site/tokens/ +ARTIFACT_SITE_TOKEN= # Set explicitly in the environment for both report build and packaging. WEBFETCH_REPORT_PUBLIC_URL= diff --git a/.gitignore b/.gitignore index 48c8dfc48..3fcfaa2df 100644 --- a/.gitignore +++ b/.gitignore @@ -29,6 +29,7 @@ run.log *.egg-info/ .pytest_cache/ .ruff_cache/ -.browser-use-eval-venv/ +artifacts/trajectory_replay/ artifacts/browser_use/ -/docs/reports/browser-use-evaluation-* +/docs/reports/trajectory-replay-evaluation-* +.ai/topics/ diff --git a/AGENTS.md b/AGENTS.md index d32233a38..65b923bc8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,7 +11,7 @@ - Independent result review is never automatic. Start it only after explicit user instruction; default to machine failures, and require a separate explicit full-review instruction before reviewing passes or the entire matrix. Publish the exact reviewed/total scope and accepted unreviewed pass count. - A reference, scoring or acceptance correction must fix the shared cause, not patch stored verdicts or one provider. Add paired executable controls to `evaluation/scoring-regressions.json` where both error directions are plausible, run them through the public `score_attempt` and task predicate, and bind the root cause plus regression IDs in every new reference/scoring revision receipt. A revision without this provenance fails before evidence is changed. - A heterogeneous same-window latency cohort keeps every predeclared method-task call. Report the common-denominator all-call distribution separately from each method's conditional successful-call distribution; an all-method common-pass intersection is diagnostic only and must never drive post-outcome task selection or a ranking. -- Evaluation credentials come only from the repository `.env`. Artifact Hub publishing is the sole user-scoped exception: read explicit `ARTIFACT_HUB_TOKEN`, then `~/.config/artifact-hub/tokens/`; never load `.env` for publishing. Save new authorizations per target host without overwriting another deployment's token. Keep `.env.example` current and publish only account requirements, official acquisition links and variable names. +- Evaluation credentials come only from the repository `.env`. Artifact Site publishing is the sole user-scoped exception: read explicit `ARTIFACT_SITE_TOKEN`, then `~/.config/artifact-site/tokens/`; the previous `ARTIFACT_HUB_TOKEN` and `~/.config/artifact-hub/` locations are read-only migration fallbacks. Never load `.env` for publishing. Save new authorizations per target host without overwriting another deployment's token. Keep `.env.example` current and publish only account requirements, official acquisition links and variable names. - Chinese and English are two localized views of one report model. Their HTML route inventories, section order, tables, values, filters and evidence links must match one-to-one. Localize interface and explanatory prose for each language; keep task names, IDs, URLs, references, provider output and other measured evidence unchanged. - Web, dynamic, popular-site and document text use shared `score_attempt` orchestration and one task-result predicate. Task-specific requirements live in case configuration, not provider- or track-specific scoring branches. Binary acceptance and text diagnostics remain separate; missing reference coverage is unavailable, not zero or perfect. - Dynamic-content reference capture follows `harness/web_fetch/README.md`: retain paired initial HTTP bytes and independently rendered visible text, bound completion and the full declared scope. Distinguish embedded state from follow-up fetching; never require JavaScript as the only valid route. @@ -19,3 +19,23 @@ - Keep pass, fail and all result filters, defaulting to fail, with failure classification. Every ranking must state its task family, denominator, acceptance meaning, typical input-to-output route and local limits. Preserve every registered provider's interface examples and adoption statistics. - When a current report has CPU, memory or network evidence, show the same grouped resource comparison on the home page and interpretation page. Lead with the evidence-derived winner for every metric where lower consumption has a clear meaning, mark winning cells in the tables, and state that resource leadership is separate from task pass rate. Do not label utilization or transfer rate as “better” merely because it is lower. Release checks must fail if the winner summary, grouped tables or winner markers are missing in either language. - Maintain `research/report-changes.json` for substantive releases and `research/provider-pricing.json` for public pricing. Release notes explain reader-visible or result-visible effects without narrating assistant workflow; prices retain verification dates, commitments, billing units and unknowns. +- Use `Browser Use` for the autonomous track and `Browser Trajectory Replay` (Chinese: `轨迹重放`) for the fixed BrowserTool-action track. Their implementation roots are `harness/browser_use/` and `harness/trajectory_replay/`; do not add aliases or compatibility paths using the retired names. +- Browser Trajectory Replay canonical inputs must not contain run-local snapshot refs. Compile each ref to a hash-bound stable selector while preserving the original ref action semantics; do not silently replace a DOM ref click/fill/select with BrowserTool's Playwright selector fallback because that adds different viewport, overlay and navigation-wait behavior. +- Browser Trajectory Replay reports keep the dataset separate from measured verdicts. Every task needs a contract-and-qualified-trajectory page; every result needs a same-task all-browser comparison, a reader-facing failure locator, the exact failed-action or evaluator error, reviewed cause, complete expandable tool responses, and on-demand public evidence. The result inventory stays compact, fixed-height, filterable and linked to a diagnosis dialog. The same-task explorer uses a registry-driven baseline, multi-select comparison browsers and explicit outcome relations; group pass/fail links instead of adding one result column per future browser. Keep Chinese and English task/result routes aligned. +- A BrowserTool call that returns success does not prove its intended site mutation occurred. When the official evaluator rejects a completed trajectory for an absent required network event, locate the last plausible trigger action and publish the exact missing-event assertion, but do not mark that action as failed or change the official verdict. Repair asynchronous or position-dependent trajectory inputs only through new qualification evidence and a frozen full-scope comparison rerun. +- Trajectory Replay release checks exercise Sites-style tokenized preview navigation from a nested result back to the report root; local relative-link checks alone do not prove hosted drilldown works. +- Browser Use reports expose one manifest-authoritative logical result per browser and task. Dataset routes contain only frozen task definitions and acceptance contracts; result routes own verdicts, timings and trajectories. Keep one filterable result inventory instead of adding duplicate success/failure tables. +- Keep automatic evaluator failure location separate from reviewed root cause. A root-cause subtype and reason must come from a per-browser, per-task review bound to the current attempts hash; render both fields blank while that result is unverified, and require complete verified coverage before a formal release when the current profile enables the gate. +- Generate browser rows and comparisons from the participant registry. The same-task explorer uses one baseline browser, multi-select comparison browsers and explicit outcome relations, with browser labels linking to full results; never add one fixed table column per future browser. List Agent, model, prompt/tool contract, resources, environment and evaluator as shared controls. Standard summaries use mean, P50 and P95; compare every registered model-by-browser configuration together and mark only globally unique winners on directionally comparable metrics. For action latency, compare all configurations by per-call latency and keep unequal-workload totals and call counts neutral. +- Put a core-metrics comparison table near the start of every report home page, before secondary diagnosis or interpretation. Prefer tables over repeated cards for dense categorical summaries. Generate its rows and columns from the participant registry rather than assuming a fixed participant count. +- Use reader-facing labels throughout reports. Do not publish unexplained internal terms or shorthand; Chinese copy must be understandable to a general reader without benchmark implementation context. State the population, denominator, and outcome relation where a compact label would otherwise be ambiguous. +- Result-detail summaries contain only the verdict, concise failed-rule labels and short scalar facts. Full evaluator messages, expected/actual payloads, traces and logs belong in dedicated evidence sections, never narrow KPI cards. +- Result inventories are fixed-height, horizontally scrollable diagnosis tables. Do not place full reviewed causes, evaluator payloads, latency stacks or other paragraph-length content in inventory cells. Every failure row opens an in-page diagnosis with the failure stage, exact program error, reviewed cause, treatment and links to the relevant trajectory, screenshots and logs; the full result page remains the canonical evidence view. +- Compare every registered model-by-browser configuration in one same-task matrix generated from the participant registry. Do not introduce pairwise selectors or duplicate comparison prose when more configurations can be represented as columns. +- Every failed result detail must expose the evidenced failure stage, the programmatic evaluator classification, the exact evaluator/runtime error, the reviewed root cause, the remediation owner/status, a link to the relevant trajectory step, sanitized logs, and the last public trace frames when available. Distinguish a failed BrowserTool action from the last successful action before evaluator rejection. Provide a shared HTML evidence viewer as well as a raw JSON link because static hosts may not preview JSON reliably; load the selected JSON on demand and never duplicate complete evidence payloads into every result page. Extract frames only from ordered Playwright `screencast-frame` events, validate full-page dimensions and provenance, and never treat arbitrary trace resources as screenshots. Label frames as last observed pages, not as exact error screenshots unless the evidence proves that timing. +- Keep official scores immutable when diagnosis finds benchmark/evaluator, site, shared-environment, or frozen-reference interference. Quarantine affected tasks across all configurations in a clearly labeled post-hoc diagnostic subset, record remediation in `research/browser-use-remediation.json`, and readmit them only after a preregistered full-scope rerun. +- Split environment preparation, browser-session setup, Agent run, BrowserTool wall time, evidence finalization and evaluator time at collection. Publish the self-hosted deployment topology and image identity; claim per-execution reset/readiness only when a preparation receipt binds reset, readiness and application health. +- Formal report prose is reader-facing: no prompts, reasoning, internal work narration, conversational rebuttals or references to parallel sessions. Review pages contain review rounds and corrections, not per-case audit tables; release notes contain reader-visible effects. +- English routes may preserve measured task names, IDs and evidence verbatim, but all report-authored interface and explanatory prose must be localized. Release checks must reject known untranslated interface markers and untranslated accessibility attributes. +- Before release, inspect every core route class in Chinese and English at desktop and narrow widths in both themes. Prevent vertical table labels, clipped controls and unreadable evidence; use ImageGen raster assets or Mermaid rather than SVG for Browser Use diagrams. +- Keep routine report checks fast and deterministic; they cover structure, evidence, links, terminology, and static readability guards without launching a browser. Run the separate full release check before packaging or publishing; it adds the complete cross-language, cross-theme, desktop and narrow-width visual gate. diff --git a/Makefile b/Makefile index 8e5828173..0b5ae13aa 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,8 @@ UV ?= uv PY := $(UV) run --frozen python +TEST_PY := $(UV) run --frozen --all-extras python -.PHONY: setup import-env runtime-check test test-python test-ui fixture-smoke ci report report-check resource-image +.PHONY: setup import-env runtime-check naming-check test test-python test-ui fixture-smoke ci report report-check resource-image browser-use-review browser-use-report browser-use-report-check browser-use-visual-check browser-use-release-check trajectory-replay-report trajectory-replay-report-check trajectory-replay-visual-check trajectory-replay-release-check trajectory-replay-package trajectory-replay-publish setup: $(UV) sync --frozen --all-extras @@ -13,13 +14,18 @@ import-env: runtime-check: @$(PY) -c 'import sys; assert sys.version_info[:2] == (3, 11), sys.version' +naming-check: runtime-check + $(PY) scripts/check_track_naming.py + test-python: runtime-check - $(PY) -m unittest discover -s tests -v - $(PY) -m pytest tests/browser_use -q + $(TEST_PY) -m unittest discover -s tests -v + $(TEST_PY) -m pytest tests/browser_use -q + $(TEST_PY) -m pytest tests/trajectory_replay -q test-ui: node tests/test_report_ui.mjs node tests/test_report_loader.mjs + node tests/browser_use/test_report_ui.mjs fixture-smoke: runtime-check $(PY) -m harness.web_fetch validate --providers fixture @@ -27,7 +33,7 @@ fixture-smoke: runtime-check test: test-python test-ui -ci: test fixture-smoke +ci: naming-check test fixture-smoke report: runtime-check @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path is required' >&2; exit 2) @@ -38,5 +44,44 @@ report-check: runtime-check $(PY) scripts/check_report_links.py reports/webfetch $(PY) scripts/check_report_revision.py reports/webfetch +browser-use-report: runtime-check + @test -n "$(EVIDENCE)" || (echo 'EVIDENCE="/absolute/path [/absolute/path...]" is required' >&2; exit 2) + $(PY) scripts/build_browser_use_report.py $(foreach path,$(EVIDENCE),--evidence "$(path)") --output reports/browser-use + +browser-use-review: runtime-check + @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path is required' >&2; exit 2) + $(PY) scripts/review_browser_use_results.py --evidence "$(EVIDENCE)" + +browser-use-visual-check: runtime-check + $(PY) scripts/check_browser_use_visual.py reports/browser-use + +browser-use-report-check: runtime-check + @test -n "$(EVIDENCE)" || (echo 'EVIDENCE="/absolute/path [/absolute/path...]" is required' >&2; exit 2) + $(PY) scripts/check_browser_use_report.py reports/browser-use $(foreach path,$(EVIDENCE),--evidence "$(path)") --require-complete + $(PY) scripts/check_report_links.py reports/browser-use + +browser-use-release-check: browser-use-report-check + $(MAKE) browser-use-visual-check + +trajectory-replay-report: runtime-check + @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path is required' >&2; exit 2) + $(PY) scripts/build_webarena_replay_report.py --results "$(EVIDENCE)" + +trajectory-replay-report-check: runtime-check + $(PY) scripts/check_webarena_replay_report.py reports/trajectory-replay + +trajectory-replay-visual-check: runtime-check + $(PY) scripts/check_webarena_replay_visual.py reports/trajectory-replay + +trajectory-replay-release-check: trajectory-replay-report-check + $(MAKE) trajectory-replay-visual-check + +trajectory-replay-package: runtime-check + @test -n "$(EVIDENCE)" || (echo 'EVIDENCE=/absolute/path/effective-results.json is required' >&2; exit 2) + $(PY) scripts/package_webarena_replay_artifact_site.py --evidence "$(EVIDENCE)" + +trajectory-replay-publish: runtime-check + $(PY) scripts/publish_webarena_replay_artifact_site.py + resource-image: docker build -f examples/web_fetch/docker/resource.Dockerfile -t lex-web-fetch-resource:local . diff --git a/README.md b/README.md index caaafdb98..5a664eb8a 100644 --- a/README.md +++ b/README.md @@ -17,17 +17,17 @@ files are local build outputs and are not committed to this repository. ## Run the browser evaluation -Requirements: Docker Engine (cgroup v2), Python 3.11+ with `venv`, at least 4 -CPU cores, 6 GiB available memory, and Internet access. +Requirements: Docker Engine (cgroup v2), `uv`, Python 3.11, at least 4 CPU +cores, 6 GiB available memory, and Internet access. ```bash -./examples/browser_use/run.sh +./examples/trajectory_replay/run.sh ``` -The command builds pinned browser images, installs the controller in a local -virtual environment, runs the versioned 30-task deterministic suite, and writes: +The command builds pinned browser images, resolves the locked project +environment, runs the versioned 30-task deterministic suite, and writes: -- raw reproducibility artifacts to `artifacts/browser_use/latest/` (ignored by Git) +- raw reproducibility artifacts to `artifacts/trajectory_replay/latest/` (ignored by Git) - latest-only Markdown, HTML, and audit data to `docs/reports/` (ignored by Git) All statistics and HTML are generated by repository scripts. To rebuild the @@ -35,20 +35,52 @@ summary and report from an existing `attempts.jsonl` without rerunning the browsers, use: ```bash -uv run --frozen python examples/browser_use/rebuild_report.py +uv run --frozen python examples/trajectory_replay/rebuild_report.py ``` -See [examples/browser_use/README.md](examples/browser_use/README.md) for the +See [examples/trajectory_replay/README.md](examples/trajectory_replay/README.md) for the measurement protocol, complexity tiers, overrides, and CubeSandbox decision. ## Harness layout -- `harness/browser_use`: interactive browser-use execution, metrics, and reports +- `harness/trajectory_replay`: fixed-action trajectory-replay execution, metrics, and reports +- `harness/browser_use`: autonomous CowAgent execution with official WebArena-Verified scoring - `harness/web_fetch`: Web Fetch providers, execution, scoring, and reports -Browser Use and Web Fetch intentionally keep separate suites and metrics. +The trajectory-replay microbenchmark, autonomous browser-use evaluation, and +Web Fetch intentionally keep separate suites and metrics. Web Fetch has no legacy top-level package or compatibility imports. +## Autonomous browser-use pilot + +The second browser track runs the real CowAgent loop against pre-registered +WebArena-Verified tasks. Browsers remain the evaluated participants; models are +separate experiment layers. Each model is collected in its own evidence +directory, and the report builder merges those immutable cohorts in memory. + +```bash +./examples/browser_use/run.sh +make browser-use-review EVIDENCE="$HOME/.browser-eval/browser-use/current" +make browser-use-report EVIDENCE="$HOME/.browser-eval/browser-use/current" +make browser-use-report-check EVIDENCE="$HOME/.browser-eval/browser-use/current" +# Before packaging or publishing; adds full responsive, theme, and language visual QA. +make browser-use-release-check EVIDENCE="$HOME/.browser-eval/browser-use/current" +``` + +For a joint report, repeat `--evidence` once per model cohort; never concatenate +their `attempts.jsonl` files: + +```bash +uv run --frozen python scripts/build_browser_use_report.py \ + --evidence "$MODEL_A_EVIDENCE" --evidence "$MODEL_B_EVIDENCE" \ + --output reports/browser-use +uv run --frozen python scripts/check_browser_use_report.py reports/browser-use \ + --evidence "$MODEL_A_EVIDENCE" --evidence "$MODEL_B_EVIDENCE" +``` + +See [examples/browser_use/README.md](examples/browser_use/README.md) for the +local-site and credential boundary. + --- ## Web Fetch 评测 @@ -133,7 +165,7 @@ ARTIFACT_HUB_BASE_URL=https://artifact-hub.app.lexmount.com uv run --frozen pyth # 发布脚本只确认上传响应、新版本与 share URL;随后在浏览器打开 share_url 做页面冒烟检查。 ``` -打包输出 `publish/webfetch-sites.zip`,将完整证据去重压缩并按需加载。Sites 发布令牌依次从显式进程环境 `ARTIFACT_HUB_TOKEN`、目标服务的 `~/.config/artifact-hub/tokens/` 读取,不加载仓库 `.env`。新授权按目标 host 保存,文件权限为 0600;不同部署的令牌不通用,也不互相覆盖,host 包含显式端口(如有)。其他评测密钥仍只读取仓库配置。当前报告绑定 `https://artifact-hub.app.lexmount.com` 的 10 方法站点 `RSdbALdbdNG6`;此前的 dev 域站点 `MblD_VwSPBRX`、旧 12 方法站点 `UeZE1vfcEOrM` 和五类浏览器参考报告站点 `UUAhhQMnrGwm` 均保持不变。构建、打包与发布都从 `evaluation/current-report.json` 读取公开地址、站点和预期远端版本;回执的服务地址与 slug 共同决定更新目标,回执、请求地址或命令行版本只要不一致就拒绝更新。服务端版本标识仅用于并发保护;成功发布后,把 profile 的预期版本更新为服务端新版本。例行发布只检查提交响应、轻量站点元数据和公开页面渲染,不回下载 ZIP 或逐文件镜像;`verify_artifact_hub.py --export` 与 `--public-files` 仅用于明确的服务端排障。发布令牌和管理回执不得提交或放入站点。 +打包输出 `publish/webfetch-sites.zip`,将完整证据去重压缩并按需加载。Artifact Site 发布令牌优先读取显式进程环境 `ARTIFACT_SITE_TOKEN` 和目标服务的 `~/.config/artifact-site/tokens/`;旧产品名的变量和路径仅作迁移回退,不加载仓库 `.env`。当前十方法 Web Fetch 报告绑定 `https://artifact-hub.app.lexmount.com` 的站点 `RSdbALdbdNG6`;五类浏览器参考报告 `UUAhhQMnrGwm` 及历史站点均保持不变。构建、打包与发布从 `evaluation/current-report.json` 读取目标地址、站点和预期版本,并用服务端版本标识防止覆盖并发更新。例行发布检查提交响应、站点元数据和公开页面;完整导出或逐文件核查只在排障时按需触发。发布令牌与管理回执不得提交或放入站点。 ## 内容规范 diff --git a/evaluation/browser-use/current-report.json b/evaluation/browser-use/current-report.json new file mode 100644 index 000000000..294992029 --- /dev/null +++ b/evaluation/browser-use/current-report.json @@ -0,0 +1,45 @@ +{ + "schema": "lex_browser_use_current_report/2", + "benchmark": "WebArena-Verified", + "benchmark_version": "1.2.3", + "task_count": 90, + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "official_data_checksum": "d65275660814663375028e9017e1f929e3c38321041b125795e2713b52243d30", + "engines": [ + "chromium", + "moli" + ], + "engine_labels": { + "chromium": "Chromium", + "moli": "Moli 1.1.3" + }, + "baseline_engine": "chromium", + "models": [ + "gpt-5.6-sol", + "gpt-6-astra" + ], + "model_labels": { + "gpt-5.6-sol": "GPT-5.6 Sol", + "gpt-6-astra": "GPT-6 Astra" + }, + "reasoning_effort": "low", + "max_agent_steps": 40, + "max_task_executions": 2, + "logical_result": "score authority follows the frozen run manifest; aggregate all execution latency and cost", + "model_transport_retries": 1, + "scoring_authority": "unmodified official programmatic evaluator", + "review_kind": "deterministic evidence integrity audit", + "independent_semantic_review": false, + "failure_cause_review": { + "required_for_publication": true, + "source_path": "research/browser-use-failure-cause-review.json", + "source_sha256": "a11134919f5e7b9a65c601374211e20e7e940f82e6f363ed025c49b52336e721" + }, + "publication": { + "base_url": "https://artifact-hub.app.lexmount.com", + "site_slug": "7l77EeXPRfml", + "share_url": "https://artifact-hub.app.lexmount.com/v/Ghn1BR6PCy-njsCMqCe2LRIJ-eyJEJoa", + "expected_version": "ver_ce4dd70fde2e444ab18d24bba3ee0e93", + "state": "verified" + } +} diff --git a/evaluation/browser-use/task-exclusions.json b/evaluation/browser-use/task-exclusions.json new file mode 100644 index 000000000..44dcfbb34 --- /dev/null +++ b/evaluation/browser-use/task-exclusions.json @@ -0,0 +1,14 @@ +{ + "schema": "lex_browser_use_task_exclusions/1", + "benchmark": "WebArena-Verified 1.2.3", + "tasks": { + "68": { + "reason": "official expected.retrieved_data violates the published results_schema", + "disposition": "exclude until the official schema is corrected or a benchmark patch is preregistered" + }, + "142": { + "reason": "category membership for 'hair care and hair style' has no observable acceptance rule", + "disposition": "exclude until category membership is bound to site taxonomy or another observable predicate" + } + } +} diff --git a/evaluation/browser-use/webarena-verified-pilot.json b/evaluation/browser-use/webarena-verified-pilot.json new file mode 100644 index 000000000..c8cc361fb --- /dev/null +++ b/evaluation/browser-use/webarena-verified-pilot.json @@ -0,0 +1,111 @@ +{ + "schema": "lex_webarena_verified_pilot/1", + "metadata": { + "benchmark": "WebArena-Verified", + "benchmark_version": "1.2.3", + "difficulty_definition": "estimated minimum visible UI operations after authentication: easy 1-4, medium 5-7, hard 8+", + "selection_policy": "pre-registered before the 90-task run; provider- and result-blind; single-site tasks from the four locally verified self-hosted sites; known invalid or unobservable acceptance contracts excluded; broad intent-template and retrieve, navigate, mutate coverage", + "tier_counts": {"easy": 15, "medium": 30, "hard": 45}, + "site_counts": {"shopping": 23, "shopping_admin": 23, "reddit": 22, "gitlab": 22}, + "task_type_counts": {"RETRIEVE": 37, "NAVIGATE": 13, "MUTATE": 40}, + "continuity": "retains 28 contract-valid tasks from the prior 30-task cohort, removes tasks 68 and 142 under the prospective exclusion registry, and adds 62 tasks without using prior browser outcomes", + "excluded_sites": { + "wikipedia": "requires a large external ZIM data mount and has not completed the local execution qualification gate", + "map": "requires multiple external map datasets and persistent volumes and has not completed the local execution qualification gate" + } + }, + "tasks": [ + {"task_id": 44, "tier": "easy", "estimated_min_steps": 2, "official_hard": true, "rationale": "打开一个需要登录的 GitLab 目标页"}, + {"task_id": 41, "tier": "easy", "estimated_min_steps": 3, "official_hard": false, "rationale": "打开一份报表并读取第一个条目"}, + {"task_id": 77, "tier": "easy", "estimated_min_steps": 3, "official_hard": false, "rationale": "打开评论列表并读取一种状态的数量"}, + {"task_id": 94, "tier": "easy", "estimated_min_steps": 4, "official_hard": false, "rationale": "搜索一张发票并读取总金额"}, + {"task_id": 404, "tier": "easy", "estimated_min_steps": 4, "official_hard": false, "rationale": "进入一个论坛、选择最新帖子并点赞"}, + {"task_id": 313, "tier": "easy", "estimated_min_steps": 2, "official_hard": false, "rationale": "从商城页面直接读取客服号码"}, + {"task_id": 274, "tier": "easy", "estimated_min_steps": 3, "official_hard": false, "rationale": "提交一个商品关键词并停留在搜索结果页"}, + {"task_id": 260, "tier": "easy", "estimated_min_steps": 2, "official_hard": false, "rationale": "从商城导航进入指定品类页"}, + {"task_id": 386, "tier": "easy", "estimated_min_steps": 4, "official_hard": false, "rationale": "搜索一个明确商品并读取评分"}, + {"task_id": 374, "tier": "easy", "estimated_min_steps": 3, "official_hard": false, "rationale": "在后台导航到指定主题设置页"}, + {"task_id": 157, "tier": "easy", "estimated_min_steps": 3, "official_hard": true, "rationale": "在后台打开全部客户详情视图"}, + {"task_id": 595, "tier": "easy", "estimated_min_steps": 4, "official_hard": false, "rationale": "从指定论坛热门帖子页完成订阅"}, + {"task_id": 645, "tier": "easy", "estimated_min_steps": 4, "official_hard": true, "rationale": "在已知论坛发布只有标题的新帖子"}, + {"task_id": 156, "tier": "easy", "estimated_min_steps": 2, "official_hard": true, "rationale": "打开分配给当前用户的合并请求页"}, + {"task_id": 258, "tier": "easy", "estimated_min_steps": 2, "official_hard": false, "rationale": "打开 GitLab 公共项目列表"}, + + {"task_id": 45, "tier": "medium", "estimated_min_steps": 5, "official_hard": false, "rationale": "进入项目议题列表并应用排序和状态筛选"}, + {"task_id": 118, "tier": "medium", "estimated_min_steps": 5, "official_hard": false, "rationale": "把需求转成商品查询并打开匹配详情页"}, + {"task_id": 117, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "进入订单历史,经翻页或排序后读取最早日期"}, + {"task_id": 132, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "找到项目贡献者活动页并限定一个日期"}, + {"task_id": 168, "tier": "medium", "estimated_min_steps": 5, "official_hard": false, "rationale": "打开个人项目并检查星标与项目标识"}, + {"task_id": 173, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "按标题和更新时间筛选个人议题,再读取状态"}, + {"task_id": 183, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "按库存量筛选后台表格并收集 SKU"}, + {"task_id": 188, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "按状态筛选订单,打开最新一单并读取总额"}, + {"task_id": 198, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "筛选并排序订单,再打开最新的匹配客户"}, + {"task_id": 399, "tier": "medium", "estimated_min_steps": 5, "official_hard": true, "rationale": "打开个人资料设置,修改简介并保存"}, + {"task_id": 225, "tier": "medium", "estimated_min_steps": 5, "official_hard": false, "rationale": "定位商品评论并提取低评分评论标题"}, + {"task_id": 231, "tier": "medium", "estimated_min_steps": 5, "official_hard": false, "rationale": "筛选订单状态并读取最近订单号"}, + {"task_id": 298, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "筛选最近已完成订单并打开详情页"}, + {"task_id": 465, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "搜索明确商品并加入愿望清单"}, + {"task_id": 279, "tier": "medium", "estimated_min_steps": 7, "official_hard": false, "rationale": "筛选品牌商品并汇总名称与价格区间"}, + {"task_id": 284, "tier": "medium", "estimated_min_steps": 7, "official_hard": true, "rationale": "按容量和价格比较商品并打开目标详情页"}, + {"task_id": 288, "tier": "medium", "estimated_min_steps": 7, "official_hard": false, "rationale": "从取消订单历史中定位客户并读取邮箱"}, + {"task_id": 243, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "定位指定商品最低评分客户并读取邮箱"}, + {"task_id": 676, "tier": "medium", "estimated_min_steps": 5, "official_hard": false, "rationale": "按欺诈状态筛选后台订单列表"}, + {"task_id": 470, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "搜索指定订单并执行取消"}, + {"task_id": 605, "tier": "medium", "estimated_min_steps": 7, "official_hard": false, "rationale": "选择相关论坛并发布一个问题"}, + {"task_id": 620, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "在关系论坛填写标题和正文并发帖"}, + {"task_id": 625, "tier": "medium", "estimated_min_steps": 7, "official_hard": true, "rationale": "为讨论主题选择相关论坛并发帖"}, + {"task_id": 630, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "在已知论坛按给定标题和正文发帖"}, + {"task_id": 714, "tier": "medium", "estimated_min_steps": 5, "official_hard": false, "rationale": "进入论坛历史榜并踩排名第一的帖子"}, + {"task_id": 475, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "创建一个命名明确的空仓库"}, + {"task_id": 339, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "在当前项目打开并筛选带缺陷标签的议题"}, + {"task_id": 418, "tier": "medium", "estimated_min_steps": 5, "official_hard": true, "rationale": "打开个人设置并更新 GitLab 状态"}, + {"task_id": 389, "tier": "medium", "estimated_min_steps": 6, "official_hard": false, "rationale": "定位指定合并请求并发表评论"}, + {"task_id": 448, "tier": "medium", "estimated_min_steps": 5, "official_hard": false, "rationale": "打开个人资料并修改主页网址"}, + + {"task_id": 28, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "找到最新论坛帖子,并按作者和票数条件检查多条评论"}, + {"task_id": 50, "tier": "hard", "estimated_min_steps": 9, "official_hard": true, "rationale": "检查一整年的分页订单并汇总数量和支出"}, + {"task_id": 66, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "检查十个排序帖子,并根据描述归类推荐内容"}, + {"task_id": 108, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "应用日期和状态条件,并汇总五个月度数量"}, + {"task_id": 110, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "应用日期和状态条件,并汇总十一个月度数量"}, + {"task_id": 124, "tier": "hard", "estimated_min_steps": 9, "official_hard": true, "rationale": "搜索商品、检查结果页并计算价格区间"}, + {"task_id": 127, "tier": "hard", "estimated_min_steps": 9, "official_hard": true, "rationale": "结合搜索词排名与独立的商品可用性检查"}, + {"task_id": 184, "tier": "hard", "estimated_min_steps": 9, "official_hard": true, "rationale": "筛选库存、检查变体并返回成对商品字段"}, + {"task_id": 193, "tier": "hard", "estimated_min_steps": 8, "official_hard": true, "rationale": "排序已完成订单、检查两份详情并合计付款"}, + {"task_id": 196, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "检查两组各四笔订单,并计算付款差额"}, + {"task_id": 204, "tier": "hard", "estimated_min_steps": 9, "official_hard": true, "rationale": "找到最新完成订单,并规范化、排序多个行项目"}, + {"task_id": 506, "tier": "hard", "estimated_min_steps": 12, "official_hard": false, "rationale": "清空购物车,按预算和评分筛选品类,选择商品并结账"}, + {"task_id": 441, "tier": "hard", "estimated_min_steps": 11, "official_hard": true, "rationale": "定位项目文件、修改标题、提交到默认分支并验证"}, + {"task_id": 163, "tier": "hard", "estimated_min_steps": 8, "official_hard": true, "rationale": "遍历商品评论并收集全部低评分标题"}, + {"task_id": 320, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "筛选指定月份的取消订单并汇总含运费退款"}, + {"task_id": 334, "tier": "hard", "estimated_min_steps": 8, "official_hard": false, "rationale": "在历史订单中跨条目定位描述匹配商品的最近日期"}, + {"task_id": 528, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "从订单历史提取订单号和金额,再填写但不提交退款表单"}, + {"task_id": 571, "tier": "hard", "estimated_min_steps": 8, "official_hard": true, "rationale": "进入账户地址簿并完整更新邮寄地址"}, + {"task_id": 585, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "从购买历史定位商品并提交指定评分和评论"}, + {"task_id": 510, "tier": "hard", "estimated_min_steps": 12, "official_hard": false, "rationale": "按品类、评论数、评分和价格比较商品并完成结账"}, + {"task_id": 464, "tier": "hard", "estimated_min_steps": 10, "official_hard": false, "rationale": "统计商品高评分评论后更新商品描述"}, + {"task_id": 547, "tier": "hard", "estimated_min_steps": 10, "official_hard": false, "rationale": "编辑可配置商品并为指定尺码增加颜色选项"}, + {"task_id": 694, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "按属性集、库存、尺码、颜色和价格创建商品"}, + {"task_id": 699, "tier": "hard", "estimated_min_steps": 10, "official_hard": false, "rationale": "创建面向注册客户的全站折扣价格规则"}, + {"task_id": 771, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "筛选全部高评分待审评论并逐项批准"}, + {"task_id": 67, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "检查十个热门帖子并精确提取单书推荐标题"}, + {"task_id": 580, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "创建论坛并配置名称、标题、描述和侧栏"}, + {"task_id": 610, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "发布书评后继续定位新帖子并追加评论"}, + {"task_id": 615, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "从来源帖子提取图片地址并转载到另一个论坛"}, + {"task_id": 640, "tier": "hard", "estimated_min_steps": 8, "official_hard": true, "rationale": "在指定论坛创建包含日期信息的活动通知"}, + {"task_id": 719, "tier": "hard", "estimated_min_steps": 10, "official_hard": false, "rationale": "筛选指定用户在论坛的全部投稿并逐项点赞"}, + {"task_id": 725, "tier": "hard", "estimated_min_steps": 10, "official_hard": false, "rationale": "筛选指定用户在论坛的全部投稿并逐项点踩"}, + {"task_id": 731, "tier": "hard", "estimated_min_steps": 8, "official_hard": false, "rationale": "定位自己的指定帖子并在正文末尾追加一行"}, + {"task_id": 29, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "定位最新帖子并按作者和票数条件统计评论"}, + {"task_id": 635, "tier": "hard", "estimated_min_steps": 8, "official_hard": false, "rationale": "判断最合适论坛并发布包含预算的推荐请求"}, + {"task_id": 600, "tier": "hard", "estimated_min_steps": 8, "official_hard": false, "rationale": "自主选择最可能获得回答的论坛并发布问题"}, + {"task_id": 349, "tier": "hard", "estimated_min_steps": 8, "official_hard": true, "rationale": "进入仓库成员设置并列出其他可访问用户"}, + {"task_id": 784, "tier": "hard", "estimated_min_steps": 8, "official_hard": false, "rationale": "定位仓库分支贡献者并读取提交最多者邮箱"}, + {"task_id": 303, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "按年度范围检查提交历史并汇总指定用户次数"}, + {"task_id": 314, "tier": "hard", "estimated_min_steps": 9, "official_hard": false, "rationale": "对仓库贡献者排序并提取前三名完整姓名"}, + {"task_id": 658, "tier": "hard", "estimated_min_steps": 12, "official_hard": true, "rationale": "创建议题并配置受理人和远期截止日期"}, + {"task_id": 666, "tier": "hard", "estimated_min_steps": 12, "official_hard": false, "rationale": "选择源分支和目标分支创建合并请求并指定审核人"}, + {"task_id": 590, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "在项目中创建并配置起止日期里程碑"}, + {"task_id": 576, "tier": "hard", "estimated_min_steps": 10, "official_hard": true, "rationale": "定位仓库并以指定角色添加两名成员"}, + {"task_id": 415, "tier": "hard", "estimated_min_steps": 9, "official_hard": true, "rationale": "定位合并请求、检查最后评论作者并条件化回复"} + ] +} diff --git a/evaluation/trajectory-replay/webarena-verified/README.md b/evaluation/trajectory-replay/webarena-verified/README.md new file mode 100644 index 000000000..a724d207f --- /dev/null +++ b/evaluation/trajectory-replay/webarena-verified/README.md @@ -0,0 +1,56 @@ +# WebArena-Verified verified trajectories + +`trajectories.json` is the compact, versioned trajectory portion of the benchmark dataset. +Each trajectory is one verified feasible BrowserTool path, not the task's only correct path. + +`current-run.json` is the run authority for the next formal replay. It freezes the same 90-task +selection used by the current Browser Use evaluation, the two registered browser methods, resource +limits, and the isolation schedule. The private ready-to-run seed manifest must contain exactly one +canonical candidate for each of those 90 tasks. The current inventory is assembled from 60 clean +autonomous successes and 30 reviewed repairs; successful executions from different methods are +candidate evidence, not additional tasks. + +- Browser Trajectory Replay consumes `tool_calls` as fixed inputs and uses + `evaluation_input` only when invoking the official evaluator. +- Autonomous Browser Use snapshots assign run-local numeric element refs. Seed generation compiles + those refs from the bound snapshot text into fixed selectors; a canonical trajectory containing + a raw ref is invalid. The compiled call records that it came from a snapshot ref and preserves + BrowserTool's original DOM click, focused keyboard fill, or change-event select semantics. It + must not silently switch to Playwright's selector fallback, whose viewport, overlay and navigation + waiting rules would change the measured trajectory. +- Browser Use may use the inventory as hidden solvability and diagnostic evidence. The Agent + must not receive the trajectory or evaluator input, and its score remains determined by the + official task outcome contract. +- A Browser Use success is only a candidate. A reviewed expert repair may also provide a candidate + when every autonomous run failed or a successful trace contains failed actions. Expert candidates + retain their repair basis and repair-spec hash and may not contain ephemeral accessibility refs. +- Before comparison, promotion requires each canonical candidate to replay without a model call and + pass both execution and evaluation on the frozen reference browser. The measured Chromium and + Moli runs consume that already-qualified trajectory set; their failures remain benchmark outcomes + and never remove a task from the comparison. Where `task-corrections.json` records a proven + upstream contract defect, the qualification proof binds its correction ID, catalog hash and + corrected task hash; historical Browser Use scores remain unchanged. +- Full HAR, screenshots, traces, resource samples and replay receipts remain outside Git under + `BROWSER_EVAL_DATA_HOME`; this directory stores deterministic actions and provenance hashes. +- The current self-hosted WebArena deployment has one fixed-port site stack, so a formal suite owns + the physical environment exclusively. It runs every task for one browser before starting the next + browser, restores the official fixture before every task, and tears down both the browser and site + containers between browsers. Parallel methods are valid only after separate site stacks also have + separate containers, data stores, ports, gateways, and lock namespaces; multiple browser + containers against one site stack are not independent environments. +- Formal execution requires clean browser-eval and CowAgent checkouts and binds their commits, the + resolved browser and site image identities, and the frozen CPU, memory, and sampling limits into + the resumable receipt. A resume with different runtime identity is rejected. Browser storage, + cache, cookies, permissions, service workers, and IndexedDB are cleared before every task; failed + teardown aborts the suite. Any pre-action context recovery writes a new physical-evidence + directory instead of overwriting the first one. +- More than one trajectory may be retained for a task when paths are materially different and + independently satisfy the same promotion gate. + +Run the frozen suite with `scripts/run_webarena_trajectory_replay.py`. Diagnostic subsets are +explicitly non-promotable. `--qualification-engine chromium` runs the complete frozen task set on +the reference browser and is the only scope that may promote the standard trajectory dataset. +After all 90 qualify, promote that receipt with `scripts/promote_webarena_trajectories.py`, then run +the measured method matrix with `--qualified-dataset` and no scope option. The runner rejects formal +execution when that dataset is missing, incomplete, or differs from the frozen profile, seeds, or +canonical actions. Never edit a verdict or verification claim by hand. diff --git a/evaluation/trajectory-replay/webarena-verified/current-run.json b/evaluation/trajectory-replay/webarena-verified/current-run.json new file mode 100644 index 000000000..16d1c274a --- /dev/null +++ b/evaluation/trajectory-replay/webarena-verified/current-run.json @@ -0,0 +1,65 @@ +{ + "schema": "lex_webarena_trajectory_replay_profile/1", + "benchmark": "WebArena-Verified", + "benchmark_version": "1.2.3", + "task_selection": { + "path": "evaluation/browser-use/webarena-verified-pilot.json", + "sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "task_count": 90 + }, + "seed_inventory": { + "base_kind": "successful Browser Use executions", + "expert_repairs_path": "evaluation/trajectory-replay/webarena-verified/expert-repairs.json", + "expert_repairs_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "required_ready_tasks": 90, + "canonical_trajectories_per_task": 1 + }, + "methods": [ + { + "id": "chromium", + "engine": "chromium", + "label": "Chromium" + }, + { + "id": "moli-1.1.3", + "engine": "moli", + "label": "Moli 1.1.3" + } + ], + "qualification": { + "reference_engine": "chromium", + "required_task_count": 90 + }, + "runs_per_task": 1, + "resources": { + "cpus": 2.0, + "memory": "2g", + "sample_interval_ms": 100 + }, + "execution": { + "environment_pool_count": 1, + "method_parallelism": 1, + "task_parallelism": 1, + "schedule": "complete one method before starting the next", + "reset_before_every_task": true, + "teardown_between_methods": true, + "trusted_source_site_hosts": [ + "192.168.5.2", + "host.docker.internal" + ], + "site_stack": "one fixed-port self-hosted WebArena environment" + }, + "scoring": { + "authority": "official programmatic evaluator or a hash-bound replay correction", + "promotion_gate": "the canonical trajectory passes execution and official evaluation on the frozen reference browser before any measured browser comparison", + "model_calls": 0 + }, + "publication": { + "base_url": "https://artifact-hub.app.lexmount.com", + "title": "WebArena-Verified 浏览器轨迹重放评测", + "state": "verified", + "site_slug": "7dF7uNK5SKIx", + "share_url": "https://artifact-hub.app.lexmount.com/v/hkZt38QFBpNIKtThKnqjpnBQk8At19PM", + "expected_version": "ver_fcd443ac19fe403f8baa3442262246e1" + } +} diff --git a/evaluation/trajectory-replay/webarena-verified/expert-repairs.json b/evaluation/trajectory-replay/webarena-verified/expert-repairs.json new file mode 100644 index 000000000..b460bf76a --- /dev/null +++ b/evaluation/trajectory-replay/webarena-verified/expert-repairs.json @@ -0,0 +1,1498 @@ +{ + "schema": "lex_webarena_expert_seed_repairs/1", + "repairs": [ + { + "task_id": 67, + "candidate_id": "expert__task-67__books-hot-v2", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected autonomous trajectory clicked into the Books forum and immediately issued a second navigation while the click navigation was still in flight. Both browsers therefore interrupted the explicit navigation.", + "strategy": "Open the Books hot-post listing directly, then read the first ten unique post documents from that stable listing without overlapping document navigations." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/f/Books" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(async()=>{const links=[...new Set([...document.querySelectorAll('main article a[href*=\"/f/books/\"]')].map(a=>a.getAttribute('href')).filter(Boolean))].slice(0,10);if(links.length!==10)throw new Error('expected ten hot-post links, found '+links.length);return Promise.all(links.map(async href=>{const response=await fetch(href);if(!response.ok)throw new Error('post fetch failed '+response.status+' '+href);const page=new DOMParser().parseFromString(await response.text(),'text/html');return {href,text:page.querySelector('main article')?.textContent?.replace(/\\s+/g,' ').trim()||null};}));})()" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "The Hobbit", + "A Christmas Carol" + ], + "error_details": null + } + }, + { + "task_id": 124, + "candidate_id": "expert-repair-a__task-124", + "source": { + "kind": "expert_authored_browser_trajectory", + "basis": "current self-hosted One Stop Market catalog", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Autonomous runs stopped after inspecting only the first result page and therefore missed the true upper bound.", + "strategy": "Use Magento's canonical quoted-search sort route for the minimum, then verify the known upper-bound wireless in-ear product on its product page." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/catalogsearch/result/index/?product_list_order=price&q=%22wireless%20earphone%22&product_list_dir=asc" + }, + { + "action": "evaluate", + "script": "(()=>{const rows=[...document.querySelectorAll('.product-item')].map(e=>({name:e.querySelector('.product-item-link')?.textContent.trim(),price:e.querySelector('[data-price-type=finalPrice] .price, .price-box .price')?.textContent.trim()})).filter(x=>x.name&&x.price);if(!rows.length)throw new Error('No earphone products rendered');return {boundary:'minimum',product:rows[0],visible_count:rows.length};})()" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sony-wi-1000xm2-industry-leading-noise-canceling-wireless-behind-neck-in-ear-headset-headphones-with-mic-for-phone-call-with-alexa-voice-control-black.html" + }, + { + "action": "evaluate", + "script": "(()=>{const product={name:document.querySelector('h1.page-title')?.textContent.trim(),price:document.querySelector('[data-price-type=finalPrice] .price, .product-info-price .price')?.textContent.trim(),description:document.querySelector('.product.attribute.description')?.textContent.replace(/\\s+/g,' ').trim()};if(!product.name||!product.price||!/wireless/i.test(product.description||'')||!/(in-ear|in ear|behind-neck)/i.test(product.name+' '+(product.description||'')))throw new Error('Upper-bound product is not a wireless in-ear model');return {boundary:'maximum',product};})()" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "min": 0.01, + "max": 298.0 + } + ], + "error_details": null + } + }, + { + "task_id": 225, + "candidate_id": "expert-repair-a__task-225", + "source": { + "kind": "expert_authored_browser_trajectory", + "basis": "current self-hosted One Stop Market product pages", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The score-bearing autonomous trajectories contained stale accessibility references even though their final answer passed.", + "strategy": "Open both matching Sephora brush products by stable URL and inspect their rendered review collections directly." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sephora-collection-skin-perfecting-brush-set.html" + }, + { + "action": "evaluate", + "script": "(()=>({product:document.querySelector('h1.page-title')?.textContent.trim(),reviews:[...document.querySelectorAll('.review-item')].map(e=>({title:e.querySelector('.review-title')?.textContent.trim(),rating:e.querySelector('.rating-result')?.getAttribute('title')}))}))()" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sephora-collection-pro-featherweight-powder-brush-91.html" + }, + { + "action": "evaluate", + "script": "(()=>({product:document.querySelector('h1.page-title')?.textContent.trim(),reviews:[...document.querySelectorAll('.review-item')].map(e=>({title:e.querySelector('.review-title')?.textContent.trim(),rating:e.querySelector('.rating-result')?.getAttribute('title')}))}))()" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "NOT_FOUND_ERROR", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 258, + "candidate_id": "expert-repair-a__task-258", + "source": { + "kind": "expert_authored_browser_trajectory", + "basis": "current self-hosted GitLab public project route", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Autonomous runs remained on the signed-in dashboard instead of opening the public-project filter.", + "strategy": "Navigate directly to GitLab Explore with visibility_level=20, the route required by the official network contract." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/explore?visibility_level=20" + } + ], + "expected_response": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 279, + "candidate_id": "expert-repair-a__task-279", + "source": { + "kind": "expert_authored_browser_trajectory", + "basis": "current self-hosted One Stop Market catalog and hash-bound replay correction wv-279-reference-title-drift-20260911", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Autonomous runs either sampled an incomplete search page or included a non-Bluetooth Sony model; the frozen reference also differed from the current seeded title by one closing parenthesis.", + "strategy": "Load the Sony catalog context, then inspect the twelve contract-bound product pages and return their current rendered names, prices, and stock state." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/catalogsearch/result/?q=Sony&product_list_limit=36" + }, + { + "action": "evaluate", + "timeout": 200000, + "script": "(async()=>{const paths=['sony-wh1000xm3-bluetooth-wireless-noise-canceling-headphones-silver-wh-1000xm3-s-renewed.html','sony-wh-ch710n-h-wireless-bluetooth-noise-cancelling-headphones.html','sony-wh-1000xm3b-wireless-bluetooth-noise-canceling-over-ear-headphones-black-basic-headphone-bundle-kit-with-stylus.html','sony-wireless-headphones-wh-ch510-wireless-bluetooth-on-ear-headset-with-mic-for-phone-call-black.html','sony-whch710n-wireless-bluetooth-noise-canceling-over-the-ear-headphones-black-with-kratos-18w-pd-two-port-power-adapter-and-kratos-6-feet-nylon-braided-usb-c-cable-bundle-3-items.html','sony-wi-sp500-wireless-in-ear-sports-headphones-white-wisp500-w.html','sony-wi-sp510-extra-bass-wireless-in-ear-headset-headphones-with-mic-for-phone-call-sports-ipx5-bluetooth-black-wisp510-b.html','sony-mdras600bt-active-sports-bluetooth-headset-black.html','sony-wh-1000xm4-wireless-noise-canceling-over-ear-headphones-black-with-sony-wla-ns7-wireless-tv-adapter-bundle-2-items.html','sony-wi-c300-wireless-in-ear-headphones-red-wic300-r.html','sony-xb950n1-extra-bass-wireless-noise-canceling-headphones-black.html','sony-h900n-hi-res-noise-cancelling-wireless-headphone-grayish-black-renewed.html'];const out=[];for(const path of paths){const response=await fetch('/'+path);if(!response.ok)throw new Error(path+': HTTP '+response.status);const doc=new DOMParser().parseFromString(await response.text(),'text/html');const name=doc.querySelector('h1.page-title')?.textContent.trim();const price=doc.querySelector('[data-price-type=finalPrice] .price, .product-info-price .price')?.textContent.trim();const stock=doc.querySelector('.stock')?.textContent.trim();if(!name||!price)throw new Error(path+': product facts missing');out.push({name,price,stock});}return out;})()" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "names": [ + "SONY WH1000XM3 Bluetooth Wireless Noise Canceling Headphones Silver WH-1000XM3/S (Renewed)", + "Sony WH-CH710N/H Wireless Bluetooth Noise Cancelling Headphones", + "Sony WH-1000XM3B Wireless Bluetooth Noise-Canceling Over-Ear Headphones (Black) Basic Headphone Bundle Kit with Stylus", + "Sony Wireless Headphones WH-CH510: Wireless Bluetooth On-Ear Headset with Mic for Phone-Call, Black", + "Sony WHCH710N Wireless Bluetooth Noise Canceling Over-The-Ear Headphones (Black) with Kratos 18W PD Two-Port Power Adapter and Kratos 6-Feet Nylon Braided USB-C Cable Bundle (3 Items)", + "Sony WI-SP500 Wireless in-Ear Sports Headphones, White (WISP500/W)", + "Sony WI-SP510 Extra BASS Wireless in-Ear Headset/Headphones with mic for Phone Call Sports IPX5 Bluetooth, Black (WISP510/B)", + "Sony MDRAS600BT Active Sports Bluetooth Headset (Black)", + "Sony WH-1000XM4 Wireless Noise Canceling Over-Ear Headphones (Black) with Sony WLA-NS7 Wireless TV Adapter Bundle (2 Items)", + "Sony WI-C300 Wireless In-Ear Headphones, Red (WIC300/R))", + "Sony XB950N1 Extra Bass Wireless Noise Canceling Headphones, Black", + "SONY - H900N Hi-Res Noise Cancelling Wireless Headphone Grayish Black Renewed" + ], + "min": 18.99, + "max": 406 + } + ], + "error_details": null + } + }, + { + "task_id": 339, + "candidate_id": "expert__task-339__opened-bug-issues", + "source": { + "task_revision": 2, + "contract_correction_id": "wv-339-issues-route-slash-equivalence-20260911", + "evidence": "Official task contract plus historical Chromium and Moli network traces" + }, + "repair_basis": { + "root_cause": "The autonomous runs reached the right filtered issue list, but the upstream route and GraphQL evidence selectors rejected the canonical route/request.", + "strategy": "Navigate directly to the stable project issues route with explicit opened-state and bug-label query parameters." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/a11yproject/a11yproject.com/-/issues?state=opened&label_name%5B%5D=bug" + }, + { + "action": "wait", + "seconds": 1 + } + ], + "expected_response": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 464, + "candidate_id": "expert__task-464__short-description", + "source": { + "task_revision": 2, + "evidence": "Official task contract and seeded product-review records for product 1796" + }, + "repair_basis": { + "root_cause": "The autonomous runs edited the long Page Builder description instead of the product short_description field required by the evaluator.", + "strategy": "Open product 1796 directly, set the short-description form control to the verified one-review result, and submit the native product form." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/edit/id/1796/" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const title=document.querySelector('[data-index=\"content\"] > .fieldset-wrapper-title'); if(!title) throw new Error('Content accordion title not found'); const ko=require('ko'); const context=ko.contextFor(title); const viewModel=context&&context.$data; if(!viewModel||typeof viewModel.toggleOpened!=='function') throw new Error('Content accordion binding not found'); viewModel.toggleOpened(); return typeof viewModel.opened==='function'?viewModel.opened():true;" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "const fields=[...document.querySelectorAll('textarea')].filter(el=>el.name==='short_description'||el.name==='product[short_description]'); if(fields.length!==1) throw new Error('expected one live short_description field, found '+fields.length); const field=fields[0]; field.value='

1 customer(s) love it!

'; field.dispatchEvent(new Event('input',{bubbles:true})); field.dispatchEvent(new Event('change',{bubbles:true})); return {name:field.name,value:field.value};" + }, + { + "action": "evaluate", + "script": "const form=require('uiRegistry').get('product_form.product_form'); const source=form?.source; if(!source?.client||typeof source.client._save!=='function') throw new Error('Magento product data-source save component not found'); source.set('data.product.short_description','

1 customer(s) love it!

'); setTimeout(()=>source.client._save(source.get('data'),{redirect:false,ajaxSave:false,attributes:{id:form.namespace}}),500); return {component:source.name,action:'save'};" + }, + { + "action": "wait", + "seconds": 3 + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 470, + "candidate_id": "expert__task-470__cancel-order-302", + "source": { + "task_revision": 2, + "evidence": "Chromium canary passed the official evaluator after Magento dependency readiness was enforced" + }, + "repair_basis": { + "root_cause": "Earlier cancellations raced Magento search readiness; Moli additionally timed out while checking actionability of the confirmation button.", + "strategy": "Open order 302 directly and use DOM clicks for both the stable cancel control and its confirmation dialog." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales/order/view/order_id/302/" + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('#order-view-cancel-button'); if(!button) throw new Error('cancel button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('.modal-popup._show .action-primary.action-accept'); if(!button) throw new Error('confirmation button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 3 + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 475, + "candidate_id": "expert__task-475__empty-project", + "source": { + "task_revision": 2, + "contract_correction_id": "wv-475-unchecked-checkbox-is-false-20260911", + "evidence": "Official task contract plus browser-standard unchecked-checkbox serialization" + }, + "repair_basis": { + "root_cause": "The upstream evaluator required an explicit zero for an unchecked checkbox even though browsers omit unchecked controls from form submissions.", + "strategy": "Create the project through GitLab's native form with the README checkbox explicitly unchecked and verify the resulting repository is empty." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/projects/new#blank_project" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const set=(selector,value)=>{const el=document.querySelector(selector);if(!el)throw new Error(selector+' not found');el.value=value;el.dispatchEvent(new Event('input',{bubbles:true}));el.dispatchEvent(new Event('change',{bubbles:true}));}; set('input[name=\"project[name]\"]','chatgpt_plugin'); set('input[name=\"project[path]\"]','chatgpt_plugin'); const ns=document.querySelector('select[name=\"project[namespace_id]\"]'); if(ns){ns.value='2505';ns.dispatchEvent(new Event('change',{bubbles:true}));} const readme=document.querySelector('input[name=\"project[initialize_with_readme]\"]'); if(readme) readme.checked=false; return true;" + }, + { + "action": "evaluate", + "script": "const form=document.querySelector('form[action$=\"/projects\"]'); if(!form) throw new Error('project form not found'); if(form.requestSubmit) form.requestSubmit(); else form.submit(); return true;" + }, + { + "action": "wait", + "seconds": 3 + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 510, + "candidate_id": "expert-repair-c-510", + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The autonomous attempts checked out a different product, and the authenticated fixture starts with unrelated items in its persistent cart; the evaluator requires a one-item cart containing product 75640.", + "strategy": "Clear the authenticated cart through the storefront form, add exactly one unit of product 75640, then complete checkout through stable DOM selectors." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/checkout/cart/" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "(() => { const quantities=[...document.querySelectorAll('input[name^=\"cart[\"][name$=\"[qty]\"]')]; if(!quantities.length) return 'cart already empty'; for(const field of quantities) field.value='0'; const form=quantities[0].form; if(!form || !/checkout\\/cart\\/updatePost\\/?$/.test(form.action)) throw new Error('cart update form missing'); form.submit(); return `submitted removal of ${quantities.length} cart item(s)`; })()" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/atlantic-technology-fs-7-0-glb-7-channel-surround-bar-gloss-black-discontinued-by-manufacturer.html" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "click", + "selector": "#product-addtocart-button" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/checkout/cart/" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "(() => { const rows=[...document.querySelectorAll('#shopping-cart-table tbody.cart.item')]; if(rows.length!==1) throw new Error(`expected one cart item, found ${rows.length}`); const name=(rows[0].querySelector('.product-item-name')?.textContent||rows[0].textContent||'').replace(/\\s+/g,' ').trim(); if(!name.includes('Atlantic Technology FS-7.0-GLB 7-channel Surround Bar')) throw new Error(`wrong cart item: ${name}`); return name; })()" + }, + { + "action": "click", + "selector": "button[data-role='proceed-to-checkout']" + }, + { + "action": "wait", + "seconds": 6 + }, + { + "action": "evaluate", + "script": "(() => { const shipping=document.querySelector('input[type=radio][name=ko_unique_1]') || document.querySelector('input[type=radio][value=flatrate_flatrate]'); if(shipping && !shipping.checked) shipping.click(); const next=document.querySelector('button[data-role=opc-continue]'); if(!next) throw new Error('shipping continue button missing'); next.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 6 + }, + { + "action": "evaluate", + "script": "(() => { const radio=document.querySelector('input[type=radio][value=checkmo]'); if(!radio) throw new Error('checkmo payment method missing'); if(!radio.checked) radio.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "(() => { const place=[...document.querySelectorAll('button')].find(e=>e.textContent.trim()==='Place Order' && !e.disabled); if(!place) throw new Error('place order button missing'); place.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 8 + }, + { + "action": "get_text", + "selector": ".checkout-success" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 528, + "candidate_id": "expert__task-528__refund-form-dom", + "source": { + "task_revision": 2, + "contract_correction_id": "wv-528-dom-state-without-submit-20260911", + "evidence": "Official task contract, seeded March 2023 order, and corrected final-DOM acceptance contract" + }, + "repair_basis": { + "root_cause": "The upstream contract simultaneously required an artificial POST and prohibited the real contact-form POST, contradicting the instruction to leave the form unsubmitted.", + "strategy": "Open Contact Us directly, set the three required DOM values, and leave the native form unsubmitted for review." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/contact" + }, + { + "action": "evaluate", + "script": "const values={\"input[name='name']\":'Emma Lopez',\"input[name='email']\":'emma.lopez@gmail.com',\"textarea[name='comment']\":'I need a refund. Order: 000000180. Amount: $12.99. Reason: It broke after three days of use.'}; for(const [selector,value] of Object.entries(values)){const el=document.querySelector(selector);if(!el)throw new Error(selector+' not found');el.value=value;el.dispatchEvent(new Event('input',{bubbles:true}));el.dispatchEvent(new Event('change',{bubbles:true}));} return Object.fromEntries(Object.keys(values).map(selector=>[selector,document.querySelector(selector).value]));" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 547, + "candidate_id": "expert-repair-c-547", + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Brown already exists as global color option 51; prior attempts created or edited a child but did not save the parent configurable-product matrix.", + "strategy": "Use the product configuration wizard to create and persist the missing S/Brown variation on parent product 1130." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/edit/id/1130/" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "evaluate", + "script": "(() => { const button=[...document.querySelectorAll('button')].find(e=>e.textContent.trim()==='Edit Configurations'); if(!button) throw new Error('Edit Configurations button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const dialog=[...document.querySelectorAll('aside[role=dialog]')].find(e=>e.offsetParent!==null); const next=[...(dialog||document).querySelectorAll('button')].find(e=>e.textContent.trim()==='Next' && !e.disabled); if(!next) throw new Error('configuration Next button missing'); next.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const boxes=[...document.querySelectorAll('input[type=checkbox]')]; const sizeValues=new Set(['55 cm','XS','65 cm','S','75 cm','M','6 foot','L','8 foot','XL','10 foot','28','29','30','31','32','33','34','36','38']); const colorValues=new Set(['Black','Blue','Brown','Gray','Green','Lavender','Multi','Orange','Purple','Red','White','Yellow']); const seen=[]; for(const box of boxes){const label=(box.labels?.[0]?.textContent||'').trim(); if(!sizeValues.has(label)&&!colorValues.has(label)) continue; const should=label==='S'||label==='Brown'; if(box.checked!==should) box.click(); if(should) seen.push(label);} if(!seen.includes('S')||!seen.includes('Brown')) throw new Error(`missing Size/Color options: ${seen.join(',')}`); const next=[...document.querySelectorAll('button')].filter(e=>e.textContent.trim()==='Next'&&!e.disabled).at(-1); if(!next) throw new Error('Attribute Values Next button missing'); next.click(); return seen; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const price=document.querySelector('#apply-single-price-radio'); const quantity=document.querySelector('#apply-single-inventory-radio'); if(!price||!quantity) throw new Error('bulk price or inventory controls missing'); price.click(); quantity.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "(() => { const set=(name,value)=>{const field=document.querySelector(`input[name=${name}]`); if(!field) throw new Error(`missing ${name}`); field.value=value; field.dispatchEvent(new Event('input',{bubbles:true})); field.dispatchEvent(new Event('change',{bubbles:true}));}; set('ko_unique_7','59'); set('ko_unique_11','100'); const next=[...document.querySelectorAll('button')].filter(e=>e.textContent.trim()==='Next'&&!e.disabled).at(-1); if(!next) throw new Error('Bulk Images and Price Next button missing'); next.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const button=[...document.querySelectorAll('button')].filter(e=>e.textContent.trim()==='Generate Products'&&!e.disabled).at(-1); if(!button) throw new Error('Generate Products button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 5 + }, + { + "action": "evaluate", + "script": "(() => { const button=document.querySelector('#save-button')||[...document.querySelectorAll('button')].find(e=>e.offsetParent!==null&&e.textContent.trim()==='Save'); if(!button) throw new Error('parent Save button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 8 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 183, + "candidate_id": "expert__task-183__quantity-ten", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected autonomous trajectory assumed a persisted Magento grid filter and unconditionally clicked Clear all, which is absent after a clean task reset.", + "strategy": "Open the product grid, clear an existing filter only when present, then apply the exact quantity range through stable form controls." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/" + }, + { + "action": "evaluate", + "script": "const clear=[...document.querySelectorAll('button')].find(e=>e.offsetParent!==null&&e.textContent.trim()==='Clear all'); if(clear){clear.click(); return 'cleared';} return 'already-clear';" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('button[data-action=\"grid-filter-expand\"]'); if(!button) throw new Error('filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "for(const [name,value] of [['qty[from]','10'],['qty[to]','10']]){const input=document.querySelector(`input[name=\"${name}\"]`); if(!input) throw new Error(`missing ${name}`); input.value=value; input.dispatchEvent(new Event('input',{bubbles:true})); input.dispatchEvent(new Event('change',{bubbles:true}));} const button=document.querySelector('button[data-action=\"grid-filter-apply\"]'); if(!button) throw new Error('apply filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "NOT_FOUND_ERROR", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 198, + "candidate_id": "expert__task-198__order-136-v2", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The dynamic repair inspected Magento's asynchronously loaded order grid after a fixed one-second wait. The grid could still show a persisted Complete filter, so no canceled row existed at that intermediate state. Four independent successful Browser Use executions and the official evaluator response all bind the fixed benchmark answer to order 136.", + "strategy": "Open the evidence-confirmed order 136 directly, wait for its detail page, and capture the page used by the official evaluator. This removes the transient grid-filter dependency without changing the frozen answer." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales/order/view/order_id/136/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "harrypotterfan1@gmail.com" + ], + "error_details": null + } + }, + { + "task_id": 28, + "candidate_id": "expert__task-28__latest-worcester-post", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory issued a second navigation while the forums click was still navigating, visited two deterministic 404 routes, and hard-coded one post ID as the newest post.", + "strategy": "Open the WorcesterMA newest-post listing directly and follow its first visible post rather than relying on an old post ID." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/f/WorcesterMA/new" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const link=document.querySelector('main article h1 a, main article h2 a, article a[href*=\"/post/\"]'); if(!link) throw new Error('newest WorcesterMA post link not found'); link.click(); return {href:link.href,text:link.textContent.trim()};" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "username": "mineinhusdson", + "post_title": "Best place for a foot rub?", + "count": 0 + } + ], + "error_details": null + } + }, + { + "task_id": 50, + "candidate_id": "expert__task-50__annual-complete-orders", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Magento's account-page redirect was still navigating when the source trajectory issued a second goto, so Playwright canceled the order-history navigation before replay began.", + "strategy": "Establish Magento's canonical origin, navigate to order history from within that document, and verify the full 50-row view before returning the previously evaluator-approved answer." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "location.assign(location.origin + '/sales/order/history/?limit=50'); return true;" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "const rows=[...document.querySelectorAll('table tbody tr')].map(r=>[...r.querySelectorAll('td')].map(c=>c.innerText.trim())); if(rows.length<21) throw new Error(`incomplete order history: ${rows.length}`); const eligible=rows.filter(r=>r[3]==='Complete'&&new Date(r[1])>=new Date('2022-06-12')&&new Date(r[1])<=new Date('2023-06-12')); return {order_count:eligible.length,amount:eligible.reduce((sum,row)=>sum+Math.round(Number(row[2].replace(/[$,]/g,''))*100),0)/100};" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "order_count": 21, + "amount": 6560.69 + } + ], + "error_details": null + } + }, + { + "task_id": 184, + "candidate_id": "expert__task-184__zero-simple-products", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory unconditionally clicked Clear all because its source run inherited a quantity filter; the control is absent after a clean reset.", + "strategy": "Open the product grid without inherited filters and apply the complete zero-quantity simple-product filter through stable form controls." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('button[data-action=\"grid-filter-expand\"]'); if(!button) throw new Error('filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "for(const [name,value] of [['qty[from]','0'],['qty[to]','0']]){const input=document.querySelector(`input[name=\"${name}\"]`); if(!input) throw new Error(`missing ${name}`); input.value=value; input.dispatchEvent(new Event('input',{bubbles:true})); input.dispatchEvent(new Event('change',{bubbles:true}));} const type=document.querySelector('select[name=\"type_id\"]'); if(!type) throw new Error('type filter not found'); type.value='simple'; type.dispatchEvent(new Event('change',{bubbles:true})); const button=document.querySelector('button[data-action=\"grid-filter-apply\"]'); if(!button) throw new Error('apply filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "name": "Cronus Yoga Pant -33-Blue", + "color": "Blue" + } + ], + "error_details": null + } + }, + { + "task_id": 284, + "candidate_id": "expert__task-284__shoe-storage-product", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The Colima gateway rebound dropped the Magento deep-link path and returned the storefront root, so the contract product was never present.", + "strategy": "Establish Magento's canonical browser origin first, then navigate from inside the page to the exact contract-bound product path." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "location.assign(location.origin + '/onlyeasy-over-the-door-shoe-storage-organizer-hanging-shoe-rack-holder-with-24-large-fabric-pockets-22-1-x-61-4-herringbone-grey-mxrodsb1p.html'); return true;" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 320, + "candidate_id": "expert__task-320__february-refunds", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory started another navigation before the order-detail click had finished navigating.", + "strategy": "Visit the two contract-bearing canceled orders by stable URLs with an explicit settling interval between documents." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sales/order/view/order_id/156/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sales/order/view/order_id/158/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [406.53], + "error_details": null + } + }, + { + "task_id": 334, + "candidate_id": "expert__task-334__last-cornbread-order", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory queried a pagination control before the order-history navigation had completed.", + "strategy": "Open the contract-bearing order directly and wait for the document before reading it." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sales/order/view/order_id/180/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": ["2023-03-11"], + "error_details": null + } + }, + { + "task_id": 506, + "candidate_id": "expert__task-506__meat-substitute-checkout", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The shopping reset left three unrelated products in the account cart, so checkout contained four items and violated the evaluator contract.", + "strategy": "Clear every server-backed cart item through its own Magento removal contract before adding and purchasing the required product." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/checkout/cart/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(async()=>{const links=[...document.querySelectorAll('a.action-delete, a[title=\"Remove item\"]')]; const formKey=document.querySelector('input[name=\"form_key\"]')?.value; if(links.length&&!formKey) throw new Error('Magento form_key missing'); for(const link of links){const raw=link.getAttribute('data-post'); if(!raw) throw new Error('cart removal contract missing'); const request=JSON.parse(raw); const response=await fetch(request.action,{method:'POST',credentials:'same-origin',headers:{'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8','X-Requested-With':'XMLHttpRequest'},body:new URLSearchParams({...request.data,form_key:formKey})}); if(!response.ok) throw new Error(`cart removal failed: ${response.status}`);} return {removed:links.length};})()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/beyond-meat-beef-beefy-crumble-5-pound-2-per-case.html" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Add to Cart\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/checkout/cart/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const names=[...document.querySelectorAll('.cart.item .product-item-name, .cart.item strong.product-item-name')].map(e=>e.textContent.trim()); if(names.length!==1 || !names[0].includes('Beyond Meat Beef Beefy Crumble')) throw new Error(`cart isolation failed: ${JSON.stringify(names)}`); return names;" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Proceed to Checkout\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "click", + "selector": "input[name=\"ko_unique_1\"]", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Next\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "click", + "selector": ".payment-method._active button.action.primary.checkout:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 20 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 585, + "candidate_id": "expert__task-585__five-star-review", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory addressed the fifth visible rating input before the Reviews panel had rendered, making both timing and position part of the selector.", + "strategy": "Open the contract product, wait for the Reviews panel, and select the five-star value by its form value." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/quoizel-tf9404m-grove-park-tiffany-multi-color-floor-lamp.html" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "click", + "selector": "a[href=\"#reviews\"]", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "click", + "selector": "input[name=\"ratings[4]\"][value=\"20\"]", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "selector": "input[name=\"nickname\"]", + "text": "Emma Lopez", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "selector": "input[name=\"title\"]", + "text": "Good purchase", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "selector": "textarea[name=\"detail\"]", + "text": "I like it", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Submit Review\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 605, + "candidate_id": "expert__task-605__fresh-console-submission", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The replay submitted the correct title and forum but the site returned the form with HTTP 200 instead of the successful 302 redirect; same-task historical evidence identifies an invalid CSRF token.", + "strategy": "Load a fresh forum-bound submission form and submit immediately, with no stale intermediate page or duplicate-click retry." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/submit/consoles" + }, + { + "action": "fill", + "selector": "textarea[name=\"submission[title]\"]", + "text": "what is the recommended console to buy these days", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 600, + "candidate_id": "expert__task-600__fresh-console-question", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory navigated away while a gaming submission-form click was still completing, before it reached the intended consoles form.", + "strategy": "Load the consoles submission form directly and submit the exact title immediately." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/submit/consoles" + }, + { + "action": "fill", + "selector": "textarea[name=\"submission[title]\"]", + "text": "what is the recommended console to buy these days?", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 725, + "candidate_id": "expert__task-725__massachusetts-submissions-downvote", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory started a forum navigation while the preceding forums click was still navigating and later addressed the downvote button by page position.", + "strategy": "Open the author's submissions directly, bind the target article to its massachusetts forum link, and click that article's downvote control." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/user/RickyDontLoseThat/submissions" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const article=[...document.querySelectorAll('main article')].find(e=>e.querySelector('a[href=\"/f/massachusetts\"]')); const button=article?.querySelector('form button:has(img[alt=\"down\"])'); if(!button) throw new Error('massachusetts submission downvote control not found'); button.click(); return {post:article.querySelector('h1,h2')?.textContent.trim()};" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 415, + "candidate_id": "expert__task-415__verification-reminder", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The comment became visible in the page, but the trajectory finalized evidence immediately after the click, before the required notes POST was captured.", + "strategy": "Open the contract merge request directly, submit the required author reminder, and allow the request to settle before evidence finalization." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/byteblaze/a11y-webring.club/-/merge_requests/40" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "fill", + "selector": "textarea[name=\"note[note]\"]:visible", + "text": "@davepgreene", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Comment\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 576, + "candidate_id": "expert__task-576__invite-two-developers", + "replaces_canonical": true, + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory addressed the third visible text input, which was the expiration-date field after reset, then waited for a user menu that could never appear.", + "strategy": "Scope member lookup to the Invite members dialog, select each named user by text, set Developer, and submit once." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/byteblaze/a11y-webring.club/-/project_members" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Invite members\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const heading=[...document.querySelectorAll('h4')].find(e=>e.textContent.trim()==='Invite members'); const root=heading?.closest('[role=dialog],.modal,.gl-modal')||heading?.parentElement?.parentElement; const input=[...(root?.querySelectorAll('input[type=text]')||[])].find(e=>e.placeholder!=='YYYY-MM-DD'); if(!input) throw new Error('member lookup input not found'); const setter=Object.getOwnPropertyDescriptor(HTMLInputElement.prototype,'value').set; setter.call(input,'abisubramanya27'); input.dispatchEvent(new Event('input',{bubbles:true})); return true;" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const button=[...document.querySelectorAll('button[role=menuitem],button')].find(e=>e.offsetParent!==null&&e.textContent.includes('abisubramanya27')); if(!button) throw new Error('abisubramanya27 suggestion not found'); button.click(); return true;" + }, + { + "action": "evaluate", + "script": "const heading=[...document.querySelectorAll('h4')].find(e=>e.textContent.trim()==='Invite members'); const root=heading?.closest('[role=dialog],.modal,.gl-modal')||heading?.parentElement?.parentElement; const input=[...(root?.querySelectorAll('input[type=text]')||[])].find(e=>e.placeholder!=='YYYY-MM-DD'); if(!input) throw new Error('member lookup input not found'); const setter=Object.getOwnPropertyDescriptor(HTMLInputElement.prototype,'value').set; setter.call(input,'lahwaacz'); input.dispatchEvent(new Event('input',{bubbles:true})); return true;" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const button=[...document.querySelectorAll('button[role=menuitem],button')].find(e=>e.offsetParent!==null&&e.textContent.includes('lahwaacz')); if(!button) throw new Error('lahwaacz suggestion not found'); button.click(); return true;" + }, + { + "action": "select", + "selector": ".modal select:visible", + "value": "30", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": ":nth-match(button:visible:has-text(\"Invite\"), 3)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 676, + "candidate_id": "expert__task-676__fraud-orders", + "source": { + "task_revision": 2, + "contract_correction_id": "wv-676-match-filter-xhr-20260911", + "evidence": "Historical successful Moli filter interaction and corrected XHR-selection contract" + }, + "repair_basis": { + "root_cause": "The upstream evaluator selected the last document navigation instead of the matching Magento grid XHR and also required a browser-specific Referer.", + "strategy": "Open the orders grid directly and drive its stable filter controls to emit the fraud-status MUI request." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales/order/" + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('button[data-action=\"grid-filter-expand\"]'); if(!button) throw new Error('filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "for(const name of ['created_at[from]','created_at[to]']){const input=document.querySelector(`input[name=\"${name}\"]`); if(input){input.value=''; input.dispatchEvent(new Event('input',{bubbles:true})); input.dispatchEvent(new Event('change',{bubbles:true}));}} const select=document.querySelector('select[name=\"status\"]'); if(!select) throw new Error('status filter not found'); select.value='fraud'; select.dispatchEvent(new Event('input',{bubbles:true})); select.dispatchEvent(new Event('change',{bubbles:true})); const button=document.querySelector('button[data-action=\"grid-filter-apply\"]'); if(!button) throw new Error('apply filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + } + ], + "expected_response": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 694, + "candidate_id": "expert-repair-c-694", + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Earlier saves raced Magento dependency readiness; Moli also timed out while waiting for actionability on fields that were already present.", + "strategy": "Open the Top attribute-set simple-product form directly, set every evaluator-bearing field through DOM events, and invoke the save button through evaluate." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/new/set/9/type/simple/" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "evaluate", + "script": "(() => { const values={'product[name]':'Energy-Bulk Women Shirt','product[sku]':'energy-bulk-women-shirt','product[price]':'60','product[quantity_and_stock_status][qty]':'50','product[quantity_and_stock_status][is_in_stock]':'1','product[weight]':'1','product[size]':'167','product[color]':'50'}; for(const [name,value] of Object.entries(values)){const el=document.querySelector(`[name=\"${name}\"]`); if(!el) throw new Error(`missing field ${name}`); el.value=value; el.dispatchEvent(new Event('input',{bubbles:true})); el.dispatchEvent(new Event('change',{bubbles:true}));} const status=document.querySelector('input[name=\"product[status]\"]'); if(status && !status.checked) status.click(); return Object.fromEntries(Object.keys(values).map(name=>[name,document.querySelector(`[name=\"${name}\"]`).value])); })()" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "(() => { const button=document.querySelector('#save-button'); if(!button) throw new Error('save button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 8 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + }, + { + "task_id": 699, + "candidate_id": "expert-repair-c-699", + "source": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Prior attempts created a Catalog Price Rule; the task and evaluator require a Cart Price Rule, and all registered groups are IDs 1, 2, and 3.", + "strategy": "Open the Cart Price Rule form directly, select website 1 and registered groups 1/2/3, choose percent discount, and save." + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales_rule/promo_quote/new/" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "evaluate", + "script": "(() => { const setValue=(selector,value)=>{const el=document.querySelector(selector);if(!el)throw new Error(`missing ${selector}`);el.value=value;el.dispatchEvent(new Event('input',{bubbles:true}));el.dispatchEvent(new Event('change',{bubbles:true}));return el;}; setValue('input[name=name]','spring sale'); const website=document.querySelector('select[name=\"website_ids\"]'); if(!website) throw new Error('missing website_ids'); [...website.options].forEach(o=>o.selected=o.value==='1'); website.dispatchEvent(new Event('change',{bubbles:true})); const groups=document.querySelector('select[name=\"customer_group_ids\"]'); if(!groups) throw new Error('missing customer_group_ids'); [...groups.options].forEach(o=>o.selected=['1','2','3'].includes(o.value)); groups.dispatchEvent(new Event('change',{bubbles:true})); const active=document.querySelector('input[name=is_active]'); if(active && !active.checked) active.click(); return {website:[...website.selectedOptions].map(o=>o.value),groups:[...groups.selectedOptions].map(o=>o.value)}; })()" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "(() => { const panel=document.querySelector('[data-index=\"actions\"]'); const title=panel?.querySelector('.admin__collapsible-title') || [...document.querySelectorAll('strong')].find(e=>e.textContent.trim()==='Actions'); if(!title) throw new Error('Actions accordion missing'); title.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const action=document.querySelector('select[name=simple_action]'); const discount=document.querySelector('input[name=discount_amount]'); if(!action || !discount) throw new Error(`Actions fields missing: simple_action=${!!action}, discount_amount=${!!discount}`); action.value='by_percent'; action.dispatchEvent(new Event('change',{bubbles:true})); discount.value='20'; discount.dispatchEvent(new Event('input',{bubbles:true})); discount.dispatchEvent(new Event('change',{bubbles:true})); return {action:action.value,discount:discount.value}; })()" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "(() => { const button=document.querySelector('#save_and_apply') || [...document.querySelectorAll('button')].find(e=>e.textContent.trim()==='Save'); if(!button) throw new Error('save button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 8 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "expected_response": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + } + } + ] +} diff --git a/evaluation/trajectory-replay/webarena-verified/task-corrections.json b/evaluation/trajectory-replay/webarena-verified/task-corrections.json new file mode 100644 index 000000000..54ff761b9 --- /dev/null +++ b/evaluation/trajectory-replay/webarena-verified/task-corrections.json @@ -0,0 +1,315 @@ +{ + "schema": "lex_webarena_replay_task_corrections/1", + "scope": "Browser Trajectory Replay standard-trajectory validation only", + "historical_browser_use_scores_immutable": true, + "upstream": { + "benchmark": "WebArena-Verified", + "package_version": "1.2.3", + "dataset_sha256": "d65275660814663375028e9017e1f929e3c38321041b125795e2713b52243d30" + }, + "corrections": [ + { + "task_id": 279, + "official_revision": 2, + "correction_id": "wv-279-reference-title-drift-20260911", + "rationale": "The current seeded catalog title contains two closing parentheses; the frozen reference contains one.", + "preconditions": [ + { + "path": "/eval/0/expected/retrieved_data/0/names/9", + "equals": "Sony WI-C300 Wireless In-Ear Headphones, Red (WIC300/R)" + } + ], + "operations": [ + { + "op": "replace", + "path": "/eval/0/expected/retrieved_data/0/names/9", + "value": "Sony WI-C300 Wireless In-Ear Headphones, Red (WIC300/R))" + } + ] + }, + { + "task_id": 339, + "official_revision": 2, + "correction_id": "wv-339-issues-route-slash-equivalence-20260911", + "rationale": "GitLab's canonical issues collection has no trailing slash, while the upstream regex requires one; navigate-task GET selection also checks the last document before matching the expected route.", + "preconditions": [ + { + "path": "/eval/1/expected/url", + "equals": "^__GITLAB__/a11yproject/a11yproject.com/-/issues/.*$" + }, + { + "path": "/eval/2/expected/headers/referer", + "equals": "__GITLAB__/a11yproject/a11yproject.com/-/issues/?state=opened&label_name%5B%5D=bug" + } + ], + "operations": [ + { + "op": "replace", + "path": "/eval/1/expected/url", + "value": "__GITLAB__/a11yproject/a11yproject.com/-/issues" + }, + { + "op": "replace", + "path": "/eval/2/expected/headers", + "value": null + }, + { + "op": "replace", + "path": "/eval/2/expected/post_data", + "value": { + "$.variables.state": "opened", + "$.variables.labelName": "bug" + } + }, + { + "op": "replace", + "path": "/eval/2/last_event_only", + "value": false + } + ], + "replay_assertions": [ + { + "kind": "network_event_selection", + "policy": "match_expected_url_method_and_query_before_recency", + "target": "canonical issues document and filtered issues GraphQL request" + } + ] + }, + { + "task_id": 448, + "official_revision": 2, + "correction_id": "wv-448-match-profile-save-before-recency-20260915", + "rationale": "GitLab emits a successful profile-save POST followed by another profile event. The acceptance contract must match the POST carrying user[website_url] before applying event recency.", + "preconditions": [ + { + "path": "/eval/1/expected/url", + "equals": "__GITLAB__/-/profile" + }, + { + "path": "/eval/1/expected/post_data/user[website_url]", + "equals": "https://egg.tart.com" + }, + { + "path": "/eval/1/last_event_only", + "equals": true + } + ], + "operations": [ + { + "op": "replace", + "path": "/eval/1/last_event_only", + "value": false + } + ] + }, + { + "task_id": 475, + "official_revision": 2, + "correction_id": "wv-475-unchecked-checkbox-is-false-20260911", + "rationale": "An unchecked HTML checkbox is omitted from a form submission; omission is the browser-standard representation of false.", + "preconditions": [ + { + "path": "/eval/1/expected/post_data/project[initialize_with_readme]", + "equals": "0" + } + ], + "operations": [ + { + "op": "remove", + "path": "/eval/1/expected/post_data/project[initialize_with_readme]" + } + ], + "replay_assertions": [ + { + "kind": "gitlab_project_state", + "project_path": "byteblaze/chatgpt_plugin", + "empty_repository": true + } + ] + }, + { + "task_id": 510, + "official_revision": 2, + "correction_id": "wv-510-freeze-target-and-use-cart-dom-20260912", + "rationale": "The prose combines rating and price without an ordering rule, and the cart-totals response body is not available through every browser evidence interface. Replay freezes the intended product and verifies the one-item cart in the rendered page while retaining add-to-cart, payment and success network checks.", + "preconditions": [ + { + "path": "/intent", + "equals": "Buy the best rating product from \"Home Audio Speaker\" category with at least 5 reviews and the product is least expensive. Choose any available variant." + }, + { + "path": "/eval/1/expected/post_data/product", + "equals": "75640" + }, + { + "path": "/eval/2/expected/response_content/items_qty", + "equals": 1 + } + ], + "operations": [ + { + "op": "replace", + "path": "/intent", + "value": "Buy the least expensive product in the Home Audio Speaker category among products with at least 5 reviews. If prices tie, choose the product with the higher rating. Buy Atlantic Technology FS-7.0-GLB 7-channel Surround Bar (Gloss Black) (Discontinued by Manufacturer), product 75640, and choose any available variant." + }, + { + "op": "add", + "path": "/instantiation_dict/selection_policy", + "value": "minimum price among products with at least 5 reviews; rating descending only as a tie-break" + }, + { + "op": "add", + "path": "/instantiation_dict/target_product_id", + "value": "75640" + }, + { + "op": "remove", + "path": "/eval/2" + } + ] + }, + { + "task_id": 528, + "official_revision": 2, + "correction_id": "wv-528-dom-state-without-submit-20260911", + "rationale": "The task requires a filled but unsubmitted form, so a synthetic POST cannot be positive evidence; final DOM values plus absence of the real submit request express the contract.", + "preconditions": [ + { + "path": "/eval/1/expected/url", + "equals": "^http://.*/dummy_bin$" + }, + { + "path": "/eval/2/should_not_exist", + "equals": true + } + ], + "operations": [ + { + "op": "remove", + "path": "/eval/1" + } + ], + "replay_assertions": [ + { + "kind": "final_dom_form_state", + "page_url": "__SHOPPING__/contact", + "form": "#contact-form", + "fields": { + "input[name='name']": "Emma Lopez", + "input[name='email']": "emma.lopez@gmail.com", + "textarea[name='comment']": "I need a refund. Order: 000000180. Amount: $12.99. Reason: It broke after three days of use." + }, + "forbid_network_event": { + "url": "__SHOPPING__/contact/index/post", + "http_method": "POST" + } + } + ] + }, + { + "task_id": 547, + "official_revision": 2, + "correction_id": "wv-547-existing-brown-option-20260912", + "rationale": "Brown already exists as global color option 51 in the seeded fixture. The parent product save request already proves creation of the S-Brown matrix row, so replay removes only the redundant attribute-save requirement.", + "preconditions": [ + { + "path": "/eval/1/expected/url", + "equals": "__SHOPPING_ADMIN__/catalog/product_attribute/save/attribute_id/93" + } + ], + "operations": [ + { + "op": "remove", + "path": "/eval/1" + } + ] + }, + { + "task_id": 676, + "official_revision": 2, + "correction_id": "wv-676-match-filter-xhr-20260911", + "rationale": "The acceptance event is the MUI grid XHR carrying filters[status]=fraud, not the surrounding sales/order document navigation.", + "preconditions": [ + { + "path": "/eval/1/expected/url", + "equals": "^__SHOPPING_ADMIN__/mui/index/render/.*$" + }, + { + "path": "/eval/1/last_event_only", + "equals": true + }, + { + "path": "/eval/1/expected/headers/referer", + "equals": "__SHOPPING_ADMIN__/sales/order/" + } + ], + "operations": [ + { + "op": "replace", + "path": "/eval/1/last_event_only", + "value": false + }, + { + "op": "replace", + "path": "/eval/1/expected/headers", + "value": null + } + ], + "replay_assertions": [ + { + "kind": "network_event_selection", + "policy": "match_expected_url_method_and_query_before_recency", + "target": "mui/index/render XHR with filters[status]=fraud" + } + ] + }, + { + "task_id": 699, + "official_revision": 2, + "correction_id": "wv-699-browser-form-array-fields-20260912", + "rationale": "Magento submits multi-select values as indexed HTML form fields. The acceptance contract must check website_ids[0] and customer_group_ids[0..2] directly; the upstream unindexed keys cannot be produced by this form.", + "preconditions": [ + { + "path": "/eval/1/expected/post_data/customer_group_ids", + "equals": [ + 1 + ] + }, + { + "path": "/eval/1/post_data_schema", + "equals": null + } + ], + "operations": [ + { + "op": "remove", + "path": "/eval/1/expected/post_data/website_ids" + }, + { + "op": "remove", + "path": "/eval/1/expected/post_data/customer_group_ids" + }, + { + "op": "add", + "path": "/eval/1/expected/post_data/website_ids[0]", + "value": "1" + }, + { + "op": "add", + "path": "/eval/1/expected/post_data/customer_group_ids[0]", + "value": "1" + }, + { + "op": "add", + "path": "/eval/1/expected/post_data/customer_group_ids[1]", + "value": "2" + }, + { + "op": "add", + "path": "/eval/1/expected/post_data/customer_group_ids[2]", + "value": "3" + } + ] + } + ] +} diff --git a/evaluation/trajectory-replay/webarena-verified/trajectories.json b/evaluation/trajectory-replay/webarena-verified/trajectories.json new file mode 100644 index 000000000..a396a261b --- /dev/null +++ b/evaluation/trajectory-replay/webarena-verified/trajectories.json @@ -0,0 +1,7749 @@ +{ + "schema": "lex_webarena_verified_trajectory_dataset/1", + "benchmark": "WebArena-Verified", + "benchmark_version": "1.2.3", + "official_data_checksum": "d65275660814663375028e9017e1f929e3c38321041b125795e2713b52243d30", + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "semantics": { + "trajectory_is_one_verified_feasible_path": true, + "trajectory_is_the_only_correct_path": false, + "browser_use_scoring_authority": "official task outcome contract", + "browser_use_agent_visibility": "hidden", + "trajectory_replay_role": "fixed BrowserTool action input", + "evaluation_input_role": "official evaluator input only; never an Agent prompt" + }, + "promotion_policy": { + "qualification_browser": "chromium", + "measured_browsers_are_not_promotion_gates": true, + "process_status": "pass", + "evaluator_status": "pass", + "model_calls": 0 + }, + "provenance": { + "seed_manifest_sha256": "bb5e95bba930d8ec6e45cc20531b4f3aff860e6970c982b99171c6a23a022567", + "profile_sha256": "3ca5d91cfec570490df202c9cb332957f7e850b70fc5af80b684709e97ef0893", + "replay_receipts": [ + { + "sha256": "bca4dc72e547fbe9b100ffbf24d90d5008507104bc2b2970da30b7d28c5a9e1a", + "source_seed_manifest_sha256": "fc1cc571c48feac7514845a2dc81f73877fbaac4602812e7aad8e9fe43655234", + "profile_sha256": "5322bd5f100d630686639893de08472be717e955e8cfa75df6a9cd146c141b04" + }, + { + "sha256": "d2b2817940fe3c0a849278d707fcd32ec9992f5fc126c258ed7cf637006a1dee", + "source_seed_manifest_sha256": "d5b3edffceec850892225d5eb6e96ddc82f0a46af742eb917037e001e3da63b6", + "profile_sha256": "f5ef0dd6f2d3f1775093149d53f425b562c0bb1d13a579aab19a4a271112d0d9" + }, + { + "sha256": "0671d3a390a97cd8d8fcf58642c8e86b9dbc8fb41f4a5e8bee745a1be9ac46d1", + "source_seed_manifest_sha256": "76ea9bce6eccd877cc2939eaeeed2a8489b4078f3b02f892e3d741b15e13b2ac", + "profile_sha256": "50488ded31860e13377fcbc424d5cf361968e7557cda6c65760801c2679ada92" + }, + { + "sha256": "f758a9bdcb1cb81347046badecfb1a41f6794c45ef9753938d1783a6dc36ea6b", + "source_seed_manifest_sha256": "231d1b5cbadc5fc83431bc0a5484a69c15b9f12ccbd58cf99d1d49d5d4a9f290", + "profile_sha256": "4b9cca016055bf626d77aab1996efec4cc29e9ba3bb7379b3da54593b7c43be3" + }, + { + "sha256": "37779197d4051833d83f7489d9f513448db3ce0f8fd09f1c2ee33975f49713c6", + "source_seed_manifest_sha256": "56499afea9aeb96641e815da824d3cc60b8c2313cbbc891c5886cac5f9fe28a4", + "profile_sha256": "2b26d3df29f8662576b9b52715df91abc8cd4a55718c84af80d3e7215e96bce7" + }, + { + "sha256": "f3910e7715137714db6dbeba54fde9f012dc01680fc85d0af8737443f06d5494", + "source_seed_manifest_sha256": "c84876cedb21d7c7e32eee5cb77689ce8d1cbc26806d2353fa64b3fbb9f54a98", + "profile_sha256": "d252c299914d61fef61c3c248b710c533d2c20f05c104081a478bfa5d6815570" + }, + { + "sha256": "f59d6568105587acb21768d7fb28d8b592c81f9c5f0b07046c70ed8a27ad677b", + "source_seed_manifest_sha256": "bb5e95bba930d8ec6e45cc20531b4f3aff860e6970c982b99171c6a23a022567", + "profile_sha256": "3ca5d91cfec570490df202c9cb332957f7e850b70fc5af80b684709e97ef0893" + } + ] + }, + "counts": { + "verified_tasks": 90, + "verified_trajectories": 90 + }, + "trajectories": [ + { + "id": "webarena-verified-28-trajectory-1", + "task_id": 28, + "task": { + "intent": "In the Worcester forum, get the username and post title of the most recent post, and count the number of comments on that post that are not from the author and have more downvotes than upvotes. Return a list of objects with keys \"username\", \"post_title\", and \"count\".", + "intent_template_id": 33, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/f/WorcesterMA/new" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const link=document.querySelector('main article h1 a, main article h2 a, article a[href*=\"/post/\"]'); if(!link) throw new Error('newest WorcesterMA post link not found'); link.click(); return {href:link.href,text:link.textContent.trim()};" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "username": "mineinhusdson", + "post_title": "Best place for a foot rub?", + "count": 0 + } + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-28__latest-worcester-post", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory issued a second navigation while the forums click was still navigating, visited two deterministic 404 routes, and hard-coded one post ID as the newest post.", + "strategy": "Open the WorcesterMA newest-post listing directly and follow its first visible post rather than relying on an old post ID." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-29-trajectory-1", + "task_id": 29, + "task": { + "intent": "In the DIY forum, get the username and post title of the most recent post, and count the number of comments on that post that are not from the author and have more downvotes than upvotes. Return a list of objects with keys \"username\", \"post_title\", and \"count\".", + "intent_template_id": 33, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:9999/f/DIY" + }, + { + "action": "click", + "selector": "button[aria-label=\"Sort by: Hot\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/DIY/new\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "username": "ziostraccette", + "post_title": "How can I bring an HDMI cable from my pc downstairs to my TV upstairs?", + "count": 0 + } + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-29", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "d45ad8af17b459778a306b4248985922d203f0a39ed1a3daddaea56391c3e558", + "response_sha256": "a0bb7efc94340a501b9ca76425c1a118d86a128c0fdf2c3fd150dadb211df07f", + "evaluation_sha256": "78211058a546985022e4a35b4363bcd51ca61785717134e0458a56322cfafb5c" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-41-trajectory-1", + "task_id": 41, + "task": { + "intent": "Get the top 1 search term(s) in my store", + "intent_template_id": 285, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "hollister" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-41", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "4d6f174990189a64add34b969c43ff7e87f671756642b54d30b2d3aeedb24e42", + "response_sha256": "419e1a97f7e0959373082754252f6c26971c91f50997211c9f3f4da7364baa47", + "evaluation_sha256": "49d1071240a2a66ada503cac2c7d9e5cdc209339f9761602bfefbc84d0a4802d" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-44-trajectory-1", + "task_id": 44, + "task": { + "intent": "Open my todos page", + "intent_template_id": 303, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": "a[href=\"/dashboard/todos\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-44", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "eab05ae2e7ca04df24aed3d437412a9e2dec06f83bdc7a6fff679a8e4ac5203d", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "4971d2ec4b2a99e4f3a978921400bf2b93c31f2bc698722192388d16a43cec07" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-45-trajectory-1", + "task_id": 45, + "task": { + "intent": "Open the issues page for the current project filtered to the most recent open issues", + "intent_template_id": 300, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__/a11yproject/a11yproject.com" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023/a11yproject/a11yproject.com" + }, + { + "action": "click", + "selector": "a[href=\"/a11yproject/a11yproject.com/-/issues\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button[aria-label=\"Sort direction: Ascending\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-45", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "41fecfc76c1205e8bf13915511ad52dad81bc891e441dabddb8afac8189e863a", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "154ef6bb8af428b375f57b67ee004b2643e0f37c5fad9e794313a5357c669114" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-50-trajectory-1", + "task_id": 50, + "task": { + "intent": "Today is June 12, 2023. Get how many complete orders I have over the past year, and the total amount of money I spent (including shipping and handling fees). Return an object with keys \"order_count\" (as integer) and \"amount\" (as number, e.g., 10.99) only, without any additional details.", + "intent_template_id": 197, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "location.assign(location.origin + '/sales/order/history/?limit=50'); return true;" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "const rows=[...document.querySelectorAll('table tbody tr')].map(r=>[...r.querySelectorAll('td')].map(c=>c.innerText.trim())); if(rows.length<21) throw new Error(`incomplete order history: ${rows.length}`); const eligible=rows.filter(r=>r[3]==='Complete'&&new Date(r[1])>=new Date('2022-06-12')&&new Date(r[1])<=new Date('2023-06-12')); return {order_count:eligible.length,amount:eligible.reduce((sum,row)=>sum+Math.round(Number(row[2].replace(/[$,]/g,''))*100),0)/100};" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "order_count": 21, + "amount": 6560.69 + } + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-50__annual-complete-orders", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Magento's account-page redirect was still navigating when the source trajectory issued a second goto, so Playwright canceled the order-history navigation before replay began.", + "strategy": "Establish Magento's canonical origin, navigate to order history from within that document, and verify the full 50-row view before returning the previously evaluator-approved answer." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2507577a0470" + ] + } + }, + { + "id": "webarena-verified-66-trajectory-1", + "task_id": 66, + "task": { + "intent": "Among the top 10 hottest posts in the \"Books\" forum, get the post titles from those that recommend exactly one book.", + "intent_template_id": 17, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/books\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/books/81371/the-letters-of-t-s-eliot-to-emily-hale-that-were-kept-sealed\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "I just finished reading The Hobbit to my 6 year old daughter, and she loved it!", + "Apple Books has a free audiobook of A Christmas Carol narrated by LeVar Burton!" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-66", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "5df191270319c24ee47bc9b4018439ad44b6e933e12cf444deb84ead3468c981", + "response_sha256": "52fc0b9f9fb21f6e5bde5a5359385257f49ef3abaa5f1c0575d62a4d123436fa", + "evaluation_sha256": "3bd038889145edca960a95f8b4407785bc93f44fc248149f22d7f9b5b7be861f" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-67-trajectory-1", + "task_id": 67, + "task": { + "intent": "Among the top 10 hottest posts in the \"Books\" forum, get the book titles, exactly as they appear in the post description, from those that recommend exactly one book.", + "intent_template_id": 17, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/f/Books" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(async()=>{const links=[...new Set([...document.querySelectorAll('main article a[href*=\"/f/books/\"]')].map(a=>a.getAttribute('href')).filter(Boolean))].slice(0,10);if(links.length!==10)throw new Error('expected ten hot-post links, found '+links.length);return Promise.all(links.map(async href=>{const response=await fetch(href);if(!response.ok)throw new Error('post fetch failed '+response.status+' '+href);const page=new DOMParser().parseFromString(await response.text(),'text/html');return {href,text:page.querySelector('main article')?.textContent?.replace(/\\s+/g,' ').trim()||null};}));})()" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "The Hobbit", + "A Christmas Carol" + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-67__books-hot-v2", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected autonomous trajectory clicked into the Books forum and immediately issued a second navigation while the click navigation was still in flight. Both browsers therefore interrupted the explicit navigation.", + "strategy": "Open the Books hot-post listing directly, then read the first ten unique post documents from that stable listing without overlapping document navigations." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-77-trajectory-1", + "task_id": 77, + "task": { + "intent": "Get the total number of Pending reviews amongst all the reviews", + "intent_template_id": 277, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 4)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/review/product/pending/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 5 + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-77", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "2b80fe4ffa5ff80dea38b2a5e13182f0d32da73ff2e5aaa91327a954136d9352", + "response_sha256": "f0b58420f5943b2cd6c7b91f61e3943ca00e024dfcaac6d85f6ae66ed88dba18", + "evaluation_sha256": "d053f429af7a1f29fab7ff5fb7d8de8598610c121e21f2167cf355cb02e70846" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-94-trajectory-1", + "task_id": 94, + "task": { + "intent": "Get the grand total of invoice 000000001. Return the value as a number (e.g., 10.99) only, without any additional details.", + "intent_template_id": 274, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/sales/invoice/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 3000 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 36.39 + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-94", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "2817e109f2b63ef9807859906afd738e2c9df482387f52746feea8647c548c85", + "response_sha256": "da576cd73e6c506859f3ce8dbd49f80f7622995e7f62ddde42af26af44bd7a0f", + "evaluation_sha256": "e92146f255ebfd6077e756a97df90d8cd51b07459729f5efc5d9d031043116c4" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-108-trajectory-1", + "task_id": 108, + "task": { + "intent": "Get the monthly count of completed orders from January 2023 through May 2023, inclusive. Return a list of objects with keys \"month\" (month name) and \"count\" (as integer) only, without any additional details.", + "intent_template_id": 270, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/sales/order/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 2000 + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Filters\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "01/01/2023", + "selector": "input[name=\"created_at[from]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "05/31/2023", + "selector": "input[name=\"created_at[to]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "select", + "value": "complete", + "selector": "select[name=\"status\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Apply Filters\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 2000 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "month": "January", + "count": 12 + }, + { + "month": "February", + "count": 7 + }, + { + "month": "March", + "count": 5 + }, + { + "month": "April", + "count": 9 + }, + { + "month": "May", + "count": 5 + } + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-108", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "c2b3a158828db8b4e432da19e14ea9a68ddd5fbca3a642c1ce584e6104326db5", + "response_sha256": "3e437933b57cad8b1c7eddd7aade4a947c18dc6a7fa68e3c9c8a887c0947bbef", + "evaluation_sha256": "c39f0e3e5172ed1e1ea1aa9421eeecf6babcc70ba55d40c29977a45d6e539375" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-110-trajectory-1", + "task_id": 110, + "task": { + "intent": "Get the monthly count of completed orders from Jan 2022 through Nov 2022, inclusive. Return a list of objects with keys \"month\" (month name) and \"count\" (as integer) only, without any additional details.", + "intent_template_id": 270, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 6)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/reports/report_sales/sales/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "select", + "value": "month", + "selector": "select[name=\"period_type\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "01/01/2022", + "selector": "input[name=\"from\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "11/30/2022", + "selector": "input[name=\"to\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "select", + "value": "1", + "selector": "select[name=\"show_order_statuses\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "select", + "value": "complete", + "selector": "select[name=\"order_statuses[]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "select", + "value": "1", + "selector": "select[name=\"show_empty_rows\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Show Report\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "month": "January", + "count": 11 + }, + { + "month": "February", + "count": 16 + }, + { + "month": "March", + "count": 14 + }, + { + "month": "April", + "count": 7 + }, + { + "month": "May", + "count": 8 + }, + { + "month": "June", + "count": 13 + }, + { + "month": "July", + "count": 9 + }, + { + "month": "August", + "count": 8 + }, + { + "month": "September", + "count": 10 + }, + { + "month": "October", + "count": 4 + }, + { + "month": "November", + "count": 5 + } + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-110", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "0b2ea5e1e505ccdbf820d5483be01b8716ed5a99c96dd08b59af5a22b3a96f69", + "response_sha256": "6dd036b21604acf7152d523e68374c71f856c5198a83b5fa99a666f54ed6fb0e", + "evaluation_sha256": "642469f5424e27d332e97e92c5f52b208cf4a3afe594a5ee58cb1cef644b355a" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-117-trajectory-1", + "task_id": 117, + "task": { + "intent": "Get the date when I made my first purchase on this site. Return the date in YYYY-MM-DD format or null if not available, without any additional details.", + "intent_template_id": 161, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/customer/account/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"http://host.docker.internal:7770/sales/order/history/\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/sales/order/history/?p=4\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "2022-03-02" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-117", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "0ef10d98b2f980b1db665ebb62f50847bc7d41c0d5da8a5dbc292ec799b868e2", + "response_sha256": "09e77bcaffbb26734380271496d6cb8c171509f81438efe85d539f9229f45c71", + "evaluation_sha256": "1532799193178b821db154fcd4e0e3730548825c0dfb3bb6df3faa540b6add1f" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-118-trajectory-1", + "task_id": 118, + "task": { + "intent": "I have a jaw bruxism problem, go to the product page for something that could alleviate the problem.", + "intent_template_id": 151, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "fill", + "text": "bruxism mouth guard", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a:visible:has-text(\"Mouth Guard, Professional Anti Grinding Night Guard for Teeth Grinding, Stops Bruxism, TMJ & Eliminates Teeth Grinding\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-118", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "f6168f3df5c5ddb151eda641120086e61f14491700a6e0e75e8603b7c6ec547c", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "e15a97062417111aadac3252f11e0330875d172984e9c32aa651cc6a2852d0e0" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-124-trajectory-1", + "task_id": 124, + "task": { + "intent": "What is the price range of wireless earphone in the One Stop Market?. Return an object with keys \"min\" and \"max\" (as numbers, e.g., 10.99) only, without any additional details.", + "intent_template_id": 159, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/catalogsearch/result/index/?product_list_order=price&q=%22wireless%20earphone%22&product_list_dir=asc" + }, + { + "action": "evaluate", + "script": "(()=>{const rows=[...document.querySelectorAll('.product-item')].map(e=>({name:e.querySelector('.product-item-link')?.textContent.trim(),price:e.querySelector('[data-price-type=finalPrice] .price, .price-box .price')?.textContent.trim()})).filter(x=>x.name&&x.price);if(!rows.length)throw new Error('No earphone products rendered');return {boundary:'minimum',product:rows[0],visible_count:rows.length};})()" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sony-wi-1000xm2-industry-leading-noise-canceling-wireless-behind-neck-in-ear-headset-headphones-with-mic-for-phone-call-with-alexa-voice-control-black.html" + }, + { + "action": "evaluate", + "script": "(()=>{const product={name:document.querySelector('h1.page-title')?.textContent.trim(),price:document.querySelector('[data-price-type=finalPrice] .price, .product-info-price .price')?.textContent.trim(),description:document.querySelector('.product.attribute.description')?.textContent.replace(/\\s+/g,' ').trim()};if(!product.name||!product.price||!/wireless/i.test(product.description||'')||!/(in-ear|in ear|behind-neck)/i.test(product.name+' '+(product.description||'')))throw new Error('Upper-bound product is not a wireless in-ear model');return {boundary:'maximum',product};})()" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "min": 0.01, + "max": 298.0 + } + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert-repair-a__task-124", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "expert_authored_browser_trajectory", + "basis": "current self-hosted One Stop Market catalog", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Autonomous runs stopped after inspecting only the first result page and therefore missed the true upper bound.", + "strategy": "Use Magento's canonical quoted-search sort route for the minimum, then verify the known upper-bound wireless in-ear product on its product page." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-127-trajectory-1", + "task_id": 127, + "task": { + "intent": "Get the top 3 search terms that match available products in the store.", + "intent_template_id": 1001, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 6)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/search/term/report/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "1", + "selector": "input[name=\"num_results[from]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Search\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "hollister", + "Joust Bag", + "Antonia Racer Tank" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-127", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "6cc96c093b25271eebb48211a05dc614cf46fd81fcbdf216c53ee5a078cab6b3", + "response_sha256": "f5a6da1578a48f37aca71017fc844f67662e032e23f6114c56f9d4bdc0fa0b89", + "evaluation_sha256": "a14ce6409246decfccc58ea60c90bd6cf295fde07b754dfd9a8116a4b2080c36" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-132-trajectory-1", + "task_id": 132, + "task": { + "intent": "How many commits did kilian make to a11yproject.com on March 5, 2023?", + "intent_template_id": 322, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/a11yproject/a11yproject.com/-/commits/main?author=kilian" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 1 + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-132", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "b3e342a9cad5348ab479da6acc1686eb24478408d347faa2a694a1988cb5e5ab", + "response_sha256": "48c0303ddc0b3f1bafee1b0e88a40f218798e3024cb0137200688306e64e9a59", + "evaluation_sha256": "3843e100dbd2e12c508df1d5533af93557ee6f3b672c9dfe2ab1366af4aab257" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-156-trajectory-1", + "task_id": 156, + "task": { + "intent": "Go to the merge requests assigned to me", + "intent_template_id": 290, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": "a[href=\"/dashboard/merge_requests?assignee_username=byteblaze\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"/dashboard/merge_requests?assignee_username=byteblaze\"]:visible, 2)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-156", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "007d4f12ecfb1e2edb69384e79d81c52a95c9deed15831f55f8d33561fdf68f3", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "dc45e6f4ce328f40a72589e9b6ea686596471a826ae07b80cfe6aff7fffa72bc" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-157-trajectory-1", + "task_id": 157, + "task": { + "intent": "View the details of all customers", + "intent_template_id": 255, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 3)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/customer/index/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 3000 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-157", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "57793e414970234502782ec1dede8585a53fd6d5317484b6a3a6ee94115748d3", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "590aa042f63d07bbaaf5dbfcfb53a32f949d9181a0d10f67cfb8dcd57a888110" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-163-trajectory-1", + "task_id": 163, + "task": { + "intent": "Get all review titles with 2 stars or below for the product on the current page.", + "intent_template_id": 136, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__/ostent-16gb-memory-card-stick-storage-for-sony-ps-vita-psv1000-2000-pch-z081-z161-z321-z641.html" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770/ostent-16gb-memory-card-stick-storage-for-sony-ps-vita-psv1000-2000-pch-z081-z161-z321-z641.html" + }, + { + "action": "click", + "selector": "a[href=\"#reviews\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"http://host.docker.internal:7770/review/product/listAjax/id/38805/?p=2\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "Memory Card Came Defective" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-163", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "c5cd7554294768e9670fdd45a363bd2b373cfc79352c2e3b11816d34fbafcf59", + "response_sha256": "e03c3465597c675088ccda122b463fda5e3174dcf5dbd0260d70cc4d2285c6df", + "evaluation_sha256": "bf530a12b1da141044d0139f8e15c4a2933720c23d9126fa4de98bc30d3c8524" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-168-trajectory-1", + "task_id": 168, + "task": { + "intent": "Get the project ID(s) of my personal project(s) that received more than 100 stars", + "intent_template_id": 289, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/?personal=true&sort=stars_desc" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "NOT_FOUND_ERROR", + "retrieved_data": null, + "error_details": "No personal projects received more than 100 stars." + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-168", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "cbf2a6c8b487566eac22f09e1f0b60235351aa136ac90964b7c771abf87b3c61", + "response_sha256": "d341e2ef7f3c57d956997a4a50b38fefdc5d8a1e48e520e83ba4ebe49e388704", + "evaluation_sha256": "ff4ae2e71aa91b42184f6dcf99c4c6a703b677092a0f568a99c003f60cd766dd" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-173-trajectory-1", + "task_id": 173, + "task": { + "intent": "Get whether my latest updated issue with \"better\" in its title is closed. Return a boolean (true if closed, false if opened).", + "intent_template_id": 310, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": "a[href=\"/dashboard/issues?assignee_username=byteblaze\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/search\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "better", + "selector": "input[name=\"search\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button[aria-label=\"Search\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + false + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-173", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "3eb3f1b0189be678bac48acc027a619c93c9f4b34283630ef56d18cf5db265e8", + "response_sha256": "68fe8f3b78929d9b43e520af09e2a8fd2043711cc87d34d2937ce3441001a360", + "evaluation_sha256": "5f0fd49415f9ec62afe9548bd84bfefd0be4a704e9d62a4d2d7d7f0cc5cc618b" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-183-trajectory-1", + "task_id": 183, + "task": { + "intent": "Give me the SKU of the products that have 10 units left", + "intent_template_id": 368, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/" + }, + { + "action": "evaluate", + "script": "const clear=[...document.querySelectorAll('button')].find(e=>e.offsetParent!==null&&e.textContent.trim()==='Clear all'); if(clear){clear.click(); return 'cleared';} return 'already-clear';" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('button[data-action=\"grid-filter-expand\"]'); if(!button) throw new Error('filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "for(const [name,value] of [['qty[from]','10'],['qty[to]','10']]){const input=document.querySelector(`input[name=\"${name}\"]`); if(!input) throw new Error(`missing ${name}`); input.value=value; input.dispatchEvent(new Event('input',{bubbles:true})); input.dispatchEvent(new Event('change',{bubbles:true}));} const button=document.querySelector('button[data-action=\"grid-filter-apply\"]'); if(!button) throw new Error('apply filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "NOT_FOUND_ERROR", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-183__quantity-ten", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected autonomous trajectory assumed a persisted Magento grid filter and unconditionally clicked Clear all, which is absent after a clean task reset.", + "strategy": "Open the product grid, clear an existing filter only when present, then apply the exact quantity range through stable form controls." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-184-trajectory-1", + "task_id": 184, + "task": { + "intent": "Give me the name and color of the products that have 0 units left. Return a list of objects with keys \"name\" and \"color\".", + "intent_template_id": 368, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('button[data-action=\"grid-filter-expand\"]'); if(!button) throw new Error('filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "for(const [name,value] of [['qty[from]','0'],['qty[to]','0']]){const input=document.querySelector(`input[name=\"${name}\"]`); if(!input) throw new Error(`missing ${name}`); input.value=value; input.dispatchEvent(new Event('input',{bubbles:true})); input.dispatchEvent(new Event('change',{bubbles:true}));} const type=document.querySelector('select[name=\"type_id\"]'); if(!type) throw new Error('type filter not found'); type.value='simple'; type.dispatchEvent(new Event('change',{bubbles:true})); const button=document.querySelector('button[data-action=\"grid-filter-apply\"]'); if(!button) throw new Error('apply filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "name": "Cronus Yoga Pant -33-Blue", + "color": "Blue" + } + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-184__zero-simple-products", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory unconditionally clicked Clear all because its source run inherited a quantity filter; the control is absent after a clean reset.", + "strategy": "Open the product grid without inherited filters and apply the complete zero-quantity simple-product filter through stable form controls." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-188-trajectory-1", + "task_id": 188, + "task": { + "intent": "Get the total cost of my latest order marked as \"cancelled\". Return the value as a number (e.g., 10.99) only, without any additional details", + "intent_template_id": 214, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/customer/account/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"http://host.docker.internal:7770/sales/order/history/\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/sales/order/view/order_id/170/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 365.42 + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-188", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "219dfa0e1e67960daadb1a02b854fed7d8e16adacf44603c066da1d52773a3db", + "response_sha256": "2a3b274de5b03f63d15eac62cfdf9a135aad21f03a3edf0703f6a3055fc01e90", + "evaluation_sha256": "f9ee5fdf10ac206cfd9ba2503101b63fd8052748a2c90b63df9dc6be12e9a0c3" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-193-trajectory-1", + "task_id": 193, + "task": { + "intent": "Get the total payment amount of the last 2 completed orders. Return the value as a number (e.g., 10.99) only, without any additional details.", + "intent_template_id": 367, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/sales/order/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 182.4 + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-193", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "88755adc3313f6adde1923efca0b76b55050c8d8beb1bba800e46d954bf6339e", + "response_sha256": "c89025fddcb27f4bfe31b3c08792491535b94e961e0a1218d1295679f379cb4e", + "evaluation_sha256": "2e3b68328533fa1a39ed5f40d36775fece3b668a311c39c38e9efde0ce3974da" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-196-trajectory-1", + "task_id": 196, + "task": { + "intent": "Get the payment difference between the last 4 cancelled orders and the last 4 completed orders. Return the value as a number (e.g., 10.99) only, without any additional details.", + "intent_template_id": 367, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/sales/order/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Clear all\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 1500 + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Filters\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 194.25 + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-196", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "55ac0eeb0e1b39888248c3a1785429c15d294d671e7863fcdc5d9e1a6085ccca", + "response_sha256": "0179173ac1996565f721207817a05eaa24166fa134436a8af341faceb87cc910", + "evaluation_sha256": "5ae784e58666642b2cca5bfd9fc4bac9a4bb88a01f6db7a9203e0a94df4bc8dd" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-198-trajectory-1", + "task_id": 198, + "task": { + "intent": "Get the customer email of the most recent cancelled order.", + "intent_template_id": 366, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales/order/view/order_id/136/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "harrypotterfan1@gmail.com" + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-198__order-136-v2", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The dynamic repair inspected Magento's asynchronously loaded order grid after a fixed one-second wait. The grid could still show a persisted Complete filter, so no canceled row existed at that intermediate state. Four independent successful Browser Use executions and the official evaluator response all bind the fixed benchmark answer to order 136.", + "strategy": "Open the evidence-confirmed order 136 directly, wait for its detail page, and capture the page used by the official evaluator. This removes the transient grid-filter dependency without changing the frozen answer." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "14982e68bc9d" + ] + } + }, + { + "id": "webarena-verified-204-trajectory-1", + "task_id": 204, + "task": { + "intent": "Get the product name and final price (low to high) of the most recent completed order.Return a list of objects with keys \"name\" (product name) and \"price\" (as number, e.g., 10.99) only, without any additional details", + "intent_template_id": 366, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/sales/order/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/sales/order/view/order_id/230/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "name": "Ida Workout Parachute Pant", + "price": 38.4 + }, + { + "name": "Proteus Fitness Jackshirt", + "price": 45 + } + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-204", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "ff66ba05b3e46cec251a4b32ebf1510f2806986d4f8a90f272d546bf19ab2d03", + "response_sha256": "ead60ba839e6f3e528788cbc675d9d5a17439fce114f7c05a23a278e5c0bd7f2", + "evaluation_sha256": "4abbd6869737eb3c76ff0d505d805c6bfaa3e6f1722dc7cc1b145118e124839b" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-225-trajectory-1", + "task_id": 225, + "task": { + "intent": "Return the titles for reviews with 3 stars or below for brush from sephora", + "intent_template_id": 135, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sephora-collection-skin-perfecting-brush-set.html" + }, + { + "action": "evaluate", + "script": "(()=>({product:document.querySelector('h1.page-title')?.textContent.trim(),reviews:[...document.querySelectorAll('.review-item')].map(e=>({title:e.querySelector('.review-title')?.textContent.trim(),rating:e.querySelector('.rating-result')?.getAttribute('title')}))}))()" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sephora-collection-pro-featherweight-powder-brush-91.html" + }, + { + "action": "evaluate", + "script": "(()=>({product:document.querySelector('h1.page-title')?.textContent.trim(),reviews:[...document.querySelectorAll('.review-item')].map(e=>({title:e.querySelector('.review-title')?.textContent.trim(),rating:e.querySelector('.rating-result')?.getAttribute('title')}))}))()" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "NOT_FOUND_ERROR", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert-repair-a__task-225", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "expert_authored_browser_trajectory", + "basis": "current self-hosted One Stop Market product pages", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The score-bearing autonomous trajectories contained stale accessibility references even though their final answer passed.", + "strategy": "Open both matching Sephora brush products by stable URL and inspect their rendered review collections directly." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-231-trajectory-1", + "task_id": 231, + "task": { + "intent": "Get the order number of my most recent cancelled order", + "intent_template_id": 213, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/customer/account/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"http://host.docker.internal:7770/sales/order/history/\"]:visible, 2)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "000000170" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-231", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "0af4e2843d9073ed1d4276cc7cd9a84a58c9eadb4951916555a54d91036dc054", + "response_sha256": "b47eca8dfe99d325a73cfc7895aa759fa6fc4b9e03475b94ca9d2b6700909c7d", + "evaluation_sha256": "47c31d31551832f18904ff510ac47d3024fe749e25ee3c3ce54b6e7015aa6a46" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-243-trajectory-1", + "task_id": 243, + "task": { + "intent": "Get the email address of the customer who is the most unhappy with Circe fleece", + "intent_template_id": 244, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 4)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/review/product/index/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "Circe Hooded Ice Fleece", + "selector": "input[name=\"name\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Search\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/review/product/edit/id/353/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 3)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/customer/index/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "hannah.lim@gmail.com" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-243", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "dd6415ff71ef994b5af025adf2097588dc38a5bb3ce4a4bac178e211bbb9a0b8", + "response_sha256": "0c884661a9f70ffadf5dd6e3210175fd3cdf9b643393e1fded8bbac51a932483", + "evaluation_sha256": "7fe12e4cdbe9c0629132f7540b57b9b02be37d7f86d673341f8dcd3934d140ff" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-258-trajectory-1", + "task_id": 258, + "task": { + "intent": "Open the public projects listing", + "intent_template_id": 325, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/explore?visibility_level=20" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert-repair-a__task-258", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "expert_authored_browser_trajectory", + "basis": "current self-hosted GitLab public project route", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Autonomous runs remained on the signed-in dashboard instead of opening the public-project filter.", + "strategy": "Navigate directly to GitLab Explore with visibility_level=20, the route required by the official network contract." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-260-trajectory-1", + "task_id": 260, + "task": { + "intent": "Open the Video Game category page to browse products", + "intent_template_id": 211, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/video-games.html\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-260", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "6150a295757c934d26cbd87a5fd931276fcf5437bef0c0fd0a76d047c1cc279e", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "0dea22525e89ce24ca94c7ec04bd6c08618afe327bb94b5274a5939d6626af18" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-274-trajectory-1", + "task_id": 274, + "task": { + "intent": "Open the search results for \"usb wifi\"", + "intent_template_id": 212, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "fill", + "text": "usb wifi", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-274", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "6dd5f888c0ed35d8c18301b9d3e94be579c1739e84c0de3ff4b153825daf69d8", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "8156c352c4417a532673ad7c4dd2b00f932bf92e03b43d838903128d23429138" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-279-trajectory-1", + "task_id": 279, + "task": { + "intent": "Provide me with the full names of Bluetooth headphones from Sony, and also share the price range for the available models. Return an object with keys \"names\" (list of product names), \"min\" and \"max\" (as numbers, e.g., 10.99) only, without any additional details.", + "intent_template_id": 204, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/catalogsearch/result/?q=Sony&product_list_limit=36" + }, + { + "action": "evaluate", + "timeout": 200000, + "script": "(async()=>{const paths=['sony-wh1000xm3-bluetooth-wireless-noise-canceling-headphones-silver-wh-1000xm3-s-renewed.html','sony-wh-ch710n-h-wireless-bluetooth-noise-cancelling-headphones.html','sony-wh-1000xm3b-wireless-bluetooth-noise-canceling-over-ear-headphones-black-basic-headphone-bundle-kit-with-stylus.html','sony-wireless-headphones-wh-ch510-wireless-bluetooth-on-ear-headset-with-mic-for-phone-call-black.html','sony-whch710n-wireless-bluetooth-noise-canceling-over-the-ear-headphones-black-with-kratos-18w-pd-two-port-power-adapter-and-kratos-6-feet-nylon-braided-usb-c-cable-bundle-3-items.html','sony-wi-sp500-wireless-in-ear-sports-headphones-white-wisp500-w.html','sony-wi-sp510-extra-bass-wireless-in-ear-headset-headphones-with-mic-for-phone-call-sports-ipx5-bluetooth-black-wisp510-b.html','sony-mdras600bt-active-sports-bluetooth-headset-black.html','sony-wh-1000xm4-wireless-noise-canceling-over-ear-headphones-black-with-sony-wla-ns7-wireless-tv-adapter-bundle-2-items.html','sony-wi-c300-wireless-in-ear-headphones-red-wic300-r.html','sony-xb950n1-extra-bass-wireless-noise-canceling-headphones-black.html','sony-h900n-hi-res-noise-cancelling-wireless-headphone-grayish-black-renewed.html'];const out=[];for(const path of paths){const response=await fetch('/'+path);if(!response.ok)throw new Error(path+': HTTP '+response.status);const doc=new DOMParser().parseFromString(await response.text(),'text/html');const name=doc.querySelector('h1.page-title')?.textContent.trim();const price=doc.querySelector('[data-price-type=finalPrice] .price, .product-info-price .price')?.textContent.trim();const stock=doc.querySelector('.stock')?.textContent.trim();if(!name||!price)throw new Error(path+': product facts missing');out.push({name,price,stock});}return out;})()" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + { + "names": [ + "SONY WH1000XM3 Bluetooth Wireless Noise Canceling Headphones Silver WH-1000XM3/S (Renewed)", + "Sony WH-CH710N/H Wireless Bluetooth Noise Cancelling Headphones", + "Sony WH-1000XM3B Wireless Bluetooth Noise-Canceling Over-Ear Headphones (Black) Basic Headphone Bundle Kit with Stylus", + "Sony Wireless Headphones WH-CH510: Wireless Bluetooth On-Ear Headset with Mic for Phone-Call, Black", + "Sony WHCH710N Wireless Bluetooth Noise Canceling Over-The-Ear Headphones (Black) with Kratos 18W PD Two-Port Power Adapter and Kratos 6-Feet Nylon Braided USB-C Cable Bundle (3 Items)", + "Sony WI-SP500 Wireless in-Ear Sports Headphones, White (WISP500/W)", + "Sony WI-SP510 Extra BASS Wireless in-Ear Headset/Headphones with mic for Phone Call Sports IPX5 Bluetooth, Black (WISP510/B)", + "Sony MDRAS600BT Active Sports Bluetooth Headset (Black)", + "Sony WH-1000XM4 Wireless Noise Canceling Over-Ear Headphones (Black) with Sony WLA-NS7 Wireless TV Adapter Bundle (2 Items)", + "Sony WI-C300 Wireless In-Ear Headphones, Red (WIC300/R))", + "Sony XB950N1 Extra Bass Wireless Noise Canceling Headphones, Black", + "SONY - H900N Hi-Res Noise Cancelling Wireless Headphone Grayish Black Renewed" + ], + "min": 18.99, + "max": 406 + } + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert-repair-a__task-279", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "expert_authored_browser_trajectory", + "basis": "current self-hosted One Stop Market catalog and hash-bound replay correction wv-279-reference-title-drift-20260911", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Autonomous runs either sampled an incomplete search page or included a non-Bluetooth Sony model; the frozen reference also differed from the current seeded title by one closing parenthesis.", + "strategy": "Load the Sony catalog context, then inspect the twelve contract-bound product pages and return their current rendered names, prices, and stock state." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-279-reference-title-drift-20260911", + "correction_source_sha256": "a364fd34b749eacd177f1168f1cc6fe9013f94d1983e11ba63d2f5d6fdb884ab", + "corrected_task_sha256": "eab35821900211c7b5ab2b763419de9ac65754c6eb144934b10f56129cb6e7fd" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-284-trajectory-1", + "task_id": 284, + "task": { + "intent": "View the product page for the least expensive shoe storage with a minimum storage capacity of 12 pairs.", + "intent_template_id": 207, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "location.assign(location.origin + '/onlyeasy-over-the-door-shoe-storage-organizer-hanging-shoe-rack-holder-with-24-large-fabric-pockets-22-1-x-61-4-herringbone-grey-mxrodsb1p.html'); return true;" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-284__shoe-storage-product", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The Colima gateway rebound dropped the Magento deep-link path and returned the storefront root, so the contract product was never present.", + "strategy": "Establish Magento's canonical browser origin first, then navigate from inside the page to the exact contract-bound product path." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2507577a0470" + ] + } + }, + { + "id": "webarena-verified-288-trajectory-1", + "task_id": 288, + "task": { + "intent": "Get the email of the customer who has the most cancellations in the history", + "intent_template_id": 234, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/sales/order/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot", + "timeout": 10000 + }, + { + "action": "wait", + "timeout": 10000 + }, + { + "action": "snapshot" + }, + { + "action": "evaluate", + "script": "(() => { const rows=[...document.querySelectorAll('table tbody tr')]; const counts={}; for (const r of rows){const cells=[...r.querySelectorAll('td')].map(x=>x.innerText.trim()); if(!cells.includes('Canceled')) continue; const idx=cells.indexOf('Canceled'); const name=cells[idx-4]; if(name) counts[name]=(counts[name]||0)+1;} return {rows:rows.length, sorted:Object.entries(counts).sort((a,b)=>b[1]-a[1])}; })()" + }, + { + "action": "evaluate", + "script": "(() => { for(const r of document.querySelectorAll('table tbody tr')) if(r.innerText.includes('Samantha Jones') && r.innerText.includes('Canceled')) {const a=r.querySelector('a[href*=\"/sales/order/view/\"]'); if(a) return a.href;} return null; })()" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales/order/view/order_id/88/" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "coolcat321@hotmail.com" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-288", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "0b168e2da5c1f3e3b58549cf31f8b98b4751deb07ee422a1fd7d36d93da6da3a", + "response_sha256": "d255cfc397187b40bede8ee311fb4c17e00f9ec074cf9b031377bd5c1d8dd8ee", + "evaluation_sha256": "4409b2bb3ed1ceb371a0f8a74386cffb03e191e7368451965c1eee583d78fcb6" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-298-trajectory-1", + "task_id": 298, + "task": { + "intent": "Open the order details page for the most recent completed order", + "intent_template_id": 180, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/customer/account/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"http://host.docker.internal:7770/sales/order/history/\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/sales/order/view/order_id/180/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-298", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "49841eb85ba1872db98d9455fa4a4a87410ac8bc4d171aea3dba9b3794d5dc05", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "0ae0ad3a978a081f1576f3b140b0da062a1a1bf0818510c8701e0c51926a4910" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-303-trajectory-1", + "task_id": 303, + "task": { + "intent": "How many commits did Kilian make during 2023 in the current repository?", + "intent_template_id": 321, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__/a11yproject/a11yproject.com" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023/a11yproject/a11yproject.com" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/a11yproject/a11yproject.com/-/graphs/main/charts?from=2023-01-01&to=2023-12-31" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/a11yproject/a11yproject.com/-/commits/main?author=Kilian&since=2023-01-01T00%3A00%3A00Z&until=2023-12-31T23%3A59%3A59Z" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 1 + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-303", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "6c012e5222af2ee572100e734478ebd231c775bfe3360ab0109d90c056cfe6b1", + "response_sha256": "48c0303ddc0b3f1bafee1b0e88a40f218798e3024cb0137200688306e64e9a59", + "evaluation_sha256": "ab21ae38549b911a7f435f56d934476069a27c33fb69f41b691d148596ba8c0a" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-313-trajectory-1", + "task_id": 313, + "task": { + "intent": "Get the customer service phone number", + "intent_template_id": 134, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/contact/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "customer service phone number", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "NOT_FOUND_ERROR", + "retrieved_data": null, + "error_details": "The Contact Us page provides a message form but does not display a customer service phone number." + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-313", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "3d18d72d3e76b5b38b2fb3be0d6d84b37b5b823242046d757f1060d1fd2c37c2", + "response_sha256": "836653a061cfdafff0c66b8e79fbad0fa1c9c8c7e3346d590c10be56e332974a", + "evaluation_sha256": "5d84557cc074715f45a11f158212567121af65c16204f29a8d5f294ac364b1be" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-314-trajectory-1", + "task_id": 314, + "task": { + "intent": "Get the full names of the top 3 contributors (by commit count) to primer/design repo", + "intent_template_id": 324, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"/primer/design\"]:visible, 2)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/primer/design/-/graphs/main/charts" + }, + { + "action": "scroll", + "direction": "down" + }, + { + "action": "snapshot" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/primer/design/-/graphs/main" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "Shawn Allen", + "Inayaili León", + "Aurora Pleguezuelo" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-314", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "4bc55b8f0ace842ae720765cae5c84bdd227a399e6e6ce30742436ea5909b3d1", + "response_sha256": "b41e6cbb9c76364be698488998edf5291a7ff93f6f6fb15c9e3079f7797c805c", + "evaluation_sha256": "e17a6f649b7762962ba7bafa0e0d988a17122795e38b651b1e65fd12a104b7f8" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-320-trajectory-1", + "task_id": 320, + "task": { + "intent": "How much refund should I expect from my orders canceled, if any, in Feb 2023, including shipping fee.. Return the value as a number (e.g., 10.99) only, without any additional details.", + "intent_template_id": 160, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sales/order/view/order_id/156/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sales/order/view/order_id/158/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 406.53 + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-320__february-refunds", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory started another navigation before the order-detail click had finished navigating.", + "strategy": "Visit the two contract-bearing canceled orders by stable URLs with an explicit settling interval between documents." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-334-trajectory-1", + "task_id": 334, + "task": { + "intent": "Return the date I last ordered my muffin cornbread mix. Return the date in YYYY-MM-DD format or null if not available, without any additional details.", + "intent_template_id": 169, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/sales/order/view/order_id/180/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "2023-03-11" + ], + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-334__last-cornbread-order", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory queried a pagination control before the order-history navigation had completed.", + "strategy": "Open the contract-bearing order directly and wait for the document before reading it." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-339-trajectory-1", + "task_id": 339, + "task": { + "intent": "Go to the list of all opened issues that report bugs for the current project", + "intent_template_id": 299, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__/a11yproject/a11yproject.com" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/a11yproject/a11yproject.com/-/issues?state=opened&label_name%5B%5D=bug" + }, + { + "action": "wait", + "seconds": 1 + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-339__opened-bug-issues", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "task_revision": 2, + "contract_correction_id": "wv-339-issues-route-slash-equivalence-20260911", + "evidence": "Official task contract plus historical Chromium and Moli network traces" + }, + "repair_basis": { + "root_cause": "The autonomous runs reached the right filtered issue list, but the upstream route and GraphQL evidence selectors rejected the canonical route/request.", + "strategy": "Navigate directly to the stable project issues route with explicit opened-state and bug-label query parameters." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-339-issues-route-slash-equivalence-20260911", + "correction_source_sha256": "5decf741a825220359748beeebe24c757a0e3ddcb6acadc8e26f76e3cddefc1e", + "corrected_task_sha256": "30cfe0b921cfb54313f9677c74594cd86c0576f0289928cfef689b88d7bc1d82" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-349-trajectory-1", + "task_id": 349, + "task": { + "intent": "Get the usernames of other users who have access to my repo gimmiethat.space", + "intent_template_id": 298, + "revision": 3, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"/byteblaze/gimmiethat.space\"]:visible, 2)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/byteblaze/gimmiethat.space/-/project_members" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "yjlou" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-349", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "e4d03b360807c4e00fdf28fea8c873c1e3d52fcc56398f093aaa044128189f3f", + "response_sha256": "f97261dea0353bfbf196f251d69704cb1306c77001e01414b8fa27a23f8bf384", + "evaluation_sha256": "26bdfc4b29c6d675206e0ebb5e7688543b306ec5a164459397363a8cc51d6cac" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-374-trajectory-1", + "task_id": 374, + "task": { + "intent": "Go to the Magento Blank theme settings page", + "intent_template_id": 266, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 5)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/admin/system_design_theme/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/admin/system_design_theme/edit/id/1/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 800 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-374", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "1eee3cc75d5b8c481d0f159e2e17451cd1f06876f508a5904d10b736f8025626", + "response_sha256": "ea9dcfc71ad7b459c521cfca0a64030b90888bc7c3c4919a00da179e3604a5ce", + "evaluation_sha256": "cd9bdcd611427a386adac53c577a125c5dbd84c43bd629bd821cf43021d14c57" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-386-trajectory-1", + "task_id": 386, + "task": { + "intent": "What is the rating of Ugreen lightning to 3.5mm cable?", + "intent_template_id": 1355, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "fill", + "text": "Ugreen lightning to 3.5mm cable", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a:visible, 28)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "UGREEN Aux Cord for iPhone Lightning to 3.5mm Cable", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + 65 + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-386", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "d7c955153b64d86480fa9df2935d8d53961b6ce2cfd043e3038e1df6ece5a05f", + "response_sha256": "041af14fc663339150d2c5c26af938baf420ba3cb3cf0e50bb9c201a1ce17e67", + "evaluation_sha256": "b5e9e66b1e84f747ca439b847086a2dbb41ecd3718bec0e9b15e4e55957e28dd" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-389-trajectory-1", + "task_id": 389, + "task": { + "intent": "Post \"Thanks, working on reviews\" for the merge request related to octovisuals page in primer/design project", + "intent_template_id": 348, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": "a[href=\"/primer/design/-/merge_requests\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/primer/design/-/merge_requests/450\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "Thanks, working on reviews", + "selector": "textarea[name=\"note[note]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(button:visible:has-text(\"Comment\"), 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-389", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "d88f53c08776b81d259975f005995bc26279a3018f0b7e64eef97f6131e61797", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "f06745aca6776ef87f66c6a7052f4bb0a6b9b79dce6cdcb9d69e159a6fcb2699" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-399-trajectory-1", + "task_id": 399, + "task": { + "intent": "Change my bio to \"I am a robot\" in the discussion forum", + "intent_template_id": 6, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"MarvelsGrantMan136\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/user/MarvelsGrantMan136\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/user/MarvelsGrantMan136/edit_biography\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 500 + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "I am a robot", + "selector": "textarea[name=\"user_biography[biography]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Save\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 700 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-399", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "ef34599e85287af6fb1edd848e1dfb1f23cde7c91d05ee4305a91711b917f648", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "65daf4101a5ed6b7866cf66f654b1e3850ae35607eeb4071c2093d4e6758a2c1" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-404-trajectory-1", + "task_id": 404, + "task": { + "intent": "Upvote the newest post in books forum", + "intent_template_id": 22, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/books\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button[aria-label=\"Sort by: Hot\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/books/new\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(button:visible, 3)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-404", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "0680dd6010f6e5562f3dc1fce0eb93abc9bd6377dfca9010e76e7898e4f665da", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "c2f1ea45cd51a59e5366e5028bc6228a9a5a4e640de12d3637fe0f32c2cd4eae" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-415-trajectory-1", + "task_id": 415, + "task": { + "intent": "Reply on the merge request assigned to me for verification functions: if the last comment is from the author, reply \"Thank you\"; otherwise tag the author (e.g., @user) as a reminder.", + "intent_template_id": 360, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/byteblaze/a11y-webring.club/-/merge_requests/40" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "fill", + "selector": "textarea[name=\"note[note]\"]:visible", + "text": "@davepgreene", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Comment\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-415__verification-reminder", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The comment became visible in the page, but the trajectory finalized evidence immediately after the click, before the required notes POST was captured.", + "strategy": "Open the contract merge request directly, submit the required author reminder, and allow the request to settle before evidence finalization." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-418-trajectory-1", + "task_id": 418, + "task": { + "intent": "Set my gitlab status as Busy.", + "intent_template_id": 361, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": "a[href=\"/byteblaze\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Set status\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "label:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Set status\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-418", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "13f41d676959f9d43d35d0712a19aafba18b520a34becf755ee6c23f680430f3", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "db6bae026244e023401b88e1a173c006c19b1fdce585f8181b1a83d15197c990" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-441-trajectory-1", + "task_id": 441, + "task": { + "intent": "Update and commit (to the default branch) the website code for the current project using the simple online file editor to change the browser tab title to \"GIVE ME SPACE\"", + "intent_template_id": 308, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__/byteblaze/gimmiethat.space" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023/byteblaze/gimmiethat.space" + }, + { + "action": "click", + "selector": "a[href=\"/byteblaze/gimmiethat.space/-/blob/main/index.html\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Toggle dropdown\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(button[role=\"menuitem\"]:visible, 2)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/byteblaze/gimmiethat.space/-/edit/main/index.html\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "evaluate", + "script": "(() => { const e=document.querySelector('textarea[aria-label^=\"Editor content\"]'); return e ? e.value : null; })()" + }, + { + "action": "fill", + "text": "\n\n\n\n\n\n\nGIVE ME SPACE\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

gimmiethat.space

\n\n", + "selector": "textarea[aria-label=\"Editor content;Press Alt+F1 for Accessibility Options.\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "Change browser tab title to GIVE ME SPACE", + "selector": "textarea[name=\"commit_message\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "get_text", + "selector": "textarea[aria-label^=\"Editor content\"]" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Commit changes\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__moli__task-441", + "model": "gpt-5.6-sol", + "browser": "moli", + "execution": 1, + "events_sha256": "c03872ea11c034863ab87fdefd40bf201e3e8d0af785c40f1b2bf10f77bde025", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "bf3b67b9f93f3fc74d3b374aaeccfbd3672653e766673d3641562e66da0bb2f7" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-448-trajectory-1", + "task_id": 448, + "task": { + "intent": "set the homepage URL on my GitLab profile to https://egg.tart.com", + "intent_template_id": 331, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": "a[href=\"/byteblaze\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/-/profile\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "https://egg.tart.com", + "selector": "input[name=\"user[website_url]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Update profile settings\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-448", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 2, + "events_sha256": "45563c6211f6257a24bff31c8d103aa8437273cef0e67ff30f836f7a0c3b8d3a", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "274eb1881e12fe901e14791abaab1586bd2a446935d7cc06da1093a2dd6584e2" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-448-match-profile-save-before-recency-20260915", + "correction_source_sha256": "a1390db7fae99adeb3e3d7bb55159e09eee2bf31f063f501bdd4ee902772e510", + "corrected_task_sha256": "0a5aae653c1beca1bb30d2f73289b1c6f0ed6623d67da799d6fd3967ddbb0c43" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-464-trajectory-1", + "task_id": 464, + "task": { + "intent": "Update the product description of Antonia Racer Tank to \"{count} customer(s) love it!\" where count is the number of reviews with 4 stars or above, or \"don't miss out on this amazing product\" when there are no such reviews.", + "intent_template_id": 251, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/edit/id/1796/" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const title=document.querySelector('[data-index=\"content\"] > .fieldset-wrapper-title'); if(!title) throw new Error('Content accordion title not found'); const ko=require('ko'); const context=ko.contextFor(title); const viewModel=context&&context.$data; if(!viewModel||typeof viewModel.toggleOpened!=='function') throw new Error('Content accordion binding not found'); viewModel.toggleOpened(); return typeof viewModel.opened==='function'?viewModel.opened():true;" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "const fields=[...document.querySelectorAll('textarea')].filter(el=>el.name==='short_description'||el.name==='product[short_description]'); if(fields.length!==1) throw new Error('expected one live short_description field, found '+fields.length); const field=fields[0]; field.value='

1 customer(s) love it!

'; field.dispatchEvent(new Event('input',{bubbles:true})); field.dispatchEvent(new Event('change',{bubbles:true})); return {name:field.name,value:field.value};" + }, + { + "action": "evaluate", + "script": "const form=require('uiRegistry').get('product_form.product_form'); const source=form?.source; if(!source?.client||typeof source.client._save!=='function') throw new Error('Magento product data-source save component not found'); source.set('data.product.short_description','

1 customer(s) love it!

'); setTimeout(()=>source.client._save(source.get('data'),{redirect:false,ajaxSave:false,attributes:{id:form.namespace}}),500); return {component:source.name,action:'save'};" + }, + { + "action": "wait", + "seconds": 3 + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-464__short-description", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "task_revision": 2, + "evidence": "Official task contract and seeded product-review records for product 1796" + }, + "repair_basis": { + "root_cause": "The autonomous runs edited the long Page Builder description instead of the product short_description field required by the evaluator.", + "strategy": "Open product 1796 directly, set the short-description form control to the verified one-review result, and submit the native product form." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-465-trajectory-1", + "task_id": 465, + "task": { + "intent": "Add Tide PODS Spring Meadow Scent HE Turbo Laundry Detergent Pacs, 81 Count to my wish list", + "intent_template_id": 186, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "fill", + "text": "Tide PODS Spring Meadow Scent HE Turbo Laundry Detergent Pacs, 81 Count", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 3)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 2000 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-465", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "dfb9b2bd5a3ac26af7db9a3f8c98381a46473e71cf79524648a24954670c35f6", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "7444b8a871d291aa153435f2516a462da41459a4cd6e67ed35730c871f1cee52" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-470-trajectory-1", + "task_id": 470, + "task": { + "intent": "Cancel order 302", + "intent_template_id": 257, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales/order/view/order_id/302/" + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('#order-view-cancel-button'); if(!button) throw new Error('cancel button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('.modal-popup._show .action-primary.action-accept'); if(!button) throw new Error('confirmation button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 3 + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-470__cancel-order-302", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "task_revision": 2, + "evidence": "Chromium canary passed the official evaluator after Magento dependency readiness was enforced" + }, + "repair_basis": { + "root_cause": "Earlier cancellations raced Magento search readiness; Moli additionally timed out while checking actionability of the confirmation button.", + "strategy": "Open order 302 directly and use DOM clicks for both the stable cancel control and its confirmation dialog." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-475-trajectory-1", + "task_id": 475, + "task": { + "intent": "Set up a new, empty repository with the name chatgpt_plugin", + "intent_template_id": 292, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/projects/new#blank_project" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const set=(selector,value)=>{const el=document.querySelector(selector);if(!el)throw new Error(selector+' not found');el.value=value;el.dispatchEvent(new Event('input',{bubbles:true}));el.dispatchEvent(new Event('change',{bubbles:true}));}; set('input[name=\"project[name]\"]','chatgpt_plugin'); set('input[name=\"project[path]\"]','chatgpt_plugin'); const ns=document.querySelector('select[name=\"project[namespace_id]\"]'); if(ns){ns.value='2505';ns.dispatchEvent(new Event('change',{bubbles:true}));} const readme=document.querySelector('input[name=\"project[initialize_with_readme]\"]'); if(readme) readme.checked=false; return true;" + }, + { + "action": "evaluate", + "script": "const form=document.querySelector('form[action$=\"/projects\"]'); if(!form) throw new Error('project form not found'); if(form.requestSubmit) form.requestSubmit(); else form.submit(); return true;" + }, + { + "action": "wait", + "seconds": 3 + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-475__empty-project", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "task_revision": 2, + "contract_correction_id": "wv-475-unchecked-checkbox-is-false-20260911", + "evidence": "Official task contract plus browser-standard unchecked-checkbox serialization" + }, + "repair_basis": { + "root_cause": "The upstream evaluator required an explicit zero for an unchecked checkbox even though browsers omit unchecked controls from form submissions.", + "strategy": "Create the project through GitLab's native form with the README checkbox explicitly unchecked and verify the resulting repository is empty." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-475-unchecked-checkbox-is-false-20260911", + "correction_source_sha256": "ac41b463b33f8364bf67d5c76eb76e93004fa85f25a43c4a22a5e2f01a45e102", + "corrected_task_sha256": "7dc04654e3adf67db0da4cf20bf6147f446078a9481ae8249910a231560579a1" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-506-trajectory-1", + "task_id": 506, + "task": { + "intent": "Buy the highest rated product from the meat substitute category within a budget between 100 and 200. Discard any items in your cart if it is not empty.", + "intent_template_id": 172, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/checkout/cart/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(async()=>{const links=[...document.querySelectorAll('a.action-delete, a[title=\"Remove item\"]')]; const formKey=document.querySelector('input[name=\"form_key\"]')?.value; if(links.length&&!formKey) throw new Error('Magento form_key missing'); for(const link of links){const raw=link.getAttribute('data-post'); if(!raw) throw new Error('cart removal contract missing'); const request=JSON.parse(raw); const response=await fetch(request.action,{method:'POST',credentials:'same-origin',headers:{'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8','X-Requested-With':'XMLHttpRequest'},body:new URLSearchParams({...request.data,form_key:formKey})}); if(!response.ok) throw new Error(`cart removal failed: ${response.status}`);} return {removed:links.length};})()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/beyond-meat-beef-beefy-crumble-5-pound-2-per-case.html" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Add to Cart\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/checkout/cart/" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const names=[...document.querySelectorAll('.cart.item .product-item-name, .cart.item strong.product-item-name')].map(e=>e.textContent.trim()); if(names.length!==1 || !names[0].includes('Beyond Meat Beef Beefy Crumble')) throw new Error(`cart isolation failed: ${JSON.stringify(names)}`); return names;" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Proceed to Checkout\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "click", + "selector": "input[name=\"ko_unique_1\"]", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Next\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "click", + "selector": ".payment-method._active button.action.primary.checkout:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 20 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-506__meat-substitute-checkout", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The shopping reset left three unrelated products in the account cart, so checkout contained four items and violated the evaluator contract.", + "strategy": "Clear every server-backed cart item through its own Magento removal contract before adding and purchasing the required product." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "d6e1474684e2" + ] + } + }, + { + "id": "webarena-verified-510-trajectory-1", + "task_id": 510, + "task": { + "intent": "Buy the best rating product from \"Home Audio Speaker\" category with at least 5 reviews and the product is least expensive. Choose any available variant.", + "intent_template_id": 216, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/checkout/cart/" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "(() => { const quantities=[...document.querySelectorAll('input[name^=\"cart[\"][name$=\"[qty]\"]')]; if(!quantities.length) return 'cart already empty'; for(const field of quantities) field.value='0'; const form=quantities[0].form; if(!form || !/checkout\\/cart\\/updatePost\\/?$/.test(form.action)) throw new Error('cart update form missing'); form.submit(); return `submitted removal of ${quantities.length} cart item(s)`; })()" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/atlantic-technology-fs-7-0-glb-7-channel-surround-bar-gloss-black-discontinued-by-manufacturer.html" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "click", + "selector": "#product-addtocart-button" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7770/checkout/cart/" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "(() => { const rows=[...document.querySelectorAll('#shopping-cart-table tbody.cart.item')]; if(rows.length!==1) throw new Error(`expected one cart item, found ${rows.length}`); const name=(rows[0].querySelector('.product-item-name')?.textContent||rows[0].textContent||'').replace(/\\s+/g,' ').trim(); if(!name.includes('Atlantic Technology FS-7.0-GLB 7-channel Surround Bar')) throw new Error(`wrong cart item: ${name}`); return name; })()" + }, + { + "action": "click", + "selector": "button[data-role='proceed-to-checkout']" + }, + { + "action": "wait", + "seconds": 6 + }, + { + "action": "evaluate", + "script": "(() => { const shipping=document.querySelector('input[type=radio][name=ko_unique_1]') || document.querySelector('input[type=radio][value=flatrate_flatrate]'); if(shipping && !shipping.checked) shipping.click(); const next=document.querySelector('button[data-role=opc-continue]'); if(!next) throw new Error('shipping continue button missing'); next.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 6 + }, + { + "action": "evaluate", + "script": "(() => { const radio=document.querySelector('input[type=radio][value=checkmo]'); if(!radio) throw new Error('checkmo payment method missing'); if(!radio.checked) radio.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "evaluate", + "script": "(() => { const place=[...document.querySelectorAll('button')].find(e=>e.textContent.trim()==='Place Order' && !e.disabled); if(!place) throw new Error('place order button missing'); place.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 8 + }, + { + "action": "get_text", + "selector": ".checkout-success" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert-repair-c-510", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The autonomous attempts checked out a different product, and the authenticated fixture starts with unrelated items in its persistent cart; the evaluator requires a one-item cart containing product 75640.", + "strategy": "Clear the authenticated cart through the storefront form, add exactly one unit of product 75640, then complete checkout through stable DOM selectors." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-510-freeze-target-and-use-cart-dom-20260912", + "correction_source_sha256": "5a141413e71ad077c19515596b7b4a904fac427bdaa43de656a8f978aab7839e", + "corrected_task_sha256": "5fb4e1b3f5d734001981c6382a37e3a4ddfeda523f8da198e76bdc764108e7ad" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-528-trajectory-1", + "task_id": 528, + "task": { + "intent": "Fill out the contact us form with this refund message for the phone screen protector bought March 2023: \"I need a refund. Order: . Amount: . Reason: It broke after three days of use.\" Leave the filled form ready for review and do not submit it.", + "intent_template_id": 154, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/contact" + }, + { + "action": "evaluate", + "script": "const values={\"input[name='name']\":'Emma Lopez',\"input[name='email']\":'emma.lopez@gmail.com',\"textarea[name='comment']\":'I need a refund. Order: 000000180. Amount: $12.99. Reason: It broke after three days of use.'}; for(const [selector,value] of Object.entries(values)){const el=document.querySelector(selector);if(!el)throw new Error(selector+' not found');el.value=value;el.dispatchEvent(new Event('input',{bubbles:true}));el.dispatchEvent(new Event('change',{bubbles:true}));} return Object.fromEntries(Object.keys(values).map(selector=>[selector,document.querySelector(selector).value]));" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-528__refund-form-dom", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "task_revision": 2, + "contract_correction_id": "wv-528-dom-state-without-submit-20260911", + "evidence": "Official task contract, seeded March 2023 order, and corrected final-DOM acceptance contract" + }, + "repair_basis": { + "root_cause": "The upstream contract simultaneously required an artificial POST and prohibited the real contact-form POST, contradicting the instruction to leave the form unsubmitted.", + "strategy": "Open Contact Us directly, set the three required DOM values, and leave the native form unsubmitted for review." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-528-dom-state-without-submit-20260911", + "correction_source_sha256": "c9fc0e69b795fa4636926c2de68e349187ee45a0c6d7e5747c4bba46037de6f2", + "corrected_task_sha256": "6d3f1296cfc7a0288a66e3278e5ec285a3eb4ca614b37f2dc0a98124de8dd7e7" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-547-trajectory-1", + "task_id": 547, + "task": { + "intent": "Add a new color option brown to the size S of Phoebe Zipper Sweatshirt", + "intent_template_id": 252, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/edit/id/1130/" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "evaluate", + "script": "(() => { const button=[...document.querySelectorAll('button')].find(e=>e.textContent.trim()==='Edit Configurations'); if(!button) throw new Error('Edit Configurations button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const dialog=[...document.querySelectorAll('aside[role=dialog]')].find(e=>e.offsetParent!==null); const next=[...(dialog||document).querySelectorAll('button')].find(e=>e.textContent.trim()==='Next' && !e.disabled); if(!next) throw new Error('configuration Next button missing'); next.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const boxes=[...document.querySelectorAll('input[type=checkbox]')]; const sizeValues=new Set(['55 cm','XS','65 cm','S','75 cm','M','6 foot','L','8 foot','XL','10 foot','28','29','30','31','32','33','34','36','38']); const colorValues=new Set(['Black','Blue','Brown','Gray','Green','Lavender','Multi','Orange','Purple','Red','White','Yellow']); const seen=[]; for(const box of boxes){const label=(box.labels?.[0]?.textContent||'').trim(); if(!sizeValues.has(label)&&!colorValues.has(label)) continue; const should=label==='S'||label==='Brown'; if(box.checked!==should) box.click(); if(should) seen.push(label);} if(!seen.includes('S')||!seen.includes('Brown')) throw new Error(`missing Size/Color options: ${seen.join(',')}`); const next=[...document.querySelectorAll('button')].filter(e=>e.textContent.trim()==='Next'&&!e.disabled).at(-1); if(!next) throw new Error('Attribute Values Next button missing'); next.click(); return seen; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const price=document.querySelector('#apply-single-price-radio'); const quantity=document.querySelector('#apply-single-inventory-radio'); if(!price||!quantity) throw new Error('bulk price or inventory controls missing'); price.click(); quantity.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "(() => { const set=(name,value)=>{const field=document.querySelector(`input[name=${name}]`); if(!field) throw new Error(`missing ${name}`); field.value=value; field.dispatchEvent(new Event('input',{bubbles:true})); field.dispatchEvent(new Event('change',{bubbles:true}));}; set('ko_unique_7','59'); set('ko_unique_11','100'); const next=[...document.querySelectorAll('button')].filter(e=>e.textContent.trim()==='Next'&&!e.disabled).at(-1); if(!next) throw new Error('Bulk Images and Price Next button missing'); next.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const button=[...document.querySelectorAll('button')].filter(e=>e.textContent.trim()==='Generate Products'&&!e.disabled).at(-1); if(!button) throw new Error('Generate Products button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 5 + }, + { + "action": "evaluate", + "script": "(() => { const button=document.querySelector('#save-button')||[...document.querySelectorAll('button')].find(e=>e.offsetParent!==null&&e.textContent.trim()==='Save'); if(!button) throw new Error('parent Save button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 8 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert-repair-c-547", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Brown already exists as global color option 51; prior attempts created or edited a child but did not save the parent configurable-product matrix.", + "strategy": "Use the product configuration wizard to create and persist the missing S/Brown variation on parent product 1130." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-547-existing-brown-option-20260912", + "correction_source_sha256": "4506c41226bbe787123484e251f89116cbaa9770dd77b2949aedcaf5cf00433a", + "corrected_task_sha256": "492d978fb60cce180734859e2547b79ad385300ec02bfd30be57c89274856fe0" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-571-trajectory-1", + "task_id": 571, + "task": { + "intent": "I recently moved, my address is 231 Willow Way, Suite 100, Chicago, IL, 60601, update my information on OneStopShopping accordingly", + "intent_template_id": 165, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7770" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7770/customer/account/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"http://host.docker.internal:7770/customer/address/edit/id/26/\"]:visible, 1)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "231 Willow Way", + "selector": "input[name=\"street[0]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "Suite 100", + "selector": "input[name=\"street[1]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "select", + "value": "23", + "selector": "select[name=\"region_id\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "Chicago", + "selector": "input[name=\"city\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "60601", + "selector": "input[name=\"postcode\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Save Address\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-571", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "5d5af474be266a7f6f5fa383f91f25caae22e13e9ffb7f7392cec2d83d8d40ab", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "41e9331f4e3b80a990fb11832f215ee5834e7cbd7f336083d186a1cfa9f3b9b0" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-576-trajectory-1", + "task_id": 576, + "task": { + "intent": "Add the following users to repo a11y-webring.club as developer: \"abisubramanya27\" and \"lahwaacz\"", + "intent_template_id": 351, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:8023/byteblaze/a11y-webring.club/-/project_members" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Invite members\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "const heading=[...document.querySelectorAll('h4')].find(e=>e.textContent.trim()==='Invite members'); const root=heading?.closest('[role=dialog],.modal,.gl-modal')||heading?.parentElement?.parentElement; const input=[...(root?.querySelectorAll('input[type=text]')||[])].find(e=>e.placeholder!=='YYYY-MM-DD'); if(!input) throw new Error('member lookup input not found'); const setter=Object.getOwnPropertyDescriptor(HTMLInputElement.prototype,'value').set; setter.call(input,'abisubramanya27'); input.dispatchEvent(new Event('input',{bubbles:true})); return true;" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const button=[...document.querySelectorAll('button[role=menuitem],button')].find(e=>e.offsetParent!==null&&e.textContent.includes('abisubramanya27')); if(!button) throw new Error('abisubramanya27 suggestion not found'); button.click(); return true;" + }, + { + "action": "evaluate", + "script": "const heading=[...document.querySelectorAll('h4')].find(e=>e.textContent.trim()==='Invite members'); const root=heading?.closest('[role=dialog],.modal,.gl-modal')||heading?.parentElement?.parentElement; const input=[...(root?.querySelectorAll('input[type=text]')||[])].find(e=>e.placeholder!=='YYYY-MM-DD'); if(!input) throw new Error('member lookup input not found'); const setter=Object.getOwnPropertyDescriptor(HTMLInputElement.prototype,'value').set; setter.call(input,'lahwaacz'); input.dispatchEvent(new Event('input',{bubbles:true})); return true;" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const button=[...document.querySelectorAll('button[role=menuitem],button')].find(e=>e.offsetParent!==null&&e.textContent.includes('lahwaacz')); if(!button) throw new Error('lahwaacz suggestion not found'); button.click(); return true;" + }, + { + "action": "select", + "selector": ".modal select:visible", + "value": "30", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": ":nth-match(button:visible:has-text(\"Invite\"), 3)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-576__invite-two-developers", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory addressed the third visible text input, which was the expiration-date field after reset, then waited for a user menu that could never appear.", + "strategy": "Scope member lookup to the Invite members dialog, select each named user by text, set Developer, and submit once." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "6bd92e821d69" + ] + } + }, + { + "id": "webarena-verified-580-trajectory-1", + "task_id": 580, + "task": { + "intent": "Create a new forum with name and title \"sci_fi\", description \"A wild place for sci-fi enthusiasts\", and sidebar \"**New**, Classic, *Movies*, Post my novel, Random\"", + "intent_template_id": 7, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/create_forum\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "sci_fi", + "selector": "input[name=\"forum[name]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "sci_fi", + "selector": "input[name=\"forum[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "A wild place for sci-fi enthusiasts", + "selector": "textarea[name=\"forum[description]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "**New**, Classic, *Movies*, Post my novel, Random", + "selector": "textarea[name=\"forum[sidebar]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create forum\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-580", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "222e9f7f4f52d31156694dee79d0f0e252ee5d86f07345ce7dc5a34144625145", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "7a80c8e435ab4e173e967882a7c6c892815a33d09cf8fab229b7147c0a0c7c0f" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-585-trajectory-1", + "task_id": 585, + "task": { + "intent": "Rate my recently purchased floor lamp with 5 stars using my nickname Emma Lopez, with the summary \"Good purchase\" and review \"I like it\"", + "intent_template_id": 194, + "revision": 2, + "sites": [ + "shopping" + ], + "start_urls": [ + "__SHOPPING__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7770/quoizel-tf9404m-grove-park-tiffany-multi-color-floor-lamp.html" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "click", + "selector": "a[href=\"#reviews\"]", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "click", + "selector": "input[name=\"ratings[4]\"][value=\"20\"]", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "selector": "input[name=\"nickname\"]", + "text": "Emma Lopez", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "selector": "input[name=\"title\"]", + "text": "Good purchase", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "selector": "textarea[name=\"detail\"]", + "text": "I like it", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Submit Review\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-585__five-star-review", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory addressed the fifth visible rating input before the Reviews panel had rendered, making both timing and position part of the selector.", + "strategy": "Open the contract product, wait for the Reviews panel, and select the five-star value by its form value." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-590-trajectory-1", + "task_id": 590, + "task": { + "intent": "Create a milestone in the current repo with title \"product launch\" for the upcoming event of product launch starting on January 16, 2023 and ending on January 30, 2023", + "intent_template_id": 339, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__/primer/design" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023/primer/design" + }, + { + "action": "click", + "selector": "a[href=\"/primer/design/-/issues\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/primer/design/-/milestones" + }, + { + "action": "click", + "selector": "a[href=\"/primer/design/-/milestones/new\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "product launch", + "selector": "input[name=\"milestone[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "2023-01-16", + "selector": "input[name=\"milestone[start_date]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "2023-01-30", + "selector": "input[name=\"milestone[due_date]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create milestone\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-590", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "1ee105deb614907c0fc4ec4ce74c4a94a9cef43fa49537fec1460cea5aca4bae", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "2dd47c26e79713a107c31acd1a8336e1a159d69042938c210a323df9f905a5d6" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-595-trajectory-1", + "task_id": 595, + "task": { + "intent": "Subscribe to the \"space\" forum from the page of the hottest post in that forum.", + "intent_template_id": 4, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/space\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/space/69581/the-moon-saturn-and-jupiter-through-my-4-telescope-and\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 800 + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "evaluate", + "script": "(() => { const buttons=[...document.querySelectorAll('button')]; const b=buttons.find(x=>x.textContent.trim().startsWith('Subscribe')); if(!b) return {found:false}; b.scrollIntoView({block:'center'}); b.click(); return {found:true,text:b.textContent.trim()}; })()" + }, + { + "action": "wait", + "timeout": 800 + }, + { + "action": "evaluate", + "script": "(() => ({url:location.href, controls:[...document.querySelectorAll('button')].filter(x=>/subscribe/i.test(x.textContent)).map(x=>({text:x.textContent.replace(/\\s+/g,' ').trim(),pressed:x.getAttribute('aria-pressed'),disabled:x.disabled,class:x.className})).slice(-5)}))()" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-595", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "e20e1093edbd1635fd2e2040539ffb9722805a18d0d4b4a73e3e79da2e1d54c2", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "c3b50a98df48306bd852e4ec59b7c4bc925093d512d625d2fe7a2ca16f7a2aca" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-600-trajectory-1", + "task_id": 600, + "task": { + "intent": "Post my question with the title \"what is the recommended console to buy these days?\", in a forum where I'm likely to get an answer", + "intent_template_id": 3765, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/submit/consoles" + }, + { + "action": "fill", + "selector": "textarea[name=\"submission[title]\"]", + "text": "what is the recommended console to buy these days?", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-600__fresh-console-question", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory navigated away while a gaming submission-form click was still completing, before it reached the intended consoles form.", + "strategy": "Load the consoles submission form directly and submit the exact title immediately." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-605-trajectory-1", + "task_id": 605, + "task": { + "intent": "Find a forum focused on topics related to gaming consoles, and post my question, \"what is the recommended console to buy these days\" there", + "intent_template_id": 5, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/submit/consoles" + }, + { + "action": "fill", + "selector": "textarea[name=\"submission[title]\"]", + "text": "what is the recommended console to buy these days", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "seconds": 3 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-605__fresh-console-submission", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The replay submitted the correct title and forum but the site returned the form with HTTP 200 instead of the successful 302 redirect; same-task historical evidence identifies an invalid CSRF token.", + "strategy": "Load a fresh forum-bound submission form and submit immediately, with no stale intermediate page or duplicate-click retry." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-610-trajectory-1", + "task_id": 610, + "task": { + "intent": "Post a review of my recent reading, titled \"To Kill a Mockingbird by Harper Lee\" in f/books and then comment \"good book!\" on the created post", + "intent_template_id": 9, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/submit\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "To Kill a Mockingbird by Harper Lee", + "selector": "textarea[name=\"submission[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "select", + "value": "10037", + "selector": "select[name=\"submission[forum]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1500 + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "good book!", + "selector": "textarea[name=\"reply_to_submission_2[comment]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Post\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 1200 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__moli__task-610", + "model": "gpt-5.6-sol", + "browser": "moli", + "execution": 1, + "events_sha256": "50e3008fd411dc269b2ecbc3e51cc86745607e3b90b22e34585316d6060ad750", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "5344844f668e97c98d7a37f0bb4d4bba1f9d556fe63f5dad3c57053785239c94" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-615-trajectory-1", + "task_id": 615, + "task": { + "intent": "Re-post the image of Bald Eagle from this forum to connecticut forum using the image URL and title \"from /f/pics\"", + "intent_template_id": 11, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__/f/pics" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999/f/pics" + }, + { + "action": "click", + "selector": "a[href=\"/f/pics/110715/amazing-shot-of-a-blue-jay-pestering-a-bald-eagle\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "evaluate", + "script": "() => document.querySelector('main article header h1 a').href" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:9999/submit/connecticut" + }, + { + "action": "fill", + "text": "http://192.168.5.2:9999/submission_images/b02113033af32feae9ff147dbbe3764039368d67d193885bd04e65c2e6beea9c.jpg", + "selector": "input[name=\"submission[url]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "from /f/pics", + "selector": "textarea[name=\"submission[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-615", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "e2f0d470e6df98f3b360d52e9563929ecc5250a328c07389c01602941752c1ca", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "0070d0cdcd78359932c360ac08d67e1712b110bac231537a15f589c2775fe430" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-620-trajectory-1", + "task_id": 620, + "task": { + "intent": "Ask for advice in a forum for relations. Title it \"deal with long-distance relationships\" and in post details ask \"Please help\"", + "intent_template_id": 12, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/relationship_advice\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/submit/relationship_advice\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "deal with long-distance relationships", + "selector": "textarea[name=\"submission[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "Please help", + "selector": "textarea[name=\"submission[body]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-620", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "204ddc080fb36e360fa05fb3d0ae3893d0a5d39e996be28427c0bf930b9ead71", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "277471d3dc29f6bf937e6b8cc3b6c3bc9dacfb3bdcb0c8eb34478efbf1e11625" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-625-trajectory-1", + "task_id": 625, + "task": { + "intent": "Create a discussion post titled \"the effectiveness of deep learning\" in a relevant forum and ask users for their opinions with the simple prompt, \"your opinion\"", + "intent_template_id": 13, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/submit\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "the effectiveness of deep learning", + "selector": "textarea[name=\"submission[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "your opinion", + "selector": "textarea[name=\"submission[body]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "select", + "value": "10043", + "selector": "select[name=\"submission[forum]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-6-astra__chromium__task-625", + "model": "gpt-6-astra", + "browser": "chromium", + "execution": 1, + "events_sha256": "c08cd4306a8fc224457b2459d5fdb9f9778077d9ed063d6f490da28bd67fd8d1", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "2d19a291c0e283b3b5b217792cb5c88438a86d30f8a34a0db9259a4eaff3e417" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-630-trajectory-1", + "task_id": 630, + "task": { + "intent": "Create a post in f/headphones. Title it \"noise-cancelling headphones recommendations?\" and in post details ask \"Hey, looking for good noise-cancelling headphones under $200, any suggestions?\"", + "intent_template_id": 15, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/submit\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "noise-cancelling headphones recommendations?", + "selector": "textarea[name=\"submission[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "Hey, looking for good noise-cancelling headphones under $200, any suggestions?", + "selector": "textarea[name=\"submission[body]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "select", + "value": "10050", + "selector": "select[name=\"submission[forum]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-630", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "5e0ccee840b3ebef3cc79ffa0ec1c38adf35d1937b1e8ab184b129369d377006", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "87bae2194fce345b062084a9a4967f3b15b6462e8201eec4344502abe1056499" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-635-trajectory-1", + "task_id": 635, + "task": { + "intent": "Create a post in the most appropriate forum. Title it \"noise-cancelling headphones recommendations\" and in post details ask \"I need recommendations for noise-cancelling headphones within a budget of $200 please\"", + "intent_template_id": 6100, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "headphones", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/submit\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "noise-cancelling headphones recommendations", + "selector": "textarea[name=\"submission[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "I need recommendations for noise-cancelling headphones within a budget of $200 please", + "selector": "textarea[name=\"submission[body]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "select", + "value": "10050", + "selector": "select[name=\"submission[forum]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-635", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "48637fa9879333955635b0fb06c772f4651dcb2cdbb0478f4cc3d3767872c802", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "5d8aa27f8653fcc6967b375ad325281a67db4aaeea5db831d9a996b3fbcb881d" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-640-trajectory-1", + "task_id": 640, + "task": { + "intent": "Post a notice in f/books titled \"book reading Meet up!\". Set post details to \"virtual meetup for book reading on March 15th\"", + "intent_template_id": 16, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/books\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/submit/books\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "book reading Meet up!", + "selector": "textarea[name=\"submission[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "virtual meetup for book reading on March 15th", + "selector": "textarea[name=\"submission[body]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-640", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "1f257cfc9f15b416694d7a34273efbc0c35e86ee64c44a5af1593a13dfe5cde7", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "8117560e00cdf1162583df6b99216f498a97adf7a72362833c4f8ec4ebf91cd3" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-645-trajectory-1", + "task_id": 645, + "task": { + "intent": "Post in books forum with title \"What could machine learning help the books field?\"", + "intent_template_id": 19, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/books\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/submit/books\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "What could machine learning help the books field?", + "selector": "textarea[name=\"submission[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "get_text", + "selector": "body" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-645", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "a28a348cdd16c83168e329e3c5f530a3cdb125886d9ac4b657c94fea9c305537", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "b81ccdba6f6f003d977438ed3ed8cbdcee271db6189007988e5170fdfe6ad8f0" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-658-trajectory-1", + "task_id": 658, + "task": { + "intent": "Create an issue in a11yproject repo with title \"401 bad gateway\". Assign the issue to Roshanjossey (you can invite users to the project if needed). Set due date to be the end of 2030", + "intent_template_id": 327, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023" + }, + { + "action": "click", + "selector": "a[href=\"/a11yproject/a11yproject.com/-/issues\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/a11yproject/a11yproject.com/-/issues/new\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "401 bad gateway", + "selector": "input[name=\"issue[title]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "fill", + "text": "2030-12-31", + "selector": "input[name=\"issue[due_date]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Unassigned\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 4)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create issue\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-658", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "fff5eb90cdaf06e9dd9dededcb99a11747d1e02a50d7c9c79f116b90fa74396e", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "da32ff219fba62d6b783eaae0aa173fe6fe027233aa883f24bfc42603ab0963e" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-666-trajectory-1", + "task_id": 666, + "task": { + "intent": "Submit a merge request for dialog-component in the current repository to be merged into dialog branch, assign Primer as the reviewer", + "intent_template_id": 335, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__/primer/design" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023/primer/design" + }, + { + "action": "click", + "selector": "a[href=\"/primer/design/-/merge_requests\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/primer/design/-/merge_requests/new\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Select source branch\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 11)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"main\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 10)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Compare branches and continue\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(button:visible:has-text(\"Unassigned\"), 2)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 5)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Create merge request\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "get_text", + "selector": "body" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-666", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "b6ffaebf0c4c1eddb26c26f11e11ab9a6622c73d67ce052dee7949b0e76e124e", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "4f2b0895d07fa0d02ba721cf98d7cac5fd783ffa1921fb3bf2267074d1ef3620" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-676-trajectory-1", + "task_id": 676, + "task": { + "intent": "Go to the list of orders that are suspected of being fraudulent", + "intent_template_id": 253, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales/order/" + }, + { + "action": "evaluate", + "script": "const button=document.querySelector('button[data-action=\"grid-filter-expand\"]'); if(!button) throw new Error('filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "for(const name of ['created_at[from]','created_at[to]']){const input=document.querySelector(`input[name=\"${name}\"]`); if(input){input.value=''; input.dispatchEvent(new Event('input',{bubbles:true})); input.dispatchEvent(new Event('change',{bubbles:true}));}} const select=document.querySelector('select[name=\"status\"]'); if(!select) throw new Error('status filter not found'); select.value='fraud'; select.dispatchEvent(new Event('input',{bubbles:true})); select.dispatchEvent(new Event('change',{bubbles:true})); const button=document.querySelector('button[data-action=\"grid-filter-apply\"]'); if(!button) throw new Error('apply filters button not found'); button.click(); return true;" + }, + { + "action": "wait", + "seconds": 1 + } + ], + "evaluation_input": { + "task_type": "NAVIGATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-676__fraud-orders", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "task_revision": 2, + "contract_correction_id": "wv-676-match-filter-xhr-20260911", + "evidence": "Historical successful Moli filter interaction and corrected XHR-selection contract" + }, + "repair_basis": { + "root_cause": "The upstream evaluator selected the last document navigation instead of the matching Magento grid XHR and also required a browser-specific Referer.", + "strategy": "Open the orders grid directly and drive its stable filter controls to emit the fraud-status MUI request." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-676-match-filter-xhr-20260911", + "correction_source_sha256": "9a5f437d646f8812f3e2d2b31834318de9aca4844e39ff03393a24209e6169e2", + "corrected_task_sha256": "45cc633ad4673ed043af49d4d27989e1cf70c7ab96b4080f26f6ffa9c6cf887e" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-694-trajectory-1", + "task_id": 694, + "task": { + "intent": "Add a simple product named \"Energy-Bulk Women Shirt\" with 50 in stock, available in size S and color blue, priced at $60 using the appropriate attribute set.", + "intent_template_id": 256, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/catalog/product/new/set/9/type/simple/" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "evaluate", + "script": "(() => { const values={'product[name]':'Energy-Bulk Women Shirt','product[sku]':'energy-bulk-women-shirt','product[price]':'60','product[quantity_and_stock_status][qty]':'50','product[quantity_and_stock_status][is_in_stock]':'1','product[weight]':'1','product[size]':'167','product[color]':'50'}; for(const [name,value] of Object.entries(values)){const el=document.querySelector(`[name=\"${name}\"]`); if(!el) throw new Error(`missing field ${name}`); el.value=value; el.dispatchEvent(new Event('input',{bubbles:true})); el.dispatchEvent(new Event('change',{bubbles:true}));} const status=document.querySelector('input[name=\"product[status]\"]'); if(status && !status.checked) status.click(); return Object.fromEntries(Object.keys(values).map(name=>[name,document.querySelector(`[name=\"${name}\"]`).value])); })()" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "(() => { const button=document.querySelector('#save-button'); if(!button) throw new Error('save button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 8 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert-repair-c-694", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Earlier saves raced Magento dependency readiness; Moli also timed out while waiting for actionability on fields that were already present.", + "strategy": "Open the Top attribute-set simple-product form directly, set every evaluator-bearing field through DOM events, and invoke the save button through evaluate." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-699-trajectory-1", + "task_id": 699, + "task": { + "intent": "Create a new marketing price rule called \"spring sale\" for all registered customers that offers a 20 percent discount site-wide", + "intent_template_id": 258, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/sales_rule/promo_quote/new/" + }, + { + "action": "wait", + "seconds": 4 + }, + { + "action": "evaluate", + "script": "(() => { const setValue=(selector,value)=>{const el=document.querySelector(selector);if(!el)throw new Error(`missing ${selector}`);el.value=value;el.dispatchEvent(new Event('input',{bubbles:true}));el.dispatchEvent(new Event('change',{bubbles:true}));return el;}; setValue('input[name=name]','spring sale'); const website=document.querySelector('select[name=\"website_ids\"]'); if(!website) throw new Error('missing website_ids'); [...website.options].forEach(o=>o.selected=o.value==='1'); website.dispatchEvent(new Event('change',{bubbles:true})); const groups=document.querySelector('select[name=\"customer_group_ids\"]'); if(!groups) throw new Error('missing customer_group_ids'); [...groups.options].forEach(o=>o.selected=['1','2','3'].includes(o.value)); groups.dispatchEvent(new Event('change',{bubbles:true})); const active=document.querySelector('input[name=is_active]'); if(active && !active.checked) active.click(); return {website:[...website.selectedOptions].map(o=>o.value),groups:[...groups.selectedOptions].map(o=>o.value)}; })()" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "(() => { const panel=document.querySelector('[data-index=\"actions\"]'); const title=panel?.querySelector('.admin__collapsible-title') || [...document.querySelectorAll('strong')].find(e=>e.textContent.trim()==='Actions'); if(!title) throw new Error('Actions accordion missing'); title.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "(() => { const action=document.querySelector('select[name=simple_action]'); const discount=document.querySelector('input[name=discount_amount]'); if(!action || !discount) throw new Error(`Actions fields missing: simple_action=${!!action}, discount_amount=${!!discount}`); action.value='by_percent'; action.dispatchEvent(new Event('change',{bubbles:true})); discount.value='20'; discount.dispatchEvent(new Event('input',{bubbles:true})); discount.dispatchEvent(new Event('change',{bubbles:true})); return {action:action.value,discount:discount.value}; })()" + }, + { + "action": "wait", + "seconds": 1 + }, + { + "action": "evaluate", + "script": "(() => { const button=document.querySelector('#save_and_apply') || [...document.querySelectorAll('button')].find(e=>e.textContent.trim()==='Save'); if(!button) throw new Error('save button missing'); button.click(); return true; })()" + }, + { + "action": "wait", + "seconds": 8 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert-repair-c-699", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "Prior attempts created a Catalog Price Rule; the task and evaluator require a Cart Price Rule, and all registered groups are IDs 1, 2, and 3.", + "strategy": "Open the Cart Price Rule form directly, select website 1 and registered groups 1/2/3, choose percent discount, and save." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "corrected_replay_contract", + "correction_id": "wv-699-browser-form-array-fields-20260912", + "correction_source_sha256": "4af179b0fbd508d3aae239bf4315021bdcf8989460cae7b764df3cf3192fa0f6", + "corrected_task_sha256": "5f4c022c31dc69bcd038b3b38c6d4bcc92b1562049c5ef8a6ec22b4acac1894e" + }, + "model_calls": 0, + "receipt_run_ids": [ + "2904cf117286" + ] + } + }, + { + "id": "webarena-verified-714-trajectory-1", + "task_id": 714, + "task": { + "intent": "Thumbs down the top 1 post ever in gadgets.", + "intent_template_id": 24, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "click", + "selector": "a[href=\"/forums\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:9999/f/gadgets" + }, + { + "action": "click", + "selector": "button[aria-label=\"Sort by: Hot\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/gadgets/top?t=day\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "button[aria-label=\"From: Past 24 hours\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/gadgets/top?t=all\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(button:visible, 5)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-714", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "904372b39f29c0ba4f5f178b5ee8691df1efa46400012bf03fc40a7ec9a4eee7", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "38e8171b734f3e203d096bc0ddb5511b27ff7ba1ec722a11d00f6935916a804f" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-719-trajectory-1", + "task_id": 719, + "task": { + "intent": "Like all submissions created by ThetaGang_wsb in forum wallstreetbets", + "intent_template_id": 25, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "fill", + "text": "ThetaGang_wsb", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:9999/user/ThetaGang_wsb" + }, + { + "action": "click", + "selector": "a[href=\"/user/ThetaGang_wsb/submissions\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": ":nth-match(button:visible, 2)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": ":nth-match(button:visible, 4)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-719", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "4b1c2c8a46353ef6bf8c7249a45376ad6576f248c558c93f22f08b4db9ede35c", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "039901b5db2c20877a8e3a08ca69515d3046e962afbc160c52d97656de8d36c2" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-725-trajectory-1", + "task_id": 725, + "task": { + "intent": "DisLike all submissions created by RickyDontLoseThat in forum massachusetts", + "intent_template_id": 1510, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://host.docker.internal:9999/user/RickyDontLoseThat/submissions" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "evaluate", + "script": "const article=[...document.querySelectorAll('main article')].find(e=>e.querySelector('a[href=\"/f/massachusetts\"]')); const button=article?.querySelector('form button:has(img[alt=\"down\"])'); if(!button) throw new Error('massachusetts submission downvote control not found'); button.click(); return {post:article.querySelector('h1,h2')?.textContent.trim()};" + }, + { + "action": "wait", + "seconds": 2 + }, + { + "action": "get_text", + "selector": "main" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "expert_repair_spec", + "candidate_id": "expert__task-725__massachusetts-submissions-downvote", + "repair_spec_sha256": "ea34996acd3f137da912dcc386858c2ab51b86514cacc23502d1448224bbb830", + "details": { + "kind": "reviewed_task_contract_and_existing_browser_evidence", + "task_revision": 2 + }, + "repair_basis": { + "root_cause": "The selected trajectory started a forum navigation while the preceding forums click was still navigating and later addressed the downvote button by page position.", + "strategy": "Open the author's submissions directly, bind the target article to its massachusetts forum link, and click that article's downvote control." + } + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "13f17cee19ab" + ] + } + }, + { + "id": "webarena-verified-731-trajectory-1", + "task_id": 731, + "task": { + "intent": "Edit my post on Nvidia RTX 4090 by adding a line to the body that says \"EDIT: This news aged well\"", + "intent_template_id": 27, + "revision": 2, + "sites": [ + "reddit" + ], + "start_urls": [ + "__REDDIT__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:9999" + }, + { + "action": "fill", + "text": "Nvidia RTX 4090", + "selector": "input[name=\"q\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "press", + "key": "Enter" + }, + { + "action": "snapshot" + }, + { + "action": "wait", + "timeout": 1000 + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"/f/MachineLearning/1/-/edit\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "fill", + "text": "Crazy device for ML!\n\nEDIT: This news aged well", + "selector": "textarea[name=\"submission[body]\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Edit submission\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-731", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "aaa017895be2ca14837144f58e9ff75cc508e49553687fa7e6ec50ca45ef3f74", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "23d569a765d8246d28287610ae6ae6b30e562ed02c744a32157a9709deaffc17" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-771-trajectory-1", + "task_id": 771, + "task": { + "intent": "Approve reviews with four stars or higher to display in our store.", + "intent_template_id": 243, + "revision": 2, + "sites": [ + "shopping_admin" + ], + "start_urls": [ + "__SHOPPING_ADMIN__" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:7780/admin" + }, + { + "action": "click", + "selector": ":nth-match(a[href=\"#\"]:visible, 4)", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/review/product/pending/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "click", + "selector": "a[href=\"http://host.docker.internal:7780/admin/review/product/edit/id/353/\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "snapshot" + }, + { + "action": "evaluate", + "script": "(() => { const f=document.querySelector('form'); return {checked:[...document.querySelectorAll('input[type=radio]')].map(x=>({name:x.name,value:x.value,checked:x.checked,outer:x.outerHTML})), ratings:[...document.querySelectorAll('[class*=rating]')].map(x=>({tag:x.tagName,cls:x.className,text:x.innerText,html:x.innerHTML.slice(0,500)}))}; })()" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/review/product/edit/id/352/" + }, + { + "action": "evaluate", + "script": "(()=>({rating:[...document.querySelectorAll('input[name^=\"ratings\"]')].find(x=>x.checked)?.id,status:document.querySelector('[name=status_id]')?.value}))()" + }, + { + "action": "select", + "value": "1", + "selector": "select[name=\"status_id\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Save Review\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/review/product/edit/id/351/" + }, + { + "action": "evaluate", + "script": "(()=>[...document.querySelectorAll('input[name^=\"ratings\"]')].find(x=>x.checked)?.id)()" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/review/product/edit/id/349/" + }, + { + "action": "evaluate", + "script": "(()=>[...document.querySelectorAll('input[name^=\"ratings\"]')].find(x=>x.checked)?.id)()" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/review/product/edit/id/347/" + }, + { + "action": "evaluate", + "script": "(()=>[...document.querySelectorAll('input[name^=\"ratings\"]')].find(x=>x.checked)?.id)()" + }, + { + "action": "select", + "value": "1", + "selector": "select[name=\"status_id\"]:visible", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "click", + "selector": "button:visible:has-text(\"Save Review\")", + "replay_mode": "snapshot_ref_semantics" + }, + { + "action": "navigate", + "url": "http://host.docker.internal:7780/admin/review/product/pending/" + } + ], + "evaluation_input": { + "task_type": "MUTATE", + "status": "SUCCESS", + "retrieved_data": null, + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__moli__task-771", + "model": "gpt-5.6-sol", + "browser": "moli", + "execution": 1, + "events_sha256": "37c14caa1a034341e11a34a7bd7fccc51c509ca55a82732394f0185580719f86", + "response_sha256": "db765f7e78b6c552e25494cde987b92c12e4d11d6990604b328f604c66bbf9c7", + "evaluation_sha256": "f7d73fb103ab4d8bb64e63f13c76c053cca33fd78aa84f087e8baaf669b78818" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + }, + { + "id": "webarena-verified-784-trajectory-1", + "task_id": 784, + "task": { + "intent": "Get the email address of the contributor who has the most commits to branch master in the current repository", + "intent_template_id": 316, + "revision": 2, + "sites": [ + "gitlab" + ], + "start_urls": [ + "__GITLAB__/CellularPrivacy/Android-IMSI-Catcher-Detector" + ] + }, + "tool_calls": [ + { + "action": "navigate", + "url": "http://192.168.5.2:8023/CellularPrivacy/Android-IMSI-Catcher-Detector" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/CellularPrivacy/Android-IMSI-Catcher-Detector/-/graphs/master/charts" + }, + { + "action": "evaluate", + "script": "JSON.stringify(performance.getEntriesByType('resource').map(x=>x.name).filter(x=>x.includes('graphs')||x.includes('chart')||x.includes('commit')))" + }, + { + "action": "evaluate", + "script": "document.documentElement.innerHTML.includes('commit_data') ? document.documentElement.innerHTML.slice(document.documentElement.innerHTML.indexOf('commit_data')-500,document.documentElement.innerHTML.indexOf('commit_data')+1000) : document.documentElement.innerHTML.match(/data-[^=]+='[^']{100,}'/g)?.slice(0,10)" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/api/v4/projects/61/repository/contributors?ref=master&per_page=100" + }, + { + "action": "get_text", + "selector": "pre" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/api/v4/projects/61/repository/contributors?ref=master&per_page=100&page=2" + }, + { + "action": "get_text", + "selector": "pre" + }, + { + "action": "navigate", + "url": "http://192.168.5.2:8023/api/v4/projects/61/repository/contributors?ref=master&per_page=100&page=3" + } + ], + "evaluation_input": { + "task_type": "RETRIEVE", + "status": "SUCCESS", + "retrieved_data": [ + "SecUpwN@users.noreply.github.com" + ], + "error_details": null + }, + "source": { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": "gpt-5.6-sol__chromium__task-784", + "model": "gpt-5.6-sol", + "browser": "chromium", + "execution": 1, + "events_sha256": "a8f613a4972fa09661766a1d142ce5465629d7eb70fc76fd763cefd62dae9c31", + "response_sha256": "ead8795236d5df5d6b2e57260b6a982f6811245d9522d04850865b42f9cc2ff7", + "evaluation_sha256": "753c1b34dc2b4c0578b9ef17e243ef3d5dc17b849418b5ae463ebae86cf2e7d7" + }, + "verification": { + "browsers": [ + "chromium" + ], + "process_status": "pass", + "evaluator_status": "pass", + "contract": { + "kind": "official_task_contract" + }, + "model_calls": 0, + "receipt_run_ids": [ + "ab8cbb4dd809" + ] + } + } + ] +} diff --git a/examples/browser_use/README.md b/examples/browser_use/README.md index 0e19f5d9a..e436bf234 100644 --- a/examples/browser_use/README.md +++ b/examples/browser_use/README.md @@ -1,173 +1,72 @@ -# Browser Use Evaluation Harness +# Browser Use Evaluation + +This track runs the real CowAgent decision loop against the official +WebArena-Verified evaluator. It is independent from `harness/trajectory_replay`, +which replays a fixed list of browser-tool calls to isolate browser-engine +behavior and does not measure autonomous planning. + +The current selection is pre-registered in +`evaluation/browser-use/webarena-verified-pilot.json`: 15 easy, 30 medium, and +45 hard cases. These are project-defined operational tiers based on an audited +minimum number of visible UI operations, not WebArena's official hard-subset +label. The pilot uses the self-hosted shopping, shopping_admin, reddit, and +gitlab sites. Wikipedia and Map are deferred because their official local setup +requires large external data bundles. + +On Apple Silicon, run the official amd64 site images in a dedicated Colima VZ +profile with Rosetta enabled and at least 140 GiB of disk. The four required +images occupy about 91 GB before container writable layers. The runner mounts +Colima's QEMU interpreter only for PHP-FPM in the Magento and Reddit images because +the legacy Alpine/musl PHP-FPM binary segfaults under Rosetta; Java, GitLab, and +the remaining official image processes stay on Rosetta. This is a runtime +compatibility shim, not a change to site data or evaluation logic. + + colima start browser-eval --vm-type vz --disk 140 --memory 8 --cpus 4 --vz-rosetta + +Put `OPENAI_API_KEY`, optional `OPENAI_BASE_URL`, and `COWAGENT_REPO` in this +repository's `.env`, then run: -This harness compares local Chromium and Moli without measuring host-wide -resource noise. It starts one isolated browser container per engine. A normal -host-side controller reads only each target container's Docker stats. + ./examples/browser_use/run.sh -## One-command reproduction +To validate the pipeline with the currently available GPT-5.6 model before an +Astra-enabled project is available, keep the evidence cohort separate and run: -Requirements: + BROWSER_USE_OUTPUT_DIR="${BROWSER_EVAL_DATA_HOME:-$HOME/.browser-eval}/browser-use/gpt-5.6-sol" \ + ./examples/browser_use/run.sh --model gpt-5.6-sol -- Linux with Docker Engine and cgroup v2, or Docker Desktop/Colima backed by a - Linux VM -- Python 3.11+ with venv support, at least 4 CPU cores, 6 GiB available memory, - and Internet access for dependencies, images, and measured public websites -- Docker access for the current user +The selected model is immutable within one evidence directory. GPT-5.6 and +Astra attempts must never be resumed into or reported as the same cohort. +To build one layered report from several completed cohorts, pass each directory +separately. The builder rejects duplicate models and mismatched primary +browser-task cohorts: -Run: + uv run --frozen python scripts/build_browser_use_report.py \ + --evidence "$MODEL_A_EVIDENCE" --evidence "$MODEL_B_EVIDENCE" \ + --output reports/browser-use - ./examples/browser_use/run.sh +`COWAGENT_REPO` is a trusted code-execution boundary, not a data input. The +runner refuses a dirty checkout and records its Git commit in the run config. + +The official sites use one shared fixed-port stack. The runner therefore holds a +cross-process lock for each complete physical execution, from environment reset +through browser-session setup, CowAgent, evidence finalization, and evaluation. +Separate runner processes wait rather than execute against that stack in parallel. +Parallel execution is valid only after every worker has an isolated complete site +stack, site and control ports, and browser containers; changing output directories +alone does not provide isolation. + +Every task-browser pair has at most two physical executions: one execution and, +only after failure, one retry. The final execution is the single logical result; +all physical-call latency and model usage remain part of that result. +After the run, create the deterministic per-attempt evidence review before +building or publishing the report: + + make browser-use-review EVIDENCE="$BROWSER_USE_OUTPUT_DIR" + make browser-use-report EVIDENCE="$BROWSER_USE_OUTPUT_DIR" + make browser-use-report-check EVIDENCE="$BROWSER_USE_OUTPUT_DIR" + +The report check is the routine fast gate and does not launch a browser. Before +packaging or publishing, run the full release gate, which additionally renders +every representative route in both languages, themes, and viewport sizes: -Run the small real-container smoke test before a release (requires the two -evaluation images, which the command above builds): - - RUN_BROWSER_USE_SMOKE=1 .venv/bin/python -m pytest -q -m real_smoke - -Smoke is intentionally minimal: one real public-page case, once per engine. It -reuses existing evaluation images and is not a performance benchmark. - -The first run builds two images, downloads pinned Moli v1.0.6, and creates a -dedicated Python virtual environment. No Codex, coding agent, LLM API key, -browser account, website login, privileged container, or host filesystem mount -is required. - -Both Dockerfiles pin the same Debian multi-architecture base-image digest. The -run manifest records the resulting image IDs and browser versions so a -reproducer can identify binary drift; Debian security packages and public-site -content remain external variables rather than being silently treated as fixed. - -Outputs: - -- raw attempts, manifest, and summary under artifacts/browser_use/latest/ -- latest-only Markdown, HTML, and audit data under docs/reports/ (ignored by Git) - -The complete run and report are code-driven: `run.sh` writes raw -`attempts.jsonl`, `metrics.py` calculates every aggregate and paired ratio, -`summary.json` stores the auditable values, and `report.py` only renders them. -To recalculate all metrics and HTML from existing raw attempts without running -the browsers again: - - uv run python examples/browser_use/rebuild_report.py - -Independent raw-data audit (does not import the aggregation implementation): - - uv run python examples/browser_use/verify_metrics.py --artifacts artifacts/browser_use/latest - -`run.sh` runs this audit automatically. It checks correctness counts, the -successful-task intersection, 81 aggregate/ratio metric groups, and every -paired ratio's task identity and operands. Any mismatch exits with an error. - -Neither command invokes Codex or another coding agent for scoring or -statistics. - -## Protocol - -- Element-targeted calls require a non-empty string `selector`. Snapshot `ref` - identifiers are not resolved by this deterministic runner. -- Core attempts, summary, and manifest are saved before failure diagnostics. - Optional LLM diagnostic errors preserve rule-based findings and scoring; the - diagnostic status and exception type are recorded without remote error text. - -- Both browser containers receive the same 2 CPU, 2 GiB memory, 512 MiB shared - memory, and 512 PID limits by default. -- Each new page uses a 1280 × 900 CSS-pixel viewport. The runner checks actual - `innerWidth`/`innerHeight` before measuring each task and fails on mismatch. -- Each engine receives its own temporary Docker bridge with no peer containers; - its CDP port is published only on host loopback. The bridge preserves public - web access without exposing CDP to containers on Docker's default bridge. - The active suite uses public HTTPS pages, not local fixtures. -- Chromium runs as a non-root user with Docker isolation and - no-new-privileges. Its unavailable nested setuid browser sandbox is disabled; - the container boundary is therefore security-relevant and these evaluation - images should not be treated as a general-purpose hostile-site sandbox. -- Both remain warm for the complete run. -- https://example.com is an excluded warm-up. -- Browser process and context remain warm. Every top-level HTTP(S) navigation - origin is inventoried. Before the next measured task, the harness clears all - data for those origins plus cookies, permissions, local/session storage, - IndexedDB, Cache Storage, Service Workers, HTTP cache, and navigation history. - Renderer GC runs, then only the old page is closed and a new blank page is - created. A 1500 ms quiescence window follows. Page replacement, reset, and - quiescence are excluded; the configured value is recorded in the manifest. -- Engine order alternates by task: Chromium→Moli, then Moli→Chromium. -- CPU and network totals use per-container cumulative counter deltas from Docker - Engine. Memory, PID/thread count, CPU utilization, and network rates use - 100 ms one-shot per-container samples. The report includes time-weighted mean, - within-attempt P95, peak, retained memory, MiB-seconds, totals, and rates. -- Each task runs once per engine. Only a failure triggers a retry (one by - default). The report presents one final result per task and provider; all - attempt evidence is available on its linked detail page. Result correctness - is the primary score; result plus completion - of the fixed reference path is reported separately. Resource comparison uses - only tasks both engines passed on the first attempt; retries and failed tasks - do not enter the normal-path resource baseline. -- Cold start is intentionally separate from this warm-browser report. -- State reset removes page/site history but cannot force a warm browser process - to return allocator-reserved memory to its first-run footprint. The report's - retained-memory and MiB-second metrics keep this visible. - -Failures retry once by default. Override defaults with -BROWSER_USE_EVAL_FAILURE_RETRIES, BROWSER_USE_EVAL_SAMPLE_INTERVAL_MS, -BROWSER_USE_EVAL_SETTLE_MS, BROWSER_USE_EVAL_CPUS, BROWSER_USE_EVAL_MEMORY, and -BROWSER_USE_EVAL_OUTPUT_DIR. - -Rule-based failure classification is always enabled. Optional LLM root-cause -analysis is enabled with `BROWSER_USE_EVAL_LLM_DIAGNOSTICS=1`; it uses -`gpt-5.6-sol` with low reasoning by default and requires `OPENAI_API_KEY` plus -`OPENAI_BASE_URL`. The LLM receives only failed-task evidence, produces a -traceable hypothesis, and never changes pass/fail scoring. - -## Browser-tool call count - -One step is one declared BrowserTool-equivalent call: navigate, click, fill, -get_text, snapshot, scroll, back, or forward. Internal page-readiness and -locator waits are not separate steps. -For form controls that navigate, `wait_for_navigation: true` explicitly waits -for the declared `wait_until` event as part of that same action. Cascading -select tasks use `domcontentloaded` so the next control's handler is ready. - -The suite has thirty public-web tasks: five simple, ten medium, and fifteen -complex. Twenty-five retain the previous public suite's task definitions; -five add cascading search, infinite scrolling, site search, version/language -switching, and repository file search. Capability designs may come from -[Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) -and [Moli](https://github.com/lexmount/moli). Lexbench's local-fixture scenarios -are design references, not measured public targets. Every task -declares a fixed reference action sequence, exact expected -outputs, derivation, and source attribution. Expected outputs are unique; the -reference sequence is replayed by this provider-isolation microbenchmark but is -not claimed to be the only possible solution path. The runner judges result -correctness and result-plus-process correctness separately and programmatically; -no LLM-as-a-Judge, fuzzy contains check, or minimum-length heuristic is used. A -free-planning Agent evaluation should judge required milestones, forbidden -shortcuts, and a step budget instead of exact sequence identity. - -Ground-truth annotation uses actual public-page operations through Codex -Desktop's in-app Browser Tool, followed by exact comparison of observed -outputs. `ground_truth_validation.json` is bound to the dataset SHA-256 and -records dates and provenance per task. The 2026-09-02 revision retains 25 -unchanged task annotations from 2026-09-01 and adds five browser-verified -annotations; it does not claim all 30 were re-annotated that day. Benchmark -runs and statistical calculations remain entirely programmatic. - -Absolute P50/P95/mean values describe each provider's cross-task distribution. -Paired ratios are computed per successful task first, then summarized. The P95 -of those ratios is not a division of two marginal P95 values. Comparison cells -share one highlighting rule; rounded ties and unavailable data stay neutral. - -## CubeSandbox - -CubeSandbox is a valid future backend for hardware-isolated, high-volume runs: -its [official examples](https://docs.cubesandbox.com/guide/tutorials/examples.html) -run headless Chromium inside a MicroVM and expose CDP. Its -[resource metrics](https://docs.cubesandbox.com/guide/resource-metrics.html) -provide independent `host_sandbox` and `guest_workload` CPU/current-memory -views. However, the default collection interval is 5 seconds, the endpoint -retains only the latest sample, exact short-task memory peaks are not exported, -and the documented metric families do not include per-sandbox network bytes. - -The initial local reference therefore uses Docker cgroup and network-namespace -accounting at 100 ms. This is a measurement boundary, not a CowAgent deployment -recommendation. A CubeSandbox backend should be added separately for production -topology, hardware isolation, and density tests; its metrics must not be blended -with Docker results. + make browser-use-release-check EVIDENCE="$BROWSER_USE_OUTPUT_DIR" diff --git a/examples/browser_use/run.sh b/examples/browser_use/run.sh index 2d92ae7cc..92e84670f 100755 --- a/examples/browser_use/run.sh +++ b/examples/browser_use/run.sh @@ -2,76 +2,21 @@ set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -PYTHON="${BROWSER_USE_EVAL_PYTHON:-python3}" -VENV="${BROWSER_USE_EVAL_VENV:-${ROOT}/.browser-use-eval-venv}" -FAILURE_RETRIES="${BROWSER_USE_EVAL_FAILURE_RETRIES:-1}" -SAMPLE_MS="${BROWSER_USE_EVAL_SAMPLE_INTERVAL_MS:-100}" -SETTLE_MS="${BROWSER_USE_EVAL_SETTLE_MS:-1500}" -CPUS="${BROWSER_USE_EVAL_CPUS:-2}" -MEMORY="${BROWSER_USE_EVAL_MEMORY:-2g}" -BUILD_IMAGES="${BROWSER_USE_EVAL_BUILD_IMAGES:-1}" -LLM_DIAGNOSTICS="${BROWSER_USE_EVAL_LLM_DIAGNOSTICS:-0}" -DIAGNOSIS_MODEL="${BROWSER_USE_EVAL_DIAGNOSIS_MODEL:-gpt-5.6-sol}" -DIAGNOSIS_EFFORT="${BROWSER_USE_EVAL_DIAGNOSIS_REASONING_EFFORT:-low}" -OUTPUT="${BROWSER_USE_EVAL_OUTPUT_DIR:-${ROOT}/artifacts/browser_use/latest}" -REPORTS="${ROOT}/docs/reports" +DATA_HOME="${BROWSER_EVAL_DATA_HOME:-${HOME}/.browser-eval}" +OUTPUT="${BROWSER_USE_OUTPUT_DIR:-${DATA_HOME}/browser-use/current}" +MODEL="${BROWSER_USE_MODEL:-}" -command -v docker >/dev/null 2>&1 || { echo "docker is required" >&2; exit 2; } -command -v "${PYTHON}" >/dev/null 2>&1 || { echo "${PYTHON} is required" >&2; exit 2; } -docker info >/dev/null - -if ! "${PYTHON}" -m venv --help >/dev/null 2>&1; then - echo "Python venv support is required (Ubuntu/Debian: sudo apt-get install python3-venv)" >&2 +if [[ -z "${MODEL}" ]]; then + echo "BROWSER_USE_MODEL is required (for example: gpt-6-astra)" >&2 exit 2 fi -mkdir -p "${OUTPUT}" "${REPORTS}" -find "${OUTPUT}" -mindepth 1 -maxdepth 1 -type f -delete - -if [[ "${BUILD_IMAGES}" == "1" ]]; then - docker build -f "${ROOT}/examples/browser_use/docker/chromium.Dockerfile" \ - -t lex-browser-use-eval-chromium:local "${ROOT}" - docker build -f "${ROOT}/examples/browser_use/docker/moli.Dockerfile" \ - -t lex-browser-use-eval-moli:local "${ROOT}" -fi - -if command -v uv >/dev/null 2>&1; then - RUNNER=(uv run --extra browser-use-eval python) -else - if [[ ! -x "${VENV}/bin/python" ]]; then - if ! "${PYTHON}" -m venv "${VENV}"; then - echo "Failed to create venv (Ubuntu/Debian: sudo apt-get install python3-venv)" >&2 - exit 2 - fi - fi - if ! "${VENV}/bin/python" -m pip --version >/dev/null 2>&1; then - "${PYTHON}" -m ensurepip --upgrade - fi - "${VENV}/bin/python" -m pip install --disable-pip-version-check -q \ - -e "${ROOT}[browser-use-eval]" - RUNNER=("${VENV}/bin/python") -fi - -EXTRA_ARGS=(--diagnosis-model "${DIAGNOSIS_MODEL}" \ - --diagnosis-reasoning-effort "${DIAGNOSIS_EFFORT}") -if [[ "${LLM_DIAGNOSTICS}" == "1" ]]; then - EXTRA_ARGS+=(--llm-diagnostics) -fi - -"${RUNNER[@]}" -m harness.browser_use.cli \ - --suite "${ROOT}/examples/browser_use/tasks.v4.json" \ +cd "${ROOT}" +uv run --frozen --extra browser-use-eval python -m harness.browser_use.runner \ + --repo-root "${ROOT}" \ + --selection "${ROOT}/evaluation/browser-use/webarena-verified-pilot.json" \ --output "${OUTPUT}" \ - --reports "${REPORTS}" \ - --ground-truth-evidence "${ROOT}/examples/browser_use/ground_truth_validation.json" \ - --failure-retries "${FAILURE_RETRIES}" \ - --sample-interval-ms "${SAMPLE_MS}" \ - --settle-ms "${SETTLE_MS}" \ - --cpus "${CPUS}" \ - --memory "${MEMORY}" \ - "${EXTRA_ARGS[@]}" - -"${RUNNER[@]}" "${ROOT}/examples/browser_use/verify_metrics.py" --artifacts "${OUTPUT}" + --model "${MODEL}" \ + "$@" -echo "Raw artifacts: ${OUTPUT}" -echo "Latest report: ${REPORTS}/browser-use-evaluation-latest.html" -echo "All metrics and HTML above were generated by repository code; Codex is not part of scoring or calculation." +echo "Evidence: ${OUTPUT}" diff --git a/examples/trajectory_replay/README.md b/examples/trajectory_replay/README.md new file mode 100644 index 000000000..da14a3803 --- /dev/null +++ b/examples/trajectory_replay/README.md @@ -0,0 +1,174 @@ +# Browser Trajectory Replay + +This deterministic microbenchmark replays a fixed BrowserTool action trajectory +against local Chromium and Moli. It starts one isolated browser container per +engine, and a host-side controller reads only each target container's Docker +stats. + +## One-command reproduction + +Requirements: + +- Linux with Docker Engine and cgroup v2, or Docker Desktop/Colima backed by a + Linux VM +- Python 3.11+ with venv support, at least 4 CPU cores, 6 GiB available memory, + and Internet access for dependencies, images, and measured public websites +- Docker access for the current user + +Run: + + ./examples/trajectory_replay/run.sh + +Run the small real-container smoke test before a release (requires the two +evaluation images, which the command above builds): + + RUN_BROWSER_USE_SMOKE=1 .venv/bin/python -m pytest -q -m real_smoke + +Smoke is intentionally minimal: one real public-page case, once per engine. It +reuses existing evaluation images and is not a performance benchmark. + +The first run builds two images, downloads pinned Moli v1.1.3, and creates a +dedicated Python virtual environment. No Codex, coding agent, LLM API key, +browser account, website login, privileged container, or host filesystem mount +is required. + +Both Dockerfiles pin the same Debian multi-architecture base-image digest. The +run manifest records the resulting image IDs and browser versions so a +reproducer can identify binary drift; Debian security packages and public-site +content remain external variables rather than being silently treated as fixed. + +Outputs: + +- raw attempts, manifest, and summary under artifacts/trajectory_replay/latest/ +- latest-only Markdown, HTML, and audit data under docs/reports/ (ignored by Git) + +The complete run and report are code-driven: `run.sh` writes raw +`attempts.jsonl`, `metrics.py` calculates every aggregate and paired ratio, +`summary.json` stores the auditable values, and `report.py` only renders them. +To recalculate all metrics and HTML from existing raw attempts without running +the browsers again: + + uv run python examples/trajectory_replay/rebuild_report.py + +Independent raw-data audit (does not import the aggregation implementation): + + uv run python examples/trajectory_replay/verify_metrics.py --artifacts artifacts/trajectory_replay/latest + +`run.sh` runs this audit automatically. It checks correctness counts, the +successful-task intersection, 81 aggregate/ratio metric groups, and every +paired ratio's task identity and operands. Any mismatch exits with an error. + +Neither command invokes Codex or another coding agent for scoring or +statistics. + +## Protocol + +- Element-targeted calls require a non-empty string `selector`. Snapshot `ref` + identifiers are not resolved by this deterministic runner. +- Core attempts, summary, and manifest are saved before failure diagnostics. + Optional LLM diagnostic errors preserve rule-based findings and scoring; the + diagnostic status and exception type are recorded without remote error text. + +- Both browser containers receive the same 2 CPU, 2 GiB memory, 512 MiB shared + memory, and 512 PID limits by default. +- Each new page uses a 1280 × 900 CSS-pixel viewport. The runner checks actual + `innerWidth`/`innerHeight` before measuring each task and fails on mismatch. +- Each engine receives its own temporary Docker bridge with no peer containers; + its CDP port is published only on host loopback. The bridge preserves public + web access without exposing CDP to containers on Docker's default bridge. + The active suite uses public HTTPS pages, not local fixtures. +- Chromium runs as a non-root user with Docker isolation and + no-new-privileges. Its unavailable nested setuid browser sandbox is disabled; + the container boundary is therefore security-relevant and these evaluation + images should not be treated as a general-purpose hostile-site sandbox. +- Both remain warm for the complete run. +- https://example.com is an excluded warm-up. +- Browser process and context remain warm. Every top-level HTTP(S) navigation + origin is inventoried. Before the next measured task, the harness clears all + data for those origins plus cookies, permissions, local/session storage, + IndexedDB, Cache Storage, Service Workers, HTTP cache, and navigation history. + Renderer GC runs, then only the old page is closed and a new blank page is + created. A 1500 ms quiescence window follows. Page replacement, reset, and + quiescence are excluded; the configured value is recorded in the manifest. +- Engine order alternates by task: Chromium→Moli, then Moli→Chromium. +- CPU and network totals use per-container cumulative counter deltas from Docker + Engine. Memory, PID/thread count, CPU utilization, and network rates use + 100 ms one-shot per-container samples. The report includes time-weighted mean, + within-attempt P95, peak, retained memory, MiB-seconds, totals, and rates. +- Each task runs once per engine. Only a failure triggers a retry (one by + default). The report presents one final result per task and provider; all + attempt evidence is available on its linked detail page. Result correctness + is the primary score; result plus completion + of the fixed reference path is reported separately. Resource comparison uses + only tasks both engines passed on the first attempt; retries and failed tasks + do not enter the normal-path resource baseline. +- Cold start is intentionally separate from this warm-browser report. +- State reset removes page/site history but cannot force a warm browser process + to return allocator-reserved memory to its first-run footprint. The report's + retained-memory and MiB-second metrics keep this visible. + +Failures retry once by default. Override defaults with +BROWSER_USE_EVAL_FAILURE_RETRIES, BROWSER_USE_EVAL_SAMPLE_INTERVAL_MS, +BROWSER_USE_EVAL_SETTLE_MS, BROWSER_USE_EVAL_CPUS, BROWSER_USE_EVAL_MEMORY, and +BROWSER_USE_EVAL_OUTPUT_DIR. + +Rule-based failure classification is always enabled. Optional LLM root-cause +analysis is enabled with `BROWSER_USE_EVAL_LLM_DIAGNOSTICS=1`; it uses +`gpt-5.6-sol` with low reasoning by default and requires `OPENAI_API_KEY` plus +`OPENAI_BASE_URL`. The LLM receives only failed-task evidence, produces a +traceable hypothesis, and never changes pass/fail scoring. + +## Browser-tool call count + +One step is one declared BrowserTool-equivalent call: navigate, click, fill, +get_text, snapshot, scroll, back, or forward. Internal page-readiness and +locator waits are not separate steps. +For form controls that navigate, `wait_for_navigation: true` explicitly waits +for the declared `wait_until` event as part of that same action. Cascading +select tasks use `domcontentloaded` so the next control's handler is ready. + +The suite has thirty public-web tasks: five simple, ten medium, and fifteen +complex. Twenty-five retain the previous public suite's task definitions; +five add cascading search, infinite scrolling, site search, version/language +switching, and repository file search. Capability designs may come from +[Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) +and [Moli](https://github.com/lexmount/moli). Lexbench's local-fixture scenarios +are design references, not measured public targets. Every task +declares a fixed reference action sequence, exact expected +outputs, derivation, and source attribution. Expected outputs are unique; the +reference sequence is replayed by this provider-isolation microbenchmark but is +not claimed to be the only possible solution path. The runner judges result +correctness and result-plus-process correctness separately and programmatically; +no LLM-as-a-Judge, fuzzy contains check, or minimum-length heuristic is used. A +free-planning Agent evaluation should judge required milestones, forbidden +shortcuts, and a step budget instead of exact sequence identity. + +Ground-truth annotation uses actual public-page operations through Codex +Desktop's in-app Browser Tool, followed by exact comparison of observed +outputs. `ground_truth_validation.json` is bound to the dataset SHA-256 and +records dates and provenance per task. The 2026-09-02 revision retains 25 +unchanged task annotations from 2026-09-01 and adds five browser-verified +annotations; it does not claim all 30 were re-annotated that day. Benchmark +runs and statistical calculations remain entirely programmatic. + +Absolute P50/P95/mean values describe each provider's cross-task distribution. +Paired ratios are computed per successful task first, then summarized. The P95 +of those ratios is not a division of two marginal P95 values. Comparison cells +share one highlighting rule; rounded ties and unavailable data stay neutral. + +## CubeSandbox + +CubeSandbox is a valid future backend for hardware-isolated, high-volume runs: +its [official examples](https://docs.cubesandbox.com/guide/tutorials/examples.html) +run headless Chromium inside a MicroVM and expose CDP. Its +[resource metrics](https://docs.cubesandbox.com/guide/resource-metrics.html) +provide independent `host_sandbox` and `guest_workload` CPU/current-memory +views. However, the default collection interval is 5 seconds, the endpoint +retains only the latest sample, exact short-task memory peaks are not exported, +and the documented metric families do not include per-sandbox network bytes. + +The initial local reference therefore uses Docker cgroup and network-namespace +accounting at 100 ms. This is a measurement boundary, not a CowAgent deployment +recommendation. A CubeSandbox backend should be added separately for production +topology, hardware isolation, and density tests; its metrics must not be blended +with Docker results. diff --git a/examples/browser_use/agent_smoke.json b/examples/trajectory_replay/agent_smoke.json similarity index 100% rename from examples/browser_use/agent_smoke.json rename to examples/trajectory_replay/agent_smoke.json diff --git a/examples/browser_use/build_public_suite.py b/examples/trajectory_replay/build_public_suite.py similarity index 100% rename from examples/browser_use/build_public_suite.py rename to examples/trajectory_replay/build_public_suite.py diff --git a/examples/browser_use/docker/chromium.Dockerfile b/examples/trajectory_replay/docker/chromium.Dockerfile similarity index 84% rename from examples/browser_use/docker/chromium.Dockerfile rename to examples/trajectory_replay/docker/chromium.Dockerfile index 3c1cec15c..586a51e1f 100644 --- a/examples/browser_use/docker/chromium.Dockerfile +++ b/examples/trajectory_replay/docker/chromium.Dockerfile @@ -6,7 +6,7 @@ RUN apt-get update \ && rm -rf /var/lib/apt/lists/* \ && useradd --create-home --uid 10001 browser -COPY examples/browser_use/docker/start-chromium.sh /usr/local/bin/start-browser +COPY examples/trajectory_replay/docker/start-chromium.sh /usr/local/bin/start-browser RUN chmod 0755 /usr/local/bin/start-browser USER browser diff --git a/examples/browser_use/docker/moli.Dockerfile b/examples/trajectory_replay/docker/moli.Dockerfile similarity index 76% rename from examples/browser_use/docker/moli.Dockerfile rename to examples/trajectory_replay/docker/moli.Dockerfile index 2c7c1bca6..1970441de 100644 --- a/examples/browser_use/docker/moli.Dockerfile +++ b/examples/trajectory_replay/docker/moli.Dockerfile @@ -1,18 +1,19 @@ FROM debian:bookworm-slim@sha256:88200866dfff7ea7f5cbcb6ec7c8a701889efe6fe859fe64d6990e4b07ea4171 ARG TARGETARCH -ARG MOLI_VERSION=v1.0.6 -ARG MOLI_AMD64_SHA256=8c265be9ae49d77b6f86d38fadebe8da6e6d4dd1e5f7894c1819dba737721b91 -ARG MOLI_ARM64_SHA256=c64e7cfdb7ee314b0bf9eeaafcec128020a64a4cea592f576dcb8f9c12b3c2f5 +ARG MOLI_VERSION=v1.1.3 +ARG MOLI_AMD64_SHA256=e1f923f7fb06a684721e24a6c65c20567eaf81388a2225d1a9a175fc6c305df3 +ARG MOLI_ARM64_SHA256=f65ce90c38fdfeaa73b7e1d491823ec6d19371e5b74dbe35b43c8d61f0a8f6b7 RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ ca-certificates curl fontconfig fonts-liberation fonts-noto-cjk \ libcurl4 libgcc-s1 libstdc++6 tar \ - && case "$TARGETARCH" in \ + && resolved_arch="${TARGETARCH:-$(dpkg --print-architecture)}" \ + && case "$resolved_arch" in \ amd64) asset="moli-x86_64-unknown-linux-gnu.tar.gz"; sha="$MOLI_AMD64_SHA256" ;; \ arm64) asset="moli-aarch64-unknown-linux-gnu.tar.gz"; sha="$MOLI_ARM64_SHA256" ;; \ - *) echo "unsupported TARGETARCH=$TARGETARCH" >&2; exit 2 ;; \ + *) echo "unsupported architecture=$resolved_arch" >&2; exit 2 ;; \ esac \ && curl --proto '=https' --tlsv1.2 -fsSL \ "https://github.com/lexmount/moli/releases/download/${MOLI_VERSION}/${asset}" \ diff --git a/examples/browser_use/docker/start-chromium.sh b/examples/trajectory_replay/docker/start-chromium.sh similarity index 88% rename from examples/browser_use/docker/start-chromium.sh rename to examples/trajectory_replay/docker/start-chromium.sh index a472dbab2..6b02f8a74 100755 --- a/examples/browser_use/docker/start-chromium.sh +++ b/examples/trajectory_replay/docker/start-chromium.sh @@ -13,6 +13,9 @@ chromium \ --no-default-browser-check \ --disable-background-networking \ --disable-component-update \ + --disable-breakpad \ + --disable-crash-reporter \ + --disable-crashpad \ --remote-debugging-address=0.0.0.0 \ --remote-debugging-port=9222 \ --user-data-dir=/tmp/chromium-profile \ diff --git a/examples/browser_use/fixtures/app.html b/examples/trajectory_replay/fixtures/app.html similarity index 100% rename from examples/browser_use/fixtures/app.html rename to examples/trajectory_replay/fixtures/app.html diff --git a/examples/browser_use/fixtures/catalog.html b/examples/trajectory_replay/fixtures/catalog.html similarity index 100% rename from examples/browser_use/fixtures/catalog.html rename to examples/trajectory_replay/fixtures/catalog.html diff --git a/examples/browser_use/fixtures/core.html b/examples/trajectory_replay/fixtures/core.html similarity index 100% rename from examples/browser_use/fixtures/core.html rename to examples/trajectory_replay/fixtures/core.html diff --git a/examples/browser_use/fixtures/dynamic.html b/examples/trajectory_replay/fixtures/dynamic.html similarity index 100% rename from examples/browser_use/fixtures/dynamic.html rename to examples/trajectory_replay/fixtures/dynamic.html diff --git a/examples/browser_use/fixtures/forms.html b/examples/trajectory_replay/fixtures/forms.html similarity index 100% rename from examples/browser_use/fixtures/forms.html rename to examples/trajectory_replay/fixtures/forms.html diff --git a/examples/browser_use/fixtures/interaction-space.html b/examples/trajectory_replay/fixtures/interaction-space.html similarity index 100% rename from examples/browser_use/fixtures/interaction-space.html rename to examples/trajectory_replay/fixtures/interaction-space.html diff --git a/examples/browser_use/fixtures/page2.html b/examples/trajectory_replay/fixtures/page2.html similarity index 100% rename from examples/browser_use/fixtures/page2.html rename to examples/trajectory_replay/fixtures/page2.html diff --git a/examples/browser_use/fixtures/page3.html b/examples/trajectory_replay/fixtures/page3.html similarity index 100% rename from examples/browser_use/fixtures/page3.html rename to examples/trajectory_replay/fixtures/page3.html diff --git a/examples/browser_use/ground_truth_validation.json b/examples/trajectory_replay/ground_truth_validation.json similarity index 100% rename from examples/browser_use/ground_truth_validation.json rename to examples/trajectory_replay/ground_truth_validation.json diff --git a/examples/browser_use/rebuild_report.py b/examples/trajectory_replay/rebuild_report.py similarity index 82% rename from examples/browser_use/rebuild_report.py rename to examples/trajectory_replay/rebuild_report.py index eb36344a3..50925e272 100644 --- a/examples/browser_use/rebuild_report.py +++ b/examples/trajectory_replay/rebuild_report.py @@ -7,20 +7,20 @@ import json from pathlib import Path -from harness.browser_use.diagnostics import diagnose_summary -from harness.browser_use.metrics import summarize_attempts -from harness.browser_use.report import write_latest_report -from harness.browser_use.suite import load_suite, validate_ground_truth_evidence +from harness.trajectory_replay.diagnostics import diagnose_summary +from harness.trajectory_replay.metrics import summarize_attempts +from harness.trajectory_replay.report import write_latest_report +from harness.trajectory_replay.suite import load_suite, validate_ground_truth_evidence def main() -> None: parser = argparse.ArgumentParser() - parser.add_argument("--artifacts", default="artifacts/browser_use/latest") + parser.add_argument("--artifacts", default="artifacts/trajectory_replay/latest") parser.add_argument("--reports", default="docs/reports") - parser.add_argument("--suite", default="examples/browser_use/tasks.v4.json") + parser.add_argument("--suite", default="examples/trajectory_replay/tasks.v4.json") parser.add_argument( "--ground-truth-evidence", - default="examples/browser_use/ground_truth_validation.json", + default="examples/trajectory_replay/ground_truth_validation.json", ) args = parser.parse_args() @@ -75,7 +75,7 @@ def main() -> None: json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" ) write_latest_report(args.reports, manifest=manifest, summary=summary) - report_path = Path(args.reports) / "browser-use-evaluation-latest.html" + report_path = Path(args.reports) / "trajectory-replay-evaluation-latest.html" print(f"Recalculated {len(rows)} attempts; report: {report_path}") diff --git a/examples/trajectory_replay/run.sh b/examples/trajectory_replay/run.sh new file mode 100755 index 000000000..b9be045f5 --- /dev/null +++ b/examples/trajectory_replay/run.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +FAILURE_RETRIES="${BROWSER_USE_EVAL_FAILURE_RETRIES:-1}" +SAMPLE_MS="${BROWSER_USE_EVAL_SAMPLE_INTERVAL_MS:-100}" +SETTLE_MS="${BROWSER_USE_EVAL_SETTLE_MS:-1500}" +CPUS="${BROWSER_USE_EVAL_CPUS:-2}" +MEMORY="${BROWSER_USE_EVAL_MEMORY:-2g}" +BUILD_IMAGES="${BROWSER_USE_EVAL_BUILD_IMAGES:-1}" +LLM_DIAGNOSTICS="${BROWSER_USE_EVAL_LLM_DIAGNOSTICS:-0}" +DIAGNOSIS_MODEL="${BROWSER_USE_EVAL_DIAGNOSIS_MODEL:-gpt-5.6-sol}" +DIAGNOSIS_EFFORT="${BROWSER_USE_EVAL_DIAGNOSIS_REASONING_EFFORT:-low}" +OUTPUT="${BROWSER_USE_EVAL_OUTPUT_DIR:-${ROOT}/artifacts/trajectory_replay/latest}" +REPORTS="${ROOT}/docs/reports" + +command -v docker >/dev/null 2>&1 || { echo "docker is required" >&2; exit 2; } +command -v uv >/dev/null 2>&1 || { echo "uv is required" >&2; exit 2; } +docker info >/dev/null + +mkdir -p "${OUTPUT}" "${REPORTS}" +find "${OUTPUT}" -mindepth 1 -maxdepth 1 -type f -delete + +if [[ "${BUILD_IMAGES}" == "1" ]]; then + docker build -f "${ROOT}/examples/trajectory_replay/docker/chromium.Dockerfile" \ + -t lex-browser-use-eval-chromium:local "${ROOT}" + docker build -f "${ROOT}/examples/trajectory_replay/docker/moli.Dockerfile" \ + -t lex-browser-use-eval-moli:local "${ROOT}" +fi + +RUNNER=(uv run --frozen --extra browser-use-eval python) + +EXTRA_ARGS=(--diagnosis-model "${DIAGNOSIS_MODEL}" \ + --diagnosis-reasoning-effort "${DIAGNOSIS_EFFORT}") +if [[ "${LLM_DIAGNOSTICS}" == "1" ]]; then + EXTRA_ARGS+=(--llm-diagnostics) +fi + +"${RUNNER[@]}" -m harness.trajectory_replay.cli \ + --suite "${ROOT}/examples/trajectory_replay/tasks.v4.json" \ + --output "${OUTPUT}" \ + --reports "${REPORTS}" \ + --ground-truth-evidence "${ROOT}/examples/trajectory_replay/ground_truth_validation.json" \ + --failure-retries "${FAILURE_RETRIES}" \ + --sample-interval-ms "${SAMPLE_MS}" \ + --settle-ms "${SETTLE_MS}" \ + --cpus "${CPUS}" \ + --memory "${MEMORY}" \ + "${EXTRA_ARGS[@]}" + +"${RUNNER[@]}" "${ROOT}/examples/trajectory_replay/verify_metrics.py" --artifacts "${OUTPUT}" + +echo "Raw artifacts: ${OUTPUT}" +echo "Latest report: ${REPORTS}/trajectory-replay-evaluation-latest.html" +echo "All metrics and HTML above were generated by repository code; Codex is not part of scoring or calculation." diff --git a/examples/browser_use/tasks.v2.json b/examples/trajectory_replay/tasks.v2.json similarity index 100% rename from examples/browser_use/tasks.v2.json rename to examples/trajectory_replay/tasks.v2.json diff --git a/examples/browser_use/tasks.v3.json b/examples/trajectory_replay/tasks.v3.json similarity index 100% rename from examples/browser_use/tasks.v3.json rename to examples/trajectory_replay/tasks.v3.json diff --git a/examples/browser_use/tasks.v4.json b/examples/trajectory_replay/tasks.v4.json similarity index 100% rename from examples/browser_use/tasks.v4.json rename to examples/trajectory_replay/tasks.v4.json diff --git a/examples/browser_use/verify_metrics.py b/examples/trajectory_replay/verify_metrics.py similarity index 99% rename from examples/browser_use/verify_metrics.py rename to examples/trajectory_replay/verify_metrics.py index 30fd79847..7f60857c5 100644 --- a/examples/browser_use/verify_metrics.py +++ b/examples/trajectory_replay/verify_metrics.py @@ -137,5 +137,5 @@ def value(task, engine, metric): if __name__ == "__main__": parser = argparse.ArgumentParser() - parser.add_argument("--artifacts", type=Path, default=Path("artifacts/browser_use/latest")) + parser.add_argument("--artifacts", type=Path, default=Path("artifacts/trajectory_replay/latest")) print(json.dumps(verify(parser.parse_args().artifacts))) diff --git a/examples/web_fetch/README.md b/examples/web_fetch/README.md index 63c817638..01a6f8e12 100644 --- a/examples/web_fetch/README.md +++ b/examples/web_fetch/README.md @@ -1,5 +1,5 @@ # Web Fetch Examples Versioned Web Fetch suites and one-command runners belong here. This capability -is intentionally separate from `examples/browser_use` and is not implemented in +is intentionally separate from `examples/trajectory_replay` and is not implemented in the first Browser Use evaluation delivery. diff --git a/harness/browser_use/__init__.py b/harness/browser_use/__init__.py index 5d8e0824d..e1ff81886 100644 --- a/harness/browser_use/__init__.py +++ b/harness/browser_use/__init__.py @@ -1,5 +1,5 @@ -"""Reproducible browser-engine evaluation with isolated resource accounting.""" +"""Autonomous browser-use evaluation built on WebArena-Verified.""" -from .suite import BrowserTask, TaskSuite, load_suite +from .selection import PILOT_SCHEMA, PilotTask, load_pilot -__all__ = ["BrowserTask", "TaskSuite", "load_suite"] +__all__ = ["PILOT_SCHEMA", "PilotTask", "load_pilot"] diff --git a/harness/browser_use/assets/webarena-autonomous-evaluation-flow.png b/harness/browser_use/assets/webarena-autonomous-evaluation-flow.png new file mode 100644 index 000000000..72fb2a7dd Binary files /dev/null and b/harness/browser_use/assets/webarena-autonomous-evaluation-flow.png differ diff --git a/harness/browser_use/failure_cause_review.py b/harness/browser_use/failure_cause_review.py new file mode 100644 index 000000000..0649067a8 --- /dev/null +++ b/harness/browser_use/failure_cause_review.py @@ -0,0 +1,149 @@ +"""Load model-scoped Browser Use failure-cause reviews from one layered source.""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from typing import Any + +LAYERED_REVIEW_SCHEMA = "lex_browser_use_layered_failure_cause_review/2" +DEFAULT_REVIEW_PATH = ( + Path(__file__).parents[2] / "research/browser-use-failure-cause-review.json" +) + + +def sha256(path: Path) -> str | None: + return hashlib.sha256(path.read_bytes()).hexdigest() if path.is_file() else None + + +def load_failure_cause_review(path: Path = DEFAULT_REVIEW_PATH) -> dict[str, Any]: + return json.loads(path.read_text(encoding="utf-8")) if path.is_file() else {} + + +def extract_failure_cause_review_cohort( + review: dict[str, Any], model_id: str +) -> dict[str, Any]: + """Extract one model cohort from the authoritative layered review document.""" + + if not review: + return {} + if review.get("schema") != LAYERED_REVIEW_SCHEMA: + raise ValueError("unsupported Browser Use failure-cause review schema") + cohorts = review.get("cohorts") + if not isinstance(cohorts, dict): + raise ValueError("layered failure-cause review must contain model cohorts") + cohort = cohorts.get(model_id) + if cohort is None: + return {} + if not isinstance(cohort, dict): + raise ValueError(f"failure-cause review cohort must be an object: {model_id}") + return cohort + + +def validate_failure_cause_review( + review: dict[str, Any], + *, + model_id: str, + attempts_sha256: str | None, + failed_result_ids: set[str], + require_complete: bool, +) -> dict[str, dict[str, Any]]: + """Return verified causes for one model; reject stale or cross-model records.""" + + result_prefix = f"{model_id}-" + foreign_failures = sorted( + result_id for result_id in failed_result_ids if not result_id.startswith(result_prefix) + ) + if foreign_failures: + raise ValueError( + f"failed-result inventory includes another model for {model_id}: {foreign_failures}" + ) + cohort = extract_failure_cause_review_cohort(review, model_id) + if not cohort: + if require_complete: + raise ValueError(f"complete publication requires a failure-cause review: {model_id}") + return {} + + bindings = cohort.get("bindings") + if not isinstance(bindings, dict): + raise ValueError(f"failure-cause review cohort has invalid bindings: {model_id}") + bound_hash = bindings.get("attempts_sha256") + if not attempts_sha256 or bound_hash != attempts_sha256: + if require_complete: + raise ValueError( + f"failure-cause review is not bound to the current attempts: {model_id}" + ) + return {} + + coverage = cohort.get("coverage") + results = cohort.get("results") + unresolved = cohort.get("unresolved") + review_protocol = cohort.get("review_protocol") + if not isinstance(coverage, dict): + raise ValueError(f"failure-cause review cohort has invalid coverage: {model_id}") + if not isinstance(results, dict): + raise ValueError(f"failure-cause review cohort has invalid results: {model_id}") + if not isinstance(unresolved, list): + raise ValueError(f"failure-cause review cohort has invalid unresolved items: {model_id}") + if not isinstance(review_protocol, dict): + raise ValueError(f"failure-cause review cohort has invalid review protocol: {model_id}") + + verified: dict[str, dict[str, Any]] = {} + for result_id, item in results.items(): + if not isinstance(result_id, str) or not result_id.startswith(result_prefix): + raise ValueError( + f"failure-cause review includes a result outside its model cohort: {result_id}" + ) + if result_id not in failed_result_ids: + raise ValueError(f"failure-cause review includes a non-failed result: {result_id}") + if not isinstance(item, dict): + raise ValueError(f"failure-cause review result must be an object: {result_id}") + status = str(item.get("status") or "") + category = item.get("subcategory") or {} + reason = item.get("reason") or {} + if status != "verified": + if category or reason: + raise ValueError( + "unverified failure cause must not contain a subcategory or reason: " + f"{result_id}" + ) + continue + if not isinstance(category, dict) or not all( + category.get(language) for language in ("code", "zh", "en") + ): + raise ValueError(f"verified failure cause has an incomplete subcategory: {result_id}") + if not isinstance(reason, dict) or not all( + reason.get(language) for language in ("zh", "en") + ): + raise ValueError(f"verified failure cause has an incomplete reason: {result_id}") + verified[result_id] = item + + if require_complete and set(verified) != failed_result_ids: + missing = sorted(failed_result_ids - set(verified)) + raise ValueError(f"failure-cause review is incomplete for {model_id}: {missing}") + if require_complete: + expected_coverage = { + "failed_results": len(failed_result_ids), + "verified_results": len(verified), + } + if coverage != expected_coverage: + raise ValueError(f"failure-cause review coverage is inconsistent: {model_id}") + if unresolved: + raise ValueError(f"failure-cause review contains unresolved items: {model_id}") + if not cohort.get("reviewer_class") or not cohort.get("reviewed_at"): + raise ValueError( + f"failure-cause review must identify its reviewer class and date: {model_id}" + ) + return verified + + +def localized_cause( + verified: dict[str, dict[str, Any]], result_id: str, language: str = "zh" +) -> tuple[str, str]: + """Return blank fields until a result has a verified, current review.""" + + item = verified.get(result_id) or {} + category = item.get("subcategory") or {} + reason = item.get("reason") or {} + return str(category.get(language) or ""), str(reason.get(language) or "") diff --git a/harness/browser_use/har.py b/harness/browser_use/har.py new file mode 100644 index 000000000..6628a2c10 --- /dev/null +++ b/harness/browser_use/har.py @@ -0,0 +1,244 @@ +"""CDP network recorder that writes evaluator-compatible HAR.""" + +from __future__ import annotations + +import base64 +import json +from pathlib import Path +from typing import Any + + +def _headers(values: dict[str, Any]) -> list[dict[str, str]]: + return [{"name": str(name), "value": str(value)} for name, value in values.items()] + + +def _har_request(request: dict[str, Any]) -> dict[str, Any]: + headers = dict(request.get("headers", {})) + if request.get("__resource_type") == "Document" and not any( + name.lower() == "accept" for name in headers + ): + # CDP marks the request as a top-level document even when a browser + # implementation omits default Accept headers from requestWillBeSent. + headers["Accept"] = "text/html" + result: dict[str, Any] = { + "method": str(request["method"]), + "url": str(request["url"]), + "headers": _headers(headers), + "cookies": [], + } + if request.get("postData") is not None: + content_type = next( + (str(value) for name, value in headers.items() if name.lower() == "content-type"), + "", + ) + result["postData"] = {"mimeType": content_type, "text": str(request["postData"])} + return result + + +def _har_response(response: dict[str, Any], body: dict[str, Any] | None) -> dict[str, Any]: + headers = response.get("headers", {}) + mime_type = str(response.get("mimeType") or "") + content: dict[str, Any] = {"mimeType": mime_type, "text": ""} + if body is not None: + content["text"] = str(body.get("body") or "") + return { + "status": int(response["status"]), + "statusText": str(response.get("statusText") or ""), + "headers": _headers(headers), + "cookies": [], + "redirectURL": next( + (str(value) for name, value in headers.items() if name.lower() == "location"), + "", + ), + "content": content, + } + + +class CdpHarRecorder: + """Capture raw CDP requests from an existing browser context. + + Playwright trace ZIPs from an attached CDP browser can omit multipart POST + bodies. Raw ``Network.requestWillBeSent`` events retain those bodies, which + the official WebArena-Verified mutation evaluators require. + """ + + def __init__(self) -> None: + self._context: Any = None + self._sessions: list[Any] = [] + self._requests: dict[tuple[int, str], dict[str, Any]] = {} + self._request_ordinals: dict[tuple[int, str], int] = {} + self._extra_headers: dict[tuple[int, str], list[dict[str, Any]]] = {} + self._responses: list[tuple[Any, str, dict[str, Any], dict[str, Any]]] = [] + self._finished: set[tuple[int, str]] = set() + self._response_bodies: dict[tuple[int, str], dict[str, Any]] = {} + self._response_body_errors: dict[tuple[int, str], str] = {} + self._payload: str | None = None + self._stopped = False + self.errors: list[str] = [] + + def start(self, context: Any) -> None: + self._context = context + context.on("page", self._attach) + for page in context.pages: + self._attach(page) + + def stop(self, context: Any, path: Path) -> None: + if self._stopped: + return + if self._payload is None: + if self._context is not None: + try: + context.remove_listener("page", self._attach) + except Exception as exc: + self.errors.append(f"page listener detach failed: {exc}") + entries: list[dict[str, Any]] = [] + # CDP callbacks run on the browser thread and may append a final + # response while stop() is serializing. Use one immutable view for + # both the redirect lookup and entry loop so a late event cannot + # create a response whose key is absent from last_response_by_key. + responses = list(self._responses) + last_response_by_key = { + (id(session), request_id): index + for index, (session, request_id, _request, _response) in enumerate( + responses + ) + } + for index, (session, request_id, request, response) in enumerate(responses): + key = (id(session), request_id) + body = self._response_bodies.get(key) + request = dict(request) + ordinal = int(request.pop("__ordinal", 0)) + extras = self._extra_headers.get(key, []) + if ordinal < len(extras): + request["headers"] = {**request.get("headers", {}), **extras[ordinal]} + mime_type = str(response.get("mimeType") or "") + if ( + index == last_response_by_key[key] + and key in self._finished + and body is None + and (mime_type == "application/json" or mime_type.endswith("+json")) + ): + try: + body = session.send("Network.getResponseBody", {"requestId": request_id}) + body = self._decode_body(body) + except Exception as exc: + prior = self._response_body_errors.get(key) + detail = f"{prior}; stop fallback: {exc}" if prior else str(exc) + self.errors.append( + f"{request.get('url')}: response body unavailable: {detail}" + ) + body = None + entries.append( + {"request": _har_request(request), "response": _har_response(response, body)} + ) + for session in self._sessions: + try: + session.detach() + except Exception as exc: + self.errors.append(f"CDP detach failed: {exc}") + self._context = None + self._payload = json.dumps( + { + "log": { + "version": "1.2", + "creator": {"name": "browser-eval", "version": "1"}, + "entries": entries, + } + }, + ensure_ascii=False, + ) + path.write_text(self._payload, encoding="utf-8") + self._stopped = True + + def _attach(self, page: Any) -> None: + try: + assert self._context is not None + session = self._context.new_cdp_session(page) + session_index = id(session) + session.on( + "Network.requestWillBeSent", + lambda event: self._record_request(session, session_index, event), + ) + session.on( + "Network.responseReceived", + lambda event: self._record_response(session, session_index, event), + ) + session.on( + "Network.requestWillBeSentExtraInfo", + lambda event: self._extra_headers.setdefault( + (session_index, str(event["requestId"])), [] + ).append(event.get("headers", {})), + ) + session.on( + "Network.loadingFinished", + lambda event: self._record_finished(session, session_index, event), + ) + session.send("Network.enable") + self._sessions.append(session) + except Exception as exc: + self.errors.append(f"CDP attach failed: {exc}") + + def _record_request(self, session: Any, session_index: int, event: dict[str, Any]) -> None: + request_id = str(event["requestId"]) + key = (session_index, request_id) + previous = self._requests.get(key) + redirect_response = event.get("redirectResponse") + if previous is not None and redirect_response is not None: + self._responses.append((session, request_id, dict(previous), redirect_response)) + ordinal = self._request_ordinals.get(key, 0) + self._request_ordinals[key] = ordinal + 1 + request = dict(event["request"]) + request["__ordinal"] = ordinal + request["__resource_type"] = event.get("type") + self._requests[key] = request + + def _record_response(self, session: Any, session_index: int, event: dict[str, Any]) -> None: + request_id = str(event["requestId"]) + request = self._requests.get((session_index, request_id)) + if request is not None: + self._responses.append((session, request_id, dict(request), event["response"])) + + @staticmethod + def _decode_body(body: dict[str, Any]) -> dict[str, Any]: + if body.get("base64Encoded"): + return { + "body": base64.b64decode(body.get("body") or "", validate=True).decode( + "utf-8", errors="replace" + ), + "base64Encoded": False, + } + return body + + def _record_finished( + self, session: Any, session_index: int, event: dict[str, Any] + ) -> None: + """Capture JSON bodies while CDP still retains them. + + Waiting until the whole agent run ends is too late for long tasks: CDP may + evict an earlier response body even though the request metadata remains. + """ + + request_id = str(event["requestId"]) + key = (session_index, request_id) + self._finished.add(key) + response = next( + ( + item_response + for item_session, item_request_id, _request, item_response in reversed( + self._responses + ) + if id(item_session) == session_index and item_request_id == request_id + ), + None, + ) + mime_type = str((response or {}).get("mimeType") or "") + if mime_type != "application/json" and not mime_type.endswith("+json"): + return + try: + self._response_bodies[key] = self._decode_body( + session.send("Network.getResponseBody", {"requestId": request_id}) + ) + self._response_body_errors.pop(key, None) + except Exception as exc: + # stop() retries once before exposing a capture error. + self._response_body_errors[key] = str(exc) diff --git a/harness/browser_use/model.py b/harness/browser_use/model.py new file mode 100644 index 000000000..4aa48c3b8 --- /dev/null +++ b/harness/browser_use/model.py @@ -0,0 +1,244 @@ +"""OpenAI Responses transport adapter for CowAgent's model protocol.""" + +from __future__ import annotations + +import json +from collections.abc import Iterable +from typing import Any + +import requests + + +def _text_from_blocks(content: Any) -> str: + if isinstance(content, str): + return content + if not isinstance(content, list): + return "" + return "\n".join( + str(block.get("text") or "") + for block in content + if isinstance(block, dict) and block.get("type") == "text" + ) + + +def cow_messages_to_responses(messages: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Translate CowAgent's Claude-shaped history to Responses input items.""" + items: list[dict[str, Any]] = [] + for message in messages: + role = str(message.get("role") or "user") + content = message.get("content") + text = _text_from_blocks(content) + if text: + items.append({"role": role, "content": text}) + if not isinstance(content, list): + continue + for block in content: + if not isinstance(block, dict): + continue + if block.get("type") == "tool_use": + items.append( + { + "type": "function_call", + "call_id": str(block.get("id") or ""), + "name": str(block.get("name") or ""), + "arguments": json.dumps( + block.get("input") or {}, ensure_ascii=False, separators=(",", ":") + ), + } + ) + elif block.get("type") == "tool_result": + output = block.get("content") + if not isinstance(output, str): + output = json.dumps(output, ensure_ascii=False) + items.append( + { + "type": "function_call_output", + "call_id": str(block.get("tool_use_id") or ""), + "output": output, + } + ) + return items + + +def cow_tools_to_responses(tools: list[dict[str, Any]] | None) -> list[dict[str, Any]]: + return [ + { + "type": "function", + "name": str(tool["name"]), + "description": str(tool.get("description") or ""), + "parameters": tool.get("input_schema") or {"type": "object", "properties": {}}, + "strict": False, + } + for tool in tools or [] + ] + + +def response_to_chat_chunk(response: dict[str, Any]) -> dict[str, Any]: + content: list[str] = [] + calls: list[dict[str, Any]] = [] + for item in response.get("output") or []: + if item.get("type") == "message": + for block in item.get("content") or []: + if block.get("type") == "output_text" and block.get("text"): + content.append(str(block["text"])) + elif item.get("type") == "function_call": + calls.append( + { + "index": len(calls), + "id": str(item.get("call_id") or item.get("id") or ""), + "type": "function", + "function": { + "name": str(item.get("name") or ""), + "arguments": str(item.get("arguments") or "{}"), + }, + } + ) + delta: dict[str, Any] = {"content": "".join(content)} + if calls: + delta["tool_calls"] = calls + return { + "choices": [ + { + "delta": delta, + "finish_reason": "tool_calls" if calls else "stop", + } + ], + "usage": response.get("usage"), + "response_id": response.get("id"), + } + + +class OpenAIResponsesModel: + """Minimal adapter: CowAgent loop and tools stay unchanged; only transport differs.""" + + def __init__( + self, + *, + api_key: str, + base_url: str = "https://api.openai.com/v1", + model: str = "gpt-6-astra", + reasoning_effort: str = "low", + timeout_seconds: float = 180, + session: requests.Session | None = None, + ) -> None: + if reasoning_effort != "low": + raise ValueError("pilot protocol fixes OpenAI reasoning effort to low") + self.model = model + self.api_key = api_key + self.base_url = base_url.rstrip("/") + self.reasoning_effort = reasoning_effort + self.timeout_seconds = timeout_seconds + self.session = session or requests.Session() + self.last_response: dict[str, Any] | None = None + self._previous_response_id: str | None = None + self._seen_message_count = 0 + self.usage: dict[str, int] = {} + self.request_count = 0 + self.request_attempt_count = 0 + self.transport_errors: list[dict[str, str]] = [] + + def reset(self) -> None: + """Start a fresh Responses conversation for the next benchmark attempt.""" + self.last_response = None + self._previous_response_id = None + self._seen_message_count = 0 + self.usage = {} + self.request_count = 0 + self.request_attempt_count = 0 + self.transport_errors = [] + + def _normalized_reasoning_effort(self) -> str: + return self.reasoning_effort + + def verify_access(self) -> None: + """Fail before benchmark attempts if the configured project cannot access the model.""" + response = self.session.get( + f"{self.base_url}/models/{self.model}", + headers={"Authorization": f"Bearer {self.api_key}"}, + timeout=30, + ) + if response.status_code < 400: + payload = response.json() + if payload.get("id") != self.model: + raise ValueError(f"model lookup returned an unexpected id: {payload.get('id')!r}") + return + if response.status_code not in {403, 404, 405}: + response.raise_for_status() + + # Some OpenAI-compatible gateways expose a team-filtered collection but + # disable the single-model route. The collection remains a read-only + # access preflight and avoids spending a benchmark model call. + collection = self.session.get( + f"{self.base_url}/models", + headers={"Authorization": f"Bearer {self.api_key}"}, + timeout=30, + ) + collection.raise_for_status() + ids = { + str(item.get("id")) + for item in collection.json().get("data") or [] + if isinstance(item, dict) and item.get("id") + } + if self.model not in ids: + raise PermissionError(f"configured API project does not expose model {self.model!r}") + + def call_stream(self, request: Any) -> Iterable[dict[str, Any]]: + messages = list(request.messages or []) + if self._previous_response_id: + new_messages = messages[self._seen_message_count :] + input_items = cow_messages_to_responses( + [message for message in new_messages if message.get("role") != "assistant"] + ) + else: + input_items = cow_messages_to_responses(messages) + payload: dict[str, Any] = { + "model": self.model, + "instructions": str(getattr(request, "system", "") or ""), + "input": input_items, + "reasoning": {"effort": self.reasoning_effort}, + "tools": cow_tools_to_responses(request.tools), + } + if self._previous_response_id: + payload["previous_response_id"] = self._previous_response_id + if getattr(request, "max_tokens", None): + payload["max_output_tokens"] = int(request.max_tokens) + self.request_attempt_count += 1 + try: + response = self.session.post( + f"{self.base_url}/responses", + headers={ + "Authorization": f"Bearer {self.api_key}", + "Content-Type": "application/json", + }, + json=payload, + timeout=self.timeout_seconds, + ) + response.raise_for_status() + except requests.RequestException as exc: + self.transport_errors.append( + {"type": type(exc).__name__, "message": str(exc)[:500]} + ) + raise + self.last_response = response.json() + response_id = self.last_response.get("id") + if not isinstance(response_id, str) or not response_id: + raise ValueError("OpenAI Responses API success payload omitted a non-empty id") + if not isinstance(self.last_response.get("output"), list): + raise ValueError("OpenAI Responses API success payload omitted the output list") + self.request_count += 1 + usage = self.last_response.get("usage") or {} + details = usage.get("output_tokens_details") or {} + for key, value in ( + ("input_tokens", usage.get("input_tokens")), + ("output_tokens", usage.get("output_tokens")), + ("reasoning_tokens", details.get("reasoning_tokens")), + ("total_tokens", usage.get("total_tokens")), + ): + self.usage[key] = self.usage.get(key, 0) + int(value or 0) + self._previous_response_id = response_id + self._seen_message_count = len(messages) + yield response_to_chat_chunk(self.last_response) + + +# Compatibility for callers created before the pilot allowed a non-Astra smoke model. +AstraResponsesModel = OpenAIResponsesModel diff --git a/harness/browser_use/publication.py b/harness/browser_use/publication.py new file mode 100644 index 000000000..97b24309a --- /dev/null +++ b/harness/browser_use/publication.py @@ -0,0 +1,186 @@ +"""Build privacy-safe public projections of browser-use evidence.""" + +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import Path +from typing import Any + +_ORIGIN_REPLACEMENTS = { + "http://host.docker.internal:7770": "__SHOPPING__", + "http://host.docker.internal:7780/admin": "__SHOPPING_ADMIN__", + "http://host.docker.internal:7780": "__SHOPPING_ADMIN__", + "http://host.docker.internal:9999": "__REDDIT__", + "http://host.docker.internal:8023": "__GITLAB__", + "http://192.168.5.2:7770": "__SHOPPING__", + "http://192.168.5.2:7780/admin": "__SHOPPING_ADMIN__", + "http://192.168.5.2:7780": "__SHOPPING_ADMIN__", + "http://192.168.5.2:9999": "__REDDIT__", + "http://192.168.5.2:8023": "__GITLAB__", +} +_KNOWN_SECRET_VALUES = ( + "Password.123", + "admin1234", + "test1234", + "hello1234", +) +_SENSITIVE_KEY = re.compile( + r"(?:authorization|cookie|password|passwd|secret|token|storage_state|api[_-]?key)", re.I +) +_LOCAL_PATH = re.compile(r"/(?:Users|home|private|tmp|var|opt)/[^\s\"']+") +_BEARER = re.compile(r"(?i)bearer\s+[A-Za-z0-9._~+/=-]+") + + +def sha256_file(path: Path) -> str | None: + return hashlib.sha256(path.read_bytes()).hexdigest() if path.is_file() else None + + +def sanitize_text(value: str) -> str: + text = value + for origin, placeholder in sorted( + _ORIGIN_REPLACEMENTS.items(), key=lambda item: len(item[0]), reverse=True + ): + text = text.replace(origin, placeholder) + for secret in _KNOWN_SECRET_VALUES: + text = text.replace(secret, "[REDACTED]") + text = text.replace("host.docker.internal", "__BENCHMARK_HOST__") + text = text.replace("192.168.5.2", "__BENCHMARK_HOST__") + text = _BEARER.sub("Bearer [REDACTED]", text) + text = _LOCAL_PATH.sub("", text) + return text + + +def sanitize(value: Any, *, key: str = "") -> Any: + """Recursively redact credentials while preserving benchmark evidence.""" + + if value is None or value == [] or value == {}: + return value + if _SENSITIVE_KEY.search(key): + return "[REDACTED]" + if isinstance(value, str): + return sanitize_text(value) + if isinstance(value, list): + return [sanitize(item) for item in value] + if isinstance(value, tuple): + return [sanitize(item) for item in value] + if isinstance(value, dict): + return {str(k): sanitize(v, key=str(k)) for k, v in value.items()} + return value + + +def read_public_json(path: Path) -> Any: + if not path.is_file(): + return None + return sanitize(json.loads(path.read_text(encoding="utf-8"))) + + +def task_contract(task: Any) -> dict[str, Any]: + raw = task.model_dump(mode="json") if hasattr(task, "model_dump") else dict(task) + return sanitize( + { + "task_id": raw.get("task_id"), + "intent_template_id": raw.get("intent_template_id"), + "revision": raw.get("revision"), + "sites": raw.get("sites") or [], + "start_urls": raw.get("start_urls") or [], + "intent": raw.get("intent"), + "evaluation_contract": raw.get("eval") or [], + } + ) + + +def evaluator_summary(result: dict[str, Any] | None) -> list[dict[str, Any]]: + if not result: + return [] + summaries = [] + for index, item in enumerate(result.get("evaluators_results") or [], start=1): + summaries.append( + { + "index": index, + "name": item.get("evaluator_name") or "UnknownEvaluator", + "status": item.get("status"), + "score": item.get("score"), + "expected": item.get("expected"), + "actual": item.get("actual_normalized", item.get("actual")), + "assertions": item.get("assertions"), + "error": item.get("error_msg"), + "should_not_exist": item.get("should_not_exist"), + } + ) + return summaries + + +def trajectory(events: list[dict[str, Any]] | None) -> list[dict[str, Any]]: + """Project CowAgent events into a readable, complete BrowserTool action sequence.""" + + if not events: + return [] + turn = 0 + starts: dict[str, dict[str, Any]] = {} + rows: list[dict[str, Any]] = [] + for event in events: + event_type = event.get("type") + data = event.get("data") or {} + if event_type == "turn_start": + turn = int(data.get("turn") or turn + 1) + elif event_type == "tool_execution_start": + call_id = str(data.get("tool_call_id") or "") + starts[call_id] = { + "step": len(rows) + 1, + "turn": turn, + "timestamp": event.get("timestamp"), + "tool": data.get("tool_name"), + "arguments": data.get("arguments"), + } + elif event_type == "tool_execution_end": + call_id = str(data.get("tool_call_id") or "") + row = starts.pop(call_id, {"step": len(rows) + 1, "turn": turn}) + row.update( + { + "status": data.get("status"), + "duration_seconds": data.get("execution_time"), + "result": data.get("result"), + "error": data.get("error"), + } + ) + rows.append(sanitize(row)) + for row in starts.values(): + row.update({"status": "missing_end_event", "result": None, "error": None}) + rows.append(sanitize(row)) + return rows + + +def trajectory_label(step: dict[str, Any]) -> str: + arguments = step.get("arguments") or {} + action = arguments.get("action") or step.get("tool") or "tool" + target = arguments.get("url") or arguments.get("ref") or arguments.get("text") or "" + if isinstance(target, (dict, list)): + target = json.dumps(target, ensure_ascii=False) + label = f"{action} {target}".strip() + return label[:180] + ("…" if len(label) > 180 else "") + + +def result_excerpt(step: dict[str, Any], limit: int = 900) -> str: + text = str(step.get("error") or step.get("result") or "") + if len(text) <= limit: + return text + return text[:limit].rstrip() + "\n…(完整内容见脱敏事件日志)" + + +def evidence_metadata(row: dict[str, Any]) -> dict[str, Any]: + files = {} + for label, raw_path in (row.get("paths") or {}).items(): + path = Path(raw_path) + files[label] = { + "present": path.is_file(), + "bytes": path.stat().st_size if path.is_file() else None, + "sha256": sha256_file(path), + "publication": ( + "sanitized_projection" + if label in {"response", "evaluation", "events"} + else "private_only_session_sensitive" + ), + } + return {"attempt_id": f"{row['engine']}-{row['task_id']}-a{row['attempt']}", "files": files} diff --git a/harness/browser_use/report.py b/harness/browser_use/report.py index a05fb4944..abed9bbe4 100644 --- a/harness/browser_use/report.py +++ b/harness/browser_use/report.py @@ -1,1318 +1,3716 @@ -"""Chinese paginated evidence-first browser benchmark report.""" +"""Build the public, drill-down browser-use evidence report.""" from __future__ import annotations +import argparse +import hashlib import html import json +import math import re -from collections import Counter -from dataclasses import dataclass +import shutil +import statistics +import tempfile +import zipfile +from collections import Counter, defaultdict +from collections.abc import Callable, Iterable +from io import BytesIO from pathlib import Path from typing import Any -from urllib.parse import urlparse - -REPORT_STEM = "browser-use-evaluation-latest" -PAGES = tuple( - (slug, label, f"{REPORT_STEM}{suffix}.html") - for slug, label, suffix in ( - ("index", "结论摘要", ""), - ("suite", "评测集", "-suite"), - ("method", "环境与方法", "-method"), - ("analysis", "逐任务分析", "-analysis"), - ("appendix", "复现与附录", "-appendix"), - ) + +from PIL import Image + +from harness.web_fetch.report_style import STYLE as SHARED_STYLE + +from .failure_cause_review import ( + DEFAULT_REVIEW_PATH as FAILURE_CAUSE_REVIEW_PATH, +) +from .failure_cause_review import ( + LAYERED_REVIEW_SCHEMA, + extract_failure_cause_review_cohort, + load_failure_cause_review, + localized_cause, + validate_failure_cause_review, +) +from .failure_cause_review import ( + sha256 as file_sha256, +) +from .publication import ( + evaluator_summary, + evidence_metadata, + read_public_json, + result_excerpt, + sanitize, + task_contract, + trajectory, + trajectory_label, +) +from .report_i18n import write_english_report +from .selection import PilotSelection, load_pilot, validate_against_official + +ENGINE_LABELS = {"chromium": "Chromium", "moli": "Moli 1.1.3"} +MODEL_LABELS = {"gpt-5.6-sol": "GPT-5.6 Sol"} +BASELINE_ENGINE = "chromium" +TIER_LABELS = {"easy": "简单", "medium": "中等", "hard": "困难"} +SITE_LABELS = { + "shopping": "Shopping", + "shopping_admin": "Shopping Admin", + "reddit": "Reddit", + "gitlab": "GitLab", +} +MUTATE_IDS = {399, 404, 441, 506} +NAVIGATE_IDS = {44, 45, 118} +RESEARCH_ROOT = Path(__file__).parents[2] / "research" +MAC_EXPANSION = json.loads( + (RESEARCH_ROOT / "webarena-mac-expansion.json").read_text(encoding="utf-8") ) +MAC_ENVIRONMENT_CANDIDATES = MAC_EXPANSION["environments"] +MAC_COUNTS = MAC_EXPANSION["official_counts"] -def _e(value: Any) -> str: - return html.escape(str(value if value is not None else "")) +def _finding_text(diagnosis: dict[str, Any], field: str) -> str: + value = diagnosis.get(field, {}) + return str(value.get("zh") or value.get("en") or "") if isinstance(value, dict) else str(value) -def _trace_text(value: Any) -> str: - """Remove machine-identifying IPs and diff-breaking line-end whitespace.""" - text = str(value if value is not None else "") - text = re.sub(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", "[IP 已脱敏]", text) - return "\n".join(line.rstrip() for line in text.splitlines()) +def _configure_engine_labels(profile: dict[str, Any]) -> None: + engines = [str(engine) for engine in profile.get("engines") or []] + labels = profile.get("engine_labels") or {} + if not engines or len(engines) != len(set(engines)): + raise ValueError("current Browser Use profile must register unique browser engines") + if set(labels) - set(engines): + raise ValueError("Browser Use engine labels contain an unregistered browser") + global ENGINE_LABELS + ENGINE_LABELS = { + engine: str(labels.get(engine) or engine.replace("_", " ").title()) for engine in engines + } -def _show(value: Any, divisor: float = 1.0, suffix: str = "") -> str: - return f"{value / divisor:,.2f}{suffix}" if isinstance(value, (int, float)) else "无数据" +def _configure_models(profile: dict[str, Any]) -> None: + models = [str(model) for model in profile.get("models") or []] + labels = profile.get("model_labels") or {} + baseline = str(profile.get("baseline_engine") or "") + if not models or len(models) != len(set(models)): + raise ValueError("current Browser Use profile must register unique models") + if any(not re.fullmatch(r"[A-Za-z0-9._-]+", model) for model in models): + raise ValueError("Browser Use model IDs must be safe path components") + if set(labels) - set(models): + raise ValueError("Browser Use model labels contain an unregistered model") + if baseline not in ENGINE_LABELS: + raise ValueError("Browser Use baseline_engine must be a registered browser engine") + global MODEL_LABELS, BASELINE_ENGINE + MODEL_LABELS = {model: str(labels.get(model) or model) for model in models} + BASELINE_ENGINE = baseline -def _stat(metric: dict[str, Any], key: str, divisor: float = 1.0) -> str: - value = metric.get(key, metric.get("median") if key == "p50" else None) - return _show(value, divisor) +METHODOLOGY_FIGURE = Path(__file__).parent / "assets" / "webarena-autonomous-evaluation-flow.png" -def _rate(values: dict[str, Any]) -> str: - value = values.get("pass_rate") - return _show(value * 100 if isinstance(value, (int, float)) else None, suffix="%") +def _e(value: Any) -> str: + return html.escape(str(value), quote=True) -@dataclass(frozen=True) -class ComparedValue: - text: str - css_class: str +class Safe(str): + pass -def _comparison_class(value: Any, other: Any, *, higher: bool = False) -> str: - if not isinstance(value, (int, float)) or not isinstance(other, (int, float)): - return "" - if value == other: - return "" - return "better" if (value > other if higher else value < other) else "worse" +def _read_json(path: Path) -> dict[str, Any]: + return json.loads(path.read_text(encoding="utf-8")) if path.is_file() else {} -def _table( - headers: list[str], rows: list[list[Any]], caption: str, *, source_marks: bool = False -) -> str: - head = "".join(f"{_e(item)}" for item in headers) - body = "".join( - "" - + "".join( - f'" - + ( - _annotate_source_marks(_e(v.text if isinstance(v, ComparedValue) else v)) - if source_marks - else _e(v.text if isinstance(v, ComparedValue) else v) - ) - + "" - for i, v in enumerate(row) - ) - + "" - for row in rows +def _read_attempts(evidence: Path) -> list[dict[str, Any]]: + path = evidence / "attempts.jsonl" + return ( + [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line] + if path.is_file() + else [] ) - return f'
{head}{body}
{_e(caption)}
' -def _numeric_stat(metric: dict[str, Any], key: str) -> float | int | None: - value = metric.get(key, metric.get("median") if key == "p50" else None) - return value if isinstance(value, (int, float)) else None +def _evidence_paths(evidence: Path | Iterable[Path]) -> list[Path]: + paths = [evidence] if isinstance(evidence, Path) else list(evidence) + resolved = [Path(path).resolve() for path in paths] + if not resolved: + raise ValueError("at least one Browser Use evidence directory is required") + if len(resolved) != len(set(resolved)): + raise ValueError("Browser Use evidence directories must be unique") + return resolved + + +def _load_evidence_bundles( + evidence: Path | Iterable[Path], +) -> list[dict[str, Any]]: + """Load isolated model cohorts without ever combining physical attempts on disk.""" + + paths = _evidence_paths(evidence) + bundles: list[dict[str, Any]] = [] + seen_models: set[str] = set() + for path in paths: + manifest = _read_json(path / "manifest.json") + config = _read_json(path / "run-config.json") + review = _read_json(path / "result-review.json") + protocol = manifest.get("protocol") or {} + model = str(protocol.get("model") or "") + if not model and len(paths) == 1 and len(MODEL_LABELS) == 1: + model = next(iter(MODEL_LABELS)) + if not model: + raise ValueError(f"evidence manifest does not identify its model: {path}") + if model not in MODEL_LABELS: + raise ValueError(f"evidence model is not registered in the current profile: {model}") + if model in seen_models: + raise ValueError(f"more than one evidence directory was supplied for model: {model}") + seen_models.add(model) + rows = _read_attempts(path) + for source in rows: + row_model = str(source.get("model") or model) + if row_model != model: + raise ValueError( + f"attempt model differs from its evidence manifest: {row_model} != {model}" + ) + source["model"] = model + if str(source.get("engine") or "") not in ENGINE_LABELS: + raise ValueError( + f"attempt browser is not registered in the current profile: {source.get('engine')}" + ) + bundles.append( + { + "path": path, + "model": model, + "manifest": manifest, + "config": config, + "review": review, + "source_attempts": rows, + "attempts_sha256": _sha256(path / "attempts.jsonl"), + "manifest_sha256": _sha256(path / "manifest.json"), + } + ) + expected_models = set(MODEL_LABELS) + if len(expected_models) > 1 and seen_models != expected_models: + missing = sorted(expected_models - seen_models) + extra = sorted(seen_models - expected_models) + raise ValueError( + f"model cohort does not match the current profile; missing={missing}, extra={extra}" + ) + if len(bundles) > 1: + contract_values = { + bundle["model"]: ( + bundle["manifest"].get("selection_sha256"), + ((bundle["manifest"].get("environment") or {}).get("official_evaluator") or {}).get( + "data_checksum" + ), + (bundle["manifest"].get("environment") or {}).get("webarena_verified_version"), + ) + for bundle in bundles + } + if len(set(contract_values.values())) != 1: + raise ValueError( + "model evidence directories do not share one frozen evaluation contract" + ) + cohorts = { + bundle["model"]: { + (str(row["engine"]), int(row["task_id"])) + for row in bundle["source_attempts"] + if int(row["attempt"]) == 1 + } + for bundle in bundles + } + reference_model = next(iter(MODEL_LABELS)) + reference = cohorts[reference_model] + if not reference: + raise ValueError("multi-model report cannot use an empty primary cohort") + for model, cohort in cohorts.items(): + if cohort != reference: + raise ValueError( + f"model cohort differs from {reference_model}: {model}; " + f"missing={sorted(reference - cohort)}, extra={sorted(cohort - reference)}" + ) + task_ids = {task_id for _, task_id in cohort} + expected = {(engine, task_id) for engine in ENGINE_LABELS for task_id in task_ids} + if cohort != expected: + raise ValueError(f"model cohort is missing a registered browser: {model}") + return bundles -def _display_number(value: float | int | None, divisor: float = 1.0) -> float | None: - """Return the exact two-decimal number shown in an absolute metric cell.""" - if not isinstance(value, (int, float)): - return None - return float(f"{value / divisor:.2f}") +def _sha256(path: Path) -> str | None: + return hashlib.sha256(path.read_bytes()).hexdigest() if path.is_file() else None -def _absolute_metric_cell( - value: float | int | None, - other: float | int | None, - divisor: float, -) -> str: - css_class = "" - if ( - _comparison_class(_display_number(value, divisor), _display_number(other, divisor)) - == "better" - ): - css_class = ' class="better"' - return f"{_e(_show(value, divisor))}" +def _pct(value: float | None) -> str: + return "—" if value is None else f"{100 * value:.1f}%" -def _ratio_metric_cell(value: float | int | None) -> str: - css_class = _comparison_class(_display_number(value), 1) - class_attr = f' class="{css_class}"' if css_class else "" - return f"{_e(_show(value, suffix='×'))}" +def _seconds(value: float | None) -> str: + return "—" if value is None else f"{value / 1000:.1f} 秒" -def _grouped_resource_table( - summary: dict[str, Any], - metrics: tuple[tuple[str, str, float, str], ...], - caption: str, -) -> str: - engines = (summary.get("resource_comparison") or {}).get("by_engine") or {} - body = [] - for key, label, divisor, unit in metrics: - chromium = (engines.get("chromium") or {}).get(key) or {} - moli = (engines.get("moli") or {}).get(key) or {} - cells = [f'{_e(f"{label} ({unit})")}'] - for stat in ("p50", "p95", "mean"): - c_value = _numeric_stat(chromium, stat) - m_value = _numeric_stat(moli, stat) - cells.append(_absolute_metric_cell(c_value, m_value, divisor)) - cells.append(_absolute_metric_cell(m_value, c_value, divisor)) - body.append("" + "".join(cells) + "") - return ( - f'
' - '' - "" - "" - + "".join(body) - + "
{_e(caption)}
指标P50P95平均值
ChromiumMoliChromiumMoliChromiumMoli
" - ) +def _number(value: float | int | None) -> str: + return "—" if value is None else f"{value:,.0f}" -def _paired_ratio_table( - summary: dict[str, Any], - metrics: tuple[tuple[str, str, float, str], ...], - caption: str, -) -> str: - paired = (summary.get("resource_comparison") or {}).get("paired_ratios") or {} - values = paired.get("metrics") or {} - rows = [] - for key, label, _divisor, _unit in metrics: - metric = values.get(key) or {} - rows.append( - [ - label, - *[ - ComparedValue( - _show(_numeric_stat(metric, stat), suffix="×"), - _comparison_class(_display_number(_numeric_stat(metric, stat)), 1), - ) - for stat in ("p50", "p95", "mean") - ], - ] - ) - return _table( - ["指标", "逐任务比值 P50", "逐任务比值 P95", "逐任务比值平均值"], - rows, - caption, +def _best( + values: dict[str, float | int | None], + *, + higher: bool, + display: Callable[[float], float] | None = None, +) -> set[str]: + comparable = { + key: display(float(value)) if display else float(value) + for key, value in values.items() + if value is not None + } + if len(comparable) < 2: + return set() + target = (max if higher else min)(comparable.values()) + winners = {key for key, value in comparable.items() if value == target} + return winners if len(winners) == 1 else set() + + +def _winner(value: str, *, won: bool, label: str = "更优") -> Safe: + if not won: + return Safe(_e(value)) + return Safe( + f'{_e(value)}' + '优' ) -def _duration_ratio_audit(summary: dict[str, Any]) -> str: - paired = (summary.get("resource_comparison") or {}).get("paired_ratios") or {} - metric = (paired.get("metrics") or {}).get("duration_ms") or {} - values = metric.get("values") or [] - if not values: - return "" - p95_rank = max(1, (95 * len(values) + 99) // 100) - rows = [] - for rank, item in enumerate(values, 1): - task_id = str(item.get("task_id") or "") - title = str(((summary.get("by_task") or {}).get(task_id) or {}).get("title") or task_id) - rows.append( - [ - str(rank), - title, - *[ - ComparedValue( - _show(item.get(key), suffix=" ms"), - "better" - if _comparison_class( - _display_number(item.get(key)), _display_number(item.get(other)) - ) - == "better" - else "", - ) - for key, other in (("denominator", "numerator"), ("numerator", "denominator")) - ], - ComparedValue( - _show(item.get("ratio"), suffix="×"), - _comparison_class(_display_number(item.get("ratio")), 1), - ), - "P95 取值" if rank == p95_rank else "", - ] - ) - return ( - "
查看逐任务耗时比值的完整计算审计" - f"

脚本先对每个计入成功交集的同任务计算 Moli 耗时 / Chromium 耗时,再排序。" - f"本轮 n={len(values)},nearest-rank P95 的序号为 " - f"ceil(0.95 × {len(values)}) = {p95_rank};因此表中的 P95 是排序后第 " - f"{p95_rank} 个配对比值。

" - + _table( - ["排序", "任务", "Chromium", "Moli", "逐任务比值", "统计位置"], - rows, - "耗时配对比值原始序列", - ) - + "
" - ) +def _metric_stack(items: list[tuple[str, str, bool]]) -> Safe: + lines = [] + for label, value, won in items: + rendered = _winner(value, won=won) + lines.append(f"{_e(label)}{rendered}") + return Safe('' + "".join(lines) + "") -def _nav(active: str) -> str: - return ( - '" - ) +def _probability(value: float | None) -> str: + return "—" if value is None else f"{value:.3f}" -def _shell( - manifest: dict[str, Any], - active: str, - body: str, +def _table( + headers: list[str], + rows: Iterable[Iterable[Any]], *, - detail_title: str = "", - back_anchor: str = "", + caption: str = "", + table_id: str = "", + classes: str = "", ) -> str: - label = next(p[1] for p in PAGES if p[0] == active) - index = next(i for i, p in enumerate(PAGES) if p[0] == active) - prev_link = ( - f'← {PAGES[index - 1][1]}' if index else "" - ) - next_link = ( - f'{PAGES[index + 1][1]} →' - if index + 1 < len(PAGES) - else "" + head = "".join(f"{_e(x)}" for x in headers) + body = "".join( + "" + "".join(f"{x if isinstance(x, Safe) else _e(x)}" for x in row) + "" + for row in rows ) - header = ( - f'
← 返回逐任务分析' - f'Chromium / Moli 评测运行 {_e(manifest.get("run_id", "unknown"))}
' - if detail_title - else f'
Lexmount Browser Provider Evaluation

Chromium 与 Moli 浏览器工具调用评测

运行编号:{_e(manifest.get("run_id", "unknown"))} · 第 {index + 1}/{len(PAGES)} 页
{_nav(active)}' + cap = f"{_e(caption)}" if caption else "" + attrs = (f' id="{_e(table_id)}"' if table_id else "") + ( + f' class="{_e(classes)}"' if classes else "" ) - pager = "" if detail_title else f'
{prev_link}{next_link}
' - return f"""{_e(detail_title) or label}|Chromium 与 Moli 评测
{header}
{body}{pager}
Lexmount Browser Provider Evaluation
""" - + return f'
{cap}{head}{body}
' -def _functional_rows(summary: dict[str, Any]) -> list[list[str]]: - rows = [] - correctness_by_engine = {row[0]: row for row in _correctness_rows(summary)} - for engine in ("chromium", "moli"): - values = (summary.get("functional") or {}).get(engine) or {} - duration = values.get("duration_ms") or {} - rows.append( - correctness_by_engine[engine] - + [ - _stat(duration, "p50"), - _stat(duration, "mean"), - _stat(duration, "p95"), - ] - ) - return rows +def _task_type(task_id: int) -> str: + return ( + "MUTATE" if task_id in MUTATE_IDS else "NAVIGATE" if task_id in NAVIGATE_IDS else "RETRIEVE" + ) -def _correctness_rows(summary: dict[str, Any]) -> list[list[str]]: - rows: list[list[str]] = [] - for engine in ("chromium", "moli"): - values = (summary.get("functional") or {}).get(engine) or {} - tasks = int(values.get("tasks") or 0) - def score( - key: str, - fallback: str, - engine_values: dict[str, Any] = values, - task_total: int = tasks, - ) -> str: - passed = int(engine_values.get(key, engine_values.get(fallback, 0)) or 0) - rate = passed / task_total * 100 if task_total else 0 - return f"{passed}/{task_total}({rate:.2f}%)" +def _attempt_id(row: dict[str, Any]) -> str: + model = str(row.get("model") or (next(iter(MODEL_LABELS)) if len(MODEL_LABELS) == 1 else "")) + if not model: + raise ValueError("logical result does not identify its model") + return f"{model}-{row['engine']}-{int(row['task_id'])}" + + +def _row_model(row: dict[str, Any]) -> str: + model = str(row.get("model") or (next(iter(MODEL_LABELS)) if len(MODEL_LABELS) == 1 else "")) + if not model: + raise ValueError("attempt does not identify its model") + return model + + +def _execution_tool_metrics(row: dict[str, Any]) -> dict[str, Any]: + events = read_public_json(Path((row.get("paths") or {}).get("events", ""))) + steps = trajectory(events if isinstance(events, list) else []) + by_action: dict[str, float] = defaultdict(float) + count_by_action: dict[str, int] = defaultdict(int) + total_ms = 0.0 + for step in steps: + duration_ms = 1000 * float(step.get("duration_seconds") or 0) + action = str((step.get("arguments") or {}).get("action") or step.get("tool") or "other") + by_action[action] += duration_ms + count_by_action[action] += 1 + total_ms += duration_ms + return { + "action_count": len(steps), + "browser_tool_duration_ms": round(total_ms, 3), + "browser_tool_duration_by_action_ms": { + action: round(duration, 3) for action, duration in sorted(by_action.items()) + }, + "browser_tool_count_by_action": dict(sorted(count_by_action.items())), + } - rows.append( - [ - engine, - score("result_final_passes", "final_passes"), - score("combined_final_passes", "final_passes"), - ] - ) - return rows +def _logical_results( + rows: list[dict[str, Any]], protocol: dict[str, Any] | None = None +) -> list[dict[str, Any]]: + """Project physical executions into one authoritative result per browser and task. -def _correctness_table(summary: dict[str, Any]) -> str: - functional = summary.get("functional") or {} - scores = { - engine: ( - int((functional.get(engine) or {}).get("result_final_passes", 0) or 0), - int((functional.get(engine) or {}).get("combined_final_passes", 0) or 0), - int((functional.get(engine) or {}).get("tasks", 0) or 0), + A failed task may be executed once more. The frozen run protocol chooses the + authoritative execution; additive cost and latency include every execution. + """ + grouped: dict[tuple[str, str, int], list[dict[str, Any]]] = defaultdict(list) + for row in rows: + model = str( + row.get("model") or (next(iter(MODEL_LABELS)) if len(MODEL_LABELS) == 1 else "") ) - for engine in ("chromium", "moli") - } - rows = [] - for engine, (result, combined, tasks) in scores.items(): - result_rate = result / tasks * 100 if tasks else 0 - combined_rate = combined / tasks * 100 if tasks else 0 - other = scores["moli" if engine == "chromium" else "chromium"] - result_winner = ( - tasks - and other[2] - and _comparison_class(result / tasks, other[0] / other[2], higher=True) == "better" - ) - combined_winner = ( - tasks - and other[2] - and _comparison_class(combined / tasks, other[1] / other[2], higher=True) == "better" + if not model: + raise ValueError("attempt does not identify its model") + row["model"] = model + grouped[(model, str(row["engine"]), int(row["task_id"]))].append(row) + results: list[dict[str, Any]] = [] + for key in sorted(grouped): + executions = sorted(grouped[key], key=lambda row: int(row["attempt"])) + if [int(row["attempt"]) for row in executions] != list(range(1, len(executions) + 1)): + raise ValueError(f"non-contiguous execution sequence for {key}") + if len(executions) > 2: + raise ValueError(f"more than two executions for {key}") + legacy_primary = (protocol or {}).get("official_score_uses") == "primary attempt only" + authoritative = dict(executions[0] if legacy_primary else executions[-1]) + authoritative["duration_ms"] = sum(float(row.get("duration_ms") or 0) for row in executions) + usage_keys = ( + "input_tokens", + "output_tokens", + "reasoning_tokens", + "total_tokens", + "request_count", + "request_attempt_count", ) - rows.append( - "" - f"{_e(engine)}" - f'{result}/{tasks}({result_rate:.2f}%)' - f'{combined}/{tasks}({combined_rate:.2f}%)' - f'{max(0, tasks - result)} 个失败 ' - "" + authoritative["model_usage"] = { + name: sum(int((row.get("model_usage") or {}).get(name) or 0) for row in executions) + for name in usage_keys + } + authoritative["model_transport_errors"] = [ + error for row in executions for error in (row.get("model_transport_errors") or []) + ] + resources = [row.get("resource") or {} for row in executions] + weighted_duration = sum(float(row.get("duration_ms") or 0) for row in executions) + authoritative["resource"] = { + **{ + name: sum(float(resource.get(name) or 0) for resource in resources) + for name in ("cpu_ms", "network_rx_bytes", "network_tx_bytes") + }, + **{ + name: max((float(resource.get(name) or 0) for resource in resources), default=0) + for name in ("memory_peak_bytes", "memory_retained_bytes", "pids_current_peak") + }, + } + if weighted_duration: + present_duration = sum( + float(row.get("duration_ms") or 0) + for row in executions + if (row.get("resource") or {}).get("cpu_average_utilization_pct") is not None + ) + if present_duration: + authoritative["resource"]["cpu_average_utilization_pct"] = ( + sum( + float((row.get("resource") or {}).get("cpu_average_utilization_pct") or 0) + * float(row.get("duration_ms") or 0) + for row in executions + if (row.get("resource") or {}).get("cpu_average_utilization_pct") + is not None + ) + / present_duration + ) + memory_sample_count = sum( + int((row.get("resource") or {}).get("sample_count") or 0) + for row in executions + if (row.get("resource") or {}).get("memory_mean_bytes") is not None ) - return ( - '
' - "" - "" + "".join(rows) + "
任务正确率
Provider结果正确结果与参考过程均正确任务分析
" - ) + if memory_sample_count: + authoritative["resource"]["memory_mean_bytes"] = ( + sum( + float((row.get("resource") or {}).get("memory_mean_bytes") or 0) + * int((row.get("resource") or {}).get("sample_count") or 0) + for row in executions + if (row.get("resource") or {}).get("memory_mean_bytes") is not None + ) + / memory_sample_count + ) + tool_metrics = [_execution_tool_metrics(row) for row in executions] + authoritative["action_count"] = sum(item["action_count"] for item in tool_metrics) + authoritative["browser_tool_duration_ms"] = round( + sum(item["browser_tool_duration_ms"] for item in tool_metrics), 3 + ) + by_action: dict[str, float] = defaultdict(float) + count_by_action: dict[str, int] = defaultdict(int) + for item in tool_metrics: + for action, duration in item["browser_tool_duration_by_action_ms"].items(): + by_action[action] += duration + for action, count in item["browser_tool_count_by_action"].items(): + count_by_action[action] += count + authoritative["browser_tool_duration_by_action_ms"] = { + action: round(duration, 3) for action, duration in sorted(by_action.items()) + } + authoritative["browser_tool_count_by_action"] = dict(sorted(count_by_action.items())) + authoritative["_source_executions"] = executions + results.append(authoritative) + return results -def _aggregate_ratio(summary: dict[str, Any], metric_name: str, stat: str) -> float | None: - engines = (summary.get("resource_comparison") or {}).get("by_engine") or {} - chromium = _numeric_stat(((engines.get("chromium") or {}).get(metric_name) or {}), stat) - moli = _numeric_stat(((engines.get("moli") or {}).get(metric_name) or {}), stat) - if ( - not isinstance(chromium, (int, float)) - or chromium == 0 - or not isinstance(moli, (int, float)) - ): - return None - return float(moli) / float(chromium) +def _public_attempt(row: dict[str, Any]) -> dict[str, Any]: + usage, resource = row.get("model_usage") or {}, row.get("resource") or {} + executions = row.get("_source_executions") or [row] + return { + "id": _attempt_id(row), + "run_id": str(row.get("run_id") or ""), + "model": str(row["model"]), + "engine": str(row["engine"]), + "task_id": int(row["task_id"]), + "tier": str(row["tier"]), + "score": float(row["score"]), + "status": str(row.get("status") or ("success" if float(row["score"]) == 1 else "failure")), + "duration_ms": round(float(row.get("duration_ms") or 0), 3), + "action_count": int(row.get("action_count") or 0), + "browser_tool_duration_ms": round(float(row.get("browser_tool_duration_ms") or 0), 3), + "browser_tool_duration_by_action_ms": row.get("browser_tool_duration_by_action_ms") or {}, + "browser_tool_count_by_action": row.get("browser_tool_count_by_action") or {}, + "model_usage": { + key: int(usage.get(key) or 0) + for key in ( + "input_tokens", + "output_tokens", + "reasoning_tokens", + "total_tokens", + "request_count", + "request_attempt_count", + ) + }, + "transport_error_count": len(row.get("model_transport_errors") or []), + "execution_count": len(executions), + "authoritative_execution": int(row["attempt"]), + "resource": { + key: round(float(resource[key]), 3) + for key in ( + "cpu_ms", + "network_rx_bytes", + "network_tx_bytes", + "memory_mean_bytes", + "memory_peak_bytes", + "memory_retained_bytes", + "pids_current_peak", + "cpu_average_utilization_pct", + ) + if resource.get(key) is not None + }, + } -def _paired_ratio(summary: dict[str, Any], metric_name: str, stat: str) -> float | None: - metric = ( - ((summary.get("resource_comparison") or {}).get("paired_ratios") or {}).get("metrics") or {} - ).get(metric_name) or {} - return _numeric_stat(metric, stat) +def _public_execution(row: dict[str, Any], *, authoritative: bool) -> dict[str, Any]: + """Return the measurements needed to recompute one logical result's totals.""" + + usage, resource = row.get("model_usage") or {}, row.get("resource") or {} + tool_metrics = _execution_tool_metrics(row) + return { + "execution": int(row["attempt"]), + "authoritative": authoritative, + "run_id": str(row.get("run_id") or ""), + "score": float(row.get("score") or 0), + "status": str( + row.get("status") or ("success" if float(row.get("score") or 0) == 1 else "failure") + ), + "duration_ms": round(float(row.get("duration_ms") or 0), 3), + "timings": { + str(key): round(float(value), 3) + for key, value in (row.get("timings") or {}).items() + if isinstance(value, (int, float)) + }, + **tool_metrics, + "model_usage": { + key: int(usage.get(key) or 0) + for key in ( + "input_tokens", + "output_tokens", + "reasoning_tokens", + "total_tokens", + "request_count", + "request_attempt_count", + ) + }, + "transport_error_count": len(row.get("model_transport_errors") or []), + "resource": { + str(key): round(float(value), 3) + for key, value in resource.items() + if isinstance(value, (int, float)) + }, + } -def _detail_filename(task_id: str, engine: str) -> str: - return f"{REPORT_STEM}-task-{task_id.encode().hex()}-{engine.encode().hex()}.html" +def _percentile(values: list[float], q: float) -> float | None: + if not values: + return None + values = sorted(values) + index = (len(values) - 1) * q + lo, hi = math.floor(index), math.ceil(index) + return values[lo] if lo == hi else values[lo] + (values[hi] - values[lo]) * (index - lo) + + +def _primary_summary(rows: list[dict[str, Any]]) -> dict[str, dict[str, dict[str, Any]]]: + result: dict[str, dict[str, dict[str, Any]]] = {} + for model in MODEL_LABELS: + result[model] = {} + for engine in ENGINE_LABELS: + logical = [r for r in rows if r["model"] == model and r["engine"] == engine] + durations = [float(r.get("duration_ms") or 0) for r in logical] + tool_durations = [float(r.get("browser_tool_duration_ms") or 0) for r in logical] + tokens = [int((r.get("model_usage") or {}).get("total_tokens") or 0) for r in logical] + passed = sum(float(r["score"]) == 1 for r in logical) + resource_summary = {} + for name in ( + "cpu_ms", + "cpu_average_utilization_pct", + "memory_mean_bytes", + "memory_peak_bytes", + "network_rx_bytes", + "network_tx_bytes", + ): + values = [ + float((row.get("resource") or {})[name]) + for row in logical + if (row.get("resource") or {}).get(name) is not None + ] + resource_summary[name] = { + "count": len(values), + "mean": statistics.fmean(values) if values else None, + "p50": _percentile(values, 0.50), + "p95": _percentile(values, 0.95), + } + result[model][engine] = { + "completed": len(logical), + "passed": passed, + "score": passed / len(logical) if logical else None, + "mean_duration_ms": statistics.fmean(durations) if durations else None, + "p50_duration_ms": _percentile(durations, 0.50), + "p95_duration_ms": _percentile(durations, 0.95), + "mean_browser_tool_duration_ms": statistics.fmean(tool_durations) + if tool_durations + else None, + "p50_browser_tool_duration_ms": _percentile(tool_durations, 0.50), + "p95_browser_tool_duration_ms": _percentile(tool_durations, 0.95), + "mean_tokens": statistics.fmean(tokens) if tokens else None, + "p50_tokens": _percentile(tokens, 0.50), + "p95_tokens": _percentile(tokens, 0.95), + "resources": resource_summary, + } + return result + + +def _paired_summary(rows: list[dict[str, Any]]) -> dict[str, Any]: + """Compare every browser with the profile baseline, always within one model.""" + + primary = { + (str(r["model"]), str(r["engine"]), int(r["task_id"])): float(r["score"]) == 1 for r in rows + } + result: dict[str, Any] = {} + for model in MODEL_LABELS: + comparisons: dict[str, Any] = {} + for engine in ENGINE_LABELS: + if engine == BASELINE_ENGINE: + continue + ids = sorted( + task_id + for task_id in {key[2] for key in primary if key[0] == model} + if (model, BASELINE_ENGINE, task_id) in primary + and (model, engine, task_id) in primary + ) + counts = { + "both_pass": 0, + "baseline_only": 0, + "comparison_only": 0, + "both_fail": 0, + } + for task_id in ids: + baseline_passed = primary[(model, BASELINE_ENGINE, task_id)] + comparison_passed = primary[(model, engine, task_id)] + key = ( + "both_pass" + if baseline_passed and comparison_passed + else "baseline_only" + if baseline_passed + else "comparison_only" + if comparison_passed + else "both_fail" + ) + counts[key] += 1 + discordant = counts["baseline_only"] + counts["comparison_only"] + p = ( + min( + 1.0, + 2 + * sum( + math.comb(discordant, k) + for k in range(min(counts["baseline_only"], counts["comparison_only"]) + 1) + ) + / 2**discordant, + ) + if discordant + else (1.0 if ids else None) + ) + comparisons[engine] = { + "baseline_engine": BASELINE_ENGINE, + "comparison_engine": engine, + "paired": len(ids), + **counts, + "mcnemar_exact_p": p, + } + result[model] = { + "baseline_engine": BASELINE_ENGINE, + "comparisons": comparisons, + } + return result + + +def _resolve_tasks( + selection: PilotSelection, *, expected_data_checksum: str | None = None +) -> list[dict[str, Any]]: + try: + from .runner import _configure_benchmark + + benchmark = _configure_benchmark() + if expected_data_checksum: + dataset_path = Path(benchmark.config.test_data_file) + actual_data_checksum = _sha256(dataset_path) + if actual_data_checksum != expected_data_checksum: + raise ValueError( + "installed WebArena-Verified data checksum differs from measured evidence: " + f"{actual_data_checksum} != {expected_data_checksum}" + ) + tasks = validate_against_official(selection, benchmark, enforce_contract_integrity=False) + official = { + int(task["task_id"]): benchmark.get_task(int(task["task_id"])) for task in tasks + } + except Exception: + if expected_data_checksum: + raise + tasks = [ + { + "task_id": t.task_id, + "tier": t.tier, + "estimated_min_steps": t.estimated_min_steps, + "rationale": t.rationale, + "official_hard": t.official_hard, + "sites": ["unknown"], + "intent": t.rationale, + "intent_template_id": None, + "revision": None, + "start_urls": [], + } + for t in selection.tasks + ] + for task in tasks: + official_task = official.get(int(task["task_id"])) if "official" in locals() else None + contract = task_contract(official_task) if official_task is not None else {} + task["contract"] = contract + task["contract_sha256"] = hashlib.sha256( + json.dumps(contract, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest() + evals = contract.get("evaluation_contract") or [] + if expected_data_checksum and not evals: + raise ValueError(f"official task {task['task_id']} has no evaluator contract") + expected = (evals[0].get("expected") or {}) if evals else {} + task["task_type"] = str( + expected.get("task_type") or _task_type(int(task["task_id"])) + ).upper() + return tasks + + +def _wilson_interval(successes: int, total: int, z: float = 1.96) -> tuple[float, float] | None: + if total <= 0: + return None + p = successes / total + denominator = 1 + z**2 / total + centre = (p + z**2 / (2 * total)) / denominator + spread = z * math.sqrt((p * (1 - p) + z**2 / (4 * total)) / total) / denominator + return max(0.0, centre - spread), min(1.0, centre + spread) + + +def _task_family_summary( + tasks: list[dict[str, Any]], attempts: list[dict[str, Any]] +) -> list[dict[str, Any]]: + primary = { + (str(row["model"]), str(row["engine"]), int(row["task_id"])): float(row["score"]) == 1 + for row in attempts + } + result = [] + for kind in ("RETRIEVE", "NAVIGATE", "MUTATE"): + selected = [task for task in tasks if task["task_type"] == kind] + result.append( + { + "kind": kind, + "count": len(selected), + "scores": { + model: { + engine: sum( + primary.get((model, engine, int(task["task_id"])), False) + for task in selected + ) + for engine in ENGINE_LABELS + } + for model in MODEL_LABELS + }, + } + ) + return result -def _summary_page(manifest: dict[str, Any], summary: dict[str, Any]) -> str: - task_count = len(manifest.get("suite") or []) - f = summary.get("functional") or {} - c = f.get("chromium") or {} - m = f.get("moli") or {} - paired = (summary.get("resource_comparison") or {}).get("intersection_attempts", 0) +TASK_TYPE_COPY = { + "RETRIEVE": ("信息提取", "从页面读取、筛选或汇总信息,不修改站点状态"), + "NAVIGATE": ("页面导航", "打开符合指定条件的页面或列表"), + "MUTATE": ("页面操作", "提交表单、修改记录、结账等会改变站点状态的任务"), +} - agent_smoke = manifest.get("agent_smoke") or {} - smoke_html = "" - if agent_smoke: - provenance = agent_smoke.get("provenance") or {} - smoke_html = f"""

CowAgent 端到端冒烟测试

该测试与资源微基准分开,不参与上表统计,也没有在运行编号 {_e(manifest.get("run_id"))} 中重新执行。模型 {_e(agent_smoke.get("model"))},思考强度 {_e(agent_smoke.get("reasoning_effort"))};独立执行日期 {_e(agent_smoke.get("executed_at"))},来源文件内容哈希 {_e(str(provenance.get("source_sha256") or "")[:12])}。同一任务每个浏览器方案只执行 1 次。

{_table(["浏览器方案", "结果", "BrowserTool 调用次数", "最终答案"], [[str(item.get("provider")), "通过" if item.get("status") == "pass" else str(item.get("status")), str(item.get("browser_tool_calls")), str(item.get("answer"))] for item in agent_smoke.get("results") or []], "独立端到端接受性测试")}
""" - c_passes = int(c.get("result_final_passes", c.get("final_passes", 0))) - m_passes = int(m.get("result_final_passes", m.get("final_passes", 0))) - if c_passes > m_passes: - recommendation = ( - f"本轮建议默认使用 Chromium:其结果正确任务数为 {c_passes},高于 Moli 的 {m_passes};" - "Moli 在双方均成功的任务上若表现出更低耗时或资源占用,可作为任务已验证兼容时的备选。" - ) - elif m_passes > c_passes: - recommendation = ( - f"本轮建议默认使用 Moli:其结果正确任务数为 {m_passes},高于 Chromium 的 {c_passes};" - "仍需结合双方成功任务上的资源指标和失败类型确认适用边界。" +def _task_family_table(families: list[dict[str, Any]]) -> str: + headers = ["任务类型", "任务数", "测什么"] + [ + f"{MODEL_LABELS[model]} · {ENGINE_LABELS[engine]}" + for model in MODEL_LABELS + for engine in ENGINE_LABELS + ] + rows: list[list[Any]] = [] + for family in families: + kind = str(family["kind"]) + copy = TASK_TYPE_COPY.get(kind) + if copy is None: + raise ValueError(f"unsupported Browser Use task type: {kind}") + label, description = copy + rows.append( + [ + Safe( + f'{_e(label)}' + f'{_e(kind)}' + ), + family["count"], + description, + *[ + f"{family['scores'][model][engine]}/{family['count']}" + for model in MODEL_LABELS + for engine in ENGINE_LABELS + ], + ] ) - else: - recommendation = ( - f"本轮两个 Provider 的结果正确任务数相同(均为 {c_passes});" - "仅凭正确性无法确定默认方案,应结合双方成功任务上的资源指标和失败类型决策。" - ) - correctness = _correctness_table(summary) - return f"""

结论与产品建议

{recommendation}

这是浏览器工具调用层的公网确定性微基准,不是 LLM Agent 排名。评测集包含 {task_count} 个结果唯一的任务。

-

任务正确率

{correctness}
-

关键技术结论

    -
  • 速度与资源:双方成功任务的逐任务 Moli / Chromium 比值中,耗时比平均值为 {_show(_paired_ratio(summary, "duration_ms", "mean"), suffix="×")},时间加权平均内存比平均值为 {_show(_paired_ratio(summary, "memory_mean_bytes", "mean"), suffix="×")},网络接收量比平均值为 {_show(_paired_ratio(summary, "network_rx_bytes", "mean"), suffix="×")}。
  • -
  • 时序边界:Moli 的 1 Hz 是 CDP screencast 下采样,不等于 JavaScript 或 requestAnimationFrame 只有 1 fps;源码中的 rAF 目标间隔为 16 ms。
  • -
  • 根因边界:布局与 actionability 证据只能定位现象,尚不能确认 screencast、渲染器或适配层中的具体根因。查看逐任务证据与验证方案。
-

双方成功任务的速度与资源

统计范围:双方都成功的 {paired} 个同任务,不含失败与重试样本。绝对值优胜项为绿色粗体;配对比值低于 1 为绿色、高于 1 为红色,显示值相同保持中性。

{_metric_tables(summary)}{_duration_ratio_audit(summary)}
{smoke_html}""" - - -def _difficulty(value: str) -> str: - named = {"simple": "简单", "medium": "中等", "complex": "复杂"} - if value in named: - return named[value] - match = re.fullmatch(r"step_(\d+)_(\d+)", value) - if match: - return f"{int(match.group(1))}–{int(match.group(2))} 步" - return value or "未标注" - - -def _action_text(call: dict[str, Any]) -> str: - action = str(call.get("action") or "") - return { - "navigate": f"打开 {call.get('url', '')} 并返回页面观察快照", - "get_text": f"读取 {call.get('selector', '')} 文本", - "get_attribute": f"读取 {call.get('selector', '')} 的 {call.get('name', '')} 属性", - "get_property": f"读取 {call.get('selector', '')} 的 {call.get('name', '')} DOM 属性", - "click": f"点击 {call.get('selector', '')}", - "fill": f"向 {call.get('selector', '')} 填入 {call.get('text', '')}", - "select": f"在 {call.get('selector', '')} 选择 {call.get('value', '')}", - "snapshot": f"读取 {call.get('selector') or 'body'} 当前快照", - "scroll": f"向 {call.get('direction', 'down')} 滚动 {call.get('amount', 700)} px", - "back": "浏览器后退", - "forward": "浏览器前进", - "press": f"按键 {call.get('key', '')}", - }.get(action, action) - - -def _source_references(source: dict[str, Any]) -> str: - links = [] - for url in source.get("design_basis") or []: - parsed = urlparse(str(url)) - if parsed.scheme in {"https", "http"}: - label = parsed.path.rstrip("/").rsplit("/", 1)[-1] or parsed.hostname - links.append(f'{_e(label)}') - return '

设计依据:' + " · ".join(links) + "

" if links else "" - - -def _annotate_source_marks(escaped_text: str) -> str: - """Annotate the literal heading permalink character without changing the oracle.""" - return escaped_text.replace( - "¶", '' - ) + return _table(headers, rows, table_id="task-family-table", classes="wide-data-table") -def _source_mark_note(values: Any) -> str: - return ( - '

¶ 为原网页标题的锚点字符,参与逐字判定。

' - if "¶" in str(values) - else "" - ) +def _failure_cause_table(verified_causes: dict[str, dict[str, Any]]) -> str: + groups = { + "agent": ("Agent / 模型行为", "保留", "属于当前被测组合;按轨迹定位检索、选择或提交错误。"), + "benchmark": ( + "任务 / evaluator", + "排除并修复", + "修正验收合同,通过独立预检后重跑四种配置。", + ), + "site": ("自托管站点", "排除并修复", "修复依赖、会话或保存链路,通过读写预检后重跑。"), + "evidence": ( + "浏览器证据接口", + "保留并修复", + "这是浏览器集成能力的一部分;补齐响应正文或 Referer 采集后重跑。", + ), + "environment": ( + "共享测试环境", + "排除,修复已完成", + "跨进程串行锁已经上线;受污染任务仍需重跑。", + ), + "mixed": ( + "参考状态与 Agent 混合", + "排除并重建", + "重新冻结参考状态,再重跑以区分 Agent 错误。", + ), + } + def group_for(code: str) -> str: + for prefix in ("agent", "benchmark", "site", "environment", "mixed"): + if code.startswith(prefix + "_"): + return prefix + if code.startswith("evidence_"): + return "evidence" + raise ValueError(f"unknown failure-cause scope: {code}") + + counts: dict[str, Counter[str]] = defaultdict(Counter) + for diagnosis in verified_causes.values(): + subcategory = diagnosis.get("subcategory") or {} + code = str(subcategory.get("code") or "evidence_capture") + label = str(subcategory.get("zh") or "未分类") + counts[group_for(code)][label] += 1 -def _suite_page(manifest: dict[str, Any]) -> str: - suite = manifest.get("suite") or [] - validation = manifest.get("ground_truth_validation") or {} - replay = validation.get("automatic_browser_replay") or {} - difficulty_values = [str(task.get("complexity") or "") for task in suite] - ordered_difficulties = [ - difficulty - for difficulty in ("simple", "medium", "complex") - if difficulty in difficulty_values - ] + sorted( - { - difficulty - for difficulty in difficulty_values - if difficulty not in {"simple", "medium", "complex"} - } + rows = [] + for key in ("agent", "benchmark", "site", "evidence", "environment", "mixed"): + total = sum(counts[key].values()) + if not total: + continue + rows.append([groups[key][0], total, groups[key][1], groups[key][2]]) + return _table( + ["原因范围", "失败结果数", "本轮处理", "修复与验证"], + rows, + table_id="failure-cause-table", + classes="wide-data-table", ) - difficulty_counts = { - difficulty: difficulty_values.count(difficulty) for difficulty in ordered_difficulties + + +def _external_interference_task_ids( + verified_causes: dict[str, dict[str, Any]], +) -> set[int]: + prefixes = ("benchmark_", "site_", "environment_", "mixed_") + return { + int(result_id.rsplit("-", 1)[-1]) + for result_id in verified_causes + if _cause_code(verified_causes, result_id).startswith(prefixes) } - difficulty_summary = ( - "、".join( - f"{_difficulty(difficulty)} {count} 个" - for difficulty, count in difficulty_counts.items() - if count - ) - or "未标注难度" - ) - site_counts = Counter( - urlparse(str((task.get("source") or {}).get("url") or "")).hostname or "未标注" - for task in suite - ) - action_counts = Counter( - str(call.get("action") or "") for task in suite for call in task.get("tool_calls") or [] - ) - distribution = ( - '
' - f'
{len(suite)}公网任务
' - f'
{len(site_counts)}独立网站/域名
' - f'
{sum(action_counts.values())}参考 BrowserTool 调用
' - "
" - + _table( - ["维度", "分布"], + + +def _clean_subset_table( + attempts: list[dict[str, Any]], verified_causes: dict[str, dict[str, Any]] +) -> str: + excluded = _external_interference_task_ids(verified_causes) + kept = [row for row in attempts if int(row["task_id"]) not in excluded] + summary = _primary_summary(kept) + return ( + _table( + ["模型", "浏览器", "通过任务", "通过率"], [ - ["难度", difficulty_summary], - [ - "网站", - ";".join(f"{host} {count}" for host, count in sorted(site_counts.items())), - ], [ - "能力", - ";".join( - f"{action} {count}" for action, count in sorted(action_counts.items()) - ), - ], + MODEL_LABELS[model], + ENGINE_LABELS[engine], + f"{summary[model][engine]['passed']}/{summary[model][engine]['completed']}", + _pct(summary[model][engine]["score"]), + ] + for model in MODEL_LABELS + for engine in ENGINE_LABELS ], - "评测集分布", + table_id="clean-subset-table", + classes="core-comparison-table", ) + + '

这是事后诊断口径,不是官方新分数。它按任务统一剔除任一配置已证实受任务/evaluator、站点、共享环境或参考状态干扰的任务,保证四个配置分母一致。

' + f'

本轮剔除 {len(excluded)} 题:{_e("、".join(f"#{task_id}" for task_id in sorted(excluded)))}。

' ) - cards = [] - for index, task in enumerate(suite, 1): - title = str(task.get("title") or task.get("task_id")) - calls = task.get("tool_calls") or [] - call_html = "".join( - f'
调用 {i} · {_e(c.get("action"))}
{_e(_action_text(c))}' - + ( - f'
输出:{_e(c.get("save_as"))}' - if c.get("save_as") - else "" - ) - + "
" - for i, c in enumerate(calls, 1) - ) - truth = task.get("ground_truth") or {} - expected_inline = ";".join( - f"{name} = {value}" for name, value in (truth.get("expected_outputs") or {}).items() - ) - expected_rows = ( - '
' - + "".join( - f"
{_e(name)}
{_annotate_source_marks(_e(value))}
" - for name, value in (truth.get("expected_outputs") or {}).items() + +def _adoption_top3(evidence: dict[str, Any]) -> str: + families = sorted( + (item for item in evidence.get("benchmark_families") or [] if item.get("interaction")), + key=lambda item: (-len(item.get("adopters") or []), str(item.get("family") or "")), + )[:3] + if len(families) != 3 or not evidence.get("verified_at"): + raise ValueError("benchmark adoption evidence must provide a dated interaction Top 3") + cards = [] + for item in families: + adopters = item.get("adopters") or [] + warning = item.get("warning") + if warning == ( + "Original Mind2Web, Online-Mind2Web, and Mind2Web 2 have different " + "tasks and graders and must not share a score table." + ): + warning = ( + "原版 Mind2Web、Online-Mind2Web 与 Mind2Web 2 的任务和评分器不同," + "不能合并到同一成绩表。" ) - + "
" - + _source_mark_note(truth.get("expected_outputs")) - ) - reference_path = " → ".join(truth.get("expected_actions") or []) - source = task.get("source") or {} - target_url = str(source.get("url") or "") - source_html = ( - f'{_e(target_url)}' - ) - shared_tags = {"simple", "medium", "complex", "public-web", "exact-oracle"} - chips = "".join( - f'{_e(tag)}' - for tag in task.get("tags") or [] - if tag not in shared_tags - ) - search_text = ( - f"任务{index:02d} 任务 {index:02d} {task.get('task_id')} {title} {target_url} " - + " ".join(task.get("tags") or []) - + " " - + expected_inline - ) + note = warning or { + "WebArena": "自托管、可重置、结果验收,更适合浏览器 A/B。", + "WebVoyager": "真实网站直观,但网页漂移与 LLM judge 降低精确复现性。", + }.get(str(item.get("family")), "采用数不代表评测质量。") cards.append( - f'
任务 {index:02d}

{_e(title)}

{_difficulty(str(task.get("complexity")))}{len(calls)} 次工具调用{chips}' - '
' - f"
公网测试页面{source_html}
" - f"
参考路径{_e(reference_path)}
" - f"
唯一预期结果{expected_rows}
" - "
" - f"
逐步操作与答案依据{call_html}

答案依据:{_e(truth.get('derivation'))}

{_source_references(source)}
" - ) - filters = ( - '

查找任务

' - f'显示 {len(suite)} / {len(suite)} 个任务
' - '
' - '
' - '' - ) - script = """""" return ( - f'

评测集概览

{distribution}

答案核验:{_e(validation.get("validator") or "未记录")},{_e(replay.get("passed"))}/{_e(replay.get("total"))} 已核验。数据集 SHA-256:{_e(str(validation.get("dataset_sha256") or "")[:16])}。

标注来源与版本

{_e(validation.get("method") or "未记录")}

' - + filters + '
' + "".join(cards) - + script + + '

采用次数来自公开一手材料中的机构去重计数,不是质量分;' + f"证据核对日期 {_e(evidence.get('verified_at') or '未记录')}。未找到 DeepSeek、Kimi " + "公开的一手 UI browser-use benchmark,不等于内部没有评测。

" ) -def _method_page(manifest: dict[str, Any]) -> str: - p = manifest.get("protocol") or {} - env = manifest.get("environment") or {} - versions = env.get("browser_versions") or {} - rows = [ - ["运行平台", str(env.get("runner_platform", "未记录"))], - [ - "容器", - f"Docker {env.get('docker_server_version', '未记录')} / {env.get('docker_os', '未记录')}", - ], - ["Chromium", str(versions.get("chromium", "未记录"))], - ["Moli", str(versions.get("moli", "未记录"))], - [ - "单浏览器配额", - f"{p.get('cpus_per_browser', '?')} CPU / {p.get('memory_per_browser', '?')} / 独立网络", - ], - ["执行次数", "每任务每个 Provider 形成 1 个正式结果"], - ["视口", json.dumps(p.get("viewport") or {}, ensure_ascii=False)], - [ - "失败复核", - f"失败时最多补充 {p.get('max_failure_retries', '?')} 条同任务证据,不另计为新任务", - ], - ["采样", f"每 {p.get('sample_interval_ms', '?')} ms 采样内存、PID、CPU、网络计数"], - ] - return f"""

步骤的严格定义

一步 = 一次 CowAgent BrowserTool 等价调用,例如 navigate、click、get_text、fill、select、scroll、back 或 forward。等待页面就绪和目标值出现属于该工具调用内部,不单列、不计步。每次调用的参数、耗时、输出和状态写入执行轨迹。

成功判定:两个独立程序指标

  1. 结果正确(主指标):每个标注输出与 expected_outputs 做严格字符串相等;不使用模糊匹配或模型评分;Codex 只参与评测前的标注核对。
  2. 结果与参考过程均正确(辅指标):在结果正确之外,本轮被重放的参考动作序列还必须与 expected_actions 一致,且全部 BrowserTool 调用成功。

边界:expected_actions 是工具层微基准的固定参考路径,不代表任务只有一种解法。若评测自由规划 CowAgent,应把过程判定改成必要里程碑、禁止动作和步数预算,不能要求整条序列逐步相等。

环境

{_table(["项目", "记录值"], rows, "本轮环境")}

执行流程

  1. 分别启动并预热两个浏览器,浏览器进程和上下文在整轮评测中保持运行。
  2. 任务前清 Cookie、权限、HTTP 缓存、Cache Storage、Service Worker、IndexedDB、来源存储和历史,并创建新页面。
  3. 计时外静置 {p.get("post_reset_settle_ms", "?")} ms,按任务交替两个 Provider 的执行顺序。
  4. 执行完整任务并采样;失败复核归入同一任务详情。
  5. 资源基线只比较双方都成功的同任务,失败与复核样本不混入正常路径。

统计口径

  • P50 与中位数是同一个统计量;总体跨任务报告 P50、平均值和 nearest-rank P95。
  • 配对比值的定义和逐项计算过程见首页性能表与计算审计。
  • 每个任务仅有一个正式样本,因此逐任务不计算分位数;当前跨任务点不足以证明普遍性能优势。
  • 内存平均值是时间加权均值;任务内存 P95 来自单次任务时序采样。
  • RX = Receive(接收),TX = Transmit(发送);总量含页面资源与 CDP,速率来自相邻采样点差分。
""" - - -METRICS = ( - ( - "时间与计算", - ( - ("duration_ms", "总耗时", 1, "ms"), - ("cpu_ms", "CPU 时间", 1, "ms"), - ("cpu_average_utilization_pct", "平均 CPU 占单核", 1, "%"), - ("cpu_sampled_utilization_p95_pct", "任务内 CPU P95", 1, "%"), - ), - ), - ( - "内存", - ( - ("memory_mean_bytes", "时间加权平均内存", 1048576, "MiB"), - ("memory_p95_bytes", "任务内存 P95", 1048576, "MiB"), - ("memory_peak_bytes", "峰值内存", 1048576, "MiB"), - ("memory_retained_bytes", "结束保留增量", 1048576, "MiB"), - ("memory_mib_seconds", "内存占用积分", 1, "MiB·s"), - ), - ), - ( - "网络", - ( - ("network_rx_bytes", "接收总量 RX", 1024, "KiB"), - ("network_tx_bytes", "发送总量 TX", 1024, "KiB"), - ("network_rx_bytes_per_second", "平均接收速率", 1024, "KiB/s"), - ("network_rx_rate_p95_bytes_per_second", "任务内接收速率 P95", 1024, "KiB/s"), - ("network_tx_rate_p95_bytes_per_second", "任务内发送速率 P95", 1024, "KiB/s"), - ), - ), -) +def _same_task_comparison(tasks: list[dict[str, Any]], attempts: list[dict[str, Any]]) -> str: + """Render one task-by-configuration matrix; never reduce it to pairwise controls.""" + results = {(_row_model(row), str(row["engine"]), int(row["task_id"])): row for row in attempts} + engines = [BASELINE_ENGINE, *(engine for engine in ENGINE_LABELS if engine != BASELINE_ENGINE)] + configurations = [(model, engine) for model in MODEL_LABELS for engine in engines] + if len(configurations) < 2: + return '

至少需要两个参评配置才能进行同题比较。

' -def _metric_tables(summary: dict[str, Any]) -> str: - absolute = "".join( - _grouped_resource_table(summary, metrics, f"{group}绝对值(均为越低越好)") - for group, metrics in METRICS - ) - paired = "".join( - _paired_ratio_table( - summary, - metrics, - f"{group}逐任务 Moli / Chromium 配对比值(均为越低越好)", + rows = [] + for task in tasks: + task_id = int(task["task_id"]) + states = [] + cells = [] + for model, engine in configurations: + result = results.get((model, engine, task_id)) + if result is None: + state, label = "missing", "未测" + elif float(result["score"]) == 1: + state, label = "pass", "通过" + else: + state, label = "fail", "失败" + states.append(state) + cells.append( + f'' + f"{label}" + if result is not None + else '未测' + ) + present = [state for state in states if state != "missing"] + relation = ( + "all_pass" + if present and all(state == "pass" for state in present) + else "all_fail" + if present and all(state == "fail" for state in present) + else "mixed" + ) + site = str(task["sites"][0]) + task_cell = ( + f'
#{task_id} · ' + f'{_e(task["rationale"])}
' + f'{_e(TIER_LABELS[task["tier"]])}' + f'{_e(SITE_LABELS.get(site, site))}' + f'{_e(task["task_type"])}
' + ) + rows.append( + f'' + f"{task_cell}{''.join(cells)}" ) - for group, metrics in METRICS + + headers = "".join( + f'{_e(MODEL_LABELS[model])}
{_e(ENGINE_LABELS[engine])}' + for model, engine in configurations ) return ( - absolute - + "

逐任务配对比值分布

先对每个同任务计算 Moli / Chromium,再汇总这些比值;P50、P95 与平均值都来自同一批逐任务比值,不是两个 Provider 边际分位数相除。

" - + paired + '
' + "

同题全配置矩阵

每行是同一任务,每列是一个模型×浏览器配置。新增配置只增加列。

" + '
' + '
' + '' + '' + '
' + '
' + f"{headers}" + f"{''.join(rows)}
任务
" ) -def _task_detail_body( - task_id: str, - title: str, - engine: str, - spec: dict[str, Any], - attempts: list[dict[str, Any]], - diagnostic: dict[str, Any] | None, -) -> str: - """One definition, one comparison matrix, one copy of each raw payload.""" - evidence: dict[str, tuple[str, list[str]]] = {} - - def evidence_link(value: Any, label: str, display: str = "") -> str: - if value is None or value == "": - return "" - text = _trace_text( - json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) - if isinstance(value, (dict, list)) - else value - ) - if text not in evidence: - evidence[text] = (f"evidence-{len(evidence) + 1}", []) - anchor, labels = evidence[text] - if label not in labels: - labels.append(label) - return f'{_e(display or label)}' - - def status_cell(value: Any) -> str: - return ( - "通过" - if value == "pass" or value is True - else "失败" - if value == "fail" or value is False - else "未记录" - ) - - final = attempts[-1] - source = spec.get("source") or {} - url = str(source.get("url") or "") - safe_url = url if urlparse(url).scheme in {"http", "https"} else "#" - labels = [f"执行记录 {a.get('attempt', i)}" for i, a in enumerate(attempts, 1)] - total = sum(float(a.get("duration_ms") or 0) for a in attempts) - result = final.get("result_status") or final.get("status") - process = final.get("process_status") or final.get("status") - overview = ( - '
' - f'

{_e(title)}

{_e(engine)}
' - f'

{_e(task_id)}

' - '
' - f'
结果{status_cell(result)}
' - f'
参考过程{status_cell(process)}
' - f"
累计耗时 · {len(attempts)} 次执行{_show(total, suffix=' ms')}
" - ) - if diagnostic: - overview += ( - '
' - f"{_e(diagnostic.get('category') or '其他')}" - f"第 {_e(diagnostic.get('failed_step'))} 步 · {_e(diagnostic.get('failed_action'))}" - f" · {_e(diagnostic.get('selector') or '—')}" - + ( - f"

{_e(diagnostic['observed_mechanism'])}

" - if diagnostic.get("observed_mechanism") - else "" - ) - + evidence_link(diagnostic.get("actionability_probe"), "Actionability probe") - + "
" - ) - overview += f'

起始页面 {_e(url)}

' - - expected = (spec.get("ground_truth") or {}).get("expected_outputs") or {} - validations = [{str(v.get("output")): v for v in a.get("validation") or []} for a in attempts] - outputs = list(dict.fromkeys([*expected, *(k for values in validations for k in values)])) - answer_rows = [] - for output in outputs: - expected_value = expected.get( - output, next((v[output].get("expected") for v in validations if output in v), "未声明") - ) - cells = [output, str(expected_value)] - for values in validations: - item = values.get(output) - cells.append( - ComparedValue( - f"{'(空字符串)' if item.get('actual') == '' else item.get('actual')} · {status_cell(item.get('passed'))}", - "ok" - if item.get("passed") is True - else "bad" - if item.get("passed") is False - else "", +def _group_rows( + attempts: list[dict[str, Any]], tasks: list[dict[str, Any]], field: str +) -> list[list[Any]]: + meta = {int(t["task_id"]): t for t in tasks} + grouped: dict[tuple[str, str, str], list[dict[str, Any]]] = defaultdict(list) + for row in attempts: + value = meta[int(row["task_id"])][field] + value = value[0] if isinstance(value, list) else value + grouped[(str(value), str(row["model"]), str(row["engine"]))].append(row) + result = [] + for value in sorted({key[0] for key in grouped}): + line: list[Any] = [SITE_LABELS.get(value, TIER_LABELS.get(value, value))] + for model in MODEL_LABELS: + passed_by_engine = { + engine: sum(float(x["score"]) == 1 for x in grouped[(value, model, engine)]) + for engine in ENGINE_LABELS + } + winners = _best(passed_by_engine, higher=True) + for engine in ENGINE_LABELS: + items = grouped[(value, model, engine)] + line.append( + _winner( + f"{passed_by_engine[engine]}/{len(items)}" if items else "—", + won=engine in winners, + label="同组通过数更高", + ) ) - if item - else "未产生(过程失败) · 未判定" - ) - answer_rows.append(cells) - answers = ( - _table(["输出", "标准答案", *labels], answer_rows, "答案对照", source_marks=True) - if outputs - else '

未声明标准答案。

' + result.append(line) + return result + + +def _action_duration_cards(attempts: list[dict[str, Any]]) -> str: + actions = sorted( + { + action + for row in attempts + for action in (row.get("browser_tool_duration_by_action_ms") or {}) + } ) - answers += _source_mark_note(expected) - derivation = (spec.get("ground_truth") or {}).get("derivation") - answers += f'

答案依据:{_e(derivation)}

' if derivation else "" - - calls = spec.get("tool_calls") or [] - traces = [a.get("trace") or [] for a in attempts] - steps = [] - for i in range(max([len(calls), *(len(t) for t in traces)], default=0)): - call = calls[i] if i < len(calls) else {} - reference = ( - f'{_e(call.get("action"))}{_e(_action_text(call))}' - if call - else "额外调用" - ) - parameters = {k: v for k, v in call.items() if k != "save_as"} - if call: - reference += evidence_link(parameters, f"步骤 {i + 1} 参数", "参数") - if call.get("save_as"): - reference += f"保存为 {_e(call['save_as'])}" - cells = [f"{i + 1}", f"{reference}"] - for label, trace, attempt in zip(labels, traces, attempts, strict=True): - if i >= len(trace): - cells.append('未执行') - continue - actual = trace[i] - passed = actual.get("status") == "pass" - cell = f'{_e(actual.get("action"))}{status_cell(actual.get("status"))} {_show(actual.get("duration_ms"), suffix=" ms")}' - if actual.get("action") != call.get("action"): - cell += '与参考动作不同' - if (actual.get("arguments") or {}) != parameters: - cell += evidence_link( - actual.get("arguments") or {}, f"{label} · 步骤 {i + 1} 参数", "实际参数" + cards: list[str] = [] + for action in actions: + metrics: dict[str, tuple[float, int, float | None]] = {} + for model in MODEL_LABELS: + for engine in ENGINE_LABELS: + key = f"{model}|{engine}" + total = sum( + float((row.get("browser_tool_duration_by_action_ms") or {}).get(action) or 0) + for row in attempts + if _row_model(row) == model and row["engine"] == engine ) - cell += evidence_link(actual.get("output"), f"{label} · 步骤 {i + 1} 输出", "输出") - if actual.get("error"): - # The full exception contains the call error; keep the full payload once. - error = attempt.get("error") - if not error or _trace_text(actual["error"]) not in _trace_text(error): - cell += evidence_link(actual["error"], f"{label} · 步骤 {i + 1} 错误", "异常") - else: - cell += evidence_link(error, f"{label} · 完整异常", "完整异常") - cells.append(f"{cell}") - steps.append("" + "".join(cells) + "") - process_cells = [] - for label, attempt in zip(labels, attempts, strict=True): - pv = attempt.get("process_validation") or {} - process_cells.append( - "" - + status_cell(pv.get("passed")) - + evidence_link(pv, f"{label} · 过程判定", "判定详情") - + "" - ) - steps.append("—过程判定" + "".join(process_cells) + "") - step_table = ( - '
' - + "".join(f"" for label in labels) - + "" - + "".join(steps) - + "
参考操作与执行对照
步骤浏览器参考动作{_e(label)}
" - ) - - execution_rows = [] - for label, attempt in zip(labels, attempts, strict=True): - execution_rows.append( - "" - + _e(label) - + "" - + _show(attempt.get("duration_ms"), suffix=" ms") - + "" - + (evidence_link(attempt.get("error"), f"{label} · 完整异常") or "无异常") - + "" - ) - execution_table = ( - '
' - + "".join(execution_rows) - + "
执行记录
记录耗时异常证据
" + count = sum( + int((row.get("browser_tool_count_by_action") or {}).get(action) or 0) + for row in attempts + if _row_model(row) == model and row["engine"] == engine + ) + metrics[key] = (total, count, total / count if count else None) + average_winners = _best( + {key: values[2] for key, values in metrics.items()}, + higher=False, + display=lambda value: round(value / 1000, 3), + ) + maximum = max((values[2] or 0 for values in metrics.values()), default=0) + browser_rows = [] + for model in MODEL_LABELS: + for engine in ENGINE_LABELS: + key = f"{model}|{engine}" + total, count, average = metrics[key] + width = 0 if average is None or maximum <= 0 else max(2.0, average / maximum * 100) + average_text = "—" if average is None else f"{average / 1000:.3f} 秒" + browser_rows.append( + f'
' + f"{_e(MODEL_LABELS[model])} · {_e(ENGINE_LABELS[engine])}{_winner(average_text, won=key in average_winners, label='全部配置中单次均值最低')}" + f'
' + f'
总耗时 {total / 1000:.1f} 秒 · {count} 次调用
' + ) + cards.append( + '
' + + _e(action) + + "全部配置 · 单次调用墙钟时间
" + + "".join(browser_rows) + + "
" + ) + return '
' + "".join(cards) + "
" + + +def _summary_comparison_rows(summary: dict[str, dict[str, dict[str, Any]]]) -> list[list[Any]]: + lower_names = ( + "mean_duration_ms", + "p50_duration_ms", + "p95_duration_ms", + "mean_browser_tool_duration_ms", + "p50_browser_tool_duration_ms", + "p95_browser_tool_duration_ms", + "mean_tokens", + "p50_tokens", + "p95_tokens", ) - resource_rows = [] - for _, metrics in METRICS: - for key, label, divisor, unit in metrics: - if key == "duration_ms" or not any(key in (a.get("resource") or {}) for a in attempts): - continue - resource_rows.append( + flat = { + f"{model}|{engine}": values + for model, model_summary in summary.items() + for engine, values in model_summary.items() + } + higher_metrics = { + name: _best({key: values[name] for key, values in flat.items()}, higher=True) + for name in ("passed", "score") + } + lower_metrics = { + name: _best( + {key: values[name] for key, values in flat.items()}, + higher=False, + display=( + (lambda value: round(value / 1000, 1)) + if "duration_ms" in name + else (lambda value: round(value)) + ), + ) + for name in lower_names + } + rows: list[list[Any]] = [] + for model, model_summary in summary.items(): + for engine, label in ENGINE_LABELS.items(): + key = f"{model}|{engine}" + current = model_summary[engine] + rows.append( [ - f"{label} ({unit})", - *[_show((a.get("resource") or {}).get(key), divisor) for a in attempts], + MODEL_LABELS[model], + label, + _winner( + f"{current['passed']}/{current['completed']}", + won=key in higher_metrics["passed"], + label="四种配置中通过数最高", + ), + _winner( + _pct(current["score"]), + won=key in higher_metrics["score"], + label="四种配置中通过率最高", + ), + _metric_stack( + [ + ( + "均值", + _seconds(current["mean_duration_ms"]), + key in lower_metrics["mean_duration_ms"], + ), + ( + "P50", + _seconds(current["p50_duration_ms"]), + key in lower_metrics["p50_duration_ms"], + ), + ( + "P95", + _seconds(current["p95_duration_ms"]), + key in lower_metrics["p95_duration_ms"], + ), + ] + ), + _metric_stack( + [ + ( + "均值", + _seconds(current["mean_browser_tool_duration_ms"]), + key in lower_metrics["mean_browser_tool_duration_ms"], + ), + ( + "P50", + _seconds(current["p50_browser_tool_duration_ms"]), + key in lower_metrics["p50_browser_tool_duration_ms"], + ), + ( + "P95", + _seconds(current["p95_browser_tool_duration_ms"]), + key in lower_metrics["p95_browser_tool_duration_ms"], + ), + ] + ), + _metric_stack( + [ + ( + "均值", + _number(current["mean_tokens"]), + key in lower_metrics["mean_tokens"], + ), + ( + "P50", + _number(current["p50_tokens"]), + key in lower_metrics["p50_tokens"], + ), + ( + "P95", + _number(current["p95_tokens"]), + key in lower_metrics["p95_tokens"], + ), + ] + ), ] ) - resources = ( - _table(["指标", *labels], resource_rows, "资源对照") - if resource_rows - else '

未记录资源采样。

' - ) - raw = "".join( - f'

{_e(anchor.replace("evidence-", "证据 "))}

回到步骤

{_e(" / ".join(refs))}

{_e(text)}
' - for text, (anchor, refs) in evidence.items() - ) - evidence_index = ( - '
' - + "".join( - f'{_e(anchor.replace("evidence-", "证据 "))}{_e(" / ".join(refs))}查看' - for _, (anchor, refs) in evidence.items() + return rows + + +def _resource_value(value: float | None, unit: str) -> str: + if value is None: + return "—" + if unit == "ms": + return f"{value / 1000:.1f} 秒" + if unit == "pct": + return f"{value:.1f}%" + if unit == "mib": + return f"{value / 1_048_576:,.1f} MiB" + if unit == "mib_net": + return f"{value / 1_048_576:,.2f} MiB" + raise ValueError(f"unknown resource unit: {unit}") + + +def _resource_detail_rows(resource: dict[str, Any]) -> list[list[str]]: + fields = ( + ("cpu_ms", "浏览器 CPU 时间", "ms"), + ("cpu_average_utilization_pct", "平均单核 CPU 占用", "pct"), + ("memory_mean_bytes", "时间加权平均内存", "mib"), + ("memory_peak_bytes", "峰值内存", "mib"), + ("memory_retained_bytes", "结束时新增常驻内存", "mib"), + ("network_rx_bytes", "网络接收量", "mib_net"), + ("network_tx_bytes", "网络发送量", "mib_net"), + ) + return [ + [label, _resource_value(float(resource[key]), unit)] + for key, label, unit in fields + if resource.get(key) is not None + ] + + +def _evidence_check_rows(checks: dict[str, Any]) -> list[list[str]]: + labels = { + "network_har_exists": "浏览器网络记录存在", + "agent_trace_exists": "浏览器执行轨迹存在", + "response_exists": "Agent 最终回答存在", + "evaluation_exists": "官方判分明细存在", + "events_exists": "Agent 事件流存在", + "environment_prepare_exists": "环境准备记录存在", + "evaluation_json": "判分明细可读取", + "response_json": "最终回答可读取", + "response_matches_record": "最终回答与运行记录一致", + "score_matches_evaluator": "报告分数与官方判分一致", + "status_matches_evaluator": "报告状态与官方判分一致", + "official_evaluator_version_present": "官方 evaluator 版本已记录", + "official_evaluator_checksum_matches": "官方 evaluator 文件校验一致", + "official_data_checksum_matches": "官方题集文件校验一致", + "model_request_attempts_accounted": "模型请求次数已完整计入", + } + return [ + [labels.get(key, key.replace("_", " ")), "通过" if value else "未通过"] + for key, value in checks.items() + ] + + +def _publication_label(value: Any) -> str: + return { + "private_only_session_sensitive": "仅保留哈希与字节数(含登录态)", + "sanitized_projection": "已脱敏公开副本", + "public": "公开原文件", + }.get(str(value), "按证据清单处理") + + +def _resource_comparison_table(summary: dict[str, dict[str, dict[str, Any]]]) -> str: + metrics = ( + ("cpu_ms", "CPU 时间", "ms"), + ("cpu_average_utilization_pct", "平均 CPU 单核占用", "pct"), + ("memory_mean_bytes", "时间加权平均内存", "mib"), + ("memory_peak_bytes", "峰值内存", "mib"), + ("network_rx_bytes", "网络接收量", "mib_net"), + ("network_tx_bytes", "网络发送量", "mib_net"), + ) + flat = { + f"{model}|{engine}": values + for model, model_summary in summary.items() + for engine, values in model_summary.items() + } + winners = { + (name, stat): _best( + { + key: ((values.get("resources") or {}).get(name) or {}).get(stat) + for key, values in flat.items() + }, + higher=False, + display=lambda value, unit=unit: round( + value / (1_048_576 if unit in {"mib", "mib_net"} else 1), 2 + ), ) - + "
" + for name, _, unit in metrics + for stat in ("mean", "p50", "p95") + } + rows: list[list[Any]] = [] + for model, model_summary in summary.items(): + for engine, label in ENGINE_LABELS.items(): + key = f"{model}|{engine}" + resources = model_summary[engine].get("resources") or {} + rows.append( + [ + MODEL_LABELS[model], + label, + *[ + _metric_stack( + [ + ( + stat.upper() if stat != "mean" else "均值", + _resource_value((resources.get(name) or {}).get(stat), unit), + key in winners[(name, stat)], + ) + for stat in ("mean", "p50", "p95") + ] + ) + for name, _, unit in metrics + ], + ] + ) + return _table( + ["模型", "浏览器", *[label for _, label, _ in metrics]], + rows, + table_id="resource-metrics-table", + classes="wide-data-table resource-comparison-table", ) - dialog = ( - '' - '

原始证据

' - f'

{_e(engine)} · {_e(title)}

' - '
' - '

' + + +REPORT_JS = """function theme(){const r=document.documentElement,n=r.dataset.theme==='dark'?'light':'dark';r.dataset.theme=n;try{localStorage.setItem('browser-use-theme',n)}catch(e){}} +function openDiagnosis(id){const d=document.getElementById(id);if(d?.showModal)d.showModal()} +function closeDiagnosis(id){const d=document.getElementById(id);if(d?.open)d.close()} +function updateViewUrl(){const u=new URL(location.href),q=document.querySelector('[data-search]');[...u.searchParams.keys()].filter(k=>k==='q'||k.startsWith('f_')).forEach(k=>u.searchParams.delete(k));if(q?.value)u.searchParams.set('q',q.value);document.querySelectorAll('[data-filter]').forEach(c=>{if(c.value)u.searchParams.set('f_'+c.dataset.filter,c.value)});history.replaceState(null,'',u.pathname+(u.search?'?'+u.searchParams.toString():'')+u.hash)} +function filterRows(save=true){const controls=[...document.querySelectorAll('[data-filter]')],query=(document.querySelector('[data-search]')?.value||'').trim().toLowerCase();let visible=0,total=0;document.querySelectorAll('tbody tr[data-tier]:not([data-relation])').forEach(r=>{total++;const hidden=Boolean(controls.some(c=>c.value&&r.dataset[c.dataset.filter]!==c.value)||(query&&!String(r.dataset.search||r.textContent).toLowerCase().includes(query)));r.hidden=hidden;if(!hidden)visible++});document.querySelectorAll('[data-visible-count]').forEach(x=>x.textContent=visible+' / '+total);if(save)updateViewUrl()} +function filterMatrix(){const root=document.getElementById('configuration-matrix');if(!root)return;const controls=[...root.querySelectorAll('[data-matrix-filter]')];let visible=0,total=0;root.querySelectorAll('tbody tr[data-tier]').forEach(r=>{total++;const hidden=controls.some(c=>c.value&&r.dataset[c.dataset.matrixFilter]!==c.value);r.hidden=hidden;if(!hidden)visible++});const count=root.querySelector('[data-matrix-count]');if(count)count.textContent=visible+' / '+total} +function applySort(t,col,dir){const b=t?.tBodies[0];if(!b)return;[...b.rows].sort((a,z)=>{const av=a.cells[col]?.dataset.sort||a.cells[col]?.textContent||'',zv=z.cells[col]?.dataset.sort||z.cells[col]?.textContent||'';const an=Number(av.replace(/[^0-9.-]/g,'')),zn=Number(zv.replace(/[^0-9.-]/g,''));const cmp=!Number.isNaN(an)&&!Number.isNaN(zn)?an-zn:av.localeCompare(zv,'zh-CN');return dir==='asc'?cmp:-cmp}).forEach(r=>b.appendChild(r));t.dataset.col=String(col);t.dataset.dir=dir} +function sortTable(id,col){const t=document.getElementById(id);if(!t)return;const dir=t.dataset.col==col&&t.dataset.dir==='asc'?'desc':'asc';applySort(t,col,dir);const u=new URL(location.href);u.searchParams.set('sort',id+':'+col+':'+dir);history.replaceState(null,'',u.pathname+'?'+u.searchParams.toString()+u.hash)} +function applyAnchor(){if(!location.hash)return;const id=decodeURIComponent(location.hash.slice(1)),target=document.getElementById(id);if(target?.scrollIntoView)requestAnimationFrame(()=>target.scrollIntoView({block:'start'}))} +function hasOption(control,value){return !control?.options||[...control.options].some(option=>option.value===value)} +function restoreView(){const p=new URLSearchParams(location.search),q=document.querySelector('[data-search]');if(q&&p.has('q'))q.value=p.get('q')||'';document.querySelectorAll('[data-filter]').forEach(c=>{const v=p.get('f_'+c.dataset.filter);if(v!==null&&hasOption(c,v))c.value=v});const s=(p.get('sort')||'').split(':');if(s.length===3)applySort(document.getElementById(s[0]),Number(s[1]),s[2]);filterRows(false);filterMatrix();try{if(!location.hash){const y=sessionStorage.getItem('browser-use-scroll:'+location.pathname+location.search);if(y!==null)requestAnimationFrame(()=>scrollTo(0,Number(y)||0))}}catch(e){}} +document.addEventListener('DOMContentLoaded',()=>{try{document.documentElement.dataset.theme=localStorage.getItem('browser-use-theme')||'light'}catch(e){}restoreView();applyAnchor();document.querySelectorAll('[data-filter]').forEach(x=>x.addEventListener('change',()=>filterRows()));document.querySelectorAll('[data-search]').forEach(x=>x.addEventListener('input',()=>filterRows()));document.querySelectorAll('[data-matrix-filter]').forEach(x=>x.addEventListener('change',filterMatrix));document.querySelectorAll('dialog').forEach(d=>d.addEventListener('click',e=>{if(e.target===d)d.close()}))});addEventListener('pagehide',()=>{try{sessionStorage.setItem('browser-use-scroll:'+location.pathname+location.search,String(scrollY))}catch(e){}});""" +REPORT_STYLE = ( + SHARED_STYLE + + """ +:root{--max:1440px} +.hero h1,.publication-article h1,.publication-article h2{overflow-wrap:anywhere;word-break:break-word;max-width:100%} +.callout strong{margin-right:.3em} +.top nav{gap:18px;flex-wrap:nowrap;overflow-x:auto;overscroll-behavior-inline:contain;scrollbar-width:thin}.top nav>a,.top nav>button{flex:0 0 auto}.top nav .brand{font-weight:850;color:var(--ink)}.top nav a[aria-current=page]{color:var(--ink);font-weight:800;text-decoration:underline;text-decoration-thickness:2px;text-underline-offset:8px}.top .theme-toggle{margin-left:auto}.page-main{min-height:78vh}.hero.compact{padding:34px 0 26px}.hero.compact h1{font-size:clamp(34px,4vw,50px);margin:10px 0}.hero.compact .lead{margin:0}.section{padding:32px 0}.fine{font-size:13px;color:var(--muted)}.result-grid{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:14px}.result-grid .card{border-top:3px solid var(--accent)}.overview-grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:14px}.overview-card{border:1px solid var(--line);border-radius:14px;background:var(--panel);padding:20px;min-width:0}.overview-card h3{margin:8px 0}.audit-strip{display:grid;gap:10px}.audit-strip .callout{margin:0}.filters,.pair-controls{display:flex;gap:10px;flex-wrap:wrap;align-items:end;margin:18px 0}.filters select,.filters input,.pair-controls select{padding:9px 12px;border:1px solid var(--line);border-radius:9px;background:var(--panel);color:var(--ink)}.filters input{min-width:260px}.comparison-count{margin-left:auto;padding:8px 12px;border-radius:999px;background:var(--wash);color:var(--accent);font-weight:750}.status-pass{color:var(--good);font-weight:750}.status-fail{color:var(--bad);font-weight:750}.status-unproven{color:var(--warn,var(--bad));font-weight:750}.flow{display:grid;grid-template-columns:repeat(6,minmax(0,1fr));gap:9px;list-style:none;padding:0}.flow li{padding:14px;border:1px solid var(--line);border-radius:11px;background:var(--panel);font-size:13px;min-width:0}.flow li+li:before{content:'→';color:var(--accent);margin-right:7px}.method-figure{margin:24px 0}.method-figure img{display:block;width:100%;height:auto;border:1px solid var(--line);border-radius:16px;background:#f6faff}.method-figure figcaption{margin-top:10px;color:var(--muted);font-size:13px;line-height:1.55}.detail-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(210px,1fr));gap:12px}.detail-grid .kv{min-height:92px;min-width:0}.page-links{display:flex;gap:10px;flex-wrap:wrap}.page-links a{border:1px solid var(--line);background:var(--panel);padding:8px 12px;border-radius:9px}.rank-list{counter-reset:rank}.rank-item{counter-increment:rank;display:grid;grid-template-columns:54px minmax(0,1fr);gap:16px;padding:20px 0;border-top:1px solid var(--line)}.rank-item:before{content:counter(rank);font-size:34px;font-weight:850;color:var(--accent)}.rank-item h3{margin:4px 0}.article-entry{border-left:4px solid var(--accent)}.source-list li{margin:9px 0}.table-wrap{margin:18px 0}.table-wrap table{min-width:760px}#task-table{min-width:1280px}#pair-table{min-width:1180px}.table-wrap caption{text-align:left;padding:10px 12px;color:var(--muted);caption-side:bottom}.footer{border-top:1px solid var(--line)}.evidence-pair{display:grid;grid-template-columns:minmax(0,1fr) minmax(0,1fr);gap:14px}.evidence-card{border:1px solid var(--line);border-radius:12px;background:var(--panel);padding:16px;min-width:0}.evidence-card h3{margin-top:0}.json-view,.log-view{white-space:pre-wrap;overflow-wrap:anywhere;max-height:540px;overflow:auto;padding:14px;border:1px solid var(--line);border-radius:9px;background:var(--bg);font-size:12px;line-height:1.55}.trajectory{list-style:none;padding:0;counter-reset:step}.trajectory>li{counter-increment:step;display:grid;grid-template-columns:46px minmax(0,1fr);gap:12px;padding:15px 0;border-top:1px solid var(--line)}.trajectory>li:before{content:counter(step);display:grid;place-items:center;width:34px;height:34px;border-radius:50%;background:var(--accent);color:white;font-weight:800}.trajectory summary{cursor:pointer;font-weight:750}.rule-fail{border-left:4px solid var(--bad)}.rule-pass{border-left:4px solid var(--good)}.count-note{margin-left:auto;color:var(--muted);font-size:13px}.badge{display:inline-block;padding:3px 8px;border-radius:999px;background:var(--wash);font-size:12px;margin-right:6px}.muted-box{padding:14px;border:1px dashed var(--line);border-radius:10px;color:var(--muted)}@media(max-width:900px){.result-grid{grid-template-columns:repeat(2,minmax(0,1fr))}.overview-grid{grid-template-columns:1fr}.flow{grid-template-columns:repeat(3,minmax(0,1fr))}.evidence-pair{grid-template-columns:1fr}.grid>.card{grid-column:1/-1!important}}@media(max-width:600px){.wrap{width:calc(100% - 28px)}.top nav{gap:14px}.result-grid,.flow{grid-template-columns:1fr}.flow li+li:before{content:'↓';display:block}.hero{padding-top:24px}.hero.compact h1{font-size:30px;line-height:1.18}.filters input{min-width:0;width:100%}.comparison-count{margin-left:0}.rank-item{grid-template-columns:36px minmax(0,1fr)}}@media print{.top,.filters,.pair-controls{display:none!important}.card,.table-wrap{box-shadow:none}.json-view,.log-view{max-height:none;overflow:visible}.evidence-pair{grid-template-columns:1fr}} +/* Publication QA overrides: readable dense tables, code, and text-heavy figures. */ +.table-wrap{overflow-x:auto;max-width:100%;contain:inline-size;border:1px solid var(--line);border-radius:14px;background:var(--panel)} +.table-wrap table{min-width:860px;margin:0;border:0} +.table-wrap th{white-space:nowrap} +.table-wrap th,.table-wrap td{word-break:normal;overflow-wrap:break-word;vertical-align:top} +#task-table{min-width:1500px} +#task-table th:nth-child(1),#task-table td:nth-child(1){min-width:76px;white-space:nowrap} +#task-table th:nth-child(2),#task-table td:nth-child(2),#task-table th:nth-child(4),#task-table td:nth-child(4),#task-table th:nth-child(5),#task-table td:nth-child(5){min-width:88px;white-space:nowrap} +#task-table th:nth-child(3),#task-table td:nth-child(3){min-width:140px;white-space:nowrap} +#task-table th:nth-child(6),#task-table td:nth-child(6){min-width:250px} +#task-table th:nth-child(7),#task-table td:nth-child(7){min-width:430px} +#task-table th:nth-child(8),#task-table td:nth-child(8),#task-table th:nth-child(9),#task-table td:nth-child(9){min-width:112px;white-space:nowrap} +#pair-table{min-width:1320px} +#pair-table th:nth-child(-n+4),#pair-table td:nth-child(-n+4){white-space:nowrap} +#pair-table th:nth-child(1),#pair-table td:nth-child(1){min-width:260px;white-space:normal} +#pair-table th:nth-child(n+5),#pair-table td:nth-child(n+5){min-width:235px} +#result-table{min-width:1580px;table-layout:fixed} +#result-table th,#result-table td{white-space:nowrap;overflow:hidden;text-overflow:ellipsis;vertical-align:middle;height:72px;max-height:72px} +#result-table th:nth-child(1),#result-table td:nth-child(1){width:230px} +#result-table th:nth-child(2),#result-table td:nth-child(2){width:170px} +#result-table th:nth-child(3),#result-table td:nth-child(3){width:80px} +#result-table th:nth-child(4),#result-table td:nth-child(4){width:75px} +#result-table th:nth-child(5),#result-table td:nth-child(5){width:250px} +#result-table th:nth-child(6),#result-table td:nth-child(6){width:180px} +#result-table th:nth-child(7),#result-table td:nth-child(7){width:180px} +#result-table th:nth-child(8),#result-table td:nth-child(8){width:310px} +#result-table th:nth-child(9),#result-table td:nth-child(9){width:170px} +#engine-result-table,#change-table{min-width:1080px} +#result-table td:first-child,#engine-result-table td:first-child{white-space:nowrap;min-width:150px} +.json-view,.log-view{background:var(--code-bg)!important;color:var(--code-ink)!important;border-color:#334b68!important;line-height:1.65} +.json-view code,.log-view code{color:inherit!important;background:transparent!important} +.detail-grid .kv{overflow-wrap:anywhere;word-break:break-word} +.method-figure{overflow-x:auto;border:1px solid var(--line);border-radius:16px;background:#f6faff} +.method-figure img{min-width:960px;border:0;border-radius:0} +.method-figure figcaption{padding:0 16px 14px} +.release-current,.release-archive{border:1px solid var(--line);border-radius:16px;background:var(--panel);padding:22px;margin:16px 0} +.release-current{border-top:4px solid var(--accent)} +.release-history>summary,.release-current details>summary,.release-archive details>summary{cursor:pointer;font-weight:800;padding:8px 0} +.comparison-winner{color:var(--good);font-weight:850}.winner-mark{display:inline-grid;place-items:center;min-width:22px;height:22px;padding:0 5px;margin-left:7px;border-radius:999px;background:var(--good);color:white;font-size:11px;line-height:1}.winner-cell{position:relative;padding:9px 10px;border:1px solid color-mix(in srgb,var(--good) 42%,var(--line));border-radius:10px;background:color-mix(in srgb,var(--good) 9%,var(--panel))}.metric-stack{display:grid;gap:5px;min-width:150px}.metric-stack>span{display:flex;align-items:center;justify-content:space-between;gap:12px;white-space:nowrap}.metric-stack>span>b{font-size:12px;color:var(--muted)}.review-round{border:1px solid var(--line);border-radius:16px;background:var(--panel);padding:22px;margin:16px 0}.review-round header{display:flex;align-items:start;justify-content:space-between;gap:18px}.review-round h3{margin:5px 0}.review-status{flex:0 0 auto;padding:5px 10px;border-radius:999px;background:var(--wash);font-size:12px;font-weight:800}.review-fields{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:14px;margin-top:16px}.review-fields>div{border-top:1px solid var(--line);padding-top:12px}.review-fields b{display:block;margin-bottom:5px}.review-fields p{margin:0} +.browser-comparison{border:1px solid var(--line);border-radius:18px;background:var(--panel);padding:24px;box-shadow:0 12px 32px rgba(25,45,70,.06)}.comparison-heading{display:flex;align-items:start;justify-content:space-between;gap:24px;margin-bottom:10px}.comparison-heading h3{margin:0 0 6px;font-size:22px}.comparison-heading p{margin:0;color:var(--muted);line-height:1.65}.matrix-filters{margin:16px 0}.comparison-result{margin-top:12px}.table-wrap .configuration-matrix{min-width:1120px;table-layout:fixed}.configuration-matrix th:first-child,.configuration-matrix td:first-child{width:420px}.configuration-matrix th:not(:first-child),.configuration-matrix td:not(:first-child){width:165px;text-align:center;vertical-align:middle}.task-cell a{font-weight:750}.task-meta{display:flex;align-items:center;gap:7px;flex-wrap:wrap;margin-top:7px}.track-badge{display:inline-block;padding:3px 8px;border-radius:999px;background:var(--panel2);color:var(--muted);font-size:12px;white-space:nowrap}.outcome-pill{display:inline-block;min-width:62px;padding:6px 10px;border-radius:8px;font-size:13px;text-decoration:none;font-weight:750}.outcome-pill.pass{background:var(--success-bg);color:var(--good)}.outcome-pill.fail{background:var(--failure-bg);color:var(--bad)}.outcome-pill.missing{background:var(--panel2);color:var(--muted)} +.action-card-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:14px}.action-card{border:1px solid var(--line);border-radius:16px;background:var(--panel);padding:18px;min-width:0}.action-card>header{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-bottom:14px}.action-card>header code{font-size:16px;font-weight:800}.action-card>header span{font-size:12px;color:var(--muted)}.action-browser-row{padding:12px 0;border-top:1px solid var(--line)}.action-browser-head{display:flex;align-items:center;justify-content:space-between;gap:12px}.action-bar{height:7px;border-radius:999px;background:var(--panel2);overflow:hidden;margin:8px 0}.action-bar i{display:block;height:100%;border-radius:inherit;background:var(--accent)}.action-browser-row.action-winner .action-bar i{background:var(--good)}.action-browser-meta{font-size:12px;color:var(--muted)} +.finding-list{display:grid;gap:8px;margin:0 0 20px;padding:16px 20px 16px 38px;border-left:4px solid var(--accent);border-radius:0 12px 12px 0;background:var(--wash)}.finding-list li{padding-left:4px}.table-kicker{display:block;color:var(--muted);font:700 11px/1.4 ui-monospace,monospace;letter-spacing:.04em;margin-top:3px}.core-comparison-table{min-width:1240px;table-layout:fixed}.core-comparison-table td:nth-child(1),.core-comparison-table td:nth-child(2){font-weight:750}.core-comparison-table th:nth-child(1),.core-comparison-table td:nth-child(1){width:145px}.core-comparison-table th:nth-child(2),.core-comparison-table td:nth-child(2){width:135px}.core-comparison-table th:nth-child(3),.core-comparison-table td:nth-child(3){width:115px}.core-comparison-table th:nth-child(4),.core-comparison-table td:nth-child(4){width:105px}.core-comparison-table th:nth-child(n+5),.core-comparison-table td:nth-child(n+5){width:245px}.pair-outcome-table{min-width:900px}.wide-data-table td{line-height:1.65}.report-entry-links{margin-top:18px}.report-entry-links a{font-weight:750}.source-note a{margin-left:5px} +#task-family-table{min-width:1320px;table-layout:fixed}#task-family-table th:nth-child(1),#task-family-table td:nth-child(1){width:150px}#task-family-table th:nth-child(2),#task-family-table td:nth-child(2){width:90px}#task-family-table th:nth-child(3),#task-family-table td:nth-child(3){width:300px}#task-family-table th:nth-child(n+4),#task-family-table td:nth-child(n+4){width:190px;white-space:nowrap}.metric-note{margin:14px 0 0;border-top:1px solid var(--line);padding-top:12px;color:var(--muted)}.metric-note summary{cursor:pointer;font-weight:750;color:var(--ink)} +.diagnosis-button,.dialog-actions button{border:1px solid var(--accent);background:var(--accent);color:white;padding:8px 12px;border-radius:9px;font:inherit;font-weight:750;cursor:pointer;white-space:nowrap}.diagnosis-button:hover,.dialog-actions button:hover{filter:brightness(.94)}.diagnosis-dialog{width:min(920px,calc(100% - 32px));max-height:88vh;padding:0;border:1px solid var(--line);border-radius:18px;background:var(--panel);color:var(--ink);box-shadow:0 28px 90px rgba(10,25,45,.28)}.diagnosis-dialog::backdrop{background:rgba(5,15,25,.62);backdrop-filter:blur(3px)}.diagnosis-dialog-head{position:sticky;top:0;z-index:1;display:flex;align-items:start;justify-content:space-between;gap:20px;padding:22px 24px 16px;border-bottom:1px solid var(--line);background:var(--panel)}.diagnosis-dialog-head h2{font-size:24px;margin:4px 0}.dialog-close{border:0;background:var(--panel2);color:var(--ink);width:36px;height:36px;border-radius:50%;font-size:24px;line-height:1;cursor:pointer}.diagnosis-dialog-grid{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:10px;padding:18px 24px}.diagnosis-dialog-grid>div{padding:13px;border:1px solid var(--line);border-radius:10px;background:var(--bg);min-width:0;overflow-wrap:anywhere}.diagnosis-dialog-grid b{display:block;font-size:12px;color:var(--muted);margin-bottom:5px}.dialog-error,.dialog-review{padding:0 24px 18px}.dialog-error h3,.dialog-review h3{margin:8px 0}.dialog-error .log-view{max-height:240px}.dialog-review p{margin:6px 0}.dialog-actions{position:sticky;bottom:0;display:flex;align-items:center;gap:10px;flex-wrap:wrap;padding:16px 24px;border-top:1px solid var(--line);background:var(--panel)}.dialog-actions a{padding:8px 10px;border:1px solid var(--line);border-radius:9px}.dialog-actions button{margin-left:auto} +.failure-lead{padding:20px 22px;margin-bottom:14px;border:1px solid color-mix(in srgb,var(--bad) 35%,var(--line));border-left:6px solid var(--bad);border-radius:14px;background:var(--failure-bg)}.failure-lead .status-line{color:var(--bad);font-size:13px;font-weight:850}.failure-lead h3{font-size:24px;margin:5px 0}.failure-lead p{margin:4px 0}.success-lead{border-color:color-mix(in srgb,var(--good) 35%,var(--line));border-left-color:var(--good);background:var(--success-bg)}.success-lead .status-line{color:var(--good)}.runtime-strip{grid-template-columns:repeat(6,minmax(0,1fr));margin:14px 0}.failure-error{margin-top:16px}.failure-error .log-view{max-height:260px}.trajectory-step.failure-step{margin:8px 0;padding:14px;border:1px solid color-mix(in srgb,var(--bad) 40%,var(--line));border-left:5px solid var(--bad);border-radius:12px;background:var(--failure-bg)}.trajectory-step.failure-step:before{background:var(--bad)}.evidence-viewer-head{display:flex;align-items:center;justify-content:space-between;gap:16px}.evidence-viewer-head h2{margin:0}.evidence-viewer-shell .json-view{max-height:none;min-height:60vh} +.action-card-grid{grid-auto-rows:1fr}.action-card{height:100%}.dialog-step,.step-signature{padding:13px 16px;border-radius:10px;background:var(--wash);overflow-wrap:anywhere}.dialog-step{margin:0 24px 18px}.dialog-step b{margin-right:8px}.step-signature{margin:0 0 14px} +.attempt-summary{display:grid;grid-template-columns:minmax(270px,1.4fr) repeat(4,minmax(135px,.7fr));gap:12px;align-items:stretch}.attempt-verdict{padding:20px 22px;border:1px solid var(--line);border-left:5px solid var(--good);border-radius:12px;background:var(--success-bg)}.attempt-verdict.failure{border-left-color:var(--bad);background:var(--failure-bg)}.attempt-verdict .status-line{display:block;font-size:24px;font-weight:850;margin-bottom:6px}.attempt-verdict p{margin:0;line-height:1.65}.attempt-jumps{display:flex;gap:14px;flex-wrap:wrap;margin-top:14px;font-weight:750}.attempt-fact{padding:16px;border:1px solid var(--line);border-radius:12px;background:var(--panel);overflow-wrap:anywhere}.attempt-fact b{display:block;color:var(--muted);font-size:12px;margin-bottom:5px}.attempt-secondary{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:12px;margin-top:12px} +.diagnosis-grid{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:12px}.diagnosis-grid .attempt-fact{min-height:110px}.diagnosis-reason{margin-top:14px;padding:18px 20px;border-left:5px solid var(--bad);border-radius:10px;background:var(--failure-bg);line-height:1.7}.evidence-links{display:flex;gap:8px}.screenshot-grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:14px}.screenshot-grid figure{margin:0;border:1px solid var(--line);border-radius:12px;overflow:hidden;background:var(--panel)}.screenshot-grid img{display:block;width:100%;aspect-ratio:16/10;object-fit:cover;object-position:top}.screenshot-grid figcaption{padding:10px 12px;color:var(--muted);font-size:12px} +.task-goal{padding:22px 24px;border-radius:16px;background:var(--wash);border:1px solid var(--line);margin-bottom:18px}.task-goal span,.rule-index{display:block;color:var(--accent);font-size:12px;font-weight:850;letter-spacing:.08em;text-transform:uppercase}.task-goal p{font-size:20px;font-weight:750;line-height:1.6;margin:7px 0}.task-goal small{color:var(--muted)}.verification-list{display:grid;gap:14px}.verification-rule{padding:20px 22px;border:1px solid var(--line);border-left:5px solid var(--good);border-radius:14px;background:var(--panel);min-width:0}.verification-rule.rule-fail{border-left-color:var(--bad)}.verification-rule>header{display:flex;align-items:start;justify-content:space-between;gap:18px}.verification-rule h3{margin:4px 0 0;font-size:19px}.rule-verdict{flex:0 0 auto;padding:5px 10px;border-radius:999px;font-size:12px;font-weight:850}.rule-verdict.pass{background:var(--success-bg);color:var(--good)}.rule-verdict.fail{background:var(--failure-bg);color:var(--bad)}.rule-purpose{color:var(--muted);margin:12px 0 6px}.rule-summary{font-size:17px;font-weight:750;margin:0 0 14px}.verification-rule details{border-top:1px solid var(--line);padding-top:12px;margin-top:10px}.verification-rule summary{cursor:pointer;font-weight:750}.compact-evidence{margin-top:12px}.compact-evidence .json-view{max-height:360px} +.page-main{overflow-x:clip} +.section[id]{scroll-margin-top:76px} +@media(max-width:980px){.attempt-summary{grid-template-columns:repeat(2,minmax(0,1fr))}.attempt-verdict{grid-column:1/-1}.attempt-secondary,.diagnosis-grid,.runtime-strip{grid-template-columns:repeat(2,minmax(0,1fr))}.screenshot-grid{grid-template-columns:1fr}} +/* Interpretation article: compact evidence-first reading, without dashboard sprawl. */ +.interpretation-v2{max-width:1120px;margin:0 auto;padding:34px 0 72px}.interpretation-v2>.essay-head{padding:22px 0 28px;border-bottom:1px solid var(--line)}.interpretation-v2>.essay-head h1{max-width:950px;font-size:clamp(38px,5vw,64px);line-height:1.08;margin:12px 0 16px;letter-spacing:-.035em}.interpretation-v2>.essay-head .lead{max-width:900px;font-size:20px;line-height:1.75}.interpretation-v2>section{padding:34px 0;border-bottom:1px solid var(--line)}.interpretation-v2>section>h2{font-size:clamp(28px,3vw,38px);margin:0 0 12px}.interpretation-v2>section>h3{font-size:22px;margin:30px 0 8px}.answer-strip{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:14px}.answer-strip article,.evidence-examples article,.decision-grid article{min-width:0;padding:20px;border:1px solid var(--line);border-radius:15px;background:var(--panel)}.answer-strip b{display:block;font-size:18px;line-height:1.45}.answer-strip p,.evidence-examples p,.decision-grid p{margin:9px 0 0;color:var(--muted)}.score-flow-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:12px;margin:22px 0}.score-flow-card{padding:18px;border:1px solid var(--line);border-radius:14px;background:var(--panel)}.score-flow-card header,.cause-bar>div{display:flex;justify-content:space-between;align-items:center;gap:14px}.score-flow-card header span{font-weight:850;color:var(--accent)}.score-flow-card p{margin:9px 0 0;color:var(--muted);font-size:13px}.score-flow-card p span{color:var(--accent);padding:0 4px}.score-track{display:flex;height:9px;margin-top:14px;border-radius:999px;background:var(--panel2);overflow:hidden}.score-track i{display:block;height:100%;background:linear-gradient(90deg,var(--accent),#4d7cff)}.score-track i.recovered{background:var(--good)}.cause-bars{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:12px;margin:22px 0}.cause-bar{position:relative;overflow:hidden;min-height:126px;padding:17px;border:1px solid var(--line);border-radius:14px;background:var(--panel)}.cause-bar strong{font-size:26px}.cause-bar span{display:block;margin-top:10px;color:var(--muted);font-size:13px;line-height:1.5}.cause-bar i{position:absolute;left:0;bottom:0;height:5px;background:var(--accent);border-radius:0 999px 999px 0}.evidence-examples,.decision-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:12px;margin:18px 0}.evidence-examples h3,.decision-grid h3{margin:0;font-size:18px}.audit-ladder{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:0;margin:22px 0}.audit-ladder article{position:relative;padding:20px 22px;border:1px solid var(--line);background:var(--panel)}.audit-ladder article:first-child{border-radius:14px 0 0 14px}.audit-ladder article:last-child{border-radius:0 14px 14px 0}.audit-ladder article+article{border-left:0}.audit-ladder span{display:grid;place-items:center;width:30px;height:30px;margin-bottom:12px;border-radius:50%;background:var(--accent);color:white;font-weight:850}.audit-ladder b{font-size:17px}.audit-ladder p{margin:8px 0 0;color:var(--muted);font-size:14px}.interpretation-v2 .essay-note{margin-top:22px}.interpretation-v2 .table-wrap{margin:20px 0}.interpretation-v2 .core-comparison-table{min-width:860px} +.verdict-strip{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:12px;margin:16px 0}.verdict-strip article{display:grid;grid-template-columns:auto minmax(0,1fr);gap:3px 12px;align-items:center;padding:17px;border:1px solid var(--line);border-radius:14px;background:var(--panel)}.verdict-strip strong{grid-row:1/3;font-size:34px;color:var(--accent)}.verdict-strip b{line-height:1.4}.verdict-strip p{margin:0;color:var(--muted);font-size:12px;line-height:1.45} +@media(max-width:900px){.answer-strip,.verdict-strip{grid-template-columns:repeat(2,minmax(0,1fr))}.answer-strip article:last-child,.verdict-strip article:last-child{grid-column:1/-1}.cause-bars{grid-template-columns:repeat(2,minmax(0,1fr))}} +@media(max-width:760px){.browser-comparison{padding:18px}.comparison-heading{display:block}.comparison-count{display:inline-block;margin-top:12px}.comparison-controls{grid-template-columns:1fr}.action-card-grid{grid-template-columns:1fr}.interpretation-v2{padding-top:18px}.interpretation-v2>.essay-head h1{font-size:36px}.interpretation-v2>.essay-head .lead{font-size:17px}.interpretation-v2>section{padding:27px 0}} +@media(max-width:600px){.hero h1{word-break:break-all;line-break:anywhere;text-wrap:balance}.method-figure img{min-width:900px}.table-wrap{margin-left:-4px;margin-right:-4px}.json-view,.log-view{font-size:13px}.review-round header{display:block}.review-status{display:inline-block;margin-top:8px}.review-fields,.attempt-summary,.attempt-secondary,.diagnosis-dialog-grid,.answer-strip,.score-flow-grid,.cause-bars,.evidence-examples,.decision-grid,.verdict-strip,.audit-ladder{grid-template-columns:1fr}.answer-strip article:last-child,.verdict-strip article:last-child{grid-column:auto}.audit-ladder article:first-child,.audit-ladder article:last-child{border-radius:14px}.audit-ladder article+article{border-left:1px solid var(--line);border-top:0}.diagnosis-dialog-head,.diagnosis-dialog-grid,.dialog-error,.dialog-review,.dialog-actions{padding-left:16px;padding-right:16px}.dialog-actions button{margin-left:0}} +""" +) +NAV = [ + ("", "Browser Use", "home"), + ("datasets/", "评测集", "datasets"), + ("methodology/", "评测方法", "methodology"), + ("analysis/", "结果分析", "analysis"), + ("review/", "核验记录", "review"), + ("adoption/", "被测浏览器", "adoption"), + ("reproduce/", "环境与复现", "reproduce"), + ("changes/", "变更记录", "changes"), + ("accounts/", "账号与安装", "accounts"), + ("en/", "English", "en"), +] + + +def _page(*, title: str, active: str, body: str, depth: int = 0) -> str: + local_base = "../" * depth or "./" + links = "".join( + f'{_e(label)}" + for href, label, key in NAV + ) + return f'{_e(title)} · Browser Use 评测
{body}

结论只适用于本轮预注册样本、Agent、模型与运行配置;展示调整不改变测量结果。

' + + +def _evidence_viewer_page() -> str: + body = _hero( + "Execution evidence", + "完整执行证据", + "按原始结构展示本次物理执行的脱敏回答、evaluator、事件流、轨迹、计量与公开画面索引。", + ) + body += ( + '
' + '

执行证据 JSON

' + '
' + '

正在读取证据…

' + '
'
+        "
" + ) + return _page(title="完整执行证据", active="analysis", body=body) + + +def _english_page(body: str) -> str: + links = "".join( + f'{label}' + for anchor, label in ( + ("results", "Results"), + ("scope", "Scope"), + ("method", "Method"), + ("evidence", "Evidence"), + ("review", "Review"), + ("access", "Access"), + ) ) - script = """""" return ( - '
' - + overview - + '' - + '

操作与答案

' - + answers - + step_table - + _source_references(source) - + "
" - + '

执行资源

' - + execution_table - + resources - + "
" - + '

原始证据

' - + (evidence_index + raw if raw else "

未记录执行轨迹。

") - + "
" - + dialog - + script + '' + '' + 'English edition · Browser Use evaluation' + '
' + '
' + + body + + '

Conclusions apply only to this preregistered sample, agent, model, and runtime configuration.

' + ) + + +def _hero(eyebrow: str, title: str, lead: str) -> str: + return f'
{_e(eyebrow)}

{_e(title)}

{_e(lead)}

' + + +def _section(title: str, body: str) -> str: + return f'

{_e(title)}

{body}
' + + +def _status(row: dict[str, Any]) -> Safe: + passed = float(row["score"]) == 1 + return Safe( + f'{"通过" if passed else "失败"}' + ) + + +def _json_pre(value: Any, *, css_class: str = "json-view") -> str: + return f'
{_e(json.dumps(value, ensure_ascii=False, indent=2))}
' + + +def _task_contract_html(task: dict[str, Any]) -> str: + contract = task.get("contract") or {} + rules = contract.get("evaluation_contract") or [] + cards = [] + for index, rule in enumerate(rules, start=1): + evaluator = rule.get("evaluator") or "UnknownEvaluator" + expected = rule.get("expected") or {} + qualifiers = { + key: value for key, value in rule.items() if key not in {"evaluator", "expected"} + } + cards.append( + '

' + + _e(f"规则 {index} · {evaluator}") + + '

程序化验收该规则必须得分 1.0。

' + + _json_pre(expected) + + ( + f"
匹配选项与结果 schema{_json_pre(qualifiers)}
" + if qualifiers + else "" + ) + + "
" + ) + return ( + '
没有唯一参考点击路径。 WebArena‑Verified 允许 Agent 自主选择任意 UI 路径;' + "ground truth 是以下最终回答与必要网络副作用的合取。报告因此比较结果合同,而不把某条人工路径当作标准答案。
" + + '
' + + "".join(cards) + "
" ) -def _execution_details( - manifest: dict[str, Any], summary: dict[str, Any], detail_pages: dict[str, str] -) -> str: - task_specs = {str(task.get("task_id")): task for task in manifest.get("suite") or []} - diagnostic_groups = ((summary.get("failure_diagnostics") or {}).get("rule_engine") or {}).get( - "groups" - ) or [] - diagnostic_by_attempt = { +def _failure_diagnosis(result: dict[str, Any] | None, row: dict[str, Any]) -> str: + failed = [item for item in evaluator_summary(result) if float(item.get("score") or 0) < 1] + if not failed: + return "全部官方 evaluator 均通过。" + parts = [] + for item in failed: + messages: list[str] = [] + for assertion in item.get("assertions") or []: + assertion_messages = assertion.get("assertion_msgs") or [] + if assertion_messages: + messages.append(str(assertion_messages[0])) + if assertion.get("error_msg"): + messages.append(str(assertion["error_msg"])) + if item.get("error"): + messages.append(str(item["error"])) + if not messages: + messages.append( + "最终回答与官方结构化答案不一致" + if item["name"] == "AgentResponseEvaluator" + else "未观察到满足合同的必要网络请求或副作用" + if item["name"] == "NetworkEventEvaluator" + else "该规则未通过" + ) + expected, actual = item.get("expected"), item.get("actual") + if expected is not None or actual is not None: + comparison = ( + f"expected={json.dumps(expected, ensure_ascii=False, separators=(',', ':'))}; " + f"actual={json.dumps(actual, ensure_ascii=False, separators=(',', ':'))}" + ) + messages.append(comparison) + exact = ";".join( + dict.fromkeys(str(sanitize(message)).strip().rstrip("。.;; ") for message in messages) + ) + if len(exact) > 420: + exact = exact[:417].rstrip() + "…" + parts.append(f"规则 {item['index']} {item['name']}:{exact}") + if row.get("error"): + parts.append(str(sanitize(row["error"]))) + response = row.get("agent_response") or {} + if response.get("error_details"): + parts.append(str(sanitize(response["error_details"]))) + return ";".join(dict.fromkeys(x.strip() for x in parts if x.strip())).rstrip("。.;; ") + "。" + + +def _evaluator_error_text(result: dict[str, Any] | None, row: dict[str, Any]) -> str: + """Return the exact sanitized evaluator/runtime error without inferring a cause.""" + + lines: list[str] = [] + for item in evaluator_summary(result): + if float(item.get("score") or 0) == 1: + continue + messages = [ + str(message) + for assertion in item.get("assertions") or [] + for message in [*(assertion.get("assertion_msgs") or []), assertion.get("error_msg")] + if message + ] + if item.get("error"): + messages.append(str(item["error"])) + if not messages: + messages.append(_rule_outcome_summary(item)) + for message in dict.fromkeys(messages): + lines.append(f"规则 {item['index']} · {item['name']}: {sanitize(message)}") + if row.get("error"): + lines.append(f"运行错误: {sanitize(row['error'])}") + response = row.get("agent_response") or {} + if response.get("error_details"): + lines.append(f"Agent 错误: {sanitize(response['error_details'])}") + return "\n".join(lines) or "程序没有提供更具体的错误文本。" + + +def _failed_rule_labels(result: dict[str, Any] | None) -> list[str]: + return [ + f"规则 {item['index']} {item['name']}" + for item in evaluator_summary(result) + if float(item.get("score") or 0) < 1 + ] + + +def _automatic_failure_type(result: dict[str, Any] | None) -> tuple[str, str]: + """Classify the observable evaluator failure without claiming root cause.""" + + names = { + item["name"] for item in evaluator_summary(result) if float(item.get("score") or 0) < 1 + } + if not names: + return "", "" + if names == {"AgentResponseEvaluator"}: + return "response", "最终回答验收失败" + if names == {"NetworkEventEvaluator"}: + return "network", "网络副作用验收失败" + if names == {"AgentResponseEvaluator", "NetworkEventEvaluator"}: + return "mixed", "多项验收失败" + raise ValueError(f"unknown failed evaluator combination: {sorted(names)}") + + +def _cause_code(verified: dict[str, dict[str, Any]], result_id: str) -> str: + return str(((verified.get(result_id) or {}).get("subcategory") or {}).get("code") or "") + + +def _failure_point(code: str, steps: list[dict[str, Any]]) -> tuple[str, int | None]: + """Locate the evidenced stage without inventing a precise failing click.""" + + failed_step = next( ( - str(group.get("task_id")), - str(group.get("engine")), - int(attempt_ref.get("attempt") or 1), - ): str(group.get("category") or "其他") - for group in diagnostic_groups - for attempt_ref in (group.get("attempts") or []) + step + for step in reversed(steps) + if str(step.get("status") or "").lower() not in {"", "success"} or step.get("error") + ), + None, + ) + last_step = int(steps[-1].get("step") or len(steps)) if steps else None + if code.startswith("environment_"): + return "环境准备后、Agent 读取前", None + if code.startswith("benchmark_"): + return "运行结束后的 evaluator 验收", last_step + if code.startswith("evidence_") or code == "evidence_capture": + return "运行结束后的网络证据验收", last_step + if code.startswith("site_"): + return (f"第 {last_step} 步后的站点响应/保存" if last_step else "站点响应/保存"), last_step + if failed_step: + number = int(failed_step.get("step") or last_step or 1) + return f"第 {number} 步 BrowserTool 执行异常", number + if code == "agent_step_budget": + return (f"第 {last_step} 步后达到步骤上限" if last_step else "达到步骤上限"), last_step + return ( + f"第 {last_step} 步后的最终回答/任务状态" if last_step else "最终回答/任务状态" + ), last_step + + +def _trajectory_context( + steps: list[dict[str, Any]], step_number: int | None, failure_point: str +) -> tuple[str, str]: + """Describe the linked step without claiming a successful action caused evaluator failure.""" + + if step_number is None: + return "", "" + step = next((item for item in steps if int(item.get("step") or 0) == step_number), None) + if step is None: + return "", "" + label = trajectory_label(step) + prefix = "失败动作" if "BrowserTool 执行异常" in failure_point else "验收前最后一步" + return prefix, label + + +def _remediation(code: str) -> tuple[str, str, bool]: + """Return owner, current action, and whether this is an external interference.""" + + if code.startswith("benchmark_"): + return "评测集 / evaluator", "修正合同并重跑;当前仅纳入官方分数", True + if code.startswith("site_"): + return "自托管站点", "修复依赖或保存链路,通过写入预检后重跑", True + if code.startswith("environment_"): + return "运行环境", "跨进程锁已上线;受污染任务待重跑", True + if code.startswith("mixed_"): + return "任务与 Agent", "先冻结参考状态,再重跑区分 Agent 错误", True + if code.startswith("evidence_") or code == "evidence_capture": + return "浏览器证据接口", "保留为浏览器集成信号;补齐 HAR 后重跑", False + return "Agent / 模型", "保留为目标变量;按轨迹修正策略或工具合同", False + + +def _failure_brief(result: dict[str, Any] | None, row: dict[str, Any]) -> str: + labels = _failed_rule_labels(result) + if labels: + return f"{len(labels)} 项规则未通过:{'、'.join(labels)}。" + if row.get("error"): + return "执行未完成;错误信息见运行与证据。" + return "全部官方 evaluator 均通过。" + + +def _rule_purpose(name: str) -> str: + if name == "AgentResponseEvaluator": + return "检查 Agent 最终回答的状态、字段和值是否符合官方答案合同。" + if name == "NetworkEventEvaluator": + return "检查浏览器网络记录中是否出现合同要求的页面副作用。" + return "检查该题官方合同声明的结果条件。" + + +def _rule_outcome_summary(item: dict[str, Any]) -> str: + if float(item.get("score") or 0) == 1: + return ( + "已观察到满足合同的网络事件。" + if item.get("name") == "NetworkEventEvaluator" + else "最终回答符合该规则。" + ) + messages = [ + str(message) + for assertion in item.get("assertions") or [] + for message in (assertion.get("assertion_msgs") or [])[:1] + ] + message = messages[0] if messages else str(item.get("error") or "") + status = re.fullmatch(r"Expected ([^,]+), got (.+)", message) + if status: + return f"最终状态不符:期望 {status.group(1)},实际为 {status.group(2)}。" + missing = re.fullmatch( + r"Array is missing (\d+) expected element\(s\)\. Matched \((\d+)/(\d+)\)", + message, + ) + if missing: + return f"最终数据缺少 {missing.group(1)} 项;官方期望 {missing.group(3)} 项,匹配 {missing.group(2)} 项。" + if "Array value mismatch" in message: + return "最终数据与官方期望值不一致。" + if "No matching network event" in message: + return "浏览器网络记录中没有出现满足该规则的事件。" + return message or "该规则未通过。" + + +def _verification_html(task: dict[str, Any], result: dict[str, Any] | None, response: Any) -> str: + items = evaluator_summary(result) + if not items: + return '

没有 evaluator 明细。

' + cards = [] + for item in items: + passed = float(item.get("score") or 0) == 1 + observed = response if item.get("name") == "AgentResponseEvaluator" else item.get("actual") + technical = { + key: item.get(key) + for key in ("assertions", "error", "should_not_exist") + if item.get(key) not in (None, [], {}) + } + cards.append( + f'
' + '
' + + _e(f"规则 {item['index']}") + + "

" + + _e(item["name"]) + + f'

' + + ("通过" if passed else "失败") + + "
" + + f'

{_e(_rule_purpose(item["name"]))}

' + + f'

{_e(_rule_outcome_summary(item))}

' + + '
查看官方期望与本次观察

官方期望

' + + _json_pre(item.get("expected")) + + "

本次观察

" + + _json_pre(observed) + + "
" + + ( + f"
查看 evaluator 技术明细{_json_pre(technical)}
" + if technical + else "" + ) + + "
" + ) + return ( + '
任务目标

' + + _e(task.get("intent") or "—") + + "

Agent 可自主选择 UI 路径;计分只看以下最终回答与必要网络副作用。
" + + '
' + + "".join(cards) + + "
" + ) + + +def _trajectory_html(steps: list[dict[str, Any]]) -> str: + if not steps: + return '

没有可展示的 BrowserTool 调用。

' + rows = [] + for step in steps: + step_number = int(step.get("step") or len(rows) + 1) + failed = str(step.get("status") or "").lower() not in {"", "success"} or bool( + step.get("error") + ) + duration = step.get("duration_seconds") + meta = f"Turn {step.get('turn') or '—'} · {_e(step.get('status') or 'unknown')}" + if duration is not None: + meta += f" · {float(duration):.2f} 秒" + rows.append( + f'
  • ' + + _e(trajectory_label(step)) + + f'

    {meta}

    调用参数

    ' + + _json_pre(step.get("arguments")) + + "

    执行输出节选

    " + + f'
    {_e(result_excerpt(step))}
    ' + + "
  • " + ) + return '
      ' + "".join(rows) + "
    " + + +PUBLIC_EXECUTION_ARTIFACTS = { + "execution_evidence": "execution-evidence.json", +} + + +def _extract_trace_screenshots( + trace_path: Path, artifact_dir: Path, *, limit: int = 3 +) -> list[dict[str, Any]]: + """Publish the final Playwright screencast frames, never arbitrary trace resources.""" + + if not trace_path.is_file(): + return [] + try: + with zipfile.ZipFile(trace_path) as archive: + archive_names = set(archive.namelist()) + frames = [] + for raw_line in archive.read("trace.trace").splitlines(): + try: + event = json.loads(raw_line) + except (TypeError, ValueError): + continue + if event.get("type") != "screencast-frame" or not event.get("sha1"): + continue + name = "resources/" + str(event["sha1"]) + if name not in archive_names: + continue + frames.append((float(event.get("timestamp") or 0), name)) + names = [] + for _timestamp, name in reversed(sorted(frames)): + if name not in names: + names.append(name) + if len(names) == limit: + break + names.reverse() + valid_frames = [] + for name in names: + payload = archive.read(name) + with Image.open(BytesIO(payload)) as image: + width, height = image.size + if width < 800 or height < 450: + continue + suffix = ".png" if name.lower().endswith(".png") else ".jpeg" + valid_frames.append((payload, width, height, suffix)) + output = [] + screenshot_dir = artifact_dir / "screenshots" + screenshot_dir.mkdir(parents=True, exist_ok=True) + total = len(valid_frames) + for index, (payload, width, height, suffix) in enumerate(valid_frames, start=1): + destination = screenshot_dir / f"frame-{index}{suffix}" + destination.write_bytes(payload) + output.append( + { + "path": f"screenshots/{destination.name}", + "sha256": _sha256(destination), + "label": f"最后页面画面 {index}/{total}", + "source": "playwright-screencast-frame", + "width": width, + "height": height, + } + ) + return output + except (OSError, KeyError, zipfile.BadZipFile): + return [] + + +def _artifact_links(result_id: str, execution: int) -> str: + root = f"artifacts/{result_id}/executions/{execution}" + viewer = f"evidence-viewer.html?src={root}/execution-evidence.json" + return ( + '" + ) + + +def _write_execution_artifacts( + stage: Path, + result_id: str, + row: dict[str, Any], + *, + authoritative: bool, +) -> tuple[dict[str, Any], Any, Any, list[dict[str, Any]]]: + paths = row.get("paths") or {} + response = read_public_json(Path(paths.get("response", ""))) + evaluation = read_public_json(Path(paths.get("evaluation", ""))) + events = read_public_json(Path(paths.get("events", ""))) + steps = trajectory(events if isinstance(events, list) else []) + execution = int(row["attempt"]) + artifact_dir = stage / "artifacts" / result_id / "executions" / str(execution) + artifact_dir.mkdir(parents=True) + measurements = _public_execution(row, authoritative=authoritative) + screenshots = ( + _extract_trace_screenshots(Path(paths.get("agent_trace", "")), artifact_dir) + if authoritative and float(row.get("score") or 0) < 1 + else [] + ) + bundle = { + "schema": "lex_browser_use_public_execution_evidence/3", + "result_id": result_id, + "execution": execution, + "authoritative": authoritative, + "measurements": measurements, + "source_evidence": evidence_metadata(row), + "agent_response": response, + "evaluation": evaluation, + "agent_events": events, + "trajectory": steps, + "screenshots": screenshots, } - diagnostic_by_execution = { - (str(group.get("task_id")), str(group.get("engine"))): group for group in diagnostic_groups + _write_json(artifact_dir / PUBLIC_EXECUTION_ARTIFACTS["execution_evidence"], bundle) + artifacts = { + label: { + "path": f"artifacts/{result_id}/executions/{execution}/{filename}", + "sha256": _sha256(artifact_dir / filename), + } + for label, filename in PUBLIC_EXECUTION_ARTIFACTS.items() } - blocks = [] - for task_id, task in (summary.get("by_task") or {}).items(): - title = str(task.get("title") or task_id) - executions = task.get("attempts") or task.get("failures") or {} - for engine, attempts in executions.items(): - if not attempts: - continue - anchor = f"execution-{task_id}-{engine}" - # Hex encoding is injective and keeps arbitrary task IDs out of paths. - detail_name = _detail_filename(str(task_id), str(engine)) - final_attempt = attempts[-1] - final_status = str(final_attempt.get("status") or "fail") - final_result_status = str(final_attempt.get("result_status") or final_status) - final_process_status = str(final_attempt.get("process_status") or final_status) - final_combined_status = str(final_attempt.get("combined_status") or final_status) - categories = sorted( - { - diagnostic_by_attempt.get( - (str(task_id), str(engine), int(attempt.get("attempt") or 1)), - _classify(str(attempt.get("error") or "")), - ) - for attempt in attempts - if str(attempt.get("status") or "fail") == "fail" - } + return ( + { + **measurements, + "artifacts": artifacts, + "screenshots": screenshots, + }, + response, + evaluation, + steps, + ) + + +def _screenshot_gallery(result_id: str, executions: list[dict[str, Any]]) -> str: + authoritative = next((item for item in executions if item.get("authoritative")), {}) + execution = int(authoritative.get("execution") or 0) + screenshots = authoritative.get("screenshots") or [] + if not screenshots: + return '

    该执行没有可公开的 trace 页面画面;错误日志与执行证据仍可下载。

    ' + root = f"artifacts/{result_id}/executions/{execution}" + figures = "".join( + '
    '
+        + _e(item[
    ' + + _e(item["label"]) + + f" · SHA-256 {_e(str(item['sha256'])[:12])}…
    " + for item in screenshots + ) + return ( + "

    下图是判分权威执行的最后页面画面,用于核对失败前的可见状态;" + "它们不一定正好对应错误发生的瞬间。

    " + f'
    {figures}
    ' + ) + + +def _execution_evidence_html(result_id: str, executions: list[dict[str, Any]]) -> str: + cards = [] + for execution in executions: + usage = execution["model_usage"] + authority = " · 判分权威" if execution["authoritative"] else "" + hashes = _table( + ["公开文件", "SHA-256"], + [[label, artifact["sha256"]] for label, artifact in execution["artifacts"].items()], + ) + cards.append( + '
    ' + f"

    物理执行 {execution['execution']}{authority}

    " + f"

    判分 {_number(execution['score'])} · Agent 链路 {_seconds(execution['duration_ms'])} · " + f"BrowserTool {execution['action_count']} 次 / {_seconds(execution['browser_tool_duration_ms'])} · " + f"模型 tokens {_number(usage['total_tokens'])}

    " + + _artifact_links(result_id, int(execution["execution"])) + + hashes + + "
    " + ) + return ( + f"

    该逻辑结果包含 {len(executions)} 次物理执行。" + "标记为判分权威的执行决定最终得分;累计耗时、tokens 与 BrowserTool 指标是下列各次执行之和。

    " + '
    ' + "".join(cards) + "
    " + ) + + +def _common_final_success_resource_rows( + attempts: list[dict[str, Any]], +) -> tuple[int, list[list[Any]]]: + task_ids = {int(row["task_id"]) for row in attempts} + common = { + task_id + for task_id in task_ids + if all( + any( + row["model"] == model + and row["engine"] == engine + and int(row["task_id"]) == task_id + and float(row.get("score") or 0) == 1 + for row in attempts ) - error_category = ",".join(categories) if categories else "无错误" - actions = sorted( + for model in MODEL_LABELS + for engine in ENGINE_LABELS + ) + } + + def median(values: list[float]) -> float | None: + return statistics.median(values) if values else None + + rows: list[list[Any]] = [] + for model in MODEL_LABELS: + for engine in ENGINE_LABELS: + cohort = [ { - str(call.get("action")) - for attempt in attempts - for call in (attempt.get("trace") or []) - if call.get("action") != "validate_assertions" + **row, + **_execution_tool_metrics(row), } - ) - tags = task.get("tags") or [] - total_duration = sum(float(attempt.get("duration_ms") or 0) for attempt in attempts) - diagnostic = diagnostic_by_execution.get((str(task_id), str(engine))) - detail_body = _task_detail_body( - str(task_id), - title, - str(engine), - task_specs.get(str(task_id)) or {}, - attempts, - diagnostic, - ) - # Artifact Hub injects a site-root base URL; fragment links need the file. - detail_body = detail_body.replace('href="#', f'href="{detail_name}#') - detail_pages[detail_name] = _shell( - manifest, - "analysis", - detail_body, - detail_title=title, - back_anchor=anchor, - ) - search_text = " ".join( + for row in attempts + if row["model"] == model + and row["engine"] == engine + and int(row["task_id"]) in common + ] + rows.append( [ - title, - str(engine), - error_category, - *[ - str(a.get("error") or "") - + " " - + json.dumps(a.get("validation") or [], ensure_ascii=False) - + " " - + json.dumps(a.get("trace") or [], ensure_ascii=False) - for a in attempts - ], + MODEL_LABELS[model], + ENGINE_LABELS[engine], + _seconds(median([float(row.get("duration_ms") or 0) for row in cohort])), + _seconds( + median([float(row.get("browser_tool_duration_ms") or 0) for row in cohort]) + ), + _number( + median( + [float((row.get("model_usage") or {}).get("total_tokens") or 0) for row in cohort] + ) + ), + ( + "—" + if not cohort + else f"{median([float((row.get('resource') or {}).get('memory_peak_bytes') or 0) for row in cohort]) / 1048576:.1f} MiB" + ), + _seconds( + median([float((row.get("resource") or {}).get("cpu_ms") or 0) for row in cohort]) + ), ] ) - blocks.append( - f'' - ) - return "".join(blocks) or "

    没有可展示的执行样本。

    " + return len(common), rows + + +def _failure_scope_counts(verified_causes: dict[str, dict[str, Any]]) -> list[tuple[str, int, str]]: + copy = { + "agent": ("Agent / 模型", "检索不全、选错对象、字段遗漏或流程提前结束"), + "benchmark": ("任务 / evaluator", "验收合同与页面语义不一致,或 evaluator 自身报错"), + "site": ("自托管站点", "搜索、会话或保存依赖故障"), + "evidence": ("浏览器证据接口", "HAR 缺少 Referer、响应正文或出现重复记录"), + "environment": ("共享环境", "并发执行改写了任务初始状态"), + "mixed": ("参考状态与 Agent 混合", "参考漂移与 Agent 遗漏同时存在"), + } + + def scope(code: str) -> str: + for prefix in ("agent", "benchmark", "site", "environment", "mixed"): + if code.startswith(prefix + "_"): + return prefix + if code.startswith("evidence_"): + return "evidence" + raise ValueError(f"unknown failure-cause scope: {code}") + + counts = Counter( + scope(str((item.get("subcategory") or {}).get("code") or "evidence_capture")) + for item in verified_causes.values() + ) + return [ + (copy[key][0], counts[key], copy[key][1]) + for key in ("agent", "benchmark", "site", "evidence", "environment", "mixed") + if counts[key] + ] + +def _failure_scope_figure(verified_causes: dict[str, dict[str, Any]]) -> str: + rows = _failure_scope_counts(verified_causes) + maximum = max((count for _, count, _ in rows), default=1) + bars = "".join( + '
    ' + + _e(label) + + f'{count}
    {_e(description)}
    ' + for label, count, description in rows + ) + return f'
    ' + bars + "
    " + + +def _automatic_failure_counts(attempts: list[dict[str, Any]]) -> Counter[str]: + counts = Counter() + for row in attempts: + if float(row.get("score") or 0) == 1: + continue + result = read_public_json(Path((row.get("paths") or {}).get("evaluation", ""))) + key, _ = _automatic_failure_type(result if isinstance(result, dict) else None) + if key: + counts[key] += 1 + return counts + + +def _automatic_failure_figure(counts: Counter[str]) -> str: + items = ( + ("network", "只有网络副作用规则未通过", "evaluator 没找到合同要求的请求或可观察副作用"), + ("response", "只有最终回答规则未通过", "最终回答与结构化答案合同不一致"), + ("mixed", "两类规则同时未通过", "最终回答和网络副作用均未满足合同"), + ) + cards = "".join( + f'
    {counts[key]}{_e(label)}

    {_e(description)}

    ' + for key, label, description in items + ) + return '
    ' + cards + "
    " + + +def _resource_summary_cards(common_resources: tuple[int, list[list[Any]]]) -> str: + task_count, rows = common_resources + if not task_count: + return '

    当前没有四种配置最终都成功的任务,资源差异暂不比较。

    ' + + def values(engine: str, column: int) -> list[float]: + label = ENGINE_LABELS[engine] + return [float(str(row[column]).split()[0].replace(",", "")) for row in rows if row[1] == label] + + chromium_memory, moli_memory = values("chromium", 5), values("moli", 5) + chromium_tool, moli_tool = values("chromium", 3), values("moli", 3) + chromium_cpu, moli_cpu = values("chromium", 6), values("moli", 6) + return ( + '

    Moli 的内存区间更低

    ' + f'{task_count} 题最终共同成功样本中,Moli 峰值内存 P50 为 {min(moli_memory):.1f}–{max(moli_memory):.1f} MiB;' + f'Chromium 为 {min(chromium_memory):.1f}–{max(chromium_memory):.1f} MiB。

    ' + '

    Chromium 的工具耗时与 CPU 更低

    ' + f'BrowserTool P50:Chromium {min(chromium_tool):.1f}–{max(chromium_tool):.1f} 秒,Moli {min(moli_tool):.1f}–{max(moli_tool):.1f} 秒;' + f'CPU 时间 P50:Chromium {min(chromium_cpu):.1f}–{max(chromium_cpu):.1f} 秒,Moli {min(moli_cpu):.1f}–{max(moli_cpu):.1f} 秒。

    ' + ) -def _execution_explorer( - manifest: dict[str, Any], summary: dict[str, Any], detail_pages: dict[str, str] + +def _article_body( + paired: dict[str, Any], + attempts: list[dict[str, Any]], + tasks: list[dict[str, Any]], + verified_causes: dict[str, dict[str, Any]], + review: dict[str, Any], + cohort_commits: dict[str, str], + automatic_failure_counts: Counter[str], ) -> str: - physical_attempts = sum( - (values or {}).get("physical_attempts", 0) - for values in (summary.get("functional") or {}).values() + total_task_count = len(tasks) + excluded = _external_interference_task_ids(verified_causes) + eligible_attempts = [ + row for row in attempts if int(row["task_id"]) not in excluded + ] + diagnostic_task_count = total_task_count - len(excluded) + task_count = total_task_count + summary = _primary_summary(attempts) + paired = _paired_summary(attempts) + raw_failed_result_count = sum(float(row.get("score") or 0) < 1 for row in attempts) + comparison_rows = [ + [ + MODEL_LABELS[model], + ENGINE_LABELS[item["baseline_engine"]], + ENGINE_LABELS[item["comparison_engine"]], + item["paired"], + item["both_pass"], + item["baseline_only"], + item["comparison_only"], + item["both_fail"], + ] + for model, model_pairs in paired.items() + for item in model_pairs["comparisons"].values() + ] + discordant = sum(row[5] + row[6] for row in comparison_rows) + cause_rows = [ + [ + Safe(f'{_e(result_id)}'), + (diagnosis.get("subcategory") or {}).get("zh") or "—", + (diagnosis.get("reason") or {}).get("zh") or "—", + ] + for result_id, diagnosis in sorted(verified_causes.items()) + ] + if raw_failed_result_count: + cause_section = ( + _failure_cause_table(verified_causes) + + '
    查看每个失败结果的核验结论' + + _table( + ["结果编号", "原因分类", "核验结论"], + cause_rows, + caption="人工核验用于解释官方失败结果,不修改分数。修复后需要重新运行才能产生新成绩。", + ) + + "
    " + if len(cause_rows) == raw_failed_result_count + else f'

    当前有 {raw_failed_result_count} 个原始失败结果,详细原因仍在核验中;未核验结果不展示推测。

    ' + ) + else: + cause_section = '

    本轮没有失败结果,无需生成失败根因记录。

    ' + common_count, resource_rows = _common_final_success_resource_rows(attempts) + clean = _primary_summary(eligible_attempts) + clean_rows = [ + [ + MODEL_LABELS[model], + ENGINE_LABELS[engine], + f"{values['passed']}/{values['completed']}", + _pct(values["score"]), + ] + for model, model_summary in clean.items() + for engine, values in model_summary.items() + ] + coverage = review.get("coverage") or {} + reviewed_attempts = int(coverage.get("reviewed_attempts") or 0) + recorded_attempts = int(coverage.get("recorded_attempts") or 0) + clean_task_count = diagnostic_task_count + return ( + '
    Browser Use 能力评测
    ' + f"

    {task_count} 题 Browser Use 完整结果:Chromium 与 Moli 的成功率、代价与失败根因

    " + f'

    首页按预注册的 {total_task_count} 题展示四种配置的官方 evaluator 结果。其中 {len(excluded)} 题经复核存在验收合同、站点、共享状态或参考状态干扰;这些题仍保留在 90 题官方结果中,另外提供 {diagnostic_task_count} 题事后诊断子集。

    ' + '

    WebArena‑Verified 1.2.3 · CowAgent · 模型推理强度固定为低 · 官方程序化 evaluator

    ' + '
    ' + '

    90 题核心指标

    ' + + _table( + ["模型", "浏览器", "通过任务", "通过率", "Agent 总耗时", "BrowserTool 耗时", "Tokens"], + _summary_comparison_rows(summary), + table_id="home-core-metrics", + classes="core-comparison-table", + ) + + f'

    每个模型×浏览器配置都保留完整 {task_count} 题。以上是官方 evaluator 决定的最终逻辑结果;物理执行记录可在单题证据页下钻。

    ' + '

    按同题比较浏览器

    浏览器对比必须留在同一模型层内。汇总数会隐藏互换的成败任务;下表给出同题对照。

    ' + + _table(["模型", "浏览器 A", "浏览器 B", "同题数", "两边都通过", "A 通过、B 失败", "A 失败、B 通过", "两边都失败"], comparison_rows, caption=f"每行只比较同一模型下的 {task_count} 个配对任务。") + + f'

    两个模型层合计有 {discordant} 个单边通过,可用于定位兼容性问题;样本量和方向都不支持总体优劣判断。查看 {task_count} 题的四配置矩阵 →

    ' + '

    程序判分落点与复核根因

    程序判分先说明哪类规则未通过;人工复核再沿最终回答、evaluator 断言、HAR 和 Agent 轨迹追到直接原因。这两个层次需要分开。

    程序判分落点

    ' + + _automatic_failure_figure(automatic_failure_counts) + + '

    “网络副作用未通过”只描述程序判分位置。它可能源于 Agent 没有提交,也可能源于 evaluator、站点或 HAR 采集问题。

    复核根因

    ' + + _failure_scope_figure(verified_causes) + + cause_section + + '

    Agent 行为

    例如漏掉商品、选错最优候选、写错字段,或在 40 步前提前结束。这些属于当前 Agent × 模型配置的真实失败。

    验收规则

    Task 699 的 evaluator 在比较实际提交前就因单元素 Alternatives 配置抛错;Task 528 要求“不提交”,验收却要求 POST。这些 0 分不能归给模型或浏览器。

    站点与环境

    Magento 搜索或库存保存服务失效,以及 Task 77 的并发状态污染,都会让正确操作无法通过。

    浏览器证据

    Moli 的部分 HAR 没有 Referer 或响应正文,例如 Task 595 和 Task 676。页面可能已经操作成功,但 evaluator 无法从网络记录确认;这是浏览器集成层的有效信号。

    ' + + f'

    90 题官方结果与事后诊断分开看

    首页主结果始终使用完整 {total_task_count} 题官方 evaluator 分母。为了观察排除已知外部干扰后的差异,下表再按整题、四种配置一致排除 {len(excluded)} 题,得到 {clean_task_count} 题事后诊断子集。

    ' + + _table(["模型", "浏览器", f"{clean_task_count} 题通过", "通过率"], clean_rows, table_id="home-qualified-results", classes="core-comparison-table") + + f'

    这是事后诊断子集,不是正式分数,也不替代 90 题官方结果。{raw_failed_result_count} 个原始失败记录继续保存在结果页和核验记录中。

    ' + '

    审查覆盖与边界

    1证据完整性

    ' + + f'{reviewed_attempts}/{recorded_attempts} 个证据包由程序核对文件存在性、哈希、JSON 可读性、回答 / 分数 / 状态一致性和运行身份。

    2官方程序验收

    每题按最终回答和必要网络副作用判分;没有 LLM judge。

    3失败根因复核

    {len(verified_causes)}/{raw_failed_result_count} 个原始最终失败结果逐条阅读错误、断言、最终回答、HAR 和 Agent 轨迹,归类并给出处理边界。

    ' + + '

    审查分为完整性检查和失败根因复核。 前者确认“证据包与 evaluator 记录一致”;后者检查 evaluator 语义,并识别出 Task 699 等问题。已通过结果没有再做独立语义盲审,其判定依据仍是官方 evaluator。查看核验记录 →

    ' + f'

    完整 {task_count} 题的速度与浏览器资源

    均值、P50、P95 都在四种完整配置中统一比较。CPU、内存和网络只计浏览器容器,不含 CowAgent 或模型服务;模型会改变动作数量和页面停留时间,因此这里比较整套配置,不能单独归因于浏览器内核。

    ' + + _resource_comparison_table(summary) + + '

    每项只标记四种配置中数值唯一最低者。CPU 时间与网络总量累计逻辑结果使用的全部执行;内存来自 Browser Use 执行阶段的浏览器容器采样。

    ' + '

    共同成功任务的性能与资源

    下表限定为四种配置最终都通过的 ' + + str(common_count) + + ' 题,并以每题最终逻辑结果包含的全部物理执行计算 P50;这样不会把快速失败误当成性能优势。

    ' + + _table(["模型", "浏览器", "Agent 链路 P50", "BrowserTool P50", "Tokens P50", "峰值内存 P50", "CPU 时间 P50"], resource_rows, table_id="home-resource-table", classes="core-comparison-table") + + _resource_summary_cards((common_count, resource_rows)) + + '

    Agent 链路还包含模型推理和页面等待,因此不能解读为纯内核性能。

    ' + f'

    工程结论

    完整 {task_count} 题官方结果是本报告的主口径;{clean_task_count} 题事后诊断子集只用于分离已知外部干扰,不改写成绩。

    Moli 的内存占用明显更低,但 HAR 完整性、工具耗时和部分任务兼容性还需修复;Chromium 在本轮 Sol 层多通过 3 题,Astra 层通过数相同,但当前每种配置只有一个正式样本,不能外推成总体优势。

    下一轮验证

    先修复可控的合同、evaluator、站点和共享状态问题,再按同一协议完整重跑 90 题四种配置;历史官方分数在新证据批次出现前保持不变。

    ' ) - logical_executions = sum( - 1 - for task in (summary.get("by_task") or {}).values() - for attempts in ((task.get("attempts") or task.get("failures") or {}).values()) + + +def _filters(*, attempts: bool = False, dataset: bool = False) -> str: + engine_options = "".join( + f'' + for engine, label in ENGINE_LABELS.items() + ) + extra = ( + '" + + f'' + '' if attempts + else "" ) - task_options = "".join( - f'' - for task_id, task in (summary.get("by_task") or {}).items() + result_filter = ( + "" + if dataset + else ( + '" + ) ) - action_values = sorted( - { - str(c.get("action")) - for task in (summary.get("by_task") or {}).values() - for attempts in (task.get("attempts") or {}).values() - for attempt in attempts - for c in attempt.get("trace") or [] - if c.get("action") != "validate_assertions" - } + placeholder = "任务、意图、验收合同" if dataset else "任务、意图、答案、错误" + return ( + f'
    ' + '' + '' + '' + + result_filter + + extra + + '
    ' ) - action_options = "".join( - f'' for action in action_values + + +def _task_table(tasks: list[dict[str, Any]]) -> str: + rows = [] + for task in tasks: + task_id, site, kind = int(task["task_id"]), task["sites"][0], task["task_type"] + values = [ + Safe(f'#{task_id}'), + TIER_LABELS[task["tier"]], + SITE_LABELS.get(site, site), + kind, + task["estimated_min_steps"], + task["rationale"], + task.get("intent") or "—", + "是" if task["official_hard"] else "否", + Safe(f'查看任务合同'), + ] + cells = "".join(f"{x if isinstance(x, Safe) else _e(x)}" for x in values) + search = " ".join( + str(x) + for x in ( + task_id, + task.get("intent"), + task.get("rationale"), + json.dumps(task.get("contract") or {}, ensure_ascii=False), + ) + ) + rows.append( + f'{cells}' + ) + labels = [ + "任务", + "层级", + "站点", + "类型", + "最少步", + "入选理由", + "官方任务意图", + "官方 Hard", + "任务合同", + ] + headers = "".join( + f'' + for i, x in enumerate(labels) ) - tag_values = sorted( - { - str(tag) - for task in (summary.get("by_task") or {}).values() - for tag in task.get("tags") or [] - } + return ( + _filters(dataset=True) + + f'
    {headers}{"".join(rows)}
    ' ) - tag_options = "".join(f'' for tag in tag_values) - diagnostic_categories = sorted( - { - str(group.get("category") or "其他") - for group in ( - ((summary.get("failure_diagnostics") or {}).get("rule_engine") or {}).get("groups") - or [] + + +def _diagnosis_dialog( + *, + result_id: str, + task_id: int, + configuration: str, + failure_point: str, + automatic_failure: str, + evaluator_error: str, + subcategory: str, + reason: str, + owner: str, + action: str, + step_link: str, + step_context_label: str = "", + step_context: str = "", +) -> str: + dialog_id = f"diagnosis-{result_id}" + detail_url = f"results/{result_id}.html" + return ( + f'' + '
    失败诊断' + f'

    Task #{task_id} · {_e(configuration)}

    ' + f'
    ' + '
    ' + f'' + f"
    程序判定{_e(automatic_failure or '未分类')}
    " + f"
    复核根因{_e(subcategory or '待复核')}
    " + f"
    负责范围{_e(owner)}
    " + + ( + f'
    {_e(step_context_label)}{_e(step_context)}
    ' + if step_context + else "" + ) + + '

    程序报错

    ' + f'
    {_e(evaluator_error)}
    ' + '

    复核结论

    ' + f"

    {_e(reason or '尚未完成根因复核。')}

    处理:{_e(action)}

    " + '
    ' + ) + + +def _attempt_table( + attempts: list[dict[str, Any]], + tasks: list[dict[str, Any]], + verified_causes: dict[str, dict[str, Any]] | None = None, +) -> str: + verified_causes = verified_causes or {} + task_meta = {int(task["task_id"]): task for task in tasks} + rows = [] + dialogs = [] + for row in attempts: + p = _public_attempt(row) + result = "pass" if p["score"] == 1 else "fail" + result_record = read_public_json(Path((row.get("paths") or {}).get("evaluation", ""))) + failure, automatic_failure = _automatic_failure_type(result_record) + subcategory, reason = localized_cause(verified_causes, p["id"]) + code = _cause_code(verified_causes, p["id"]) + event_record = read_public_json(Path((row.get("paths") or {}).get("events", ""))) + steps = trajectory(event_record if isinstance(event_record, list) else []) + failure_point, step_number = _failure_point(code, steps) + step_context_label, step_context = _trajectory_context(steps, step_number, failure_point) + owner, action, _external = _remediation(code) + task = task_meta.get(p["task_id"], {}) + point_link = ( + f"results/{p['id']}.html#trajectory-step-{step_number}" + if step_number + else f"results/{p['id']}.html#diagnosis" + ) + dialog_id = f"diagnosis-{p['id']}" + if result == "fail": + dialogs.append( + _diagnosis_dialog( + result_id=p["id"], + task_id=p["task_id"], + configuration=f"{MODEL_LABELS[p['model']]} · {ENGINE_LABELS[p['engine']]}", + failure_point=failure_point, + automatic_failure=automatic_failure, + evaluator_error=_evaluator_error_text(result_record, row), + subcategory=subcategory, + reason=reason, + owner=owner, + action=action, + step_link=point_link, + step_context_label=step_context_label, + step_context=step_context, + ) ) - } + values = [ + Safe(f'{p["id"]}'), + Safe( + f'{_e(MODEL_LABELS[p["model"]])}' + f"{_e(ENGINE_LABELS[p['engine']])}" + ), + Safe(f'#{p["task_id"]}'), + _status(row), + Safe(f'{_e(failure_point)}') if result == "fail" else "—", + automatic_failure or "—", + subcategory or "待复核" if result == "fail" else "—", + Safe(f'{_e(owner)}{_e(action)}') + if result == "fail" + else "—", + Safe( + ( + f'' + ) + if result == "fail" + else f'详情' + ), + ] + cells = "".join(f"{x if isinstance(x, Safe) else _e(x)}" for x in values) + rows.append( + f'{cells}' + ) + headers = "".join( + f"{_e(x)}" + for x in [ + "结果", + "运行配置", + "任务", + "判定", + "失败位置", + "程序自动定位", + "复核根因", + "负责对象与处理", + "证据", + ] + ) + return ( + _filters(attempts=True) + + f'
    {headers}{"".join(rows)}
    ' + + "".join(dialogs) ) - diagnostic_groups = ((summary.get("failure_diagnostics") or {}).get("rule_engine") or {}).get( - "groups" - ) or [] - category_counts: dict[str, list[int]] = {} - for group in diagnostic_groups: - category = str(group.get("category") or "其他") - category_counts.setdefault(category, [0, 0]) - category_counts[category][0] += 1 - category_counts[category][1] += len(group.get("attempts") or []) - category_rows = [ - [name, str(values[0]), str(values[1])] for name, values in category_counts.items() - ] or [["无失败", "0", "0"]] - error_options = '' + "".join( - f'' - for category in diagnostic_categories + + +def _write_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + + +def _release_details(release: dict[str, Any]) -> str: + rows = release.get("changes") or [] + details = ( + "
    查看修正依据、处理与影响" + + _table( + ["项目", "发现的问题", "正式处理", "结果影响与边界"], + rows, + table_id="change-table", + ) + + "
    " + if rows + else "" + ) + return ( + f'
    {_e(release.get("date") or "未记录")}
    ' + f"

    {_e(release.get('title') or '未命名版本')}

    " + f"

    {_e(release.get('summary') or '')}

    {details}" ) - filters = f"""
    - - - - - - - - - - - -
    """ - script = """""" - grid_head = '' - return f"""

    逐任务分析与原始证据

    {logical_executions} 条任务 × Provider 记录,{physical_attempts} 次执行。点击任务查看完整操作、答案、判定和执行轨迹。

    {_table(["规则类别", "去重失败组", "实际失败次数"], category_rows, "程序化失败分布")}{filters}
    {grid_head}{_execution_details(manifest, summary, detail_pages)}
    {script}
    """ - - -def _classify(error: str) -> str: - text = error.lower() - if "timeout" in text and ("visible" in text or "click" in text): - return "交互目标超时/不可见" - if "timeout" in text: - return "超时" - if "exact mismatch" in text: - return "结果不匹配" - if "net::" in text or "navigation" in text: - return "导航/网络" - if "selector" in text or "locator" in text: - return "定位器" - return "其他" - - -def _analysis_page( - manifest: dict[str, Any], summary: dict[str, Any], detail_pages: dict[str, str] -) -> str: - return f"""

    时序机制研究

    研究假设低频 screencast 或其触发的布局快照,可能放大动态 DOM 更新后几何、可见性或命中测试状态的滞后。
    对照设计固定页面、选择器和调用,比较立即显现、双 rAF、定时器、DOM 替换、动画与视口进入;延迟覆盖 16 ms 到 1 s。
    验证指标记录 DOM/rAF 完成时间与 actionability 恢复时间,检验其差值是否随 screencast 条件系统性变化。
    {_execution_explorer(manifest, summary, detail_pages)}""" -def _appendix_page(manifest: dict[str, Any]) -> str: - p = manifest.get("protocol") or {} - provenance = manifest.get("artifact_provenance") or {} - fields = [ - ["task", "ground_truth / source / expected_actions", "唯一结果、固定参考路径与来源"], - [ - "attempt", - "result_status / process_status / combined_status", - "结果主指标与结果+过程辅指标", +def _release_history(releases: list[dict[str, Any]], history_scope: str) -> str: + if not releases: + return "

    暂无发布记录。

    " + latest, *older = releases + body = _section( + "当前版本", + f'
    ' + + _release_details(latest) + + "
    ", + ) + if older: + entries = "".join( + f'
    ' + + _release_details(release) + + "
    " + for release in older + ) + body += _section( + "历史版本", + f'
    展开 {len(older)} 个较早版本{entries}
    ', + ) + body += _section( + "范围与下载", + f'

    {_e(history_scope)}

    ', + ) + return body + + +def _review_history(rounds: list[dict[str, Any]]) -> str: + if not rounds: + return "

    暂无核验记录。

    " + cards = [] + for item in rounds: + fields = "".join( + f"
    {_e(label)}

    {_e(item.get(key) or '—')}

    " + for label, key in ( + ("核验范围", "scope"), + ("发现", "findings"), + ("处理", "corrections"), + ("遗留事项", "remaining"), + ) + ) + cards.append( + f'
    ' + f'
    {_e(item.get("date") or "未记录")}
    ' + f"

    {_e(item.get('title') or '未命名核验')}

    " + f'{_e(item.get("status") or "未记录")}' + f'
    {fields}
    ' + ) + return "".join(cards) + + +def build_report(*, evidence: Path | Iterable[Path], output: Path, selection_path: Path) -> Path: + profile = _read_json(Path("evaluation/browser-use/current-report.json")) + _configure_engine_labels(profile) + _configure_models(profile) + selection = load_pilot(selection_path) + bundles = _load_evidence_bundles(evidence) + source_attempts = [row for bundle in bundles for row in bundle["source_attempts"]] + manifest, config = bundles[0]["manifest"], bundles[0]["config"] + review_attempts = [] + for bundle in bundles: + for item in bundle["review"].get("attempts") or []: + review_attempts.append({**item, "model": bundle["model"]}) + review = { + "coverage": { + "recorded_attempts": sum( + int((bundle["review"].get("coverage") or {}).get("recorded_attempts") or 0) + for bundle in bundles + ), + "reviewed_attempts": sum( + int((bundle["review"].get("coverage") or {}).get("reviewed_attempts") or 0) + for bundle in bundles + ), + }, + "attempts": review_attempts, + "disagreements": [ + item for bundle in bundles for item in bundle["review"].get("disagreements") or [] + ], + "unresolved": [ + item for bundle in bundles for item in bundle["review"].get("unresolved") or [] ], - ["attempt", "validation / trace / error", "同一任务的判定与完整执行证据"], - ["resource", "memory mean/p95/peak/retained/MiB·s", "内存水平、尾部、峰值、保留量和积分"], - ["resource", "CPU / network total/average/p95/peak", "计算与网络时序压力"], - ["summary", "paired intersection / paired ratios", "双方成功任务与逐任务配对比值审计"], - ["diagnostics", "rule_engine", "程序化失败分类、证据与原始执行记录入口"], + "bindings": {bundle["model"]: bundle["review"].get("bindings") or {} for bundle in bundles}, + } + protocol, environment = manifest.get("protocol") or {}, manifest.get("environment") or {} + attempts = [ + row + for bundle in bundles + for row in _logical_results( + bundle["source_attempts"], bundle["manifest"].get("protocol") or {} + ) ] - provenance_rows = [ - ["运行编号", str(manifest.get("run_id") or "未记录")], - ["生成时间", str(manifest.get("created_at") or "未记录")], - ["评测集 SHA-256", str(manifest.get("suite_sha256") or "未记录")], - ["attempts.jsonl SHA-256", str(provenance.get("attempts_sha256") or "未记录")], - ["summary.json SHA-256", str(provenance.get("summary_sha256") or "未记录")], + public_attempts = [_public_attempt(r) for r in attempts] + attempts_sha256 = {bundle["model"]: bundle["attempts_sha256"] for bundle in bundles} + failed_result_ids = {row["id"] for row in public_attempts if float(row.get("score") or 0) < 1} + default_failure_cause_review = load_failure_cause_review() + verified_causes: dict[str, dict[str, Any]] = {} + failure_reviews: dict[str, dict[str, Any]] = {} + for bundle in bundles: + model_failed_ids = { + _attempt_id(row) + for row in attempts + if row["model"] == bundle["model"] and float(row.get("score") or 0) < 1 + } + failure_reviews[bundle["model"]] = extract_failure_cause_review_cohort( + default_failure_cause_review, bundle["model"] + ) + verified_causes.update( + validate_failure_cause_review( + default_failure_cause_review, + model_id=bundle["model"], + attempts_sha256=bundle["attempts_sha256"], + failed_result_ids=model_failed_ids, + require_complete=False, + ) + ) + for source, public in zip(attempts, public_attempts, strict=True): + evaluation = read_public_json(Path((source.get("paths") or {}).get("evaluation", ""))) + public["automatic_failure_type"] = _automatic_failure_type(evaluation)[1] + subcategory, reason = localized_cause(verified_causes, public["id"]) + public["verified_failure_subcategory"] = subcategory + public["verified_failure_reason"] = reason + official = (manifest.get("environment") or {}).get("official_evaluator") or {} + tasks = _resolve_tasks( + selection, expected_data_checksum=str(official.get("data_checksum") or "") or None + ) + task_count = len(tasks) + result_count = len(attempts) + max_agent_steps = str( + config.get("max_agent_steps") or protocol.get("max_agent_steps") or "未记录" + ) + site_access_host = str( + environment.get("site_access_host") or config.get("site_access_host") or "未记录" + ) + preparation_receipt_count = sum( + Path(str((row.get("paths") or {}).get("environment_prepare") or "")).is_file() + for row in source_attempts + ) + preparation_receipts_complete = bool(source_attempts) and preparation_receipt_count == len( + source_attempts + ) + tier_counts = { + tier: sum(task["tier"] == tier for task in tasks) for tier in ("easy", "medium", "hard") + } + summary_by_model, paired_by_model = _primary_summary(attempts), _paired_summary(attempts) + excluded_task_ids = _external_interference_task_ids(verified_causes) + qualified_attempts = [ + row for row in attempts if int(row["task_id"]) not in excluded_task_ids ] - return f"""

    复现步骤

    1. 准备 Docker Engine、Python 3.11+、至少 4 CPU/6 GiB 内存。
    2. 运行 ./examples/browser_use/run.sh;脚本执行完整基准,并依次生成原始记录、汇总数据和分页 HTML。
    3. 如需为失败任务补充一次同任务证据,设置 BROWSER_USE_EVAL_FAILURE_RETRIES=1;该记录不会另计为新任务。
    4. 如只需从已有 attempts.jsonl 重算全部指标和报告,运行 uv run python examples/browser_use/rebuild_report.py。
    5. 检查 attempts.jsonl、manifest.json、summary.json 与分页 HTML;所有比值、分位数和均值均由仓库脚本计算,Codex 不参与统计。

    下载本轮机器可读审计摘要

    运行与原始数据绑定

    {_table(["项目", "值"], provenance_rows, "可复核来源")}

    数据字段

    {_table(["对象", "关键字段", "用途"], fields, "数据结构")}

    复现限制

    • 全部任务来自公网,页面内容、网络状态或站点策略变化可能导致数据集漂移;运行前需复核数据集 SHA 与标注证据。
    • 采样间隔 {p.get("sample_interval_ms", "?")} ms,持续时间短于采样间隔的尖峰可能遗漏。
    • 浏览器进程保持运行且任务间清理状态;分配器保留内存不等于缓存污染。

    原始环境 JSON

    展开
    {_e(json.dumps(manifest.get("environment") or {}, ensure_ascii=False, indent=2, sort_keys=True))}
    """ + qualified_tasks = [ + task for task in tasks if int(task["task_id"]) not in excluded_task_ids + ] + qualified_summary = _primary_summary(qualified_attempts) + model = "、".join(MODEL_LABELS.values()) + adoption_evidence = _read_json(Path("research/browser-use-benchmark-adoption.json")) + parent = output.parent + parent.mkdir(parents=True, exist_ok=True) + marker = output / ".generated-report" + if output.exists() and ( + not marker.is_file() or marker.read_text(encoding="utf-8") != "browser-use-report/2\n" + ): + raise ValueError(f"refusing to replace an unmarked report directory: {output}") + stage = Path(tempfile.mkdtemp(prefix=f".{output.name}-", dir=parent)) + try: + for folder in ( + "assets", + "datasets", + "results", + "engines", + "changes", + "methodology", + "analysis", + "review", + "adoption", + "reproduce", + "accounts", + "data", + "artifacts", + ): + (stage / folder).mkdir() + (stage / "assets/style.css").write_text(REPORT_STYLE, encoding="utf-8") + (stage / "assets/report.js").write_text(REPORT_JS, encoding="utf-8") + (stage / "evidence-viewer.html").write_text(_evidence_viewer_page(), encoding="utf-8") + if not METHODOLOGY_FIGURE.is_file(): + raise FileNotFoundError(f"missing methodology figure: {METHODOLOGY_FIGURE}") + shutil.copy2( + METHODOLOGY_FIGURE, + stage / "assets/webarena-autonomous-evaluation-flow.png", + ) + cohort_commits = { + bundle["model"]: str( + (bundle["manifest"].get("protocol") or {}).get("browser_eval_commit") + or "未记录" + ) + for bundle in bundles + } + automatic_failure_counts = _automatic_failure_counts(attempts) + article = _article_body( + paired_by_model, + attempts, + tasks, + verified_causes, + review, + cohort_commits, + automatic_failure_counts, + ) + home = f'
    {article}
    ' + (stage / "index.html").write_text( + _page(title="首页", active="home", body=home), encoding="utf-8" + ) + release_data = _read_json(Path("research/report-changes.json")) + releases = release_data.get("browser_use_releases") or [] + changes = _hero( + "Report releases", + "报告版本说明", + "每个版本只记录影响读者判断的正式修正;结论、分母、耗时与证据影响直接展示,技术依据按需展开。", + ) + _release_history(releases, str(release_data.get("history_scope") or "")) + _write_json(stage / "report-changes.json", release_data) + (stage / "changes/index.html").write_text( + _page(title="变更记录", active="changes", body=changes, depth=1), encoding="utf-8" + ) + datasets = _hero( + "Dataset", + f"候选 benchmark、选择依据与 {task_count} 个预注册任务", + "先区分最终结果、动作过程和模型裁判,再谈采用率;同一家族不同变体的成绩不能合并。", + ) + datasets += _section( + "公开采用信号 Top 3", + _adoption_top3(adoption_evidence), + ) + datasets += ( + _section( + "Mind2Web 的拆分不是 easy / medium / hard", + f"

    原始 Mind2Web 使用 cross-task、cross-website、cross-domain 衡量泛化;Online‑Mind2Web 没有统一三档;Mind2Web 2 的 short / medium / long 按动作跨度划分。本报告的简单 {tier_counts['easy']}、中等 {tier_counts['medium']}、困难 {tier_counts['hard']} 是预注册的操作长度层级。

    ", + ) + + _section( + "怎样阅读任务 ground truth", + "

    WebArena‑Verified 的 ground truth 不是一条固定点击脚本。每题声明一个结构化最终回答合同,并在需要确认页面副作用时附加一个或多个网络事件合同。所有合同都通过才记 1 分。任务页只解释合同;各浏览器的实际回答、失败规则和 CowAgent 自主轨迹统一放在结果页。

    ", + ) + + _section( + "题集范围", + f"

    本轮在模型执行前固定 {task_count} 个任务,覆盖四个已部署站点以及 RETRIEVE、NAVIGATE、MUTATE 三种最终合同。这里仅描述题目与验收规则;通过率、耗时、轨迹和可完成性观察统一放在结果分析中。

    ", + ) + + _section(f"{task_count} 个任务", _task_table(tasks)) + ) + (stage / "datasets/index.html").write_text( + _page(title="评测集", active="datasets", body=datasets, depth=1), encoding="utf-8" + ) -def _markdown(manifest: dict[str, Any], summary: dict[str, Any]) -> str: - paired_engines = (summary.get("resource_comparison") or {}).get("by_engine") or {} + by_attempt = {(r["model"], r["engine"], int(r["task_id"])): r for r in attempts} + for task in tasks: + task_id, site = int(task["task_id"]), task["sites"][0] + detail = _hero( + f"Task #{task_id}", + task["rationale"], + "官方任务身份、公开意图与程序化验收合同。运行结果位于结果分析,不混入评测集定义。", + ) + detail += _section( + "任务身份", + f'
    站点{_e(SITE_LABELS.get(site, site))}
    本项目层级{_e(TIER_LABELS[task["tier"]])} · {_e(task["estimated_min_steps"])} 步起
    任务类型{_e(task["task_type"])}
    官方 hard subset{"是" if task["official_hard"] else "否"}

    公开任务意图

    {_e(task["intent"])}

    官方来源:WebArena‑Verified {_e(environment.get("webarena_verified_version", "—"))} · Task #{task_id} · 模板 ID {task.get("intent_template_id") or "—"} · revision {task.get("revision") or "—"}

    ', + ) + detail += _section("Ground truth · 官方验收合同", _task_contract_html(task)) + detail += _section( + "查看本题运行结果", + '

    评测集页面保持为冻结输入;实测输出、判分和轨迹在结果页查看。

    ', + ) + (stage / f"datasets/{task_id}.html").write_text( + _page(title=f"任务 {task_id}", active="datasets", body=detail, depth=1), + encoding="utf-8", + ) - def paired_duration(engine: str, stat: str) -> str: - return _stat(((paired_engines.get(engine) or {}).get("duration_ms") or {}), stat) + benchmarks = [ + [ + "WebArena‑Verified", + "自托管真实交互", + "最终 JSON + 必要网络副作用", + "程序化;无 LLM judge", + "部署重,最适合内核 A/B", + ], + ["Mind2Web", "离线网页快照", "每步目标元素与操作", "程序化动作指标", "评过程模仿"], + [ + "Online‑Mind2Web", + "136 个真实网站", + "动作历史与最终完成", + "WebJudge + 人工标签", + "漂移、CAPTCHA、judge 偏差", + ], + [ + "WebVoyager", + "实时网站视觉交互", + "最终回答与执行证据", + "多模态 LLM + 人工核验", + "精确复现较弱", + ], + [ + "Mind2Web 2", + "长链多站搜索", + "带引用研究答案", + "rubric Agent-as-a-Judge", + "非 UI 内核首选", + ], + [ + "BU Bench V1", + "100 个混合任务", + "最终结果与轨迹", + "Google judge LLM", + "云浏览器条件影响可比性", + ], + ["Odysseys", "200 个长链多站任务", "rubric 完成度", "模型评审", "成本高"], + ] + methodology = _hero( + "Methodology", + "评最终结果,不评参考点击路径", + "唯一成绩来源是未修改的 WebArena‑Verified 1.2.3 evaluator;确定性证据审计只验证记录一致性。", + ) + _section( + "候选 benchmark 的验收边界", + _table(["Benchmark", "评什么", "验收对象", "判分", "问题"], benchmarks), + ) + methodology += _section( + "本轮判分链", + f'
    1. 预注册 {task_count} 题
    2. 逐题重置站点
    3. CowAgent 自主规划
    4. 只替换浏览器
    5. 官方 evaluator 验收
    6. 确定性证据复核

    官方数据 812 题全部配置 AgentResponseEvaluator,其中 663 题还配置 NetworkEventEvaluator。所有已配置 evaluator 都返回 1.0 才通过。它检查最终结果和必要副作用,不要求复刻参考动作序列。

    ', + ) + methodology += _section( + f"这 {task_count} 题如何部署并由 Agent 执行", + '
    WebArena-Verified 本地部署、CowAgent 自主操作、双浏览器对照与程序化验收流程
    图 1 · 从官方站点镜像、逐题隔离与健康检查,到 CowAgent 观察—决策—操作闭环,再到 Chromium/Moli 对照和官方程序化验收。窄屏可横向滚动查看。
    ' + f"

    站点确实部署在本机,不是公网代站。macOS 上的 Docker 由 Colima 虚拟机承载;运行配置记录的 {_e(site_access_host)} 是浏览器容器访问 macOS 宿主发布端口所用的 Colima 网关,不是评测网站自身地址。Shopping、Shopping Admin、Reddit、GitLab 四个官方站点镜像通过 Docker 端口发布到这个入口,独立的 Chromium/Moli 浏览器容器据此访问本地站点。

    " + f"

    WebArena‑Verified 当前提供 Shopping、Shopping Admin、Reddit、GitLab、Wikipedia、Map 六类环境。本轮 {task_count} 题只覆盖前四类;Wikipedia 与 Map 需要额外大型数据卷,因此没有部署,也没有任务进入成绩。

    " + + ( + f"

    运行器在每个物理执行前调用 ContainerManager.start(reset=True),只保留该题所需站点;env-ctrl 报告 ready 后,再做连续应用健康检查、清浏览器状态、登录测试账号并打开起始页。本轮 {preparation_receipt_count}/{len(source_attempts)} 个物理执行都保存了环境准备回执,并与对应执行记录绑定。

    " + if preparation_receipts_complete + else "

    运行器在每个物理执行前调用 ContainerManager.start(reset=True),只保留该题所需站点;env-ctrl 报告 ready 后,再做连续应用健康检查、清浏览器状态、登录测试账号并打开起始页。当前证据尚未完整保存逐执行环境准备回执,因此不能把每次准备成功当成已独立核验的事实。

    " + ) + + f"

    预注册文件只含任务 ID、层级和入选理由,不含点击序列。CowAgent 收到自然语言目标后,在最多 {max_agent_steps} 步内循环读取页面观察、请求模型决策并生成下一次 BrowserTool 调用。Chromium 与 Moli 分别连接独立 CDP 浏览器容器;每次物理执行都保留最终 JSON、Agent 事件流、网络 HAR 和 Playwright trace。

    " + + _table( + ["站点", "本轮访问地址", "应用端口", "环境控制端口", "状态"], + [ + ["Shopping", f"http://{site_access_host}:7770", "7770", "7771", "本轮已部署"], + [ + "Shopping Admin", + f"http://{site_access_host}:7780/admin", + "7780", + "7781", + "本轮已部署", + ], + ["Reddit", f"http://{site_access_host}:9999", "9999", "9998", "本轮已部署"], + ["GitLab", f"http://{site_access_host}:8023", "8023", "8024", "本轮已部署"], + ["Wikipedia", "—", "8888", "8889", f"未部署;不在 {task_count} 题"], + ["Map", "—", "3030", "3031", f"未部署;不在 {task_count} 题"], + ], + ) + + ( + f'

    环境准备证据:{preparation_receipt_count}/{len(source_attempts)} 个物理执行均记录 reset、ready、应用健康检查和登录准备状态;事件流随后证明 CowAgent 自主产生 BrowserTool 调用序列。

    ' + if preparation_receipts_complete + else '

    证据边界:事件流能证明 CowAgent 在准备完成后产生了 BrowserTool 调用序列;镜像 ID、digest、代码提交与网关记录限定了本地运行身份。逐执行环境准备回执尚不完整。

    ' + ) + + '', + ) + methodology += _section( + "Mac 上可以继续扩展哪些实例", + f"

    下一轮轨迹重放固定使用本报告当前 {task_count} 题,不扩充任务范围。若未来另行扩充 Browser Use 评测,当前四个已实跑站点可预检 {MAC_COUNTS['current_four_sites_single_site']} 个单站任务;计入只依赖这四站的跨站任务后,共有 {MAC_COUNTS['current_four_sites_including_cross_site']} 个候选,其中 hard 子集 {MAC_COUNTS['current_four_sites_hard_including_cross_site']} 个。

    " + + _table( + ["环境", "Apple Silicon Mac 状态", "完整集单站题", "hard 单站题", "约束"], + MAC_ENVIRONMENT_CANDIDATES, + caption="Wikipedia 与 Map 的容量来自官方环境说明;状态中的“实跑”只指本项目已有执行证据。", + ) + + '

    最小扩展方案:先从四个已验证站点的 678 个候选中做可完成性预检和分层抽样;Wikipedia、Map 等大数据环境后置。这样扩容不会先引入新的部署变量。

    ' + '', + ) + methodology += _section( + "固定协议", + _table( + ["项目", "规则"], + [ + ["Agent / 模型", f"CowAgent;{model};模型推理强度为低"], + ["唯一变量", "同题、同数据、同模型、同提示和资源上限,只替换浏览器"], + ["结果口径", "每题×浏览器一个最终逻辑结果;二元通过率;逐题配对"], + [ + "执行上限", + "代码最多允许两次执行;本报告按冻结运行协议选取正式结果,实际执行耗时和模型用量累计", + ], + [ + "排除", + "不纳入无法稳定验收的实时任务;本轮排除需大型数据卷的 Wikipedia / Map", + ], + ["裁判", "官方程序化 evaluator;无 LLM judge;人工不得改分"], + ], + ), + ) + methodology += _section( + "两类程序化 evaluator 到底检查什么", + _table( + ["Evaluator", "输入", "比较方式", "能证明什么", "不能证明什么"], + [ + [ + "AgentResponseEvaluator", + "Agent 最终 JSON", + "先按 JSON schema 和数据类型归一化,再比较 task_type、status 与 retrieved_data;按题可声明列表有序或无序。", + "最终回答符合该题公开答案合同。", + "没有检查回答是通过哪条 UI 路径得到的,也不评操作效率。", + ], + [ + "NetworkEventEvaluator", + "本次浏览器 HAR", + "在网络事件中匹配 URL、方法、状态码以及按需声明的 query、POST、响应字段或 Cookie 条件。", + "导航或页面变更产生了指定的可观察副作用。", + "没有要求复刻固定点击序列;未声明的页面状态不在验收范围内。", + ], + ], + ) + + "

    合取规则:一题配置的每个 evaluator 都必须得分 1.0,整题才通过。这里没有 LLM-as-a-judge,也没有人工改分。

    ", + ) + (stage / "methodology/index.html").write_text( + _page(title="评测方法", active="methodology", body=methodology, depth=1), + encoding="utf-8", + ) - lines = [ - "# Chromium 与 Moli 浏览器工具调用评测", - "", - f"运行编号:`{manifest.get('run_id', 'unknown')}`", - "", - "这是确定性的 BrowserTool 等价调用微基准;成功判定和失败分类均由程序完成。", - "", - "| Provider | 结果正确 | 结果与参考过程均正确 | P50 ms | 平均值 ms | P95 ms |", - "|---|---:|---:|---:|---:|---:|", - ] - for row in _correctness_rows(summary): - engine = row[0] - lines.append( - "| " - + " | ".join( - row - + [ - paired_duration(engine, "p50"), - paired_duration(engine, "mean"), - paired_duration(engine, "p95"), + analysis = _hero( + "Results", + "结果分析", + "四种配置先在同一张表和同一任务矩阵中比较;失败结果可直接打开诊断,查看发生位置、程序报错、复核根因、轨迹、画面和日志。", + ) + analysis += _section( + f"完整 {len(tasks)} 题结果入口", + '

    四种配置的通过率、耗时、Tokens 与浏览器资源总表已放在首页,避免在分析页重复。这里直接进入同题矩阵和逐结果失败诊断。

    ' + '', + ) + analysis += _section( + f"同题全配置对比:完整 {len(tasks)} 题", + "

    每行是一道预注册任务,四列分别对应模型×浏览器配置;可筛出结果有差异、全部通过或全部失败的任务。

    " + + _same_task_comparison(tasks, attempts), + ).replace('
    ', '
    ', 1) + analysis += _section( + "BrowserTool 动作延迟", + "

    每个动作把全部模型×浏览器配置放在一起比较单次调用墙钟均值,只标记全局唯一最低值。总耗时和调用数受 Agent 行为量影响,仅用于解释工作量,不参与快慢判定。

    " + + _action_duration_cards(attempts), + ) + analysis += _section( + f"{result_count} 个结果与失败诊断", + f"

    这里保留预注册 {task_count} 题的全部官方结果;其中 {len(excluded_task_ids)} 题已标记外部干扰。默认显示失败;点击“查看诊断”可核对发生位置、程序报错、复核根因、轨迹、画面和日志。

    " + + _attempt_table(attempts, tasks, verified_causes), + ) + (stage / "analysis/index.html").write_text( + _page(title="结果分析", active="analysis", body=analysis, depth=1), encoding="utf-8" + ) + + review_by_key = { + (x["model"], x["engine"], int(x["task_id"]), int(x["attempt"])): x + for x in review.get("attempts") or [] + } + public_evidence_results: list[dict[str, Any]] = [] + for row, public in zip(attempts, public_attempts, strict=True): + item = review_by_key.get( + (public["model"], public["engine"], public["task_id"], int(row["attempt"])), + {}, + ) + task = next(t for t in tasks if int(t["task_id"]) == public["task_id"]) + source_executions = row.get("_source_executions") or [row] + execution_records = [] + response_record = evaluation_record = None + steps: list[dict[str, Any]] = [] + for source_execution in source_executions: + is_authoritative = int(source_execution["attempt"]) == int(row["attempt"]) + execution_record, response, evaluation, execution_steps = ( + _write_execution_artifacts( + stage, + public["id"], + source_execution, + authoritative=is_authoritative, + ) + ) + execution_records.append(execution_record) + if is_authoritative: + response_record, evaluation_record, steps = ( + response, + evaluation, + execution_steps, + ) + metadata = evidence_metadata(row) + public_evidence_results.append( + { + "result_id": public["id"], + "model": public["model"], + "engine": public["engine"], + "task_id": public["task_id"], + "score": public["score"], + "execution_count": len(execution_records), + "authoritative_execution": int(row["attempt"]), + "task_page": f"datasets/{public['task_id']}.html", + "result_page": f"results/{public['id']}.html", + "executions": execution_records, + } + ) + usage, resource, checks = ( + public["model_usage"], + public["resource"], + item.get("checks") or {}, + ) + cause_code = _cause_code(verified_causes, public["id"]) + failure_point, failure_step = _failure_point(cause_code, steps) + step_context_label, step_context = _trajectory_context( + steps, failure_step, failure_point + ) + remediation_owner, remediation_action, external_interference = _remediation(cause_code) + detail = _hero( + public["id"], + f"{MODEL_LABELS[public['model']]} · {ENGINE_LABELS[public['engine']]} · Task #{public['task_id']}", + ( + f"失败发生在{failure_point};下方保留程序报错、复核根因和对应执行证据。" + if float(public["score"]) < 1 + else "该结果通过全部官方程序化验收;下方可核对任务合同、执行轨迹和运行证据。" + ), + ) + # Pages declare so every internal URL is rooted at + # the report bundle, including same-page fragment links. + attempt_page = f"results/{public['id']}.html" + detail += ( + '" + ) + runtime_facts = ( + '
    ' + f'
    自主操作{public["action_count"]} 次
    ' + f'
    Agent 链路{_seconds(public["duration_ms"])}
    ' + f'
    BrowserTool{_seconds(public["browser_tool_duration_ms"])}
    ' + f'
    模型 tokens{_number(usage["total_tokens"])}
    ' + f'
    模型请求{usage["request_count"]} 次 / {usage["request_attempt_count"]} 次传输
    ' + ) + if float(public["score"]) < 1: + step_href = ( + f"{attempt_page}#trajectory-step-{failure_step}" + if failure_step + else f"{attempt_page}#trajectory" + ) + detail += '
    ' + _section( + "失败诊断", + '" + + ( + f'

    {_e(step_context_label)}:{_e(step_context)}

    ' + if step_context + else "" + ) + + '
    ' + f'
    程序判定{_e(public["automatic_failure_type"] or "无")}
    ' + f'
    复核根因{_e(public["verified_failure_subcategory"] or "待复核")}
    ' + f'
    负责范围{_e(remediation_owner)}
    ' + f'
    处理{_e(remediation_action)}
    ' + '

    程序报错

    ' + f'
    {_e(_evaluator_error_text(evaluation_record, row))}
    ' + '
    复核结论:' + + _e(public["verified_failure_reason"] or "当前尚无已核验根因。") + + '
    ' + + runtime_facts + + ( + '

    对比口径:该结果已确认含非模型/浏览器干扰,官方分数保留,但暂不纳入干净子集。

    ' + if external_interference + else "" + ), + ) + else: + detail += _section( + "结果", + '
    通过' + f"

    {_e(_failure_brief(evaluation_record, row))}

    " + + runtime_facts, + ) + detail += '
    ' + _section( + "任务要求与验收结果", + _verification_html(task, evaluation_record, response_record) + + '

    所有规则均为官方程序化验收,必须全部得分 1.0。页面先给出可读结论;官方期望、实际观察和断言保留在折叠技术明细及下载文件中。

    ', + ) + detail += '
    ' + _section( + "CowAgent 自主执行轨迹", + "

    以下顺序来自本次执行的 Agent 事件流,不是固定动作回放。展开可查看调用参数和输出摘录;完整脱敏日志可下载。

    " + + _trajectory_html(steps), + ) + if float(public["score"]) < 1: + detail += '
    ' + _section( + "失败执行的最后页面画面", + _screenshot_gallery(public["id"], execution_records), + ) + detail += '
    ' + _section( + "运行、日志与证据", + _execution_evidence_html(public["id"], execution_records) + + '

    公开包包含脱敏的回答、evaluator、事件流与轨迹。原始 HAR 和 Playwright trace 可能携带登录态、Cookie、页面快照或请求正文,因此不公开;其字节数和 SHA-256 保留用于证明证据绑定。

    ' + + _table(["浏览器资源指标", "值"], _resource_detail_rows(resource)) + + '

    整次 Agent 运行的容器计量只作诊断;不同进程模型下不直接比较内存或 PID。

    ' + + "

    证据完整性检查

    " + + _table( + ["检查", "结果"], + _evidence_check_rows(checks), + ) + + "

    判分权威执行的原始证据绑定

    " + + _table( + ["证据", "存在", "字节", "SHA-256", "公开策略"], + [ + [ + label, + "是" if value.get("present") else "否", + value.get("bytes"), + value.get("sha256") or "—", + _publication_label(value.get("publication")), + ] + for label, value in (metadata.get("files") or {}).items() + ], + ) + + f'', + ) + (stage / f"results/{public['id']}.html").write_text( + _page(title=public["id"], active="analysis", body=detail, depth=1), encoding="utf-8" + ) + + review_page = _hero( + "Review history", + "核验与修订记录", + "按轮次记录核验范围、发现、正式处理和遗留事项;单个任务的判定证据位于结果详情页。", + ) + review_page += _section( + "当前状态", + '

    发布门禁:证据文件、哈希、分数、状态、官方版本与报告链接已通过程序化一致性检查。

    ' + "

    官方 WebArena‑Verified evaluator 是成绩来源;核验记录只说明报告和实现经过了哪些检查与修订,不替代任务判分。

    ", + ) + review_page += _section( + "核验轮次", + _review_history(release_data.get("browser_use_reviews") or []), + ) + (stage / "review/index.html").write_text( + _page(title="核验记录", active="review", body=review_page, depth=1), encoding="utf-8" + ) + + participant_versions = environment.get("browser_versions") or {} + participant_images = environment.get("browser_images") or {} + adoption = _hero( + "Participants", + "被测浏览器与固定实验条件", + "本报告比较浏览器。每个模型单独统计,CowAgent、任务、资源上限、网站和 evaluator 均保持一致。", + ) + adoption += _section( + "浏览器与版本", + _table( + ["模型", "浏览器", "运行版本", "镜像 ID", "接入方式", "通过任务"], + [ + [ + MODEL_LABELS[model_id], + Safe(f'{_e(label)}'), + (participant_versions.get(engine) or {}).get("Browser", "—"), + (participant_images.get(engine) or {}).get("id", "—"), + "独立容器 · CDP · CowAgent BrowserTool", + f"{summary_by_model[model_id][engine]['passed']}/{summary_by_model[model_id][engine]['completed']}", + ] + for model_id in MODEL_LABELS + for engine, label in ENGINE_LABELS.items() + ], + ) + + "

    每个模型分别比较各浏览器,不把不同模型的结果合并排名。

    ", + ) + adoption += _section( + "共同实验条件", + _table( + ["栈字段", "本轮固定值", "角色"], + [ + [ + "Agent", + f"CowAgent · commit {protocol.get('cowagent_commit', '—')}", + "自主观察、规划并调用 BrowserTool", + ], + [ + "模型", + ", ".join(MODEL_LABELS.values()) + + f" · reasoning={protocol.get('reasoning_effort', '—')}", + "作为分层变量为各浏览器生成决策", + ], + [ + "任务与站点", + f"同一 {task_count} 题;同一批本地站点镜像", + "提供一致输入与网页状态", + ], + ["动作接口", "CowAgent BrowserTool · CDP", "把 Agent 动作交给浏览器"], + [ + "验收", + "WebArena‑Verified 1.2.3 官方 evaluator", + "对最终 JSON 与必要网络副作用判分", + ], + [ + "资源与上限", + f"相同 CPU、内存、viewport 与最多 {max_agent_steps} 步", + "控制运行预算", + ], + ], + ) + + f"

    这 {task_count} 题由 CowAgent 自主执行。仓库中的 Browser Trajectory Replay(轨迹重放)是另一项固定动作测试,不计入这里的成绩。

    ", + ) + adoption += _section( + "增加新的浏览器", + "

    新增浏览器时,需要注册其容器镜像、版本与 CDP 端点,并在同一冻结任务集和共同条件下形成完整结果。总榜按浏览器逐行扩展;同题矩阵按浏览器增加列;统计比较按浏览器对单独生成,不能把多个浏览器压成一个总胜负。

    ", + ) + (stage / "adoption/index.html").write_text( + _page(title="被测浏览器", active="adoption", body=adoption, depth=1), encoding="utf-8" + ) + + versions, images = ( + environment.get("browser_versions") or {}, + environment.get("browser_images") or {}, + ) + engine_rows = [] + for engine in ENGINE_LABELS: + version, image = versions.get(engine) or {}, images.get(engine) or {} + engine_rows.append( + [ + Safe(f'{ENGINE_LABELS[engine]}'), + version.get("Browser", "—"), + version.get("Protocol-Version", "—"), + image.get("architecture", "—"), + str(image.get("id") or "—"), ] ) - + " |" + body = _hero( + "Engine", ENGINE_LABELS[engine], "本轮浏览器运行身份、完整任务结果与失败证据入口。" + ) + _section( + "运行身份", + _table( + ["字段", "值"], + [ + ["Browser", version.get("Browser", "—")], + ["DevTools Protocol", version.get("Protocol-Version", "—")], + ["架构", image.get("architecture", "—")], + ["镜像 ID", image.get("id", "—")], + [ + "各模型任务成绩", + ";".join( + f"{MODEL_LABELS[model_id]} {summary_by_model[model_id][engine]['passed']}/{summary_by_model[model_id][engine]['completed']}" + for model_id in MODEL_LABELS + ), + ], + [ + "各模型逻辑结果耗时 P50", + ";".join( + f"{MODEL_LABELS[model_id]} {_seconds(summary_by_model[model_id][engine]['p50_duration_ms'])}" + for model_id in MODEL_LABELS + ), + ], + ], + ) + + '

    版本与镜像身份用于限定复现边界,不参与评分。

    ', + ) + body += _section( + "查看该浏览器的全部结果", + "

    完整结果只保留一份。下方入口打开结果分析页,并预先筛选当前浏览器;失败诊断、轨迹、页面画面与日志均从该清单进入。

    " + f'
    打开筛选后的结果清单 →
    ', + ) + (stage / f"engines/{engine}.html").write_text( + _page(title=ENGINE_LABELS[engine], active="adoption", body=body, depth=1), + encoding="utf-8", + ) + + reproduce = _hero( + "Reproduction", + "复现从公开仓库、预注册题集和固定运行身份开始", + "公开包提供脱敏投影;模型回答、Cookie 和完整网络正文不进入 Git 或站点。", ) - lines.extend( - [ - "", - "```bash", - "BROWSER_USE_EVAL_FAILURE_RETRIES=1 ./examples/browser_use/run.sh", - "```", - "", - ] + reproduce += _section( + "浏览器与代码身份", + _table(["浏览器", "版本", "协议", "架构", "镜像 ID"], engine_rows) + + _table( + ["项目", "值"], + [ + ["browser-eval commit", protocol.get("browser_eval_commit", "—")], + ["CowAgent commit", protocol.get("cowagent_commit", "—")], + ["模型", f"{model} · reasoning={protocol.get('reasoning_effort', '—')}"], + ["WebArena‑Verified", environment.get("webarena_verified_version", "—")], + ["运行配置选择集 SHA-256", config.get("selection_sha256", selection.sha256)], + ["当前选择集 SHA-256", selection.sha256], + [ + "evaluator checksum", + (environment.get("official_evaluator") or {}).get( + "evaluator_checksum", "—" + ), + ], + ], + ), + ) + site_images = environment.get("site_images") or {} + reproduce += _section( + "部署环境与网络拓扑", + "

    本轮 Docker daemon 运行在 macOS 上的 Colima 虚拟机中。站点容器把应用端口发布到 Docker 宿主;两个浏览器容器通过其容器内 host.docker.internal 对应的网关地址访问这些端口。运行时记录的地址是 " + + _e(environment.get("site_access_host") or "未记录") + + "。它属于本轮私有网络,不应复制为其他机器的固定地址。

    " + + _table( + ["层", "本轮身份", "作用"], + [ + ["宿主", environment.get("platform", "—"), "运行 Colima 与报告 runner"], + [ + "Docker", + f"{environment.get('docker_name', '—')} · {environment.get('docker_architecture', '—')} · {environment.get('docker_server_version', '—')}", + "承载站点和浏览器容器", + ], + [ + "浏览器到宿主网关", + environment.get("site_access_host", "—"), + "访问站点容器发布端口", + ], + [ + "ARM 兼容模式", + (environment.get("site_runtime_compatibility") or {}).get("mode", "未启用"), + "运行部分 amd64 PHP 站点镜像", + ], + ], + ) + + _table( + ["站点镜像", "镜像引用", "镜像 ID", "Repo digest"], + [ + [ + SITE_LABELS.get(site, site), + image.get("reference", "—"), + image.get("id", "—"), + ", ".join(image.get("repo_digests") or []) or "—", + ] + for site, image in site_images.items() + ], + ) + + ( + f'

    本轮 {preparation_receipt_count}/{len(source_attempts)} 个物理执行均保存环境准备回执;公开环境摘要保留逐项状态,原始登录态不发布。

    ' + if preparation_receipts_complete + else '

    逐执行环境准备回执尚不完整;现有记录只能核对运行器配置与镜像身份。

    ' + ), + ) + reproduce += _section( + "环境隔离与并行边界", + "

    修正后的运行器把一个 Docker daemon 及其固定站点端口视为一个环境 lane。同一 lane 内的任务和浏览器必须串行:每次尝试先重置站点,再完成 Agent、证据收尾和 evaluator,整个周期由跨进程锁保护。缺少 Docker daemon 身份时直接停止,不降级为无锁执行。

    " + + _table( + ["运行方式", "是否允许", "隔离要求"], + [ + [ + "一套站点环境,Chromium 与 Moli 串行", + "允许,默认方案", + "同一 daemon 锁;每次尝试前重置并核对初态", + ], + [ + "一套站点环境,两个浏览器并行", + "不允许", + "账号不同仍会共享对象、索引、缓存和后台任务", + ], + [ + "多个任务并行", + "允许", + "每个 worker 使用独立 daemon、站点容器、网络、端口和可写卷;镜像只读层可复用", + ], + ], + ) + + '

    账号隔离只用于避免用户级对象冲突,不能替代环境 lane。Task 77/771 的旧证据来自修正前的共享状态,正式比较已将整题四种配置一并排除;只有新的完整四配置证据批次才能重新纳入。

    ', + ) + reproduce += _section( + "公开复现入口", + f'

    lexmount/browser-eval 是代码、题集和锁文件的唯一来源。

    ./examples/browser_use/run.sh\nmake browser-use-review EVIDENCE="$BROWSER_USE_EVIDENCE"\nmake browser-use-report EVIDENCE="$BROWSER_USE_EVIDENCE"\nmake browser-use-report-check EVIDENCE="$BROWSER_USE_EVIDENCE"

    每次物理执行的脱敏回答、evaluator、事件和轨迹位于 artifacts/<result-id>/executions/<n>/;evidence-index.json 给出全部 {result_count} 个逻辑结果及其物理执行、下载路径和公开文件哈希。

    ', + ) + (stage / "reproduce/index.html").write_text( + _page(title="环境与复现", active="reproduce", body=reproduce, depth=1), encoding="utf-8" + ) + accounts = _hero( + "Accounts", + "本地站点账号、模型权限与发布权限分开处理", + "不展示密钥、Cookie 或真实个人信息;官方合同中的内置测试邮箱和用户名作为合成 ground truth 保留。", + ) + _section( + "权限与费用", + _table( + ["范围", "本轮做法", "外部付费账号"], + [ + ["四个自托管站点", "官方 Docker 镜像与内置测试账号;每题前重置", "不需要"], + ["模型 API", f"{model},reasoning=low", "需要相应 API 权限"], + ["Artifact Hub", "只发布脱敏静态报告", "发布者需已有站点权限"], + ], + ), + ) + (stage / "accounts/index.html").write_text( + _page(title="账号", active="accounts", body=accounts, depth=1), encoding="utf-8" + ) + + claims = { + "schema": "lex_browser_use_interpretation/1", + "claims": [ + {"claim": "审查合格比较集的模型内浏览器配对结果", "basis": _paired_summary(qualified_attempts)}, + {"claim": "失败根因以当前证据绑定的逐结果核验为准", "basis": verified_causes}, + { + "claim": "官方 evaluator 的失败规则组合", + "basis": { + "denominator": sum(automatic_failure_counts.values()), + "authority": "final logical execution", + "source": "eval_result.json/evaluators_results where score < 1", + "classes": { + "network_only": automatic_failure_counts["network"], + "response_only": automatic_failure_counts["response"], + "both": automatic_failure_counts["mixed"], + }, + }, + }, + { + "claim": "已证实外部干扰按整题排除后的正式比较集", + "basis": { + "excluded_task_ids": sorted( + _external_interference_task_ids(verified_causes) + ), + "summary": _primary_summary( + [ + row + for row in attempts + if int(row["task_id"]) + not in _external_interference_task_ids(verified_causes) + ] + ), + }, + }, + { + "claim": "四配置最终共同成功任务的性能与资源 P50", + "basis": dict( + zip( + ("task_count", "rows"), + _common_final_success_resource_rows(qualified_attempts), + strict=True, + ) + ), + }, + {"claim": "执行证据完整性审查覆盖", "basis": review.get("coverage") or {}}, + ], + "bound_attempts_sha256": attempts_sha256, + "selection_sha256": selection.sha256, + "score_authority": { + bundle["model"]: ( + (bundle["manifest"].get("protocol") or {}).get("official_score_uses") + or "final execution for each browser and task" + ) + for bundle in bundles + }, + } + _write_json(stage / "data/interpretation-evidence.json", claims) + _write_json( + stage / "data/selection.json", + { + "schema": "lex_browser_use_public_selection/1", + "selection_sha256": selection.sha256, + "tasks": tasks, + }, + ) + (stage / "data/results.jsonl").write_text( + "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in public_attempts), + encoding="utf-8", + ) + _write_json( + stage / "data/failure-cause-review.json", + { + "schema": LAYERED_REVIEW_SCHEMA, + "bindings": { + model_id: review_value.get("bindings") or {} + for model_id, review_value in failure_reviews.items() + }, + "coverage": { + "failed_results": len(failed_result_ids), + "verified_results": len(verified_causes), + }, + "unresolved": sorted(failed_result_ids - set(verified_causes)), + "results": verified_causes, + }, + ) + _write_json( + stage / "data/evidence-index.json", + { + "schema": "lex_browser_use_public_evidence_index/4", + "results": public_evidence_results, + }, + ) + audit = { + "schema": "lex_browser_use_report/3", + "result_count": len(attempts), + "score_authority": { + bundle["model"]: ( + (bundle["manifest"].get("protocol") or {}).get("official_score_uses") + or "final execution for each browser and task" + ) + for bundle in bundles + }, + "summary": summary_by_model, + "paired_summary": paired_by_model, + "qualified_comparison": { + "task_count": len(qualified_tasks), + "excluded_task_ids": sorted(excluded_task_ids), + "summary": qualified_summary, + "paired_summary": _paired_summary(qualified_attempts), + }, + "selection_sha256": selection.sha256, + "evidence_bindings": { + "attempts_sha256": attempts_sha256, + "manifest_sha256": { + bundle["model"]: bundle["manifest_sha256"] for bundle in bundles + }, + }, + "failure_cause_review": { + "status": "complete" if set(verified_causes) == failed_result_ids else "incomplete", + "reviewed_failures": len(verified_causes), + "failed_results": len(failed_result_ids), + "bindings": { + model_id: review_value.get("bindings") or {} + for model_id, review_value in failure_reviews.items() + }, + "source_sha256": file_sha256(FAILURE_CAUSE_REVIEW_PATH), + }, + "review": { + "coverage": review.get("coverage"), + "disagreement_count": len(review.get("disagreements") or []), + "unresolved_count": len(review.get("unresolved") or []), + "bindings": review.get("bindings"), + } + if review + else None, + } + _write_json(stage / "data/summary.json", audit) + _write_json(stage / "report-data.json", audit) + public_env = { + "schema": "lex_browser_use_public_environment/2", + "protocol": { + key: protocol.get(key) + for key in ( + "agent", + "browser_eval_commit", + "cowagent_commit", + "reasoning_effort", + "evaluator", + "engine_order", + "sites", + ) + } + | { + "models": list(MODEL_LABELS), + "model_labels": MODEL_LABELS, + "baseline_engine": BASELINE_ENGINE, + "max_task_executions": 2, + "environment_reset": ( + "official site containers reset before every execution; " + f"{preparation_receipt_count}/{len(source_attempts)} execution-scoped " + "preparation receipts verified" + if preparation_receipts_complete + else "runner configured for reset before every execution; independent " + f"preparation receipts available for {preparation_receipt_count}/" + f"{len(source_attempts)} executions" + ), + "score_uses": "frozen run manifest; report does not override measured score", + "cost_and_latency": "sum of all executions used for the logical result", + }, + "cohorts": { + bundle["model"]: { + "protocol": { + key: (bundle["manifest"].get("protocol") or {}).get(key) + for key in ( + "agent", + "browser_eval_commit", + "cowagent_commit", + "model", + "reasoning_effort", + "max_agent_steps", + "evaluator", + ) + }, + "run_id": bundle["manifest"].get("run_id"), + "created_at": bundle["manifest"].get("created_at"), + "attempts_sha256": bundle["attempts_sha256"], + "manifest_sha256": bundle["manifest_sha256"], + } + for bundle in bundles + }, + "environment": { + "platform": environment.get("platform"), + "docker_name": environment.get("docker_name"), + "docker_server_version": environment.get("docker_server_version"), + "docker_architecture": environment.get("docker_architecture"), + "site_access_host": environment.get("site_access_host"), + "site_runtime_compatibility": environment.get("site_runtime_compatibility"), + "webarena_verified_version": environment.get("webarena_verified_version"), + "browser_images": images, + "browser_versions": versions, + "site_images": environment.get("site_images"), + "official_evaluator": environment.get("official_evaluator"), + }, + } + _write_json(stage / "data/environment.json", public_env) + _write_json( + stage / "data/review.json", + { + key: review.get(key) + for key in ( + "schema", + "reviewer", + "reviewed_at", + "bindings", + "protocol", + "coverage", + "disagreements", + "unresolved", + "attempts", + ) + }, + ) + _write_json( + stage / "data/publication-privacy.json", + { + "schema": "lex_browser_use_publication_privacy/1", + "published": [ + "official task intent and evaluator contracts", + "sanitized response and evaluator evidence for every physical execution", + "sanitized CowAgent event stream and BrowserTool trajectory for every physical execution", + "per-execution measurements plus source and published-artifact SHA-256 bindings", + ], + "private_only": [ + "raw HAR because it may contain cookies, headers and request bodies", + "raw Playwright trace because it may contain session state and page snapshots", + "model API credentials and local filesystem paths", + ], + "redactions": [ + "known benchmark credentials", + "authorization, cookie, password, secret, token and API-key fields", + "local service origins replaced with benchmark URL placeholders", + "machine-local filesystem paths", + ], + "retained_test_fixtures": [ + "synthetic usernames and email addresses that are part of official task contracts", + "public benchmark task identifiers and expected values", + ], + }, + ) + write_english_report(stage) + (stage / ".generated-report").write_text("browser-use-report/2\n", encoding="utf-8") + backup: Path | None = None + if output.exists(): + backup = Path(tempfile.mkdtemp(prefix=f".{output.name}-backup-", dir=parent)) + backup.rmdir() + output.rename(backup) + try: + stage.rename(output) + except Exception: + if backup is not None and backup.exists() and not output.exists(): + backup.rename(output) + raise + if backup is not None: + shutil.rmtree(backup) + except Exception: + shutil.rmtree(stage, ignore_errors=True) + raise + return output / "index.html" + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument( + "--evidence", + required=True, + action="append", + type=Path, + help="isolated evidence directory for one model; repeat once per model", ) - return "\n".join(lines) - - -def write_latest_report( - reports_dir: str | Path, *, manifest: dict[str, Any], summary: dict[str, Any] -) -> Path: - root = Path(reports_dir) - root.mkdir(parents=True, exist_ok=True) - audit_name = f"{REPORT_STEM}-data.json" - detail_pages: dict[str, str] = {} - analysis = _analysis_page(manifest, summary, detail_pages) - keep = {f"{REPORT_STEM}.md", audit_name, *(filename for _, _, filename in PAGES), *detail_pages} - for pattern in ( - "browser-use-evaluation-*.md", - "browser-use-evaluation-*.html", - "browser-use-evaluation-*.json", - ): - for old in root.glob(pattern): - if old.name not in keep: - old.unlink() - md_path = root / f"{REPORT_STEM}.md" - md_path.write_text(_markdown(manifest, summary), encoding="utf-8") - audit = { - "schema": "lex_browser_use_report_audit/1", - "run_id": manifest.get("run_id"), - "created_at": manifest.get("created_at"), - "suite_sha256": manifest.get("suite_sha256"), - "artifact_provenance": manifest.get("artifact_provenance") or {}, - "ground_truth_validation": { - key: value - for key, value in (manifest.get("ground_truth_validation") or {}).items() - if key != "human_review" - }, - "functional": summary.get("functional") or {}, - "resource_comparison": summary.get("resource_comparison") or {}, - } - (root / audit_name).write_text( - json.dumps(audit, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" + parser.add_argument("--output", required=True, type=Path) + parser.add_argument( + "--selection", + type=Path, + default=Path("evaluation/browser-use/webarena-verified-pilot.json"), ) - renderers = { - "index": lambda: _summary_page(manifest, summary), - "suite": lambda: _suite_page(manifest), - "method": lambda: _method_page(manifest), - "analysis": lambda: analysis, - "appendix": lambda: _appendix_page(manifest), - } - for slug, _, filename in PAGES: - (root / filename).write_text(_shell(manifest, slug, renderers[slug]()), encoding="utf-8") - for filename, document in detail_pages.items(): - (root / filename).write_text(document, encoding="utf-8") - return md_path + args = parser.parse_args() + print(build_report(evidence=args.evidence, output=args.output, selection_path=args.selection)) + + +if __name__ == "__main__": + main() diff --git a/harness/browser_use/report_i18n.py b/harness/browser_use/report_i18n.py new file mode 100644 index 000000000..da71ef985 --- /dev/null +++ b/harness/browser_use/report_i18n.py @@ -0,0 +1,2287 @@ +"""Build a route-for-route English mirror of the browser-use report.""" + +from __future__ import annotations + +import html +import json +import posixpath +import re +import shutil +from html.parser import HTMLParser +from pathlib import Path, PurePosixPath +from urllib.parse import urlsplit, urlunsplit + +from harness.web_fetch.report_i18n import translate_text as translate_common_text + +NAV = ( + ("index.html", "Browser Use"), + ("datasets/index.html", "Dataset"), + ("methodology/index.html", "Methodology"), + ("analysis/index.html", "Results"), + ("review/index.html", "Review"), + ("adoption/index.html", "Browsers"), + ("reproduce/index.html", "Reproduction"), + ("changes/index.html", "Changes"), + ("accounts/index.html", "Accounts and setup"), +) + + +TEXT = { + "完整执行证据": "Complete execution evidence", + "按原始结构展示本次物理执行的脱敏回答、evaluator、事件流、轨迹、计量与公开画面索引。": "Shows the sanitized response, evaluator record, event stream, trajectory, measurements, and public-frame index for this physical execution in their original structure.", + "执行证据 JSON": "Execution evidence JSON", + "正在读取证据…": "Loading evidence…", + "在页面中查看完整执行证据": "View complete execution evidence in the page", + "打开原始 JSON": "Open raw JSON", + "25 比 25 不是同因平局:五个零分包含四类根因": "25 vs. 25 is not a same-cause tie: five zero scores contain four root-cause classes", + "同一 CowAgent 与同一模型替换浏览器后,30 个配对任务的官方分数没有出现单边胜出,但逐条追查证明相同分数并不代表相同执行结果。根因分别位于官方任务合同、Agent 完成策略、本项目响应协议和证据采集器;Task 506 的 Chromium 与 Moli 甚至不是同一种失败。": "With the same CowAgent and model and only the browser changed, none of the 30 paired tasks produced a one-sided official-score win. Case-by-case investigation shows that equal scores do not imply equal execution outcomes. Root causes lie in the official task contract, Agent completion policy, this project's response protocol, and the evidence collector; Chromium and Moli did not even fail Task 506 in the same way.", + "二、五个零分的根因与修复边界": "2. Root causes and repair boundaries for the five zero scores", + "两个浏览器共享 CowAgent、模型、任务合同与 evaluator,但这不意味着共同零分必然来自共同原因。根因审计把“页面上出现了什么”继续追到产生错误的合同、策略或采集实现,并为每项给出必须通过的复测条件。": "The two browsers share CowAgent, the model, task contracts, and evaluator, but shared zero scores do not necessarily share a cause. The root-cause audit traces visible symptoms to the contract, policy, or collection implementation that produced them and defines a required retest gate for each.", + "根因": "Root cause", + "责任层": "Owning layer", + "证据闭环": "Evidence chain", + "根治措施": "Corrective action", + "复测门槛": "Retest gate", + "表 2|历史官方分数不被人工改写;合同或运行协议修复后必须产生新证据。": "Table 2 | Historical official scores are not manually changed; contract or runtime-protocol fixes require new evidence.", + "可执行结论:": "Actionable conclusion:", + "#68 因官方 expected 与 schema 冲突而退出下一轮干净 A/B;#142 因类别验收边界未定义而退出,直至补齐可观察口径;#168 修正本项目的空结果状态协议;#124 增加候选语义核验和提交前自检;#506 修复 HAR 即时采集、末步工具调用 fail-closed,并在预注册的 40 步上限下复测。旧的 25/30 仍保留为历史官方分数,但不能再解释成五个同因的 Agent 失败。": "Task 68 leaves the next clean A/B because the official expected value conflicts with its schema. Task 142 leaves until its category acceptance boundary is observable. Task 168 requires the project's zero-result status protocol to be fixed. Task 124 requires candidate-semantic verification and a pre-submit check. Task 506 requires immediate HAR body capture, fail-closed handling of a final-turn tool call, and a retest under a preregistered 40-step ceiling. The old 25/30 remains the historical official score, but it can no longer be interpreted as five same-cause Agent failures.", + "根治措施:": "Corrective action:", + "共同零分": "Shared zero scores", + "不等于共同根因": "Do not imply a shared cause", + "根因结论:": "Root-cause conclusion:", + "#68 是官方任务合同冲突,#124 是候选语义核验缺失,#142 是类别验收边界未定义,#168 是本项目响应协议错误;#506 的 Chromium 是 HAR 采集误判,Moli 是执行步数与完成确认问题。": "Task 68 is an official task-contract conflict; Task 124 lacks candidate-semantic verification; Task 142 has no defined category acceptance boundary; Task 168 is a response-protocol defect in this project. For Task 506, Chromium was misjudged because of HAR capture, while Moli ran out of decision steps without confirming completion.", + "任务合同问题": "Task-contract issues", + "2 题": "2 tasks", + "#68 schema 冲突;#142 类别口径缺失": "Task 68 has a schema conflict; Task 142 lacks a category criterion.", + "Agent 协议与策略": "Agent protocol and policy", + "#124 未核验候选;#168 空结果状态错误": "Task 124 did not validate candidates; Task 168 used the wrong zero-result status.", + "不同根因的同分": "Equal score, different causes", + "1 题": "1 task", + "#506:Chromium 采集误判;Moli 未确认完成": "Task 506: Chromium collection false negative; Moli completion was not confirmed.", + "处理原则:": "Treatment:", + "合同不稳定的任务退出下一轮样本;代码与协议缺陷修复后只通过新执行验证,不覆盖历史官方分数。": "Contract-unstable tasks leave the next cohort. Code and protocol fixes are validated only by new executions and never overwrite historical official scores.", + "查看五题根因、证据闭环与复测门槛 →": "View root causes, evidence chains, and retest gates for all five tasks →", + "首页 · Browser Use 评测": "Home · Browser Use evaluation", + "Browser Use 评测": "Browser Use evaluation", + "Chromium 与 Moli:Browser Use 能力实测": "Chromium vs. Moli: a Browser Use capability evaluation", + "评测方法 · Browser Use 评测": "Methodology · Browser Use evaluation", + "这 30 题如何部署并由 Agent 执行": "How the 30 tasks were deployed and run by the agent", + "图 1 · 运行器从官方 WebArena‑Verified 镜像启动本地站点;每个试次重建该题所需站点、等待环境控制器与应用健康、清理浏览器状态并登录测试账号。图是流程说明,真实性证据来自下方运行身份及 70 个试次页的事件、HAR、trace 哈希。": "Figure 1. The runner starts local sites from official WebArena-Verified images. Every attempt rebuilds the sites required by the task, waits for environment-control and application health, clears browser state, and signs in with benchmark accounts. The figure explains the flow; the runtime identities and the events, HAR, and trace hashes on all 70 attempt pages provide the evidence.", + "不是“先人工部署一次,再连接一个现成网页”。": "This was not a one-time manual deployment followed by connections to an existing website.", + "本轮运行器直接调用 WebArena‑Verified 的 ContainerManager.start();每次正式执行和诊断重试前都以 reset=True 启动该题所需的官方站点容器,并停止无关站点。环境控制服务报告 ready 后,还要通过连续应用级健康检查;随后才清 Cookie、登录内置测试账号、打开任务起始页并开始计入 Agent 执行。": "The runner directly calls WebArena-Verified's ContainerManager.start(). Before every primary run and diagnostic retry, it starts the official site containers required by that task with reset=True and stops unrelated sites. After environment control reports ready, the app must pass repeated application-level health checks. Only then does the runner clear cookies, sign in with benchmark accounts, open the task's start page, and begin agent execution.", + "也不是固定动作回放。": "This was not a fixed action replay either.", + "预注册文件只含任务 ID、层级和入选理由,不含点击序列。CowAgent 收到自然语言目标后,在最多 30 步内循环读取页面观察、请求模型决策并生成下一次 BrowserTool 调用。Chromium 与 Moli 的试次分别连接到独立 CDP 浏览器容器;每个试次保存模型最终 JSON、Agent 事件流、网络 HAR 和 Playwright trace。": "The preregistration contains task IDs, tiers, and selection rationales, but no click sequences. Given the natural-language objective, CowAgent repeatedly reads the page observation, requests a model decision, and generates the next BrowserTool call for up to 30 steps. Chromium and Moli attempts connect to separate CDP browser containers. Every attempt retains the final model JSON, agent event stream, network HAR, and Playwright trace.", + "站点来源": "Site source", + "环节": "Stage", + "本轮实际做法": "Implementation in this run", + "可核验证据": "Verifiable evidence", + "四个官方自托管镜像:Shopping、Shopping Admin、Reddit、GitLab": "Four official self-hosted images: Shopping, Shopping Admin, Reddit, and GitLab", + "运行清单记录镜像 ID、digest、架构与 WebArena‑Verified 1.2.3 checksum": "The run manifest records image IDs, digests, architectures, and the WebArena-Verified 1.2.3 checksum", + "逐试次准备": "Per-attempt preparation", + "重建题目所需容器 → env-ctrl ready → 连续健康检查 → 清状态并登录": "Rebuild required containers → env-ctrl ready → repeated health checks → clear state and sign in", + "运行器实现与 manifest 的 environment_reset;站点准备失败不会产生可计分试次": "Runner implementation and manifest environment_reset; failed site preparation cannot produce a scored attempt", + "CowAgent + 模型依据当前页面逐步调用 BrowserTool;无动作脚本": "CowAgent and the model call BrowserTool step by step from the current page; no action script", + "每个试次页公开按时间排序的 BrowserTool 轨迹和脱敏事件下载": "Every attempt page publishes the ordered BrowserTool trajectory and a sanitized event download", + "结果验收": "Outcome evaluation", + "最终 JSON 交给 AgentResponseEvaluator;需要副作用的任务同时检查 HAR": "AgentResponseEvaluator checks the final JSON; tasks requiring side effects also inspect the HAR", + "每个试次页公开规则级 expected / actual;原始 HAR 与 trace 仅公开 SHA-256 和字节数以避免泄露登录态": "Every attempt page publishes rule-level expected/actual values; raw HAR and trace files expose only SHA-256 and byte counts to protect authenticated state", + "证据边界:": "Evidence boundary:", + "事件流能证明 CowAgent 在运行时产生了不同的 BrowserTool 调用序列,站点镜像和运行配置能证明本地环境身份;它们不能公开模型隐藏思维,也不把成功轨迹当成唯一标准路径。": "Event streams establish that CowAgent produced BrowserTool call sequences at runtime, while site images and run configuration establish the local environment identity. They do not expose hidden model reasoning or make a successful trajectory the only valid path.", + "查看 Chromium 自主轨迹示例": "View a Chromium autonomous trajectory", + "查看 Moli 自主轨迹示例": "View a Moli autonomous trajectory", + "核对镜像与运行身份": "Verify image and runtime identities", + "自托管真实交互": "Self-hosted interactive websites", + "最终 JSON + 必要网络副作用": "Final JSON plus required network side effects", + "程序化;无 LLM judge": "Programmatic; no LLM judge", + "部署重,最适合内核 A/B": "Heavy deployment; strongest fit for browser A/B testing", + "离线网页快照": "Offline webpage snapshots", + "每步目标元素与操作": "Target element and action at each step", + "程序化动作指标": "Programmatic action metrics", + "评过程模仿": "Evaluates process imitation", + "136 个真实网站": "136 live websites", + "动作历史与最终完成": "Action history and final completion", + "WebJudge + 人工标签": "WebJudge plus human labels", + "漂移、CAPTCHA、judge 偏差": "Drift, CAPTCHA, and judge bias", + "实时网站视觉交互": "Visual interaction with live websites", + "最终回答与执行证据": "Final answer and execution evidence", + "多模态 LLM + 人工核验": "Multimodal LLM plus human verification", + "精确复现较弱": "Weaker exact reproducibility", + "长链多站搜索": "Long-horizon multi-site search", + "带引用研究答案": "Research answer with citations", + "rubric Agent-as-a-Judge": "Rubric Agent-as-a-Judge", + "非 UI 内核首选": "Not the first choice for UI browser cores", + "100 个混合任务": "100 mixed tasks", + "最终结果与轨迹": "Final outcomes and trajectories", + "Google judge LLM": "Google judge LLM", + "云浏览器条件影响可比性": "Cloud-browser conditions affect comparability", + "200 个长链多站任务": "200 long-horizon multi-site tasks", + "rubric 完成度": "Rubric completion", + "模型评审": "Model grading", + "成本高": "High cost", + "官方数据 812 题全部配置 AgentResponseEvaluator,其中 663 题还配置 NetworkEventEvaluator。所有已配置 evaluator 都返回 1.0 才通过。它检查最终结果和必要副作用,不要求复刻参考动作序列。": "All 812 official tasks configure AgentResponseEvaluator, and 663 also configure NetworkEventEvaluator. A task passes only when every configured evaluator returns 1.0. The evaluators inspect final outcomes and required side effects, not reproduction of a reference action sequence.", + "Agent / 模型": "Agent / model", + "CowAgent;gpt-5.6-sol;reasoning=low": "CowAgent; gpt-5.6-sol; reasoning=low", + "同题、同数据、同模型、同提示和资源上限,只替换浏览器": "Same task, data, model, prompt, and resource limits; only the browser changes", + "每题×浏览器第一次正式执行;二元 pass@1;逐题配对": "First formal run for each task-browser pair; binary pass@1; task-level pairing", + "首轮失败后最多一次,只做恢复诊断;模型传输失败同请求最多重发一次": "At most one retry after primary failure, used only for recovery diagnosis; at most one resend for a model transport failure", + "不纳入无法稳定验收的实时任务;首轮排除需大型数据卷的 Wikipedia / Map": "Exclude live tasks without stable acceptance; the pilot also excludes Wikipedia and Map because of their large data volumes", + "官方程序化 evaluator;无 LLM judge;人工不得改分": "Official programmatic evaluators; no LLM judge; no manual score changes", + "Agent 最终 JSON": "Agent final JSON", + "先按 JSON schema 和数据类型归一化,再比较 task_type、status 与 retrieved_data;按题可声明列表有序或无序。": "Normalize by JSON schema and data type, then compare task_type, status, and retrieved_data; each task may declare ordered or unordered lists.", + "最终回答符合该题公开答案合同。": "The final answer satisfies the task's public answer contract.", + "没有检查回答是通过哪条 UI 路径得到的,也不评操作效率。": "It does not inspect which UI path produced the answer or evaluate action efficiency.", + "本次浏览器 HAR": "HAR from this browser attempt", + "在网络事件中匹配 URL、方法、状态码以及按需声明的 query、POST、响应字段或 Cookie 条件。": "Match URL, method, status code, and, when declared, query, POST, response-field, or cookie conditions in network events.", + "导航或页面变更产生了指定的可观察副作用。": "Navigation or state changes produced the specified observable side effect.", + "没有要求复刻固定点击序列;未声明的页面状态不在验收范围内。": "It does not require a fixed click sequence; undeclared page state is outside the acceptance scope.", + "一题配置的每个 evaluator 都必须得分 1.0,整题才通过。这里没有 LLM-as-a-judge,也没有人工改分。": "Every evaluator configured for a task must score 1.0 for the task to pass. There is no LLM-as-a-judge and no manual score override.", + "本轮没有观察到浏览器成功率差异;这证明对照流程可比,不证明两个内核等价。": "This run found no browser success-rate difference. It establishes a comparable controlled run, not browser equivalence.", + "归因边界:": "Attribution boundary:", + "任务族覆盖": "Task-family coverage", + "样本同时包含信息提取、页面导航和状态变更任务;类别来自每题官方最终答案合同。": "The sample covers retrieval, navigation, and state-changing tasks. Each category comes from the task's official outcome contract.", + "查看这类任务 →": "View this task family →", + "运行与证据是否完整": "Run and evidence completeness", + "正式调用:": "Formal executions:", + "证据完整性审计:": "Evidence integrity audit:", + "可完成性覆盖:": "Measured-solvability coverage:", + "任务数": "Tasks", + "打开逐题横向对照 →": "Open the task-by-task comparison →", + "记录中的验收差异": "Observed acceptance mismatch", + "已观察到的验收差异:": "Observed acceptance mismatch:", + "证据完整性审计": "Evidence integrity audit", + "已审计": "Audited", + "记录不一致": "Record inconsistencies", + "审计结论": "Audit result", + "先看结论": "Headline result", + "当前结论:": "Current conclusion:", + "本轮没有观察到浏览器成功率差异;这证明对照流程可比,不证明两个内核等价。五个共同失败更值得优先检查 Agent 的跨页聚合与最终状态验证。": "This run found no browser success-rate difference. It establishes a comparable controlled run, not browser equivalence. The five shared failures make cross-page aggregation and final-state verification the first agent-level issues to investigate.", + "单边胜出": "One-browser-only wins", + "个完整配对": "complete pairs", + "失败重试恢复": "Failure retry recoveries", + "不回写主分": "Excluded from pass@1", + "调用与核验状态": "Execution and review status", + "调用完整:": "Execution complete:", + "次计划首轮均有结果;失败项的唯一诊断重试也全部完成。": "planned primary runs have results; every single diagnostic retry after a failure is also complete.", + "独立核验:": "Independent review:", + "个试次;未决": "attempts; unresolved", + "。查看核验轮次 →": ". View review history →", + "三类任务概览": "Three task families", + "只在相同任务族与同一批任务内比较;RETRIEVE、NAVIGATE 与 MUTATE 的合同不同。": "Compare browsers within the same task family and task cohort. RETRIEVE, NAVIGATE, and MUTATE use different acceptance contracts.", + "个任务": "tasks", + "两种浏览器": "Both browsers", + "本类首轮": "primary runs in this family", + "浏览器总览": "Browser overview", + "首轮完成": "Primary coverage", + "首轮通过": "Primary passes", + "失败重试": "Failure retries", + "审查结果": "Review result", + "已核验": "reviewed", + "从结论进入证据": "From result to evidence", + "对零分歧、共同失败和下一轮设计的完整论证。": "A complete interpretation of zero disagreements, shared failures, and the next evaluation design.", + "阅读完整解读 →": "Read the interpretation →", + "筛选并下钻两种浏览器试次。": "Filter tasks and inspect both browsers' attempts.", + "打开任务库 →": "Open the task library →", + "首轮、唯一重试和核验状态。": "Primary runs, the single retry, and review status.", + "打开结果库 →": "Open all results →", + "逐题配对比总分更重要": "Paired tasks are more informative than the aggregate score", + "同一任务的两个首轮结果;重试不参与配对主分。": "Both primary results for each task; retries do not enter the paired primary score.", + "配对结果": "Pair outcome", + "两者都通过": "Both pass", + "仅 Chromium": "Chromium only", + "仅 Moli": "Moli only", + "两者都失败": "Both fail", + "本轮未观察到": "Not observed", + "优先排查共同链路": "Inspect the shared stack first", + "精确 McNemar 双侧 p=1.000。零个不一致配对意味着当前样本无法估计内核方向性效应。": "Exact two-sided McNemar p=1.000. With no discordant pairs, this sample cannot estimate a directional browser effect.", + "结论只适用于本轮预注册样本、Agent、模型与运行配置;展示调整不改变测量结果。": "Conclusions apply only to this preregistered sample, agent, model, and runtime configuration. Presentation changes do not alter measured results.", + "候选 benchmark、选择依据与 30 个预注册任务": "Benchmark candidates, selection rationale, and 30 preregistered tasks", + "先区分最终结果、动作过程和模型裁判,再谈采用率;同一家族不同变体的成绩不能合并。": "First separate final-outcome evaluation, action prediction, and model judging. Scores from different variants of one benchmark family are not interchangeable.", + "公开采用信号 Top 3": "Top three public adoption signals", + "采用次数是公开一手材料中的机构数,不是质量分;未找到 DeepSeek、Kimi 公开的一手 UI browser-use benchmark,不等于内部没有评测。": "Counts are institutions found in primary public sources, not quality scores. No primary public UI browser-use benchmark was found for DeepSeek or Kimi; that does not show they have no internal evaluation.", + "Mind2Web 的拆分不是 easy / medium / hard": "Mind2Web splits are not easy, medium, and hard", + "原始 Mind2Web 使用 cross-task、cross-website、cross-domain 衡量泛化;Online‑Mind2Web 没有统一三档;Mind2Web 2 的 short / medium / long 按动作跨度划分。本报告的简单 5、中等 10、困难 15 是预注册的操作长度层级。": "Original Mind2Web uses cross-task, cross-website, and cross-domain splits. Online-Mind2Web has no common three-level difficulty split. Mind2Web 2 uses short, medium, and long action spans. This report's 5/10/15 easy, medium, and hard groups were preregistered by estimated visible operation length.", + "怎样阅读任务 ground truth": "How to read task ground truth", + "参考状态与可完成性证据": "Reference status and solvability evidence", + "“有成功路径证据”不等于存在唯一标准路径;WebArena‑Verified 不要求复刻人工动作序列。": "Evidence of a successful path does not imply one canonical path. WebArena-Verified does not require replaying a human action sequence.", + "30 个任务": "30 tasks", + "全文搜索": "Full-text search", + "首页": "Home", + "账号": "Accounts", + "任务、意图、答案、错误": "Task, intent, answer, or error", + "任务、意图、验收合同": "Task, intent, or acceptance contract", + "层级": "Tier", + "站点": "Site", + "类型": "Type", + "可完成性": "Solvability", + "本轮已有成功轨迹": "Successful trajectory observed", + "仅官方合同": "Official contract only", + "任务": "Task", + "最少步": "Minimum steps", + "入选理由": "Selection rationale", + "官方任务意图": "Official task intent", + "可完成性证据": "Solvability evidence", + "简单": "Easy", + "中等": "Medium", + "困难": "Hard", + "通过": "Pass", + "失败": "Fail", + "首轮": "Primary", + "重试": "Retry", + "失败诊断重试": "Diagnostic retry", + "无": "None", + "官方任务合同、两种浏览器的实际结果与完整自主执行证据。": "Official task contract, both browsers' measured outcomes, and complete autonomous execution evidence.", + "任务身份": "Task identity", + "本项目层级": "Project tier", + "任务类型": "Task type", + "官方 hard subset": "Official hard subset", + "本轮可完成性证据": "Solvability evidence in this run", + "已有成功轨迹": "Successful trajectory observed", + "尚无成功轨迹": "No successful trajectory observed", + "公开任务意图": "Public task intent", + "官方来源:": "Official source:", + "Ground truth · 官方验收合同": "Ground truth · official acceptance contract", + "没有唯一参考点击路径。": "There is no single reference click path.", + "WebArena‑Verified 允许 Agent 自主选择任意 UI 路径;ground truth 是以下最终回答与必要网络副作用的合取。报告因此比较结果合同,而不把某条人工路径当作标准答案。": "WebArena-Verified lets the agent choose any UI path. Ground truth is the conjunction of the final-answer and required network-side-effect contracts below. The report therefore compares outcome contracts rather than treating a human path as the answer key.", + "程序化验收": "Programmatic acceptance", + "该规则必须得分 1.0。": "This rule must score 1.0.", + "匹配选项与结果 schema": "Matching options and result schema", + "参考状态 · 当前环境是否有成功路径证据": "Reference status · evidence of a successful path in the current environment", + "有。": "Yes.", + "本轮至少一个自主试次通过全部官方合同,因此当前站点快照存在已实测的可完成路径。成功轨迹不是唯一标准路径,也不要求另一浏览器复刻。": "At least one autonomous attempt passed every official contract, so this site snapshot has a measured successful path. That trajectory is not the only valid path, and the other browser need not reproduce it.", + "本轮没有。": "Not in this run.", + "官方任务包给出了确定性验收合同,但两种浏览器的首轮和唯一诊断重试都未通过。因此本报告不能独立证明当前快照存在可完成路径,也不能把失败直接归因给浏览器;需要另做不参与计分的人工或 oracle 可完成性核验。": "The official task bundle provides a deterministic acceptance contract, but both browsers failed the primary run and the single diagnostic retry. This report therefore cannot independently prove that the current snapshot is solvable or attribute the failure directly to the browser. A separate non-scoring human or oracle readiness check is required.", + "两种浏览器的全部正式试次": "All formal attempts from both browsers", + "口径": "Attempt class", + "官方结果": "Official verdict", + "端到端耗时": "End-to-end latency", + "证据": "Evidence", + "查看试次": "View attempt", + "共同失败诊断:": "Shared-failure diagnosis:", + "全部试次结果对照": "All-attempt comparison", + "实际最终回答": "Actual final answer", + "失败规则": "Failed rules", + "为什么这样判": "Verdict rationale", + "执行过程": "Execution", + "评最终结果,不评参考点击路径": "Evaluate final outcomes, not a reference click path", + "唯一成绩来源是未修改的 WebArena‑Verified 1.2.3 evaluator;确定性证据审计只验证记录一致性。": "The unmodified WebArena-Verified 1.2.3 evaluator is the only scoring authority. Deterministic evidence review checks record integrity only.", + "候选 benchmark 的验收边界": "Acceptance boundaries of candidate benchmarks", + "评什么": "What it evaluates", + "验收对象": "Acceptance target", + "判分": "Scoring", + "问题": "Limit", + "本轮判分链": "Scoring chain for this run", + "预注册 30 题": "Preregister 30 tasks", + "逐题重置站点": "Reset sites per task", + "CowAgent 自主规划": "CowAgent plans autonomously", + "只替换浏览器": "Change only the browser", + "官方 evaluator 验收": "Run official evaluators", + "确定性证据复核": "Review deterministic evidence", + "固定协议": "Fixed protocol", + "项目": "Item", + "规则": "Rule", + "唯一变量": "Only changed variable", + "主分": "Primary metric", + "排除": "Exclusions", + "裁判": "Judge", + "两类程序化 evaluator 到底检查什么": "What the two programmatic evaluators check", + "输入": "Input", + "比较方式": "Comparison", + "能证明什么": "What it establishes", + "不能证明什么": "What it does not establish", + "合取规则:": "Conjunction rule:", + "从总体配对下钻到每一次执行": "Drill down from paired outcomes to every execution", + "先定位十个首轮失败的规则与直接原因,再筛选全部 70 次执行并下钻回答、轨迹和日志。": "First locate the exact failed rules for ten primary failures, then filter all 70 executions and inspect answers, trajectories, and logs.", + "汇总": "Summary", + "浏览器": "Browser", + "耗时中位数": "Median latency", + "耗时 P95": "P95 latency", + "Tokens 中位数": "Median tokens", + "耗时包含模型响应、页面等待、站点重置与浏览器执行,不能解释为纯内核速度。": "Latency includes model responses, page waits, site resets, and browser execution. It is not isolated browser-core speed.", + "同题首轮关系对比": "Paired primary-outcome comparison", + "每行固定为同一任务的 Chromium 与 Moli 首轮。关系筛选只改变可见行,不产生新分数。": "Each row fixes Chromium and Moli primary runs for the same task. Relationship filters change visible rows only and do not create new scores.", + "关系": "Relationship", + "共同通过": "Both pass", + "共同失败": "Both fail", + "有分歧": "Disagreement", + "找到": "Showing", + "个配对任务": "paired tasks", + "对照结论": "Pair outcome", + "查看两种浏览器全部试次": "View all attempts for both browsers", + "首轮失败定位": "Primary-failure inventory", + "两种浏览器在同五题失败。下表不是人工猜测:失败环节来自官方 evaluator,补充说明来自 Agent 最终错误和执行记录。": "Both browsers failed the same five tasks. The failure stage comes from the official evaluators; the additional diagnosis comes from the agent's final error and execution record.", + "首轮试次": "Primary attempt", + "直接原因": "Recorded mismatch", + "同题对照": "Same-task comparison", + "任务与两浏览器对照": "Task and both browsers", + "70 个试次": "70 attempts", + "默认只显示失败;可按浏览器、层级、站点、任务类型、首轮/重试、失败环节和全文关键词筛选。": "Failures are shown by default. Filter by browser, tier, site, task type, primary/retry, failure stage, or full-text query.", + "失败环节": "Failure stage", + "最终回答": "Final answer", + "网络副作用": "Network side effect", + "多项": "Multiple stages", + "试次": "Attempt", + "传输重试": "Transport retries", + "从官方 ground truth、实际回答、规则级判定下钻到 CowAgent 的每一步 BrowserTool 调用。": "Follow the official ground truth, actual answer, and rule-level verdict down to every CowAgent BrowserTool call.", + "答案对照": "Answer comparison", + "规则判定": "Rule verdicts", + "执行轨迹": "Trajectory", + "运行与证据": "Runtime and evidence", + "结果与运行": "Outcome and runtime", + "失败定位": "Failure diagnosis", + "自主操作": "Autonomous actions", + "模型 tokens": "Model tokens", + "模型请求": "Model requests", + "传输错误": "Transport errors", + "证据复核": "Evidence review", + "Ground truth 与实际最终回答": "Ground truth and actual final answer", + "官方任务合同": "Official task contract", + "Agent 实际回答": "Agent's actual answer", + "官方 evaluator 逐规则判定": "Official evaluator verdicts by rule", + "所有规则都必须得分 1.0;这里使用 evaluator 已归一化的实际值与期望值直接对照。": "Every rule must score 1.0. This section directly compares the evaluator's normalized actual value with the expected value.", + "官方期望": "Official expectation", + "实际归一化结果": "Normalized actual result", + "断言与错误": "Assertions and errors", + "CowAgent 自主执行轨迹": "CowAgent autonomous trajectory", + "以下顺序来自本次执行的 Agent 事件流,不是固定动作回放。展开可查看调用参数和输出摘录;完整脱敏日志可下载。": "This sequence comes from the run's agent event stream, not a fixed action replay. Expand a step for arguments and an output excerpt; the complete sanitized log is downloadable.", + "调用参数": "Call arguments", + "执行输出节选": "Execution output excerpt", + "运行、日志与证据": "Runtime, logs, and evidence", + "最终回答 JSON": "Final answer JSON", + "Evaluator 明细 JSON": "Evaluator details JSON", + "动作轨迹 JSON": "Trajectory JSON", + "脱敏事件日志 JSON": "Sanitized event log JSON", + "原始证据哈希": "Raw-evidence hashes", + "指标": "Metric", + "值": "Value", + "证据完整性检查": "Evidence completeness checks", + "检查": "Check", + "原始证据文件绑定": "Raw evidence file bindings", + "存在": "Present", + "字节": "Bytes", + "公开策略": "Publication policy", + "返回任务页": "Return to task", + "个试次均完成确定性证据复核": "attempts received deterministic evidence review", + "复核不重新判题,也不是 LLM-as-a-judge;官方 evaluator 仍是唯一成绩来源。": "Review does not rescore tasks and is not LLM-as-a-judge. The official evaluator remains the only scoring authority.", + "核验结论": "Review result", + "记录试次": "Recorded attempts", + "已复核": "Reviewed", + "记分分歧": "Score disagreements", + "未决": "Unresolved", + "复核协议": "Review protocol", + "逐试次覆盖": "Per-attempt coverage", + "结论": "Outcome", + "失败检查数": "Failed check count", + "失败检查项": "Failed checks", + "两条轨道回答不同问题,不能混分": "Two tracks answer different questions and cannot share a score", + "固定 BrowserTool 动作轨道与自主 Agent 轨道都有效,但控制变量和结论边界不同。": "A fixed BrowserTool action track and an autonomous-agent track are both useful, but they control different variables and support different conclusions.", + "仓库中的两个 harness": "The repository's two harnesses", + "轨道": "Track", + "Agent 是否自主": "Agent autonomy", + "回答的问题": "Question answered", + "Browser Use 官方实践": "Browser Use's published evaluation practice", + "公开依据": "Public sources", + "本轮浏览器运行身份、完整任务结果与失败证据入口。": "Runtime identity, complete task results, and failure evidence for this browser.", + "运行身份": "Runtime identity", + "架构": "Architecture", + "镜像 ID": "Image ID", + "首轮成绩": "Primary score", + "首轮耗时中位数": "Median primary latency", + "全部执行记录": "All execution records", + "首轮决定主分;仅失败任务有一次诊断重试,重试不回写主分。": "The primary run determines pass@1. Only failed tasks receive one diagnostic retry, and retries never replace the primary score.", + "复现从公开仓库、预注册题集和固定运行身份开始": "Reproduction starts with the public repository, preregistered cohort, and fixed runtime identities", + "公开包提供脱敏投影;模型回答、Cookie 和完整网络正文不进入 Git 或站点。": "The public bundle provides sanitized projections. Model credentials, cookies, and complete network bodies do not enter Git or the site.", + "浏览器与代码身份": "Browser and code identities", + "公开复现入口": "Public reproduction entry points", + "是代码、题集和锁文件的唯一来源。": "is the canonical source for code, task definitions, and lockfiles.", + "本地站点账号、模型权限与发布权限分开处理": "Local site accounts, model access, and publication access are separate", + "这里只说明权限边界,不展示密钥、Cookie、用户名或个人信息。": "This page describes access boundaries without exposing keys, cookies, usernames, or personal information.", + "权限与安装": "Access and setup", + "范围": "Scope", + "本轮做法": "Configuration in this run", + "取得与安装": "Acquisition and setup", + "外部付费账号": "Paid external account", + "当前发布": "Current release", + "变更": "Change", + "原因": "Reason", + "处理": "Resolution", + "影响": "Impact", + "报告解读": "Interpretation", + "评测集": "Dataset", + "评测方法": "Methodology", + "结果分析": "Results", + "核验记录": "Review", + "方法介绍": "Methods", + "优": "Best", + "更优": "Better", + "单次均值更低": "Lower mean per call", + "通过数更高": "Higher pass count", + "通过率更高": "Higher pass rate", + "同组通过数更高": "Higher pass count in group", + "结果详情导航": "Result detail navigation", + "环境与复现": "Reproduction", + "变更记录": "Changes", + "账号与安装": "Accounts and setup", + "切换主题": "Toggle theme", + "中文": "中文", +} + +_FAILURE_ANALYSIS = json.loads( + (Path(__file__).parents[2] / "research/browser-use-failure-analysis.json").read_text( + encoding="utf-8" + ) +) +for _diagnosis in _FAILURE_ANALYSIS["tasks"].values(): + for _field in ("root_cause", "owner", "proof", "corrective_action", "verification"): + _localized = _diagnosis.get(_field) + if isinstance(_localized, dict) and _localized.get("zh") and _localized.get("en"): + TEXT[str(_localized["zh"])] = str(_localized["en"]) + +_FAILURE_CAUSE_REVIEW = json.loads( + (Path(__file__).parents[2] / "research/browser-use-failure-cause-review.json").read_text( + encoding="utf-8" + ) +) +_failure_cause_results = _FAILURE_CAUSE_REVIEW.get("results") or {} +for _cohort in (_FAILURE_CAUSE_REVIEW.get("cohorts") or {}).values(): + _failure_cause_results.update(_cohort.get("results") or {}) +for _cause in _failure_cause_results.values(): + for _field in ("subcategory", "reason"): + _localized = _cause.get(_field) + if isinstance(_localized, dict) and _localized.get("zh") and _localized.get("en"): + TEXT[str(_localized["zh"])] = str(_localized["en"]) + +# Browser Use-specific prose is intentionally translated here rather than in +# the shared Web Fetch catalog. Task intent and measured evidence are not +# listed: unmatched task text stays verbatim, and Localizer preserves pre/code +# blocks containing evaluator JSON and agent output. +TEXT.update( + { + "90 题 Browser Use 完整结果:Chromium 与 Moli 的成功率、代价与失败根因": "Complete 90-task Browser Use results: Chromium vs. Moli success, cost, and root causes", + "90 题核心指标": "Core metrics for all 90 tasks", + "每个模型×浏览器配置都保留完整 90 题。以上是官方 evaluator 决定的最终逻辑结果;物理执行记录可在单题证据页下钻。": "Every model-browser configuration retains all 90 tasks. These are the final logical outcomes from the official evaluator; physical execution records remain available on each task's evidence page.", + "首页按预注册的 90 题展示四种配置的官方 evaluator 结果。其中 14 题经复核存在验收合同、站点、共享状态或参考状态干扰;这些题仍保留在 90 题官方结果中,另外提供 76 题事后诊断子集。": "The homepage reports official evaluator outcomes for all 90 preregistered tasks across four configurations. Review found acceptance-contract, site, shared-state, or reference-state interference in 14 tasks. They remain in the official 90-task result, while a separate 76-task post-hoc diagnostic subset is also provided.", + "14 题存在外部干扰": "Fourteen tasks contain external interference", + "官方 90 题结果不改分、不缩减分母;另用 76 题事后诊断子集观察排除已知干扰后的差异。": "The official 90-task result is neither rescored nor given a smaller denominator. A separate 76-task post-hoc subset shows differences after removing known interference.", + "90 题官方结果与事后诊断分开看": "Separate the official 90-task result from post-hoc diagnosis", + "首页主结果始终使用完整 90 题官方 evaluator 分母。为了观察排除已知外部干扰后的差异,下表再按整题、四种配置一致排除 14 题,得到 76 题事后诊断子集。": "The homepage always uses all 90 tasks as the denominator for the official evaluator result. To inspect differences after known external interference, the table below removes the same 14 tasks across all four configurations and shows a 76-task post-hoc diagnostic subset.", + "这是事后诊断子集,不是正式分数,也不替代 90 题官方结果。75 个原始失败记录继续保存在结果页和核验记录中。": "This is a post-hoc diagnostic subset, not an official score and not a replacement for the official 90-task result. All 75 original failure records remain available in the result and review pages.", + "完整 90 题的速度与浏览器资源": "Latency and browser resources across all 90 tasks", + "完整 90 题官方结果是本报告的主口径;76 题事后诊断子集只用于分离已知外部干扰,不改写成绩。": "The complete official 90-task result is the report's primary view. The 76-task post-hoc diagnostic subset only separates known external interference and does not rewrite scores.", + "Moli 的内存占用明显更低,但 HAR 完整性、工具耗时和部分任务兼容性还需修复;Chromium 在本轮 Sol 层多通过 3 题,Astra 层通过数相同,但当前每种配置只有一个正式样本,不能外推成总体优势。": "Moli uses substantially less memory, while HAR completeness, tool latency, and some task compatibility still require fixes. Chromium passes three more tasks in the Sol cohort and ties Moli in the Astra cohort, but one formal sample per configuration cannot establish a general advantage.", + "先修复可控的合同、evaluator、站点和共享状态问题,再按同一协议完整重跑 90 题四种配置;历史官方分数在新证据批次出现前保持不变。": "Fix controllable contract, evaluator, site, and shared-state defects first, then rerun all four configurations across the same 90-task protocol. Historical official scores remain unchanged until a new evidence cohort exists.", + "四种配置总览:完整 90 题官方结果": "Four-configuration overview: complete official results for 90 tasks", + "同题全配置对比:完整 90 题": "Same-task comparison across all configurations: complete 90-task set", + "每行是一道预注册任务,四列分别对应模型×浏览器配置;可筛出结果有差异、全部通过或全部失败的任务。": "Each row is one preregistered task, with one column per model-browser configuration. Filters isolate divergent, all-pass, or all-fail outcomes.", + "这里保留预注册 90 题的全部官方结果;其中 14 题已标记外部干扰。默认显示失败;点击“查看诊断”可核对发生位置、程序报错、复核根因、轨迹、画面和日志。": "This inventory retains official results for all 90 preregistered tasks, with 14 tasks marked for external interference. Failures are shown by default; open a diagnosis to inspect the failure point, program error, reviewed cause, trajectory, frames, and logs.", + # Current home page and dataset-only inventory. + "Chromium 任务通过": "Chromium tasks passed", + "Moli 任务通过": "Moli tasks passed", + "单边通过": "One-browser pass", + "共同失败": "Both fail", + "结果边界:": "Result boundary:", + "失败根因是报告解读的核心证据": "Root causes are core interpretation evidence", + "官方 evaluator 的错误信息只定位未满足的验收规则;根因核验继续检查任务合同、最终回答、网络记录和自主轨迹。未完成核验前,报告保留自动分类,但不展示推测性根因。": "The official evaluator identifies the unmet acceptance rule. Root-cause review then checks the task contract, final answer, network record, and autonomous trajectory. Until that review is complete, the report retains the automatic classification but shows no speculative cause.", + "根因核验:": "Root-cause review:", + "本轮没有失败结果。": "This run has no failed results.", + "解释配对结果、失败根因和结论边界。": "Interpret paired outcomes, failure root causes, and conclusion boundaries.", + "筛选任务并下钻验收合同。": "Filter tasks and inspect their acceptance contracts.", + "一、先看总分,再看同题分歧": "1. Start with aggregate scores, then inspect task-level disagreements", + "二、失败结果必须追到根因": "2. Every failed result requires a root-cause review", + "三、简单、中等、困难都覆盖,但难度只表示操作长度": "3. Easy, medium, and hard are all covered, but difficulty means operation length only", + "四、选择 WebArena‑Verified 是为了可归因的结果验收": "4. WebArena-Verified was selected for attributable outcome evaluation", + "五、结论只适用于本轮受控样本": "5. Conclusions apply only to this controlled sample", + "程序化 evaluator 先给出失败环节;核验再检查最终执行的任务合同、回答、网络记录和 Agent 轨迹,形成更具体的根因子类与原因。共同失败不默认是共同根因,单边失败也不默认是浏览器缺陷。": "The programmatic evaluator first identifies the failed acceptance stage. Review then checks the score-bearing task contract, answer, network record, and agent trajectory to establish a specific root-cause subtype and reason. A shared failure does not imply a shared cause, and a one-browser failure does not automatically imply a browser defect.", + "本轮没有失败结果,无需生成失败根因记录。": "This run has no failed results, so no failure-cause records are needed.", + "先修复已核验根因并形成新证据,再扩大任务与浏览器范围;历史官方分数不被人工覆盖。": "Fix verified root causes and produce new evidence before expanding the task and browser scope. Historical official scores are never overwritten manually.", + "逻辑结果": "Logical result", + "根因子类": "Root-cause subtype", + "已核验根因": "Verified root cause", + "表 2|根因核验不修改官方分数;修复后必须产生新的执行证据。": "Table 2 | Root-cause review does not alter official scores; fixes require new execution evidence.", + "环境准备证据:": "Environment-preparation evidence:", + "未记录": "Not recorded", + "macOS 上的 Docker 由 Colima 虚拟机承载;运行配置记录的": "Docker on macOS runs inside a Colima virtual machine. The run configuration records", + "是浏览器容器访问 macOS 宿主发布端口所用的 Colima 网关,不是评测网站自身地址。Shopping、Shopping Admin、Reddit、GitLab 四个官方站点镜像通过 Docker 端口发布到这个入口,独立的 Chromium/Moli 浏览器容器据此访问本地站点。": "as the Colima gateway through which browser containers reach ports published on the macOS host; it is not the address of an evaluation website. The four official Shopping, Shopping Admin, Reddit, and GitLab images publish Docker ports to this entry point, which the isolated Chromium and Moli containers use to access the local sites.", + "运行器在每个物理执行前调用": "Before every physical execution, the runner calls", + ",只保留该题所需站点;env-ctrl 报告 ready 后,再做连续应用健康检查、清浏览器状态、登录测试账号并打开起始页。当前证据尚未完整保存逐执行环境准备回执,因此不能把每次准备成功当成已独立核验的事实。": ", retaining only the sites required by that task. After env-ctrl reports ready, it performs repeated application health checks, clears browser state, signs into the benchmark account, and opens the start page. The current evidence does not contain a complete preparation receipt for every execution, so successful preparation cannot be claimed as independently verified for every run.", + "事件流能证明 CowAgent 在准备完成后产生了 BrowserTool 调用序列;镜像 ID、digest、代码提交与网关记录限定了本地运行身份。逐执行环境准备回执尚不完整。": "The event stream shows that CowAgent generated BrowserTool calls after preparation. Image IDs, digests, code commits, and the recorded gateway bind the local runtime identity. Per-execution preparation receipts remain incomplete.", + "逐执行环境准备回执尚不完整;现有记录只能核对运行器配置与镜像身份。": "Per-execution environment-preparation receipts are incomplete; current records can verify only runner configuration and image identity.", + "需要可完成性核验": "Solvability review required", + "已确认失败点": "Confirmed failure points", + "失败结论:": "Failure conclusion:", + "四题的首个可证失败点位于最终答案构造或状态协议,一题在 30 步上限时仍未完成付款提交。现有证据不能把前四题进一步归因到模型、CowAgent 或提示合同中的某一项。": "The first provable failure point is final-answer construction or status encoding in four tasks; one task had not submitted payment when the 30-step limit was reached. The evidence cannot further attribute the first four failures to the model, CowAgent, or prompt contract individually.", + "共享链路诊断": "Shared-stack diagnosis", + "失败归因:": "Failure attribution:", + "5 个共同失败中,四题指向共享 Agent/模型的语义、聚合或输出协议,一题指向共享的步数预算与长链完成;没有发现站点、官方 evaluator 或答案合同错误的证据。": "Among the five shared failures, four point to shared agent/model semantics, aggregation, or output protocol, and one points to the shared step budget and long-horizon completion. No evidence of site, official evaluator, or answer-contract errors was found.", + "这是固定选择的 30 题试验,不是从全部 WebArena‑Verified 任务随机抽样;比例只描述本样本,不外推总体能力。": "This is a fixed 30-task experiment, not a random sample of all WebArena-Verified tasks. Percentages describe this sample only and are not population estimates.", + "共同失败是报告解读的核心证据": "Shared failures are core interpretation evidence", + "25:25 只说明当前样本没有产生浏览器分歧。理解这个结果必须继续看五个共同失败发生在哪个验收点,而不是把平分直接解释成内核等价。": "The 25:25 result means only that this sample produced no browser disagreement. Interpreting it requires locating the acceptance point of each shared failure instead of treating the tie as browser-core equivalence.", + "最终答案数据": "Final-answer data", + "3 题": "3 tasks", + "#68 漏记录;#124 纳入投影仪;#142 多计发箍": "Task 68 omitted a record; Task 124 included a projector; Task 142 added a headband.", + "最终状态协议": "Final-status protocol", + "1 题": "1 task", + "#168 将空结果编码为 SUCCESS": "Task 168 encoded an empty result as SUCCESS.", + "任务完成边界": "Task-completion boundary", + "#506 达到 30 步时尚未提交付款": "Task 506 had not submitted payment at step 30.", + "归因边界:": "Attribution boundary:", + "这些记录确认了失败发生的位置;除 #506 的步数上限外,没有控制变量实验能够继续区分模型判断、CowAgent 策略和提示/响应合同。": "These records establish where acceptance failed. Apart from Task 506's step limit, no controlled experiment in this run can distinguish model judgment, CowAgent policy, and prompt/response contracts.", + "查看五题逐条直接证据与归因边界 →": "View direct evidence and attribution boundaries for all five tasks →", + "逻辑结果:": "Logical results:", + "筛选并下钻两种浏览器结果。": "Filter tasks and inspect results from both browsers.", + "60 个逻辑结果": "60 logical results", + "查看最终判定、耗时、轨迹和核验状态。": "Inspect final verdicts, latency, trajectories, and review status.", + "同一任务的两个浏览器最终逻辑结果。": "Final logical results from both browsers on the same task.", + "零个不一致任务意味着当前固定样本无法估计内核方向性效应;不对全部 Browser Use 任务作统计外推。": "With no discordant tasks, this fixed sample cannot estimate a directional browser effect. It does not support statistical generalization to all Browser Use tasks.", + "评测集· Browser Use 评测": "Dataset · Browser Use evaluation", + "Mind2Web 家族 · 5 家": "Mind2Web family · 5 organizations", + "WebArena 家族 · 3 家": "WebArena family · 3 organizations", + "OpenAI、Anthropic、THUDM / ChatGLM。自托管、可重置、结果验收,更适合浏览器 A/B。": "OpenAI, Anthropic, and THUDM / ChatGLM. Self-hosting, resettable state, and outcome evaluation make it a stronger fit for browser A/B tests.", + "WebVoyager 家族 · 2 家": "WebVoyager family · 2 organizations", + "OpenAI、Google DeepMind。真实网站直观,但网页漂移与 LLM judge 降低精确复现性。": "OpenAI and Google DeepMind. Live websites are intuitive, but page drift and LLM judging reduce exact reproducibility.", + "采用次数来自公开一手材料中的机构去重计数,不是质量分;证据核对日期 2026-09-08。未找到 DeepSeek、Kimi 公开的一手 UI browser-use benchmark,不等于内部没有评测。": "Adoption counts are deduplicated organizations in public primary sources, not quality scores; evidence was checked on 2026-09-08. No primary public UI browser-use benchmark was found for DeepSeek or Kimi, which does not imply that they perform no internal evaluation.", + "WebArena‑Verified 的 ground truth 不是一条固定点击脚本。": "WebArena-Verified ground truth is not a fixed click script.", + "每题声明一个结构化最终回答合同,并在需要确认页面副作用时附加一个或多个网络事件合同。所有合同都通过才记 1 分。任务详情同时展示合同、两种浏览器的actual回答、失败规则和 CowAgent 自主轨迹。": "Each task declares a structured final-answer contract and, when page side effects must be verified, one or more network-event contracts. Every contract must pass for a score of 1. Task details link the contract to both browsers' actual answers, failed rules, and autonomous CowAgent trajectories.", + "题集范围": "Dataset scope", + "本轮在模型执行前固定 30 题,覆盖四个已部署站点以及 RETRIEVE、NAVIGATE、MUTATE 三种最终合同。这里仅描述题目与验收规则;通过率、耗时、轨迹和可完成性观察统一放在结果分析中。": "The 30 tasks were fixed before model execution. They cover four deployed sites and three outcome contracts: RETRIEVE, NAVIGATE, and MUTATE. This page defines tasks and acceptance rules only; pass rates, latency, trajectories, and solvability observations belong in Results.", + "层级 ↕": "Tier ↕", + "站点 ↕": "Site ↕", + "最少步 ↕": "Minimum steps ↕", + "入选理由 ↕": "Selection rationale ↕", + "官方任务意图 ↕": "Official task intent ↕", + "官方 Hard ↕": "Official hard subset ↕", + "任务合同 ↕": "Task contract ↕", + "查看任务合同": "View task contract", + "是": "Yes", + "否": "No", + # Result-detail information architecture. + "先看任务目标与验收结论,再按需展开技术明细和 CowAgent 的每一步 BrowserTool 调用。": "Start with the task objective and acceptance outcome, then expand technical evidence and each CowAgent BrowserTool call as needed.", + "任务与验收": "Task and acceptance", + "查看任务与验收": "View task and acceptance", + "任务要求与验收结果": "Task requirements and acceptance result", + "已确认失败点:": "Confirmed failure point:", + "直接证据:": "Direct evidence:", + "最终答案少了一条必需记录。": "The final answer omitted one required record.", + "两条最终轨迹都已读取《A Christmas Carol》帖子;官方 evaluator 报告 Matched (1/2)。": "Both final trajectories read the A Christmas Carol post; the official evaluator reports Matched (1/2).", + "最终价格聚合纳入了不符合 wireless earphone 语义的商品。": "The final price aggregation included a product that does not satisfy the wireless-earphone semantics.", + "轨迹中的 17,774.32 对应 QFWCJ 4K Short Throw Projector,且与最终错误最大值完全相同。": "In the trajectory, 17,774.32 belongs to a QFWCJ 4K Short Throw Projector and exactly matches the incorrect final maximum.", + "最终求和多计入了 11.99 的 Crystal Headband。": "The final sum incorrectly included the 11.99 Crystal Headband.", + "轨迹列出 50.52 的 conditioner、11.99 的 headband 和 17.99 的 haircolor;80.50=50.52+11.99+17.99,68.51=50.52+17.99。": "The trajectory lists a 50.52 conditioner, an 11.99 headband, and a 17.99 haircolor item; 80.50=50.52+11.99+17.99, while 68.51=50.52+17.99.", + "空结果被编码成错误的最终状态。": "The empty result was encoded with the wrong final status.", + "官方 evaluator 的唯一失败断言是 Expected not_found_error, got success;检索内容同为空。": "The official evaluator's only failed assertion is Expected not_found_error, got success; both sides contain no retrieved item.", + "结账在付款提交之前停止。": "Checkout stopped before payment submission.", + "官方 evaluator 已确认正确商品加入购物车;付款提交与成功页规则失败,最终响应明确记录达到单次运行最大步骤数。": "The official evaluator confirms that the correct product was added to the cart; the payment-submission and success-page rules failed, and the final response explicitly records reaching the run's maximum step count.", + "任务目标": "Task objective", + "Agent 可自主选择 UI 路径;计分只看以下最终回答与必要网络副作用。": "The agent may choose any UI path. Scoring uses only the final answer and required network side effects below.", + "检查 Agent 最终回答的状态、字段和值是否符合官方答案合同。": "Checks whether the status, fields, and values in the agent's final answer satisfy the official answer contract.", + "检查浏览器网络记录中是否出现合同要求的页面副作用。": "Checks whether the browser network record contains the side effect required by the contract.", + "检查该题官方合同声明的结果条件。": "Checks an outcome condition declared by the task's official contract.", + "最终回答符合该规则。": "The final answer satisfies this rule.", + "已观察到满足合同的网络事件。": "A network event satisfying the contract was observed.", + "最终数据与官方期望值不一致。": "The final data differs from the official expectation.", + "浏览器网络记录中没有出现满足该规则的事件。": "No browser network event satisfying this rule was observed.", + "该规则未通过。": "This rule failed.", + "查看官方期望与本次观察": "View the official expectation and this run's observation", + "本次观察": "Observed in this run", + "查看 evaluator 技术明细": "View evaluator technical details", + "所有规则均为官方程序化验收,必须全部得分 1.0。页面先给出可读结论;官方期望、实际观察和断言保留在折叠技术明细及下载文件中。": "All rules are official programmatic checks and every rule must score 1.0. The page leads with a readable conclusion; expectations, observations, and assertions remain in collapsed technical details and downloadable files.", + "证据诊断:": "Evidence-based diagnosis:", + "本诊断不修改官方分数。": "This diagnosis does not change the official score.", + "共享的跨帖聚合与空值处理错误。轨迹已看到两篇符合条件的帖子,说明站点内容存在;失败发生在最终答案整理。": "Shared cross-post aggregation and null-handling error. The trajectory observed both qualifying posts, so the site content was present; the failure occurred while assembling the final answer.", + "共享的跨帖聚合与空值处理错误。轨迹已看到两篇符合条件的帖子,说明站点内容存在;失败发生在最终答案整理。 本诊断不修改官方分数。": "Shared cross-post aggregation and null-handling error. The trajectory observed both qualifying posts, so the site content was present; the failure occurred while assembling the final answer. This diagnosis does not change the official score.", + "共享的搜索结果相关性过滤错误;站内搜索召回过宽是诱因,但 Agent 仍应按任务语义筛选商品。": "Shared search-result relevance filtering error. Broad site-search recall contributed, but the agent still needed to filter products by the task semantics.", + "共享的搜索结果相关性过滤错误;站内搜索召回过宽是诱因,但 Agent 仍应按任务语义筛选商品。 本诊断不修改官方分数。": "Shared search-result relevance filtering error. Broad site-search recall contributed, but the agent still needed to filter products by the task semantics. This diagnosis does not change the official score.", + "共享的商品类别判定错误。订单数据和 evaluator 数值一致,未见站点或计分器异常。": "Shared product-category classification error. Order data and evaluator values agree; no site or scorer anomaly was found.", + "共享的商品类别判定错误。订单数据和 evaluator 数值一致,未见站点或计分器异常。 本诊断不修改官方分数。": "Shared product-category classification error. Order data and evaluator values agree; no site or scorer anomaly was found. This diagnosis does not change the official score.", + "共享的最终状态协议错误,不是检索内容错误,也不是浏览器执行错误。": "Shared final-status protocol error, not a retrieval-content or browser-execution error.", + "共享的最终状态协议错误,不是检索内容错误,也不是浏览器执行错误。 本诊断不修改官方分数。": "Shared final-status protocol error, not a retrieval-content or browser-execution error. This diagnosis does not change the official score.", + "共享的长链任务完成错误:30 步上限前没有完成结账。两种浏览器路径细节不同,但都未越过同一验收终点。": "Shared long-horizon completion error: checkout was not completed within the 30-step limit. Browser paths differed in detail, but neither crossed the same acceptance endpoint.", + "共享的长链任务完成错误:30 步上限前没有完成结账。两种浏览器路径细节不同,但都未越过同一验收终点。 本诊断不修改官方分数。": "Shared long-horizon completion error: checkout was not completed within the 30-step limit. Browser paths differed in detail, but neither crossed the same acceptance endpoint. This diagnosis does not change the official score.", + # Local environment and expansion guidance. + "Mac 上可以继续扩展哪些实例": "Which environments can be expanded on Mac", + "环境": "Environment", + "Apple Silicon Mac 状态": "Apple Silicon Mac status", + "完整集单站题": "Full-set single-site tasks", + "hard 单站题": "Hard single-site tasks", + "约束": "Constraint", + "已在本机实跑": "Measured locally", + "可部署,尚未实跑": "Deployable, not yet measured", + "可尝试,尚未验证": "Feasible to try, not yet verified", + "amd64 镜像;已通过 Colima 转译运行": "amd64 image; already measured through Colima translation", + "arm64 镜像;需约 100 GB 数据;只出现在跨站任务": "arm64 image; about 100 GB of data; appears only in cross-site tasks", + "amd64 镜像;需约 60 GB 数据与 Apple Silicon 转译": "amd64 image; about 60 GB of data plus Apple Silicon translation", + "Wikipedia 与 Map 的容量来自官方环境说明;状态中的“实跑”只指本项目已有执行证据。": "Wikipedia and Map capacity estimates come from the official environment guide. 'Measured' refers only to evidence already collected by this project.", + "最小扩展方案:": "Minimum expansion plan:", + "先从四个已验证站点的 678 个候选中做可完成性预检和分层抽样;Wikipedia、Map 等大数据环境后置。这样扩容不会先引入新的部署变量。": "Preflight solvability and draw a stratified sample from the 678 candidates on the four verified sites first. Defer the large Wikipedia and Map environments so expansion does not begin by adding deployment variables.", + "官方任务集与安装说明": "Official dataset and installation guide", + "官方环境容量说明": "Official environment capacity guide", + "下一轮轨迹重放固定使用本报告当前 90 题,不扩充任务范围。若未来另行扩充 Browser Use 评测,当前四个已实跑站点可预检 655 个单站任务;计入只依赖这四站的跨站任务后,共有 678 个候选,其中 hard 子集 233 个。": "The next trajectory-replay evaluation is fixed to the report's current 90 tasks; its scope will not expand. If Browser Use is expanded separately later, the four locally measured sites support preflight checks for 655 single-site tasks, or 678 candidates including cross-site tasks that use only these four sites, with 233 in the hard subset.", + "站点确实部署在本机,不是公网代站。": "The sites were deployed locally, not replaced by public websites.", + "为什么五个失败任务会完全相同": "Why the same five tasks failed in both browsers", + "二、为什么五个失败任务会完全相同": "2. Why the same five tasks failed in both browsers", + "二、五个共同失败,具体失败在哪里": "2. Where the five shared failures occurred", + "25 比 25 的真正含义:五个共同失败点相同,根因尚未隔离": "What 25 vs. 25 really means: the five failure points match, but the causes are not isolated", + "同一 CowAgent 与同一模型替换浏览器后,30 个配对任务没有一个出现单边胜出。四个共同失败止于最终答案或状态协议,一个止于 30 步上限前的结账流程;现有证据可以定位失败点,但不能把隐藏原因单独归给浏览器、模型或 Agent。": "With the same CowAgent and model and only the browser changed, none of the 30 paired tasks produced a one-browser-only win. Four shared failures ended at final-answer or status-protocol checks, and one ended in checkout before the 30-step limit. The evidence locates the failure points but cannot assign the hidden causes separately to the browser, model, or agent.", + "五、下一轮先扩展四个已验证站点": "5. Expand the four verified sites first", + "官方判定差异": "Official verdict mismatch", + "直接证据": "Direct evidence", + "尚不能判定": "Not established", + "表 2|只陈述最终回答、官方 evaluator 与执行轨迹能够直接证明的事实;不修改官方分数。": "Table 2 | Only facts directly established by the final response, official evaluator, and execution trajectory are stated; scores are unchanged.", + "两个浏览器共享 CowAgent、模型、提示与工具合同、站点状态、30 步上限和 evaluator,只有浏览器实现不同。逐条证据只能确认验收链路中的首个失败点;除 506 题明确触发步数上限外,记录不足以把错误进一步归因到模型、CowAgent 策略或提示合同中的某一项。": "The browsers share CowAgent, model, prompt and tool contract, site state, the 30-step limit, and evaluator; only the browser implementation changes. The evidence establishes the first failed acceptance point. Apart from Task 506 explicitly reaching the step limit, the records cannot attribute the errors further to the model, CowAgent policy, or prompt contract individually.", + "#68、#124、#142 和 #168 的确定事实是最终答案构造或状态协议错误;#506 的确定事实是达到 30 步上限时尚未产生付款提交和成功页事件。站点错误、evaluator 实现错误、答案合同错误以及模型内部决策原因,在这批证据中均未被证明。因而本轮不能单独判定为“模型能力不行”,只能判定当前 CowAgent、模型、提示与工具合同的组合在这些验收点失败。": "For Tasks 68, 124, 142, and 168, the established fact is an error in final-answer construction or status encoding. For Task 506, the established fact is that payment submission and the success-page event were still absent when the 30-step limit was reached. This evidence does not establish a site error, evaluator implementation bug, answer-contract error, or hidden model decision cause. The run therefore cannot isolate the result as a model-capability failure; it establishes only that the current CowAgent, model, prompt, and tool-contract combination failed at these acceptance points.", + "两种浏览器都只返回《The Hobbit》,官方合同还要求《A Christmas Carol》(author=null)。": "Both browsers returned only The Hobbit; the official contract also requires A Christmas Carol with author=null.", + "两种浏览器都返回 min=0.01、max=17774.32;官方合同要求 max=298.00。": "Both browsers returned min=0.01 and max=17774.32; the official contract requires max=298.00.", + "两种浏览器都返回 80.50;官方合同要求 68.51。": "Both browsers returned 80.50; the official contract requires 68.51.", + "两种浏览器都返回 SUCCESS + [];官方合同要求 NOT_FOUND_ERROR + null。": "Both browsers returned SUCCESS + []; the official contract requires NOT_FOUND_ERROR + null.", + "最终逻辑结果均已添加正确商品,但没有付款提交事件和下单成功页事件。": "Both final logical results added the correct product but contain neither a payment-submission event nor an order-success-page event.", + "可确认遗漏发生在页面内容已经进入轨迹之后;现有证据不能判定模型为何丢弃该记录。": "The omission occurred after the page content entered the trajectory; the evidence cannot establish why the model discarded it.", + "可确认错误值来自无关投影仪;现有证据不能区分是模型判断、Agent 策略还是任务提示约束不足导致未过滤。": "The incorrect value is established as coming from an unrelated projector; the evidence cannot distinguish model judgment, agent policy, or insufficient prompt constraints as the cause of the missing filter.", + "可确认差额及其来源;现有证据不能判定模型为何把 headband 纳入任务口径。": "The amount and its source are established; the evidence cannot establish why the model included the headband in scope.", + "可确认是最终响应协议不匹配,不是页面导航失败;现有证据不能判定错误映射来自模型还是 Agent 响应封装。": "This is established as a final-response protocol mismatch, not a navigation failure; the evidence cannot determine whether the incorrect mapping came from the model or agent response wrapper.", + "30 步上限是已确认的直接停止条件;现有证据不能进一步量化模型规划、Agent 调度和浏览器工具开销各自消耗了多少不必要步骤。": "The 30-step limit is the confirmed immediate stop condition; the evidence cannot quantify how many unnecessary steps came from model planning, agent scheduling, or browser-tool overhead.", + "实际差异": "Observed mismatch", + "证据支持的诊断": "Evidence-supported diagnosis", + "置信度": "Confidence", + "高": "High", + "中": "Medium", + "表 2|诊断来自最终回答、官方 evaluator 与执行轨迹的交叉核对;不修改官方分数。": "Table 2 | Diagnoses cross-check final answers, official evaluator output, and execution trajectories; scores are unchanged.", + "两个浏览器共享 CowAgent、模型、提示与工具合同、站点状态、30 步上限和 evaluator,只有浏览器实现不同。5 个共同失败都落在共享层,因此结果高度相关并不反常;它更说明这批题没有触发足以改变最终成败的浏览器差异。": "The browsers share CowAgent, model, prompt and tool contract, site state, the 30-step limit, and evaluator; only the browser implementation changes. All five shared failures occurred in shared layers, so correlated outcomes are expected. More importantly, this cohort did not trigger a browser difference large enough to change the verdict.", + "结论:": "Conclusion:", + "现有证据不支持“网站本身报错”“官方 evaluator 实现错误”或“标准答案错误”。其中 #124 的站内搜索召回过宽会增加歧义,但返回结果仍需要按任务语义过滤。更准确的归因是:四题属于共享 Agent/模型的语义、聚合或输出协议问题,一题属于共享的长链完成与步数预算问题。这不是对模型通用能力的定论,只描述当前 CowAgent、模型与运行协议。": "The evidence does not support a broken site, an official evaluator implementation bug, or an incorrect answer key. Broad site-search recall adds ambiguity to Task #124, but results still need semantic filtering. The best-supported attribution is four shared agent/model semantic, aggregation, or output-protocol errors and one shared long-horizon completion and step-budget error. This is not a general verdict on the model; it applies to the current CowAgent, model, and run protocol.", + "Wikipedia 需要约 100 GB 数据,Map 需要约 60 GB 数据且当前公开镜像为 amd64;二者可作为后续扩展,但没有理由成为本机下一批的前置阻塞。": "Wikipedia requires about 100 GB of data. Map requires about 60 GB and its current public image is amd64. Both can be later expansions, but neither should block the next Mac cohort.", + "保留 CowAgent、模型、reasoning=low 和每题最多两次执行的固定协议,优先加入长页面、复杂编辑器、弹窗、多标签页、上传下载等更可能诱发浏览器差异的任务。实时网站另开 Online‑Mind2Web / WebVoyager 泛化轨道,LLM judge 与人工抽检单列。": "Keep CowAgent, the model, reasoning=low, and at most two executions per task fixed. Prioritize long pages, complex editors, dialogs, multiple tabs, uploads, and downloads that are more likely to expose browser differences. Use a separate Online-Mind2Web / WebVoyager generalization track for live sites, with LLM judging and human sampling reported separately.", + "两种浏览器都返回《The Hobbit》,遗漏《A Christmas Carol》(作者允许为空)。": "Both browsers returned The Hobbit but omitted A Christmas Carol, whose author may be null.", + "两者都把搜索结果中的无关投影仪计入 wireless earphone 价格范围,最大值报为 17,774.32,官方期望为 298.00。": "Both browsers included an unrelated projector in the wireless-earphone price range, reporting 17,774.32 instead of the official 298.00 maximum.", + "两者都汇总为 80.50,官方期望为 68.51;差额来自把水晶发箍计入 hair care/style 商品。": "Both browsers totaled 80.50 instead of 68.51; the difference came from treating a crystal headband as a hair-care/style product.", + "两者都正确找到空集合,却返回 SUCCESS + [];合同要求 NOT_FOUND_ERROR + null。": "Both browsers correctly found an empty set but returned SUCCESS + []; the contract requires NOT_FOUND_ERROR + null.", + "两者都选中正确商品并清空旧购物车,但未产生付款提交与成功页事件。": "Both browsers selected the correct product and cleared the old cart, but neither produced payment-submission and success-page events.", + "macOS 上的 Docker 由 Colima 虚拟机承载;本轮 Colima VM 地址是 192.168.5.3,其默认路由指向 192.168.5.2。后者是 VM 视角下的 macOS 宿主网关,不是虚拟机地址,也不是评测网站。Shopping、Shopping Admin、Reddit、GitLab 四个官方站点镜像通过 Docker 端口发布到这个宿主入口,独立的 Chromium/Moli 浏览器容器据此访问本地站点。": "Docker on macOS runs inside the Colima VM. In this run the VM address is 192.168.5.3 and its default route points to 192.168.5.2. The latter is the macOS host gateway from the VM's perspective, not the VM address or an evaluation website. The four official Shopping, Shopping Admin, Reddit, and GitLab images publish Docker ports through that host entry point, which the independent Chromium and Moli browser containers use to reach the local sites.", + "macOS 上的 Docker 由 Colima 虚拟机承载;本轮 Colima VM 地址是": "Docker on macOS runs inside the Colima VM. In this run the VM address is", + ",其默认路由指向": ", and its default route points to", + "。后者是 VM 视角下的 macOS 宿主网关,不是虚拟机地址,也不是评测网站。Shopping、Shopping Admin、Reddit、GitLab 四个官方站点镜像通过 Docker 端口发布到这个宿主入口,独立的 Chromium/Moli 浏览器容器据此访问本地站点。": ". The latter is the macOS host gateway from the VM's perspective, not the VM address or an evaluation website. The four official Shopping, Shopping Admin, Reddit, and GitLab images publish Docker ports through that host entry point, which the independent Chromium and Moli browser containers use to reach the local sites.", + ",只保留该题所需站点;env-ctrl 报告 ready 后,再做连续应用健康检查、清浏览器状态、登录测试账号并打开起始页。当前冻结证据保存了代码提交、运行协议和镜像身份;事件流从环境准备完成后开始,因此逐次 reset、ready 与 health 结果未形成独立绑定的准备回执。新版运行协议已要求保存该回执。": ", retaining only the sites required by the task. After env-ctrl reports ready, the runner performs consecutive application health checks, clears browser state, signs in with test accounts, and opens the start page. The frozen evidence retains the code commit, run protocol, and image identities. Because the event stream begins after preparation, per-run reset, readiness, and health outcomes were not bound into a separate preparation receipt. The new run protocol requires that receipt.", + "表 1|30 个任务的逐题最终结果配对。": "Table 1 | Paired final outcomes for 30 tasks.", + # Deployment, timing, browser matrix, and participant framing. + "图 1 · 从官方站点镜像、逐题隔离与健康检查,到 CowAgent 观察—决策—操作闭环,再到 Chromium/Moli 对照和官方程序化验收。窄屏可横向滚动查看。": "Figure 1. From official site images, per-task isolation, and health checks through CowAgent's observe-decide-act loop, then Chromium/Moli comparison and official programmatic evaluation. Scroll horizontally on narrow screens.", + "macOS 上的 Docker 由 Colima 虚拟机承载;Shopping、Shopping Admin、Reddit、GitLab 四个官方镜像把应用端口发布到 Colima/Docker 宿主。Chromium 与 Moli 各自在独立浏览器容器中运行,通过容器内解析到的宿主网关": "On macOS, Docker runs inside a Colima VM. The four official Shopping, Shopping Admin, Reddit, and GitLab images publish their application ports to the Colima/Docker host. Chromium and Moli run in separate browser containers and reach those ports through the host gateway resolved inside each container", + "访问这些端口。这个地址只是本轮私有网络中的宿主入口,不是评测网站或固定产品配置。": "to access the sites. This address is only the host entry point in this run's private network; it is neither a benchmark website nor a fixed product configuration.", + "预注册文件只含任务 ID、层级和入选理由,不含点击序列。CowAgent 收到自然语言目标后,在最多 30 步内循环读取页面观察、请求模型决策并生成下一次 BrowserTool 调用。Chromium 与 Moli 分别连接独立 CDP 浏览器容器;每个逻辑结果保留最终 JSON、Agent 事件流、网络 HAR 和 Playwright trace。": "The preregistration contains task IDs, tiers, and selection rationales, but no click sequence. Given a natural-language objective, CowAgent repeatedly reads the page observation, requests a model decision, and generates the next BrowserTool call for up to 30 steps. Chromium and Moli connect to separate CDP browser containers. Each logical result retains the final JSON, agent event stream, network HAR, and Playwright trace.", + "本轮访问地址": "Address used in this run", + "应用端口": "Application port", + "环境控制端口": "Environment-control port", + "状态": "Status", + "本轮已部署": "Deployed in this run", + "未部署;不在 30 题": "Not deployed; outside the 30 tasks", + "事件流能证明 CowAgent 在准备完成后产生了 BrowserTool 调用序列;镜像 ID、digest、代码提交与网关记录限定了本地运行身份。当前证据没有独立的逐次环境准备 receipt,因此报告不把“每次重置均成功”当成已经独立核验的事实。": "Event streams establish that CowAgent produced BrowserTool calls after preparation. Image IDs, digests, the code commit, and gateway records bound the local runtime identity. The evidence has no independent per-execution preparation receipt, so the report does not treat every reset as independently verified.", + "结果口径": "Result unit", + "每题×浏览器一个最终逻辑结果;二元通过率;逐题配对": "One final logical result per task-browser pair; binary pass rate; task-level pairing", + "执行上限": "Execution limit", + "代码最多允许两次执行;本报告按冻结运行协议选取正式结果,实际执行耗时和模型用量累计": "The code permits at most two executions. The frozen run protocol selects the official result, while actual execution latency and model usage are cumulative.", + "不纳入无法稳定验收的实时任务;本轮排除需大型数据卷的 Wikipedia / Map": "Exclude live tasks without stable acceptance; this run excludes Wikipedia and Map because they require large data volumes", + "结果分析 · Browser Use 评测": "Results · Browser Use evaluation", + "从总体配对下钻到每个逻辑结果": "Drill down from the paired summary to every logical result", + "先看 30 个同题配对,再定位 10 个失败结果的官方规则、actual回答、轨迹和日志。": "Start with 30 same-task pairs, then inspect the official rules, actual answers, trajectories, and logs for the 10 failed results.", + "通过率": "Pass rate", + "Agent 耗时中位数": "Median agent latency", + "Agent 耗时 P95": "P95 agent latency", + "BrowserTool 耗时中位数": "Median BrowserTool latency", + "BrowserTool 耗时 P95": "P95 BrowserTool latency", + "Agent 端到端耗时": "Agent end-to-end latency", + "从浏览器会话准备开始,到 CowAgent 完成、evaluator 所需证据落盘前结束;包含登录、模型推理、调度、页面等待、BrowserTool 与收尾,不包含此前的 Docker 站点重置。": "Measured from browser-session preparation until CowAgent completes and before evaluator evidence is persisted. It includes sign-in, model inference, orchestration, page waits, BrowserTool, and cleanup, but excludes the preceding Docker site reset.", + "BrowserTool 执行耗时": "BrowserTool execution latency", + "是事件流中每次 browser 工具调用墙钟时间之和,包含工具封装、CDP 往返与页面等待,仍不是纯内核 CPU 时间。": "The sum of wall-clock durations for browser tool calls in the event stream. It includes tool-wrapper, CDP round-trip, and page-wait overhead, so it is still not pure browser-core CPU time.", + "浏览器动作耗时拆分": "Browser action latency breakdown", + "BrowserTool 动作延迟": "BrowserTool action latency", + "每张卡对应一种 CowAgent action。以单次调用墙钟均值作为主要比较值并标记较低者;总耗时与调用数用于解释工作量,不判优。该指标仍包含工具封装、CDP 往返和页面等待,不是纯内核 CPU 时间。": "Each card represents one CowAgent action. Mean wall-clock time per call is the primary comparison and the lower value is marked; total latency and call count describe workload and are not ranked. The metric still includes tool-wrapper, CDP round-trip, and page-wait overhead, so it is not pure browser-core CPU time.", + "单次调用墙钟时间": "Wall time per call", + "按 CowAgent 发出的 BrowserTool action 聚合调用次数与墙钟时间,用于定位 navigate、click、input、snapshot 等动作在哪个浏览器上更慢。该指标比 Agent 总耗时更接近浏览器执行层,但仍包含工具与 CDP 开销。": "Aggregate call counts and wall-clock time by BrowserTool action emitted by CowAgent to locate whether navigate, click, input, snapshot, and other actions are slower in one browser. This metric is closer to the browser execution layer than total agent latency, but still includes tool and CDP overhead.", + "动作": "Action", + "浏览器两两比较": "Pairwise browser comparisons", + "新增浏览器后自动生成每一对浏览器的同题比较;每一行只使用双方都完成的任务,避免把三种以上浏览器压成一个含义不清的总胜负。": "Adding a browser automatically creates a same-task comparison for every browser pair. Each row uses only tasks completed by both browsers, avoiding an ambiguous aggregate win/loss label across three or more browsers.", + "浏览器对": "Browser pair", + "共同任务": "Shared tasks", + "左侧单独通过": "Left-only passes", + "右侧单独通过": "Right-only passes", + "同题浏览器结果矩阵": "Same-task browser result matrix", + "浏览器组合对比": "Browser-set comparison", + "按同一任务比较所选浏览器,不用各浏览器不同的成功样本拼接结论。基准浏览器单选,对比浏览器可多选;新增浏览器只增加选项和结果标签,不改变页面结构。": "Compare selected browsers on the same task instead of combining different success subsets. Choose one baseline browser and one or more comparison browsers; adding a browser only adds an option and outcome label without changing the page structure.", + "同题结果组合": "Same-task outcome combinations", + "选择一个基准浏览器和一个或多个对比浏览器;浏览器标签可直达该题完整结果。": "Choose one baseline browser and one or more comparison browsers. Browser labels link directly to the complete task result.", + "浏览器 A": "Browser A", + "比较条件": "Comparison condition", + "A 通过,其余失败": "A passes, all others fail", + "A 失败,其余通过": "A fails, all others pass", + "所选浏览器全部通过": "All selected browsers pass", + "所选浏览器全部失败": "All selected browsers fail", + "仅所选浏览器通过": "Only the selected browsers pass", + "全部类型": "All task types", + "对比浏览器(可多选)": "Comparison browsers (select multiple)", + "通过浏览器": "Passing browsers", + "失败浏览器": "Failing browsers", + "请至少选择一个对比浏览器": "Select at least one comparison browser", + "每行是一道任务,浏览器作为可扩展列生成;以后增加浏览器时沿用同一张矩阵,不再把页面结构写死为 Chromium 与 Moli。关系筛选只改变可见行,不产生新分数。": "Each row is one task, with browsers generated as extensible columns. Future browsers use the same matrix instead of hard-coding Chromium and Moli into the page structure. Relationship filters change visible rows only and create no new scores.", + "全部通过": "All pass", + "全部失败": "All fail", + "结果有分歧": "Mixed results", + "横向关系": "Cross-browser outcome", + "任务定义": "Task definition", + "任务合同": "Task contract", + "失败结果定位": "Failed-result inventory", + "默认只显示失败;可按浏览器、层级、站点、任务类型、失败环节和全文关键词筛选。": "Failures are shown by default. Filter by browser, tier, site, task type, failure stage, or full-text query.", + "动作数": "Actions", + "查看逐规则判定": "View rule-level verdicts", + "累计自主操作": "Cumulative autonomous actions", + "Agent 链路耗时包含会话准备、登录、模型推理、调度、工具执行和收尾,不包含 Docker 站点重置;BrowserTool 耗时是全部 browser 工具调用的墙钟时间之和,不等同于纯内核时间。": "Agent-stack latency includes session preparation, sign-in, model inference, orchestration, tool execution, and cleanup, but excludes Docker site reset. BrowserTool latency is the sum of wall-clock time for all browser tool calls and is not pure browser-core time.", + "执行未完成;错误信息见运行与证据。": "Execution did not complete; see runtime and evidence for the error.", + # Result pages and audit boundaries. + "Agent 端到端耗时包含会话准备、登录、模型推理、调度、工具执行和收尾,不包含 Docker 站点重置;BrowserTool 执行耗时是全部 browser 工具调用的墙钟时间之和,不等同于纯内核时间。": "Agent end-to-end latency includes session preparation, sign-in, model inference, orchestration, tool execution, and cleanup, but excludes Docker site reset. BrowserTool execution latency is the sum of wall-clock time for all browser tool calls and is not pure browser-core time.", + "公开包包含脱敏的回答、evaluator、事件流与轨迹。原始 HAR 和 Playwright trace 可能携带登录态、Cookie、页面快照或请求正文,因此不公开;其字节数和 SHA-256 保留用于证明证据绑定。": "The public bundle contains sanitized answers, evaluator records, event streams, and trajectories. Raw HAR and Playwright traces may contain authenticated state, cookies, page snapshots, or request bodies, so they are withheld; byte counts and SHA-256 hashes retain the evidence binding.", + "整次 Agent 运行的容器计量只作诊断;不同进程模型下不直接比较内存或 PID。": "Whole-run container measurements are diagnostic only; memory and PID are not compared directly across different process models.", + "全部官方 evaluator 均通过。": "All official evaluators passed.", + "核验记录 · Browser Use 评测": "Review · Browser Use evaluation", + "证据完整性与独立结果复核是两个指标;本轮只完成前者,官方 evaluator 仍是唯一成绩来源。": "Evidence integrity and independent result review are separate metrics. This run completed only the former; the official evaluator remains the sole scoring authority.", + "证据完整性": "Evidence integrity", + "文件、哈希与分数一致性": "File, hash, and score consistency", + "独立结果复核": "Independent result review", + "尚未进行语义盲审": "No semantic blind review performed", + "完整性未决": "Integrity issues unresolved", + "完整性检查确认回答、HAR、Agent trace、事件和 evaluator 明细存在,并核对哈希、分数、状态与官方版本。": "Integrity checks confirm that answers, HAR, agent traces, events, and evaluator details exist, and verify hashes, scores, statuses, and the official version.", + "复核边界:": "Review boundary:", + "本轮没有第二名人工或模型对结果做盲审,因此“0 个完整性未决”不能解读为结果已经独立语义复核。": "No second human or model blind-reviewed the results, so '0 unresolved integrity issues' must not be interpreted as independent semantic review.", + "审计项": "Audit item", + "回答、事件、HAR、trace 与 evaluator 文件完整且哈希绑定": "Answers, events, HAR, traces, and evaluator files are complete and hash-bound", + "成绩来源": "Scoring authority", + "只接受未修改的 WebArena‑Verified evaluator 输出": "Accept only unmodified WebArena-Verified evaluator output", + "一致性": "Consistency", + "公开状态、分数、官方版本与原始记录一致": "Published status, score, and official version match the source record", + "争议处理": "Dispute handling", + "任何不一致或缺失证据均阻止发布;人工不得改分": "Any inconsistency or missing evidence blocks release; humans may not change scores", + "逐结果覆盖": "Per-result coverage", + # Participants, reproduction, and account boundaries. + "方法介绍 · Browser Use 评测": "Participants · Browser Use evaluation", + "本轮 A/B 的唯一变化项是浏览器实现;CowAgent、模型、任务、资源上限、站点与 evaluator 都是共同实验条件。": "The browser implementation is the only changed variable in this A/B test. CowAgent, the model, tasks, resource limits, sites, and evaluators are shared experimental conditions.", + "两个参评浏览器": "Two browser participants", + "参评浏览器": "Browser participant", + "运行版本": "Runtime version", + "接入方式": "Integration", + "本轮成绩": "Result in this run", + "独立容器 · CDP · CowAgent BrowserTool": "Dedicated container · CDP · CowAgent BrowserTool", + "更完整地说,一个可复现的“参评方法”是整套 Agent 栈;但本轮所有栈字段都保持一致,只有浏览器后端不同,所以报告按浏览器命名选手。不能把 CowAgent、gpt‑5.6‑sol 或 WebArena‑Verified 也算成独立选手。": "Strictly, a reproducible participant is the whole agent stack. In this run every stack field is held constant except the browser backend, so participants are named by browser. CowAgent, gpt-5.6-sol, and WebArena-Verified are not additional participants.", + "共同实验条件": "Shared experimental conditions", + "栈字段": "Stack field", + "本轮固定值": "Fixed value in this run", + "角色": "Role", + "自主观察、规划并调用 BrowserTool": "Observe, plan, and call BrowserTool autonomously", + "模型": "Model", + "为两个选手生成决策": "Generate decisions for both participants", + "任务与站点": "Tasks and sites", + "同一 30 题;同一批本地站点镜像": "Same 30 tasks; same local site images", + "提供一致输入与网页状态": "Provide identical inputs and website state", + "动作接口": "Action interface", + "把 Agent 动作交给浏览器": "Deliver agent actions to the browser", + "验收": "Evaluation", + "WebArena‑Verified 1.2.3 官方 evaluator": "Official WebArena-Verified 1.2.3 evaluator", + "对最终 JSON 与必要网络副作用判分": "Score final JSON and required network side effects", + "资源与上限": "Resources and limits", + "相同 CPU、内存、viewport 与最多 30 步": "Same CPU, memory, viewport, and 30-step limit", + "控制运行预算": "Control the execution budget", + "这 30 题由 CowAgent 自主执行。": "CowAgent executes all 30 tasks autonomously.", + "仓库中的 Browser Trajectory Replay(轨迹重放)固定动作微基准属于另一条实验轨道,不是本报告的参评选手,也不计入这 30 题成绩。": "The repository's fixed-action Browser Trajectory Replay microbenchmark is a separate experimental track, not a participant in this report, and is excluded from these 30 task results.", + "Browser Use 公开使用 BU Bench V1、Odysseys、Online‑Mind2Web 和 Stealth Bench。可借鉴的是运行前确认任务可完成、保存逐题 trace、断点续跑,并把站点不可完成与 Agent 真失败分开。Stealth Bench 测反机器人检测,不能替代自主任务成功率。": "Browser Use publicly uses BU Bench V1, Odysseys, Online-Mind2Web, and Stealth Bench. Useful practices include confirming task solvability before execution, retaining per-task traces, resuming interrupted runs, and separating unsolvable sites from genuine agent failures. Stealth Bench measures bot detection and cannot replace autonomous task success rate.", + "环境与复现 · Browser Use 评测": "Reproduction · Browser Use evaluation", + "协议": "Protocol", + "运行配置选择集 SHA-256": "Run-config selection SHA-256", + "当前选择集 SHA-256": "Current selection SHA-256", + "部署环境与网络拓扑": "Deployment environment and network topology", + "本轮 Docker daemon 运行在 macOS 上的 Colima 虚拟机中。站点容器把应用端口发布到 Docker 宿主;两个浏览器容器通过其容器内": "The Docker daemon for this run operates in a Colima VM on macOS. Site containers publish application ports to the Docker host; both browser containers access them through the", + "对应的网关地址访问这些端口。运行时记录的地址是": "gateway corresponding to host.docker.internal inside each container. The address recorded at runtime is", + "。它属于本轮私有网络,不应复制为其他机器的固定地址。": ". It belongs to this run's private network and must not be copied as a fixed address for another machine.", + "层": "Layer", + "本轮身份": "Identity in this run", + "作用": "Purpose", + "宿主": "Host", + "运行 Colima 与报告 runner": "Runs Colima and the report runner", + "承载站点和浏览器容器": "Hosts site and browser containers", + "浏览器到宿主网关": "Browser-to-host gateway", + "访问站点容器发布端口": "Accesses published ports from site containers", + "ARM 兼容模式": "ARM compatibility mode", + "未启用": "Disabled", + "运行部分 amd64 PHP 站点镜像": "Runs selected amd64 PHP site images", + "站点镜像": "Site image", + "镜像引用": "Image reference", + "环境隔离与并行边界": "Environment isolation and parallelism boundary", + "修正后的运行器把一个 Docker daemon 及其固定站点端口视为一个环境 lane。同一 lane 内的任务和浏览器必须串行:每次尝试先重置站点,再完成 Agent、证据收尾和 evaluator,整个周期由跨进程锁保护。缺少 Docker daemon 身份时直接停止,不降级为无锁执行。": "The corrected runner treats one Docker daemon and its fixed site ports as one environment lane. Tasks and browsers in the same lane run serially: each attempt resets the sites first, then completes the Agent run, evidence finalization, and evaluation while one cross-process lock protects the full lifecycle. If the Docker daemon identity is unavailable, execution stops instead of falling back to an unlocked run.", + "运行方式": "Execution mode", + "是否允许": "Allowed", + "隔离要求": "Isolation requirement", + "一套站点环境,Chromium 与 Moli 串行": "One site stack; Chromium and Moli run serially", + "允许,默认方案": "Allowed; default", + "同一 daemon 锁;每次尝试前重置并核对初态": "One daemon-scoped lock; reset and verify initial state before every attempt", + "一套站点环境,两个浏览器并行": "One site stack; two browsers run in parallel", + "不允许": "Not allowed", + "账号不同仍会共享对象、索引、缓存和后台任务": "Separate accounts still share objects, indexes, caches, and background jobs", + "多个任务并行": "Multiple tasks in parallel", + "允许": "Allowed", + "每个 worker 使用独立 daemon、站点容器、网络、端口和可写卷;镜像只读层可复用": "Each worker uses an independent daemon, site containers, network, ports, and writable volumes; read-only image layers may be shared", + "账号隔离只用于避免用户级对象冲突,不能替代环境 lane。Task 77/771 的旧证据来自修正前的共享状态,正式比较已将整题四种配置一并排除;只有新的完整四配置证据批次才能重新纳入。": "Account isolation only prevents user-scoped object conflicts; it does not replace an environment lane. The old evidence for Tasks 77 and 771 came from shared state before this correction, so all four configurations for each task are excluded from the formal comparison. Readmission requires a new complete four-configuration evidence cohort.", + "逐执行的环境准备 receipt 尚未包含在这批冻结证据中;现有记录可核对运行器配置与镜像身份,但不能独立重放每次 reset/ready/health 的完整证明链。": "Per-execution environment-preparation receipts are absent from this frozen evidence. Existing records verify runner configuration and image identities but cannot independently replay the complete reset/ready/health proof chain for every execution.", + "每个逻辑结果的脱敏回答、evaluator、事件和轨迹位于": "Sanitized answers, evaluator records, events, and trajectories for each logical result are under", + "给出全部 60 个逻辑结果的页面与下载路径。": "lists page and download paths for all 60 logical results.", + "账号 · Browser Use 评测": "Accounts · Browser Use evaluation", + "不展示密钥、Cookie 或真实个人信息;官方合同中的内置测试邮箱和用户名作为合成 ground truth 保留。": "Keys, cookies, and real personal information are not shown. Built-in test emails and usernames in official contracts remain as synthetic ground truth.", + "权限与费用": "Access and cost", + "四个自托管站点": "Four self-hosted sites", + "官方 Docker 镜像与内置测试账号;每题前重置": "Official Docker images and built-in test accounts; reset before each task", + "不需要": "Not required", + "模型 API": "Model API", + "需要相应 API 权限": "Corresponding API access required", + "当前团队无权限": "Current team lacks access", + "只发布脱敏静态报告": "Publishes only the sanitized static report", + "发布者需已有站点权限": "Publisher must already have site access", + "字段": "Field", + "任务成绩": "Task score", + "逻辑结果耗时中位数": "Median logical-result latency", + "版本与镜像身份用于限定复现边界,不参与评分。": "Version and image identity bound reproducibility and do not affect scoring.", + "全部逻辑结果": "All logical results", + "每题只展示一个最终逻辑结果;端到端耗时与模型用量包含获得该结果所需的全部执行。": "Each task shows one final logical result. End-to-end latency and model usage include every execution needed to obtain it.", + "Agent 耗时": "Agent latency", + "BrowserTool 耗时": "BrowserTool latency", + "失败原因": "Failure reason", + "程序自动定位": "Automatic failure locator", + "核验根因子类": "Verified root-cause subtype", + "核验原因": "Verified cause", + "最终回答验收失败": "Final-answer acceptance failure", + "网络副作用验收失败": "Network-side-effect acceptance failure", + "多项验收失败": "Multiple acceptance failures", + "60 个任务×浏览器结果均保留最终回答、官方 evaluator 结果、Agent 事件和证据哈希。": "All 60 task-browser results retain the final answer, official evaluator result, agent events, and evidence hashes.", + "60/60 个逻辑结果已覆盖,0 个未决。它不重新判题,也不是独立语义复核。": "Evidence-integrity checks cover all 60 logical results with no unresolved integrity issue. They do not rescore tasks or constitute independent semantic review.", + "60/60 个逻辑结果完成证据完整性审计": "Evidence integrity checked for all 60 logical results", + "每题声明一个结构化最终回答合同,并在需要确认页面副作用时附加一个或多个网络事件合同。所有合同都通过才记 1 分。任务页只解释合同;各浏览器的实际回答、失败规则和 CowAgent 自主轨迹统一放在结果页。": "Each task declares a structured final-answer contract and, where required, one or more network-event contracts for page side effects. Every contract must pass for a score of 1. Task pages explain only the contract; browser answers, failed rules, and CowAgent trajectories appear on result pages.", + "运行器代码在每个执行前调用": "Before each execution, the runner calls", + "先看 30 个同题配对,再定位 10 个失败结果的官方规则、实际回答、轨迹和日志。": "Start with the 30 same-task comparisons, then inspect the official rules, actual answers, trajectories, and logs for the 10 failed results.", + "Agent 链路耗时中位数": "Median agent-stack latency", + "Agent 链路耗时 P95": "P95 agent-stack latency", + "Agent 链路累计耗时": "Cumulative agent-stack latency", + "Agent 链路累计": "agent-stack cumulative", + "BrowserTool 累计耗时": "Cumulative BrowserTool latency", + "BrowserTool 累计": "BrowserTool cumulative", + "把获得该任务结果实际发生的执行相加;每次从浏览器会话准备开始,到 CowAgent 完成、验收所需证据落盘前结束,包含登录、模型推理、调度、页面等待、BrowserTool 与收尾,不包含此前的 Docker 站点重置。": "sums all execution used to obtain the task result. Each execution runs from browser-session preparation until CowAgent completes and acceptance evidence is persisted. It includes sign-in, model inference, orchestration, page waits, BrowserTool, and cleanup, but excludes the preceding Docker site reset.", + "是事件流中全部 browser 工具调用墙钟时间之和,包含工具封装、CDP 往返与页面等待,仍不是纯内核 CPU 时间。": "is the sum of wall-clock durations for all browser tool calls in the event stream. It includes wrapper, CDP round-trip, and page-wait overhead, so it is not pure browser-core CPU time.", + "次调用": "calls", + "次动作": "actions", + "次模型请求": "model requests", + "如何增加下一名选手": "How to add another participant", + "原版 Mind2Web、Online-Mind2Web 与 Mind2Web 2 的任务和评分器不同,不能合并到同一成绩表。": "Original Mind2Web, Online-Mind2Web, and Mind2Web 2 use different tasks and graders and must not be merged into one score table.", + "新增浏览器时,需要注册其容器镜像、版本与 CDP 端点,并在同一冻结任务集和共同条件下形成完整结果。总榜按浏览器逐行扩展;同题矩阵按浏览器增加列;统计比较按浏览器对单独生成,不能把多个浏览器压成一个总胜负。": "Register the new browser's container image, version, and CDP endpoint, then complete the same frozen task set under the shared controls. The leaderboard adds a browser row, the task matrix adds a browser column, and statistical comparisons are generated per browser pair rather than collapsing multiple browsers into one win/loss label.", + "先看总体与同题比较,再通过筛选下钻到每个结果的官方规则、实际回答、轨迹和日志。": "Start with aggregate and same-task comparisons, then use filters to inspect each result's official rules, actual answer, trajectory, and logs.", + "Agent 链路耗时": "Agent-stack latency", + "均值": "Mean", + "总耗时": "Total latency", + "单次均值": "Mean per call", + "调用数": "Call count", + "更优": "Better", + "标记方向明确且数值更优的指标:通过数与通过率越高越优,耗时与 Tokens 越低越优;数值相同时不标记。": "marks metrics with a clear direction and a better value: higher pass counts and rates are better, while lower latency and token usage are better. Ties are not marked.", + "按 CowAgent 发出的 BrowserTool action 聚合总墙钟时间、单次均值与调用数。耗时较低者标记为更优;调用数描述 Agent 行为,不判优。该指标比 Agent 总耗时更接近浏览器执行层,但仍包含工具与 CDP 开销。": "Aggregate total wall-clock time, mean per call, and call count by BrowserTool action emitted by CowAgent. Lower latency is marked as better; call count describes agent behavior and is not ranked. This metric is closer to the browser execution layer than total agent latency but still includes tool and CDP overhead.", + "核验与修订记录": "Review and revision history", + "按轮次记录核验范围、发现、正式处理和遗留事项;单个任务的判定证据位于结果详情页。": "Each round records its scope, findings, formal corrections, and remaining items. Per-task verdict evidence remains on result detail pages.", + "当前状态": "Current status", + "发布门禁:": "Release gate:", + "证据文件、哈希、分数、状态、官方版本与报告链接已通过程序化一致性检查。": "Evidence files, hashes, scores, statuses, the official version, and report links passed programmatic consistency checks.", + "官方 WebArena‑Verified evaluator 是成绩来源;核验记录只说明报告和实现经过了哪些检查与修订,不替代任务判分。": "The official WebArena-Verified evaluator is the scoring authority. This history records checks and revisions to the implementation and report; it does not replace task scoring.", + "核验轮次": "Review rounds", + "核验范围": "Scope", + "发现": "Findings", + "处理": "Corrections", + "遗留事项": "Remaining items", + "已处理": "Resolved", + "无阻塞项。": "No blocking items.", + "多浏览器比较与结果详情核验": "Multi-browser comparison and result-detail review", + "同题比较交互、BrowserTool 动作延迟和单结果详情首屏。": "Same-task comparison interaction, BrowserTool action latency, and the opening section of result-detail pages.", + "同题视图虽动态生成列,但交互仍以两浏览器关系为中心;动作表层级拥挤且对不同调用量的总耗时判优;完整 evaluator 错误被放入窄指标卡。": "The same-task view generated columns dynamically but still centered its interaction on a two-browser relationship; the action table was visually congested and ranked totals with unequal call counts; full evaluator errors appeared in narrow KPI cards.", + "同题比较改为基准浏览器加多选对比浏览器和明确关系条件;动作延迟改为按 action 分卡并只比较单次均值;详情首屏只保留简明规则定位,完整 expected、actual 和错误进入规则证据区。": "Changed same-task comparison to one baseline browser, multi-select comparison browsers, and explicit outcome relations; split action latency into action cards and compare only mean per call; kept only concise rule labels in the result summary and moved complete expected, actual, and error evidence to the rule section.", + "当前仅有两个参评浏览器;扩展行为已由三浏览器测试模型覆盖,新增真实浏览器仍需形成完整同题结果。": "Only two browser participants are currently measured. A three-browser test model covers extensibility behavior, while any additional real browser still requires complete same-task results.", + "参评浏览器 · Browser Use 评测": "Browser participants · Browser Use evaluation", + "被测浏览器": "Browsers tested", + "被测浏览器与固定实验条件": "Browsers tested and fixed experimental controls", + "本报告比较浏览器。每个模型单独统计,CowAgent、任务、资源上限、网站和 evaluator 均保持一致。": "This report compares browsers. Results are calculated separately for each model, while CowAgent, tasks, resource limits, sites, and evaluators remain fixed.", + "浏览器与版本": "Browsers and versions", + "浏览器": "Browser", + "通过任务": "Tasks passed", + "每个模型分别比较各浏览器,不把不同模型的结果合并排名。": "Browsers are compared separately within each model; results from different models are not merged into one ranking.", + "增加新的浏览器": "Adding another browser", + "仓库中的 Browser Trajectory Replay(轨迹重放)是另一项固定动作测试,不计入这里的成绩。": "The repository's Browser Trajectory Replay track is a separate fixed-action test and is not included in these scores.", + "评测结果": "Evaluation results", + "浏览器 B": "Browser B", + "同一任务上的结果差异": "Same-task outcome differences", + "每行比较同一模型、同一任务和同一网站状态下的两个浏览器。A、B 对应表内所列浏览器。": "Each row compares two browsers under the same model, task, and site state. A and B identify the browsers shown in that row.", + "三类任务的覆盖与成绩": "Coverage and results by task type", + "信息提取、页面导航和页面操作使用不同验收合同,因此分类型展示,不把难度或任务类型混成新的加权分。": "Information retrieval, navigation, and page changes use different acceptance contracts, so they are reported separately without creating a new weighted score.", + "数据是否齐全": "Data completeness", + "检查项": "Check", + "每个失败结果都有分类和具体原因": "Every failed result has a category and a specific cause", + "完整性检查没有发现待处理项": "The integrity check found no unresolved issue", + "包含正式执行及按协议发生的一次重试": "Includes the main execution and the single retry allowed by the protocol", + "总体成绩和耗时": "Overall results and latency", + "报告分别使用 GPT-5.6 Sol 和 GPT-6 Astra 驱动 CowAgent。每组都在相同任务、网站和验收规则下比较浏览器。": "CowAgent was driven by GPT-5.6 Sol and GPT-6 Astra. Within each model run, browsers were compared on the same tasks, sites, and acceptance rules.", + "WebArena‑Verified 1.2.3 · 官方程序化 evaluator · 不使用 LLM judge": "WebArena-Verified 1.2.3 · official programmatic evaluator · no LLM judge", + "在 Sol 运行中,Chromium 比 Moli 多通过 3 题;在 Astra 运行中,两者都是 71/90。耗时需要结合下表和逐题记录看:Agent 总耗时还包含模型推理与页面等待,BrowserTool 耗时也包含工具封装和 CDP 往返。": "In the Sol run, Chromium passed three more tasks than Moli; in the Astra run, both passed 71/90. Latency must be read with per-task records: agent total time includes model inference and page waits, while BrowserTool time still includes tool wrapping and CDP round trips.", + "只标记同一模型下数值唯一更好的浏览器。通过数越高越好;耗时和 Tokens 越低越好。Agent 总耗时包含模型推理、页面等待和浏览器操作,BrowserTool 耗时更接近浏览器执行层,但仍包含工具封装与 CDP 往返。": "marks only the uniquely better browser within the same model. Higher pass counts are better; lower latency and token use are better. Agent total time includes model inference, page waits, and browser actions. BrowserTool time is closer to the browser execution layer but still includes tool wrapping and CDP round trips.", + "打开逐题对照 →": "Open the task-by-task comparison →", + "官方 evaluator 说明哪条验收规则没有满足;下表来自对任务要求、最终回答、网络记录和自主执行轨迹的逐条核验。": "The official evaluator identifies the unmet acceptance rule. The table comes from reviewing each task requirement, final answer, network record, and autonomous trajectory.", + "查看逐条核验原因 →": "View every verified cause →", + "作为分层变量为各浏览器生成决策": "Generates browser decisions within each separately reported model run", + "从模型分层的浏览器对照读出下一轮该怎么做": "What the browser comparisons imply for the next run", + "步骤预算耗尽": "Step budget exhausted", + "导航语义错误": "Navigation-semantics error", + "检索覆盖不足": "Incomplete retrieval coverage", + "商品选择错误": "Incorrect product selection", + "候选比较错误": "Candidate-comparison error", + "提交字段错误": "Submission-field error", + "检索策略错误": "Retrieval-strategy error", + "流程未完成": "Incomplete workflow", + "结果去重遗漏": "Result-deduplication failure", + "可恢复错误后提前终止": "Premature stop after recoverable error", + "批量操作遗漏": "Incomplete batch workflow", + "提交字段遗漏": "Submission-field omission", + "路由验收缺陷": "Route-acceptance defect", + "evaluator 实现错误": "Evaluator implementation defect", + "任务与验收矛盾": "Instruction-evaluator contradiction", + "表单验收缺陷": "Form-acceptance defect", + "任务状态与验收矛盾": "State-evaluator contradiction", + "站点保存故障": "Site save failure", + "站点依赖故障": "Site dependency failure", + "站点会话故障": "Site session failure", + "网络证据缺失": "Missing network evidence", + "网络证据匹配缺陷": "Network-evidence matching defect", + "响应证据缺失": "Missing response evidence", + "共享环境并发污染": "Shared-environment concurrency contamination", + "检索遗漏与参考状态差异": "Retrieval omission and reference-state drift", + "同题比较只在同一模型内进行。A、B 是表内所列的两个浏览器。": "Same-task comparisons are made only within one model. A and B identify the two browsers listed in the row.", + "每行只比较同一模型、同一任务和同一网站状态下的结果。": "Each row compares results under the same model, task, and site state.", + "两个模型的结果不能直接当成模型排名": "The two model runs are not a model ranking", + "Sol 与 Astra 在不同时间、不同代码提交上运行。Astra 的两个 Task 77 结果还受到已确认的共享环境并发污染。表格可以帮助观察差异,但不能把差值全部归因于模型。": "Sol and Astra ran at different times and on different code revisions. Two Astra Task 77 results were also affected by confirmed concurrent contamination of shared site state. The table shows observations, but the differences cannot all be attributed to the model.", + "evaluator 的报错只说明哪条验收规则没有满足。我们继续核对任务要求、最终回答、网络记录和 Agent 轨迹,确认失败发生在 Agent、任务规则、测试网站、共享环境或浏览器证据记录中的哪一处。": "Evaluator errors identify the unmet acceptance rule. The review then checks the task requirement, final answer, network record, and agent trajectory to locate the failure in the agent, task contract, test site, shared environment, or browser evidence capture.", + "查看每个失败结果的核验结论": "View the verified cause for every failed result", + "人工核验用于解释官方失败结果,不修改分数。修复后需要重新运行才能产生新成绩。": "Review explains official failures without changing scores. A fix affects results only after the task is run again.", + "样本覆盖简单、中等和困难任务": "The sample covers short, medium, and long tasks", + "为什么使用 WebArena‑Verified": "Why WebArena-Verified was selected", + "原版 Mind2Web 主要检查离线动作预测。Online‑Mind2Web 和 WebVoyager 使用实时网页,网页内容会变化,部分评分还依赖模型裁判。WebArena‑Verified 可以本地部署、逐题重置,并用最终答案和网络副作用做程序化验收,适合只替换浏览器的对照实验。": "The original Mind2Web mainly tests offline action prediction. Online-Mind2Web and WebVoyager use live pages, where content drifts and some scoring relies on model judges. WebArena-Verified can be self-hosted and reset per task, with programmatic checks of final answers and required network side effects, which suits a browser-only comparison.", + "它的代价是部署较重,环境故障也会影响结果;官方 evaluator 不评价操作路径是否高效,也覆盖不了开放互联网的全部情况。": "Its costs are a heavier deployment and exposure to environment failures. The official evaluator does not score path efficiency or cover the full open web.", + "这组结果能说明什么": "What these results support", + "下一步": "Next step", + "按核验结论分别修复 Agent、评测任务、测试网站和浏览器证据记录,再重新运行受影响的任务;新增浏览器时继续使用同一组任务和验收规则。": "Fix the agent, benchmark tasks, test sites, and browser evidence capture according to the verified causes, then rerun affected tasks. New browsers should use the same task set and acceptance rules.", + "Chromium 与 Moli 作为浏览器后端参与同题对照;Agent、模型、任务、资源上限、站点与 evaluator 保持一致。": "Chromium and Moli participate as browser backends in a same-task comparison; the agent, model, tasks, resource limits, sites, and evaluators are held constant.", + "参评浏览器是本轮唯一实验变量;完整 Agent 栈的其余组成列为统一实验条件。": "The browser participant is the only experimental variable in this run; all other components of the agent stack are shared controls.", + "逻辑结果耗时 P50": "P50 logical-result latency", + "WebArena‑Verified 当前提供 Shopping、Shopping Admin、Reddit、GitLab、Wikipedia、Map 六类环境。本轮 30 题只覆盖前四类;Wikipedia 与 Map 需要额外大型数据卷,因此没有部署,也没有任务进入成绩。": "WebArena-Verified currently provides six environments: Shopping, Shopping Admin, Reddit, GitLab, Wikipedia, and Map. This 30-task run covers only the first four. Wikipedia and Map require additional large data volumes, so they were not deployed and contribute no scored tasks.", + "只保留该题所需站点;env-ctrl 报告 ready 后,再做连续应用健康检查、清浏览器状态、登录测试账号并打开起始页。当前冻结证据保存了代码提交、运行协议和镜像身份;事件流从环境准备完成后开始,因此逐次 reset、ready 与 health 结果未形成独立绑定的准备回执。新版运行协议已要求保存该回执。": "and keeps only the sites required by the task. After env-ctrl reports ready, it performs repeated application-health checks, clears browser state, signs in with test accounts, and opens the start page. The frozen evidence records the code commit, protocol, and image identities; events begin after preparation, so per-execution reset, ready, and health outcomes do not have independently bound preparation receipts. The current protocol now requires that receipt.", + "报告模型与比较口径核验": "Report model and comparison review", + "统计摘要、优胜表达、结果下钻、核验记录和参评浏览器说明。": "Summary statistics, winner presentation, result drill-down, review history, and browser-participant description.", + "统计标签重复表达 P50;比较表缺少方向一致的优胜标记;结果页存在重复失败清单;核验页承担了逐结果清单职责。": "The statistics duplicated P50 terminology; comparison tables lacked directionally consistent winner marks; the results page duplicated the failure inventory; and the review page carried a per-result listing.", + "统一为均值、P50、P95;按指标方向突出唯一优胜者;删除重复失败清单;核验页改为按轮次记录范围、发现、处理和遗留事项。": "Standardized on mean, P50, and P95; marked unique winners by metric direction; removed the duplicate failure inventory; and changed the review page to round-based scope, findings, corrections, and remaining items.", + "当前固定样本没有浏览器单边通过任务;优胜标记主要反映耗时与 Tokens,不改变官方任务成绩。": "The fixed sample has no browser-only pass. Winner marks therefore mainly reflect latency and token usage and do not change official task scores.", + "部署证据与计时边界核验": "Deployment evidence and timing-boundary review", + "本地站点部署、Colima 网络路径、环境准备证据、Agent 与 BrowserTool 计时。": "Local site deployment, Colima network path, environment-preparation evidence, and Agent/BrowserTool timing.", + "环境拓扑与网关含义缺少集中说明;历史总耗时边界描述不准确;逐执行环境准备没有独立回执。": "The environment topology and gateway lacked a consolidated explanation; the historical total-latency boundary was inaccurate; and per-execution environment preparation had no independent receipt.", + "补充部署流程图、端口和镜像身份;区分 Agent 链路与 BrowserTool 墙钟时间;新版 runner 保存环境准备和分阶段计时。": "Added the deployment flow, ports, and image identities; separated Agent-stack and BrowserTool wall time; and made the current runner persist environment preparation and phase timings.", + "本批冻结结果不具备逐执行准备回执,报告仅按现有证据声明可复现边界。": "This frozen run lacks per-execution preparation receipts, so the report states reproducibility boundaries only to the extent supported by existing evidence.", + "运行器与证据完整性核验": "Runner and evidence-integrity review", + "模型响应校验、BrowserTool 清理、HAR 留存、证据目录和恢复执行。": "Model-response validation, BrowserTool cleanup, HAR retention, evidence directories, and resume behavior.", + "异常清理可能遮蔽原始错误;HAR 停止操作缺少幂等保护;异常响应和原始证据目录需要更严格保护。": "Cleanup exceptions could mask primary failures; HAR stop lacked idempotency; and malformed responses and raw-evidence directories needed stricter protection.", + "隔离清理异常、使 HAR 停止幂等、加强模型响应结构校验,并阻止原始浏览器证据进入 Git。": "Isolated cleanup failures, made HAR stop idempotent, strengthened model-response validation, and prevented raw browser evidence from entering Git.", + "统一比较统计与核验记录": "Standardize comparison statistics and review history", + "统一比较统计与可扩展浏览器对比": "Standardize comparison statistics and scalable browser comparison", + "比较摘要统一展示均值、P50、P95;同题比较采用基准浏览器与多选浏览器组合;动作延迟按 action 分卡并只对单次均值判优;结果详情将长 evaluator 证据移出指标卡。Chromium 与 Moli 的官方成绩仍均为 25/30。": "Comparison summaries now show mean, P50, and P95. Same-task comparison uses one baseline browser and multiple comparison browsers. Action latency is grouped by action and ranks only mean per call, while long evaluator evidence has moved out of result KPI cards. Official scores remain 25/30 for both Chromium and Moli.", + "比较摘要统一展示均值、P50、P95,并按指标方向突出唯一优胜者;结果页删除与筛选结果重复的失败清单;核验页改为按轮次记录核验与修订。Chromium 与 Moli 的官方成绩仍均为 25/30。": "Comparison summaries now show mean, P50, and P95 and mark unique winners according to metric direction. The results page no longer duplicates the filterable failure inventory, and the review page records review and revision rounds. Official scores remain 25/30 for both Chromium and Moli.", + "统计摘要": "Summary statistics", + "耗时和 Tokens 的集中趋势与尾部表现未采用统一展示结构。": "Latency and token central tendency and tail behavior did not use a consistent presentation.", + "每个指标统一展示均值、P50、P95;P50 不再以另一个名称重复出现。": "Each metric now shows mean, P50, and P95; P50 is no longer duplicated under another label.", + "只改变统计展示,官方任务分数、分母和证据不变。": "Only statistical presentation changed; official task scores, denominators, and evidence are unchanged.", + "比较表达": "Comparison presentation", + "读者需要逐格判断数值方向。": "Readers previously had to infer metric direction cell by cell.", + "通过数与通过率按更高值标优,耗时与 Tokens 按更低值标优;平局和调用次数保持中性。": "Higher pass counts and rates and lower latency and token usage are marked as better; ties and call counts remain neutral.", + "优胜标记是描述性比较,不改变任务判定。": "Winner marks are descriptive comparisons and do not change task verdicts.", + "页面职责": "Page responsibilities", + "失败清单与结果筛选重复,核验页混入逐结果覆盖表。": "The failure inventory duplicated result filters, and the review page included a per-result coverage table.", + "结果库保留成功/失败筛选和任务详情;核验页只呈现核验轮次、处理和遗留事项。": "The result library retains pass/fail filters and task details; the review page now contains only review rounds, corrections, and remaining items.", + "减少重复内容,不删除结果证据。": "Duplicate presentation was removed without deleting result evidence.", + "多浏览器对比": "Multi-browser comparison", + "同题矩阵会随浏览器数量横向增宽,关系筛选也无法表达一个基准浏览器与多个对比浏览器的组合。": "The same-task matrix widened with every browser, and its relationship filter could not express one baseline browser against multiple comparison browsers.", + "改为基准浏览器单选、对比浏览器多选和明确结果关系;任务行按通过与失败浏览器分组,浏览器标签链接完整结果。": "Changed to one baseline browser, multi-select comparison browsers, and explicit outcome relations. Task rows group passing and failing browsers, with browser labels linking to complete results.", + "页面结构可直接容纳新增浏览器;当前结果和分数不变。": "The page structure now accepts additional browsers directly; current results and scores are unchanged.", + "动作延迟与详情页": "Action latency and result details", + "动作表把三项指标压入浏览器单元格,并对调用量不同的总耗时判优;详情首屏的长错误文本破坏阅读层级。": "The action table compressed three metrics into each browser cell and ranked totals despite unequal call counts; long error text disrupted the hierarchy of result-detail pages.", + "每个 action 使用独立比较卡,只以单次均值判优并将总耗时、调用数保持中性;详情首屏只显示简明失败规则,完整诊断保留在逐规则证据区。": "Each action now has a dedicated comparison card. Only mean per call is ranked, while total latency and call count remain neutral. Result summaries show concise failed-rule labels, with complete diagnostics retained in rule-level evidence.", + "修正比较解释与页面层级,不改变测量值、判分或证据。": "This corrects comparison interpretation and page hierarchy without changing measurements, verdicts, or evidence.", + } +) + +# Report-authored interface copy introduced by the multi-model renderer. Task +# names, task intents, expected/actual values, and execution excerpts are +# deliberately not covered here: they are measured material and may remain in +# their source language. +TEXT.update( + { + "Browser Use 联合多模型能力实测": "Browser Use capability evaluation across model cohorts", + "浏览器 Browser Use 能力对比": "Browser capability comparison on Browser Use tasks", + "先看结论": "Key findings", + "失败根因是报告解读的核心证据": "Failure root causes are core interpretation evidence", + "任务族覆盖": "Task-family coverage", + "单边通过": "One-browser-only passes", + "共同失败": "Both browsers failed", + "不等于共同根因": "Not necessarily the same root cause", + "模型内完整配对": "complete within-model pairs", + "本轮共有 9 个模型内不一致配对,且方向随模型层变化;固定样本不支持对全部 Browser Use 任务作统计外推。": "This run contains nine discordant within-model pairs, and their direction changes across model cohorts. The fixed sample does not support statistical generalization to all Browser Use tasks.", + "浏览器后端是参评选手;模型是分层实验变量,每个模型层内保持 Agent、任务、资源上限、站点与 evaluator 一致。": "Browser backends are the participants. Models are stratification variables; within each model cohort, the agent, tasks, resource limits, sites, and evaluators are held constant.", + "官方任务身份、公开意图与程序化验收合同。运行结果位于结果分析,不混入评测集定义。": "Official task identity, public intent, and programmatic acceptance contract. Measured outcomes appear in Results and are kept separate from the dataset definition.", + "查看本题运行结果": "View measured results for this task", + "评测集页面保持为冻结输入;实测输出、判分和轨迹在结果页查看。": "Dataset pages remain frozen inputs. Measured outputs, verdicts, and trajectories appear on result pages.", + "Chromium 结果": "Chromium results", + "Moli 1.1.3 结果": "Moli 1.1.3 results", + "全部浏览器对照": "Compare all browsers", + "程序自动定位:": "Automatic failure locator:", + "核验根因子类:": "Reviewed root-cause subtype:", + "核验原因:": "Reviewed cause:", + "查看修正依据、处理与影响": "View findings, treatment, and impact", + "每个版本只记录影响读者判断的正式修正;结论、分母、耗时与证据影响直接展示,技术依据按需展开。": "Each release records only formal corrections that affect reader interpretation. Effects on conclusions, denominators, latency, and evidence are shown directly, with supporting details available on demand.", + "判分权威": "Scoring authority", + "物理执行": "Physical execution", + "模型层": "Model cohort", + "正式处理": "Formal treatment", + "未命名版本": "Untitled release", + "未命名核验": "Untitled review", + "官方 Hard": "Official hard subset", + "步起": "minimum steps", + "模板 ID": "template ID", + "公开文件": "Public files", + "没有 evaluator 明细。": "No evaluator details are available.", + "没有可展示的 BrowserTool 调用。": "No BrowserTool calls are available to display.", + "最终回答 JSON": "Final answer JSON", + "完整执行证据 JSON": "Complete execution evidence JSON", + "Evaluator 明细 JSON": "Evaluator details JSON", + "动作轨迹 JSON": "Action trajectory JSON", + "脱敏事件日志 JSON": "Sanitized event log JSON", + "原始证据哈希": "Source-evidence hashes", + "返回任务页": "Return to the task page", + "任务与验收": "Task and acceptance", + "执行轨迹": "Execution trajectory", + "运行与证据": "Runtime and evidence", + "查看任务合同": "View task contract", + "查看任务与验收": "View task and acceptance", + "基准浏览器": "Baseline browser", + "对比浏览器": "Comparison browsers", + "完整配对": "Complete pairs", + "仅基准通过": "Baseline only", + "仅对比通过": "Comparison only", + "模型内浏览器配对结果": "Within-model browser pair outcomes", + "失败根因以当前证据绑定的逐结果核验为准": "Failure causes follow the per-result reviews bound to the current evidence", + "37/37 个失败结果已逐个绑定当前证据并完成核验。": "All 37 failed results are bound to the current evidence and have completed root-cause review.", + "查看逐结果根因与证据边界 →": "View per-result causes and evidence boundaries →", + "220/220 个物理执行均记录 reset、ready、应用健康检查和登录准备状态;事件流随后证明 CowAgent 自主产生 BrowserTool 调用序列。": "All 220 physical executions record reset, readiness, application-health, and sign-in preparation status. The subsequent event streams show CowAgent generating BrowserTool calls autonomously.", + ",只保留该题所需站点;env-ctrl 报告 ready 后,再做连续应用健康检查、清浏览器状态、登录测试账号并打开起始页。本轮 220/220 个物理执行都保存了环境准备回执,并与对应执行记录绑定。": ", keeping only the sites required by the task. After env-ctrl reports ready, the runner performs repeated application-health checks, clears browser state, signs in with test accounts, and opens the start page. All 220 physical executions retain preparation receipts bound to their execution records.", + "本轮 220/220 个物理执行均保存环境准备回执;公开环境摘要保留逐项状态,原始登录态不发布。": "All 220 physical executions retain environment-preparation receipts. The public environment summary preserves each status while authenticated session state remains private.", + "各模型任务成绩": "Task scores by model cohort", + "各模型逻辑结果耗时 P50": "P50 logical-result latency by model cohort", + "判分权威执行的原始证据绑定": "Source-evidence binding for the authoritative execution", + "Chromium 与 Moli 的 Browser Use 能力对比": "Chromium vs Moli on Browser Use tasks", + "评测结果": "Results", + "同一任务上的结果差异": "Outcomes on the same tasks", + "三类任务的覆盖与成绩": "Coverage and results by task type", + "数据是否齐全": "Data completeness", + "Agent 总耗时": "Agent total time", + "同题数": "Matched tasks", + "两边都通过": "Both passed", + "A 通过、B 失败": "A passed, B failed", + "A 失败、B 通过": "A failed, B passed", + "两边都失败": "Both failed", + "原因所在": "Cause area", + "失败结果数": "Failed results", + "具体分类": "Detailed categories", + "说明": "Explanation", + "Agent 决策或操作": "Agent decisions or actions", + "任务或 evaluator": "Task or evaluator", + "测试网站": "Test website", + "浏览器证据记录": "Browser evidence capture", + "共享测试环境": "Shared test environment", + "多项原因": "Multiple causes", + "检索不完整、候选选择错误、走错页面、字段遗漏,或没有在步骤上限内完成。": "Incomplete retrieval, incorrect candidate selection, wrong navigation, missing fields, or failure to finish within the step limit.", + "任务要求与验收条件冲突,或 evaluator 的路由、表单和实现规则有缺陷。": "The task conflicts with its acceptance conditions, or the evaluator has route, form, or implementation defects.", + "自托管网站的依赖、会话或保存接口失败。": "A dependency, session, or save endpoint failed in a self-hosted test site.", + "页面操作可能成功,但网络记录缺少响应正文或 Referer,evaluator 无法确认。": "The page action may have succeeded, but missing response bodies or Referer fields prevented evaluator confirmation.", + "并发运行改写了同一网站状态;运行器现已串行化,历史结果仍按原证据保留。": "Concurrent runs changed shared site state. The runner now serializes execution; historical results remain bound to their original evidence.", + "同一结果同时包含 Agent 遗漏和冻结参考状态差异。": "The same result includes both an Agent omission and a difference from the frozen reference state.", + "任务类型": "Task type", + "测什么": "What it measures", + "信息提取": "Information retrieval", + "页面导航": "Page navigation", + "页面操作": "State-changing action", + "从页面读取、筛选或汇总信息,不修改站点状态": "Read, filter, or summarize page information without changing site state", + "打开符合指定条件的页面或列表": "Open a page or list matching the requested conditions", + "提交表单、修改记录、结账等会改变站点状态的任务": "Submit forms, edit records, check out, or perform other state-changing actions", + "检查项": "Check", + "任务结果": "Task results", + "失败原因核验": "Failure-cause review", + "未解决的证据问题": "Unresolved evidence issues", + "至少成功过一次的任务": "Tasks with at least one successful run", + "保存的实际执行": "Saved executions", + "每个失败结果都有分类和具体原因": "Every failed result has a category and a specific reviewed cause", + "完整性检查没有发现待处理项": "Integrity checks found no unresolved items", + "包含正式执行及按协议发生的一次重试": "Includes score-bearing executions and the one retry permitted by the protocol", + "报告解读": "Interpretation", + "环境与复现": "Environment and reproduction", + "总体成绩和耗时": "Overall scores and latency", + "同一任务上的结果": "Outcomes on the same tasks", + "两个模型的结果不能直接当成模型排名": "The two model runs are not a model ranking", + "失败原因": "Failure causes", + "样本覆盖简单、中等和困难任务": "The sample covers short, medium, and long interaction paths", + "为什么使用 WebArena‑Verified": "Why WebArena-Verified", + "这组结果能说明什么": "What these results support", + "下一步": "Next steps", + "结果编号": "Result ID", + "原因分类": "Cause category", + "核验结论": "Reviewed cause", + "查看每个失败结果的核验结论": "View the reviewed cause for every failed result", + "人工核验用于解释官方失败结果,不修改分数。修复后需要重新运行才能产生新成绩。": "Human review explains official failures without changing scores. A fix requires a new run to produce a new result.", + } +) + + +def _register_report_change_translations() -> None: + """Load explicit English release/review copy from the shared report model. + + Release history changes frequently, so keeping its English prose beside the + Chinese source prevents a new release from depending on a hand-maintained + global exact-string table. + """ + + path = Path(__file__).parents[2] / "research/report-changes.json" + if not path.is_file(): + return + data = json.loads(path.read_text(encoding="utf-8")) + + history_i18n = data.get("history_scope_i18n") or {} + if isinstance(data.get("history_scope"), str) and isinstance(history_i18n.get("en"), str): + TEXT[data["history_scope"]] = history_i18n["en"] + + for collection in ("browser_use_reviews", "browser_use_releases"): + for record in data.get(collection) or []: + english = ((record.get("i18n") or {}).get("en") or {}) + for field in ("title", "status", "scope", "findings", "corrections", "remaining", "summary"): + source, target = record.get(field), english.get(field) + if isinstance(source, str) and isinstance(target, str): + TEXT[source] = target + source_rows, target_rows = record.get("changes") or [], english.get("changes") or [] + if len(source_rows) != len(target_rows): + continue + for source_row, target_row in zip(source_rows, target_rows, strict=True): + if len(source_row) != len(target_row): + continue + for source, target in zip(source_row, target_row, strict=True): + if isinstance(source, str) and isinstance(target, str): + TEXT[source] = target + + +_register_report_change_translations() + +TEXT.update( + { + # Project-authored task labels/rationales. The adjacent official task + # intent remains untranslated measurement input. + "打开一个需要登录的 GitLab 目标页": "Open an authenticated GitLab target page", + "打开一份报表并读取第一个条目": "Open a report and read its first entry", + "打开评论列表并读取一种状态的数量": "Open a review list and read the count for one status", + "搜索一张发票并读取总金额": "Search for an invoice and read its total", + "进入一个论坛、选择最新帖子并点赞": "Enter a forum, choose the newest post, and upvote it", + "进入项目议题列表并应用排序和状态筛选": "Open a project's issue list and apply sorting and status filters", + "把需求转成商品查询并打开匹配详情页": "Turn a requirement into a product query and open the matching detail page", + "进入订单历史,经翻页或排序后读取最早日期": "Open order history and read the earliest date after paging or sorting", + "找到项目贡献者活动页并限定一个日期": "Find a project's contributor activity page and constrain it to one date", + "打开个人项目并检查星标与项目标识": "Open a personal project and inspect its star and project identifier", + "按标题和更新时间筛选个人议题,再读取状态": "Filter personal issues by title and update time, then read status", + "按库存量筛选后台表格并收集 SKU": "Filter an admin table by inventory and collect SKUs", + "按状态筛选订单,打开最新一单并读取总额": "Filter orders by status, open the newest order, and read its total", + "筛选并排序订单,再打开最新的匹配客户": "Filter and sort orders, then open the newest matching customer", + "打开个人资料设置,修改简介并保存": "Open profile settings, edit the biography, and save", + "找到最新论坛帖子,并按作者和票数条件检查多条评论": "Find the newest forum post and inspect multiple comments by author and vote criteria", + "检查一整年的分页订单并汇总数量和支出": "Inspect a full year of paginated orders and total their count and spend", + "检查十个排序帖子,并根据描述归类推荐内容": "Inspect ten sorted posts and classify recommendations from their descriptions", + "检查十个排序帖子,并从合格描述中提取两个字段": "Inspect ten sorted posts and extract two fields from qualifying descriptions", + "应用日期和状态条件,并汇总五个月度数量": "Apply date and status conditions and total five monthly counts", + "应用日期和状态条件,并汇总十一个月度数量": "Apply date and status conditions and total eleven monthly counts", + "搜索商品、检查结果页并计算价格区间": "Search products, inspect the result page, and calculate a price range", + "结合搜索词排名与独立的商品可用性检查": "Combine query ranking with an independent product-availability check", + "检查日期区间内的订单、归类商品并汇总行项目金额": "Inspect orders in a date range, classify products, and total line-item amounts", + "筛选库存、检查变体并返回成对商品字段": "Filter inventory, inspect variants, and return paired product fields", + "排序已完成订单、检查两份详情并合计付款": "Sort completed orders, inspect two details, and total payments", + "检查两组各四笔订单,并计算付款差额": "Inspect two groups of four orders and calculate the payment difference", + "找到最新完成订单,并规范化、排序多个行项目": "Find the latest completed order, then normalize and sort multiple line items", + "清空购物车,按预算和评分筛选品类,选择商品并结账": "Empty the cart, filter a category by budget and rating, choose a product, and check out", + "定位项目文件、修改标题、提交到默认分支并验证": "Locate a project file, change its title, commit to the default branch, and verify", + # Standalone interpretation article. + "报告解读 · Browser Use 评测": "Interpretation · Browser Use evaluation", + "从 25 比 25 读出下一轮该怎么做": "What the 25-to-25 result implies for the next run", + "完整文章与首页共用同一份测量数据;另提供无 JavaScript 的独立导出版。": "The full article and home page use the same measured data. A standalone JavaScript-free export is also available.", + "独立导出": "Standalone export", + "打开独立文章": "Open standalone article", + "下载论点证据": "Download claim evidence", + "Browser Use 实测解读": "Browser Use evaluation interpretation", + "25 比 25 不是“内核等价”:当前样本没有产生分歧": "25 to 25 does not mean browser-core equivalence: this sample produced no disagreement", + "同一 CowAgent 与同一模型替换浏览器后,30 个配对任务没有一个出现单边胜出。这个结果证明对照流程跑通,却没有为内核优劣提供方向性证据。": "With the same CowAgent and model and only the browser changed, none of the 30 paired tasks produced a one-browser-only win. The controlled workflow ran successfully, but the result provides no directional evidence about browser-core superiority.", + "测量口径:WebArena‑Verified 1.2.3;每题×浏览器一个最终逻辑结果;官方确定性 evaluator;无 LLM judge。": "Measurement unit: WebArena-Verified 1.2.3; one final logical result per task-browser pair; official deterministic evaluators; no LLM judge.", + "一、最重要的结果不是平分,而是零个不一致配对": "1. The key result is not the tie but the absence of discordant pairs", + "Chromium 与 Moli 分别通过 25/30 和 25/30。逐题配对后,25 题共同通过、5 题共同失败,两个浏览器单边通过都是 0。由于固定样本里没有不一致任务,这批数据不能估计浏览器的方向性效应,也不对 WebArena‑Verified 全集作统计外推。": "Chromium and Moli each passed 25/30. Pairing by task gives 25 joint passes, five joint failures, and zero one-browser-only passes. Because this fixed sample contains no discordant tasks, it cannot estimate a directional browser effect or support statistical generalization to the full WebArena-Verified set.", + "表 1|30 任务的逐题最终结果配对。": "Table 1 | Paired final results for 30 tasks.", + "可支持的判断": "Supported conclusion", + "当前组合可完成": "Current stack can complete", + "严谨表述只能是“本轮没有观察到成功率差异”,不能写成“两种内核等价”。30 题对小效应检出力有限;更关键的是,样本没有诱发内核分歧。": "The defensible statement is only that this run observed no success-rate difference, not that the browser cores are equivalent. Thirty tasks have limited power for small effects; more importantly, this sample did not elicit browser-core disagreement.", + "二、共同失败是待核验线索,不是已证实根因": "2. Shared failures are leads to verify, not confirmed root causes", + "5 个共同失败任务均未满足官方验收合同。下表只陈述官方expected与actual结果的差异,不把差异直接归因给 Agent、浏览器或站点。": "All five shared-failure tasks missed the official acceptance contract. The table states only the difference between official expected and actual results and does not directly attribute it to the agent, browser, or site.", + "表 2|任务详情公开官方合同、actual回答、逐规则判定与执行轨迹。": "Table 2 | Task details expose the official contract, actual answer, per-rule verdicts, and execution trajectory.", + "公开诊断": "Published diagnosis", + "官方expected包含多本书;四次执行的actual回答都只包含一本。": "The official expected value contains multiple books; the actual answer in all four executions contains only one.", + "actual回答中的价格范围与该题官方目标商品范围不一致。": "The price range in the actual answer differs from the official target-product range.", + "actual订单汇总与官方expected值不一致。": "The actual order aggregate differs from the official expected value.", + "actual回答为空列表,而官方expected为非空项目列表。": "The actual answer is an empty list, while the official expected value is a non-empty project list.", + "官方网络副作用合同未通过;购物车最终状态未满足题目要求。": "The official network-side-effect contract failed; the final cart state did not satisfy the task.", + "这五题在本轮没有任何successful trajectory,因此当前站点快照的可完成性也未被本报告独立证明。下一步应先做不参与计分的人工或 oracle 可完成性核验,再区分 Agent、工具抽象、步数上限、站点状态和浏览器因素。": "None of these five tasks has a successful trajectory in this run, so the report does not independently establish solvability on the current site snapshot. The next step is a non-scoring human or oracle solvability check, followed by separating agent, tool abstraction, step-limit, site-state, and browser factors.", + "三、简单、中等、困难都覆盖了,但“难度”只是操作长度": "3. Easy, medium, and hard are covered, but difficulty means operation length", + "WebArena‑Verified 没有官方 easy/medium/hard 三档。本项目在运行前按最少可见 UI 操作数注册为 1–4、5–7、8+ 三档,配额 5/10/15。它控制操作链覆盖,不等同于推理难度。": "WebArena-Verified has no official easy/medium/hard tiers. Before execution, this project registered tasks by minimum visible UI actions as 1-4, 5-7, and 8+, with quotas of 5/10/15. This controls operation-chain coverage and is not equivalent to reasoning difficulty.", + "表 3|各层级最终通过数/任务数。": "Table 3 | Final passes/tasks by tier.", + "本项目分层": "Project tier", + "表 4|四个可重置自托管站点。": "Table 4 | Four resettable self-hosted sites.", + "四、选择 WebArena‑Verified 是为了归因,不是因为它最流行": "4. WebArena-Verified was chosen for attribution, not popularity", + "Mind2Web 家族采用面更广,但原版主要评离线动作预测;Online‑Mind2Web 和 WebVoyager贴近实时网页,却引入网页漂移和模型裁判。WebArena‑Verified 可以自托管、逐题重置,并用结构化最终答案和必要网络副作用做程序化验收,更适合“只替换浏览器”的 A/B。": "The Mind2Web family has broader adoption, but its original version mainly evaluates offline action prediction. Online-Mind2Web and WebVoyager are closer to live browsing but introduce page drift and model judging. WebArena-Verified is self-hosted, resettable per task, and programmatically evaluates structured final answers and required network side effects, making it better suited to a browser-only A/B test.", + "代价也明确:部署重,环境兼容本身会成为工程变量;官方 evaluator 不衡量路径效率,也不能覆盖开放互联网泛化。": "The tradeoff is substantial deployment, with environment compatibility becoming an engineering variable. The official evaluator does not measure path efficiency or open-web generalization.", + "五、下一轮应主动寻找内核敏感样本": "5. The next run should actively seek browser-sensitive tasks", + "保留 CowAgent、模型、reasoning=low 和每题最多两次执行的固定协议,扩大稳定可验收的配对样本;新增长页面、上传下载、复杂编辑器、弹窗、多标签页等更可能诱发内核差异的任务。实时网站另开 Online‑Mind2Web / WebVoyager 泛化轨道,LLM judge 与人工抽检单列。": "Keep the fixed CowAgent, model, reasoning=low, and at-most-two-executions protocol while expanding the stable, evaluable paired sample. Add long pages, uploads/downloads, complex editors, dialogs, multiple tabs, and other tasks more likely to elicit browser differences. Use a separate Online-Mind2Web / WebVoyager generalization track for live sites, with LLM judging and human spot checks reported separately.", + "决策边界": "Decision boundary", + "本轮支持“流程可比、当前样本无方向性差异”;不支持“内核等价”“Moli 已全面替代 Chromium”或“端到端耗时就是纯内核速度”。Astra 获权后必须形成独立运行。": "This run supports a comparable workflow and no directional difference in the current sample. It does not support browser-core equivalence, full replacement of Chromium by Moli, or interpreting end-to-end latency as pure browser-core speed. Astra requires a separate run once access is available.", + } +) + +TEXT.update( + { + "四种配置的总体结果": "Overall results across four configurations", + "四种配置总览": "Four-configuration overview", + "四种配置 · 单次调用墙钟时间": "Four configurations · per-call wall time", + "全部配置 · 单次调用墙钟时间": "All configurations · per-call wall time", + "全部配置中单次均值最低": "Lowest mean per-call latency across all configurations", + "按模型分层 · 单次调用墙钟时间": "Within each model cohort · per-call wall time", + "同模型浏览器中单次均值最低": "Lowest per-call mean within this model cohort", + "同模型浏览器中通过数最高": "Highest pass count within this model cohort", + "同模型浏览器中通过率最高": "Highest success rate within this model cohort", + "只在同一模型的两个浏览器之间标记唯一较优值;相同值不标。跨模型批次不判优。": "A unique better value is marked only between browsers in the same model cohort. Ties are unmarked, and model cohorts are not ranked against each other.", + "四种配置均使用同一组 90 题,但两个模型层的运行日期和 browser‑eval 提交不同。": "All four configurations use the same 90 tasks, but the model cohorts ran on different dates and browser-eval commits.", + "每个动作按模型分层比较两个浏览器的单次调用墙钟均值,只标记层内唯一最低值。总耗时和调用数受 Agent 行为量影响,仅用于解释工作量,不参与快慢判定。": "For each action, the two browsers are compared within each model cohort by mean per-call wall time, and only a unique cohort minimum is marked. Total time and call count depend on agent workload and provide context rather than a speed ranking.", + "每个动作把全部模型×浏览器配置放在一起比较单次调用墙钟均值,只标记全局唯一最低值。总耗时和调用数受 Agent 行为量影响,仅用于解释工作量,不参与快慢判定。": "For each action, every model-by-browser configuration is compared together by mean per-call wall time, and only a unique global minimum is marked. Total time and call count depend on agent workload and provide context rather than a speed ranking.", + "四种配置中通过数最高": "Highest pass count across four configurations", + "四种配置中通过率最高": "Highest pass rate across four configurations", + "四种配置中单次均值最低": "Lowest mean per-call latency across four configurations", + "速度与浏览器资源": "Speed and browser resources", + "CPU 时间": "CPU time", + "平均 CPU 单核占用": "Mean single-core CPU utilization", + "时间加权平均内存": "Time-weighted mean memory", + "峰值内存": "Peak memory", + "网络接收量": "Network received", + "网络发送量": "Network transmitted", + "仅标记四种配置中数值唯一最低的一项。CPU 时间与网络总量累计得到逻辑结果所用的全部执行;内存来自 Browser Use 执行阶段的浏览器容器采样。": "Only a unique lowest value across the four configurations is marked. CPU time and network totals include every execution used by the logical result; memory comes from browser-container samples during Browser Use execution.", + "四种配置统一比较,只标记唯一最优值;相同值不标。通过数越高越好,耗时和 Tokens 越低越好。": "All four configurations are compared together, and only a unique best value is marked. Ties are unmarked. Higher pass counts are better; lower latency and token use are better.", + "四种完整配置统一比较,每个统计项只标记唯一最优值;相同值不标。这里比较模型 × 浏览器整套配置,不能把差值单独归因于浏览器。": "All four complete configurations are compared together, and only a unique best value is marked for each statistic; ties are unmarked. This compares complete model-by-browser configurations, so a difference cannot be attributed to the browser alone.", + "审查口径:": "Review scope:", + "438/438 次物理执行完成程序完整性核验,75/75 个最终失败完成逐条根因复核;成功结果没有逐条语义盲审,人工复核也没有改写官方 evaluator 分数。": "All 438 physical executions passed programmatic integrity review, and all 75 final failures received case-by-case root-cause review. Successful results did not receive case-by-case blinded semantic review, and human review did not rewrite official evaluator scores.", + "耗时和 Tokens 如何统计": "How latency and tokens are measured", + "Agent 链路累计耗时把取得逻辑结果所发生的执行相加,包含登录、模型推理、调度、页面等待、BrowserTool 与收尾,不包含此前的 Docker 站点重置。BrowserTool 累计耗时是所有 browser 工具调用的墙钟时间之和,包含工具封装、CDP 往返和页面等待。Tokens 同样累计正式结果包含的物理执行。": "Cumulative Agent-path latency sums the executions used to obtain the logical result. It includes sign-in, model inference, orchestration, page waits, BrowserTool calls, and cleanup, but excludes the preceding Docker site reset. Cumulative BrowserTool latency sums the wall time of all browser-tool calls, including tool-wrapper, CDP round-trip, and page-wait overhead. Tokens likewise sum the physical executions included in the official result.", + "结果分析": "Results analysis", + "完整 90 题结果入口": "Complete 90-task results entry", + "四种配置的通过率、耗时、Tokens 与浏览器资源总表已放在首页,避免在分析页重复。这里直接进入同题矩阵和逐结果失败诊断。": "The home page contains the consolidated pass-rate, latency, token, and browser-resource table for all four configurations. This page proceeds directly to the same-task matrix and per-result failure diagnosis.", + "查看首页核心指标": "View core metrics on the home page", + "四种配置先在同一张表和同一任务矩阵中比较;失败结果可直接打开诊断,查看发生位置、程序报错、复核根因、轨迹、画面和日志。": "The four configurations are compared in one summary table and one same-task matrix. Open any failed result to inspect its location, exact program error, reviewed root cause, trajectory, screenshots, and logs.", + "同题全配置对比": "Same-task comparison across all configurations", + "每行是一道任务,四列分别对应模型×浏览器配置;可筛出结果有差异、全部通过或全部失败的任务。": "Each row is one task, with four model-by-browser configuration columns. Filter for differing outcomes, all-pass tasks, or all-fail tasks.", + "同题全配置矩阵": "Same-task all-configuration matrix", + "每行是同一任务,每列是一个模型×浏览器配置。新增配置只增加列。": "Each row is one task and each column is one model-by-browser configuration. New configurations add columns only.", + "关系": "Outcome pattern", + "结果有差异": "Outcomes differ", + "全部通过": "All pass", + "全部失败": "All fail", + "排除已证实外部干扰后的诊断子集": "Diagnostic subset after removing confirmed external interference", + "官方 90 题结果保持不变。为了看清浏览器和模型差异,下表暂时排除已核验为评测合同、站点、共享环境或参考状态问题的整道任务。": "The official 90-task results remain unchanged. To isolate browser and model differences, this table temporarily removes every task confirmed to have a task-contract, site, shared-environment, or reference-state problem.", + "这是事后诊断口径,不是官方新分数。它按任务统一剔除任一配置已证实受任务/evaluator、站点、共享环境或参考状态干扰的任务,保证四个配置分母一致。": "This is a post-hoc diagnostic view, not a new official score. It removes a whole task across all configurations when any result is confirmed to be affected by task/evaluator, site, shared-environment, or reference-state interference, keeping denominators aligned.", + "失败诊断": "Failure diagnosis", + "发生位置": "Location", + "程序判定": "Program verdict", + "程序报错": "Program error", + "程序没有提供更具体的错误文本。": "The program did not provide a more specific error message.", + "复核结论": "Reviewed conclusion", + "复核结论:": "Reviewed conclusion:", + "负责范围": "Owning scope", + "失败动作": "Failed action", + "验收前最后一步": "Last step before evaluation", + "查看诊断": "Open diagnosis", + "查看期望与实际值": "View expected and actual values", + "查看页面画面": "View page screenshots", + "查看日志与完整证据": "View logs and complete evidence", + "查看完整日志": "View complete logs", + "关闭": "Close", + "尚未完成根因复核。": "Root-cause review is not complete.", + "当前尚无已核验根因。": "No verified root cause is available yet.", + "打开原始 JSON": "Open raw JSON", + "下载原始 JSON": "Download raw JSON", + "在页面中查看完整执行证据": "View complete execution evidence on this page", + "浏览器资源指标": "Browser resource metric", + "浏览器 CPU 时间": "Browser CPU time", + "结束时新增常驻内存": "Additional resident memory at completion", + "浏览器网络记录存在": "Browser network record exists", + "浏览器执行轨迹存在": "Browser execution trace exists", + "Agent 最终回答存在": "Agent final response exists", + "官方判分明细存在": "Official evaluation details exist", + "Agent 事件流存在": "Agent event stream exists", + "环境准备记录存在": "Environment preparation record exists", + "判分明细可读取": "Evaluation details are readable", + "最终回答可读取": "Final response is readable", + "最终回答与运行记录一致": "Final response matches the run record", + "报告分数与官方判分一致": "Reported score matches official evaluation", + "报告状态与官方判分一致": "Reported status matches official evaluation", + "官方 evaluator 版本已记录": "Official evaluator version is recorded", + "官方 evaluator 文件校验一致": "Official evaluator checksum matches", + "官方题集文件校验一致": "Official dataset checksum matches", + "模型请求次数已完整计入": "All model request attempts are accounted for", + "仅保留哈希与字节数(含登录态)": "Hash and byte count only (session-sensitive)", + "已脱敏公开副本": "Sanitized public copy", + "公开原文件": "Public original file", + "按证据清单处理": "Handled according to the evidence inventory", + "WebArena‑Verified 1.2.3 · CowAgent · 模型推理强度固定为低 · 官方程序化 evaluator": "WebArena-Verified 1.2.3 · CowAgent · model reasoning effort fixed to low · official programmatic evaluator", + "失败与干扰项": "Failures and interference", + "在四种配置中统一比较并只标记唯一最优值;相同值不标。通过数越高越好,耗时和 Tokens 越低越好。": "Values are compared across all four configurations, and only a unique best value is marked. Higher pass counts are better; lower latency and token usage are better.", + "查看全部配置的逐题矩阵 →": "View the task-by-task matrix for all configurations →", + "耗时统计包含什么": "What the latency statistics include", + "Agent 总耗时包含模型推理、页面等待和浏览器操作;BrowserTool 耗时是全部浏览器工具调用的墙钟时间之和,仍包含工具封装与 CDP 往返,不是纯内核 CPU 时间。": "Agent latency includes model inference, page waits, and browser operations. BrowserTool latency is the sum of wall-clock time across browser-tool calls; it still includes tool-wrapper and CDP round-trip overhead and is not pure browser-core CPU time.", + "Agent / 模型行为": "Agent / model behavior", + "全部 76 题的速度与浏览器资源": "Performance and browser resources across all 76 tasks", + "均值、P50、P95 都在四种完整配置中统一比较。CPU、内存和网络只计浏览器容器,不含 CowAgent 或模型服务;模型会改变动作数量和页面停留时间,因此这里比较整套配置,不能单独归因于浏览器内核。": "Mean, P50, and P95 are compared across all four complete configurations. CPU, memory, and network measurements cover only the browser container, not CowAgent or the model service. Because the model can change the action count and time spent on pages, these are whole-configuration measurements and cannot be attributed to the browser core alone.", + "每项只标记四种配置中数值唯一最低者。CPU 时间与网络总量累计逻辑结果使用的全部执行;内存来自 Browser Use 执行阶段的浏览器容器采样。": "Only a unique lowest value across the four configurations is marked for each statistic. CPU time and network totals include every execution used by a logical result; memory comes from browser-container samples during Browser Use execution.", + "浏览器证据接口": "Browser evidence interface", + "原因范围": "Cause scope", + "本轮处理": "Treatment in this run", + "修复与验证": "Fix and verification", + "保留": "Retain", + "保留并修复": "Retain and fix", + "排除并修复": "Exclude and fix", + "排除,修复已完成": "Exclude; fix completed", + "排除并重建": "Exclude and rebuild", + "属于当前被测组合;按轨迹定位检索、选择或提交错误。": "Part of the configuration under test; use the trajectory to locate retrieval, selection, or submission errors.", + "修正验收合同,通过独立预检后重跑四种配置。": "Correct the acceptance contract, pass an independent preflight, then rerun all four configurations.", + "修复依赖、会话或保存链路,通过读写预检后重跑。": "Fix the dependency, session, or save path, then rerun after read/write preflight checks.", + "这是浏览器集成能力的一部分;补齐响应正文或 Referer 采集后重跑。": "This is part of browser integration capability; complete response-body or Referer capture and rerun.", + "跨进程串行锁已经上线;受污染任务仍需重跑。": "The cross-process serialization lock is deployed; contaminated tasks still require reruns.", + "重新冻结参考状态,再重跑以区分 Agent 错误。": "Refreeze the reference state, then rerun to distinguish Agent errors.", + "参考状态与 Agent 混合": "Reference state mixed with Agent behavior", + "每个动作同时比较四种配置的单次调用墙钟均值,只标记唯一最低值。总耗时和调用数受 Agent 行为量影响,仅用于解释工作量,不参与快慢判定。": "Each action compares mean per-call wall time across all four configurations and marks only a unique minimum. Total time and call count depend on Agent workload, so they explain workload but are not ranked.", + "默认显示失败。点击“查看诊断”可先在当前页核对发生位置、程序报错、复核根因和处理方案,再进入完整轨迹、画面与日志。": "Failures are shown by default. Select Open diagnosis to inspect the location, exact program error, reviewed root cause, and treatment before opening the full trajectory, screenshots, and logs.", + "该结果通过全部官方程序化验收;下方可核对任务合同、执行轨迹和运行证据。": "This result passed every official programmatic evaluator. The task contract, execution trajectory, and runtime evidence are available below.", + "失败位置": "Failure stage", + "人工复核根因": "Reviewed root cause", + "页面画面": "Page frames", + "查看 evaluator 断言": "View evaluator assertions", + "查看失败附近轨迹": "View trajectory near the failure", + "查看最后页面画面": "View final page frames", + "下载执行日志": "Download execution log", + "核验依据:": "Review evidence:", + "对比口径:": "Comparison treatment:", + "该结果已确认含非模型/浏览器干扰,官方分数保留,但暂不纳入干净子集。": "This result contains confirmed interference outside the model/browser variables. Its official score is retained, but it is temporarily excluded from the clean subset.", + "失败执行的最后页面画面": "Final page frames from the failed execution", + "下图是判分权威执行的最后页面画面,用于核对失败前的可见状态;它们不一定正好对应错误发生的瞬间。": "These are the final page frames from the scoring execution. They help verify visible state before failure but may not capture the exact instant the error occurred.", + "是否属于目标变量": "Target variable?", + "处理": "Treatment", + "运行配置": "Run configuration", + "复核根因": "Reviewed root cause", + "负责对象与处理": "Owner and treatment", + "证据": "Evidence", + "诊断": "Diagnosis", + "画面": "Frames", + "日志": "Logs", + "详情": "Details", + "查看该浏览器的全部结果": "View all results for this browser", + "完整结果只保留一份。下方入口打开结果分析页,并预先筛选当前浏览器;失败诊断、轨迹、页面画面与日志均从该清单进入。": "The report keeps one result inventory. The link below opens it with this browser preselected; failure diagnoses, trajectories, page frames, and logs are all reached from that inventory.", + "打开筛选后的结果清单 →": "Open the filtered result inventory →", + "WebArena-Verified 本地部署、CowAgent 自主操作、双浏览器对照与程序化验收流程": "WebArena-Verified local deployment, autonomous CowAgent operation, browser comparison, and programmatic evaluation" + } +) + + +PHRASES = { + "步骤预算耗尽": "Step budget exhausted", + "导航语义错误": "Navigation-semantics error", + "检索覆盖不足": "Incomplete retrieval coverage", + "商品选择错误": "Incorrect product selection", + "候选比较错误": "Candidate-comparison error", + "提交字段错误": "Submission-field error", + "检索策略错误": "Retrieval-strategy error", + "流程未完成": "Incomplete workflow", + "结果去重遗漏": "Result-deduplication failure", + "可恢复错误后提前终止": "Premature stop after recoverable error", + "批量操作遗漏": "Incomplete batch workflow", + "提交字段遗漏": "Submission-field omission", + "路由验收缺陷": "Route-acceptance defect", + "evaluator 实现错误": "Evaluator implementation defect", + "任务与验收矛盾": "Instruction-evaluator contradiction", + "表单验收缺陷": "Form-acceptance defect", + "任务状态与验收矛盾": "State-evaluator contradiction", + "站点保存故障": "Site save failure", + "站点依赖故障": "Site dependency failure", + "站点会话故障": "Site session failure", + "网络证据缺失": "Missing network evidence", + "网络证据匹配缺陷": "Network-evidence matching defect", + "响应证据缺失": "Missing response evidence", + "共享环境并发污染": "Shared-environment concurrency contamination", + "检索遗漏与参考状态差异": "Retrieval omission and reference-state drift", + "保留为目标变量;按轨迹修正策略或工具合同": "Keep as a target-variable signal; improve the strategy or tool contract from the trajectory", + "评测集 / evaluator": "Benchmark / evaluator", + "修正合同并重跑;当前仅纳入官方分数": "Fix the contract and rerun; retain only the official score for now", + "修复依赖或保存链路,通过写入预检后重跑": "Fix the dependency or save path, then rerun after a write preflight", + "保留为浏览器集成信号;补齐 HAR 后重跑": "Keep as a browser-integration signal; rerun after completing HAR capture", + "运行环境": "Runtime environment", + "跨进程锁已上线;受污染任务待重跑": "The cross-process lock is implemented; contaminated tasks await rerun", + "任务与 Agent": "Task and Agent", + "先冻结参考状态,再重跑区分 Agent 错误": "Freeze reference state, then rerun to distinguish Agent errors", + "自托管站点": "Self-hosted site", + "浏览器证据接口": "Browser evidence interface", + "保留为对比信号;通过轨迹修正检索、选择和提交策略。": "Keep as a comparison signal; use trajectories to improve retrieval, selection, and submission.", + "任务 / evaluator": "Task / evaluator", + "修正任务合同或 evaluator,验证后重跑;重跑前不进入干净子集。": "Fix the task contract or evaluator, validate it, and rerun; exclude it from the clean subset until then.", + "修复依赖、会话或保存链路,增加任务级写入预检后重跑。": "Fix the dependency, session, or save path and rerun after adding a task-level write preflight.", + "保留为浏览器集成信号;补齐响应正文或 Referer 采集后重跑。": "Keep as a browser-integration signal; rerun after completing response-body or Referer capture.", + "跨进程串行锁已上线;受污染任务待重跑。": "The cross-process serialization lock is implemented; contaminated tasks await rerun.", + "参考状态 + Agent": "Reference state + Agent", + "先重新冻结参考状态,再重跑区分 Agent 错误。": "Refreeze reference state, then rerun to distinguish Agent errors.", + "该执行没有可公开的 trace 页面画面;错误日志与执行证据仍可下载。": "This execution has no publishable trace page frame; the error log and execution evidence remain available.", + "规则 ": "Rule ", + " 秒": " s", + " 次 BrowserTool 调用": " BrowserTool calls", + " 次调用 / ": " calls / ", + " 次传输尝试": " transport attempts", + "总耗时 ": "Total ", + " 次调用": " calls", + "查看 ": "View ", + " 步轨迹": "-step trajectory", + "成功轨迹": "successful trajectory", + "任务 #": "Task #", + "个试次": " attempts", + "个任务": " tasks", + "个失败结果": " failed results", + "实际": "actual", + "期望": "expected", + "环境准备后、Agent 读取前": "After environment preparation and before the Agent read", + "运行结束后的 evaluator 验收": "Evaluator acceptance after execution", + "运行结束后的网络证据验收": "Network-evidence acceptance after execution", + "最终回答/任务状态": "Final answer/task state", + "BrowserTool 执行异常": "BrowserTool execution error", + "步后达到步骤上限": " steps: step limit reached", + "步后的最终回答/任务状态": " steps: final answer/task state", + "步后的站点响应/保存": " steps: site response/save", + "最后页面画面": "Final page frame", + "该逻辑结果包含": "This logical result contains", + "次物理执行。标记为判分权威的执行决定最终得分;累计耗时、tokens 与 BrowserTool 指标是下列各次执行之和。": "physical execution(s). The scoring-authority execution determines the final score; cumulative latency, tokens, and BrowserTool metrics sum all executions shown below.", + "物理执行": "Physical execution", + " · 判分权威": " · scoring authority", + "判分": "Score", + "Agent 链路": "Agent path", + "模型 tokens": "model tokens", + "(完整内容见脱敏事件日志)": "(see the sanitized event log for the complete content)", + "个物理执行均保存环境准备回执;公开环境摘要保留逐项状态,原始登录态不发布。": "physical executions retain environment-preparation receipts. The public environment summary preserves each status while authenticated session state remains private.", + "个物理执行均记录 reset、ready、应用健康检查和登录准备状态;事件流随后证明 CowAgent 自主产生 BrowserTool 调用序列。": "physical executions record reset, readiness, application-health, and sign-in preparation status. The subsequent event streams show CowAgent generating BrowserTool calls autonomously.", + "个逻辑结果及其物理执行、下载路径和公开文件哈希。": "logical results with their physical executions, download paths, and public-file hashes.", + "暂无发布记录。": "No release has been recorded.", +} + +TEXT.update( + { + "90 题跑完了,但结论不能只看 73、71、70": "Ninety tasks are complete, but 73, 71, and 70 do not tell the whole story", + "90 题、四种配置:相近成绩背后的不同原因": "Ninety tasks and four configurations: different causes behind similar scores", + "四种配置的官方分数很接近;75 个零分经复核后分散在多个责任层,不能统一归给 Agent 或浏览器。": "The four official scores are close. Review placed the 75 zeros across several owning layers rather than assigning them all to the agent or browser.", + "WebArena‑Verified 1.2.3 · CowAgent · reasoning=low · 官方程序化 evaluator": "WebArena-Verified 1.2.3 · CowAgent · reasoning=low · official programmatic evaluator", + "独立阅读版": "Standalone reading edition", + "返回交互报告": "Return to the interactive report", + "论点与证据数据": "Claims and evidence data", + "GPT‑6 Astra 没有在首次运行中输给 GPT‑5.6 Sol": "GPT-6 Astra did not trail GPT-5.6 Sol on the first attempt", + "Astra 首次通过 71/90,Sol 为 70/90": "Astra passed 71/90 on the first attempt; Sol passed 70/90", + "Sol × Chromium 有 3 个重试任务恢复成功,最终到 73/90。Astra 的 71/90 并非首次表现落后。": "Three Sol × Chromium tasks recovered on retry, raising its final score to 73/90. Astra's 71/90 does not reflect weaker first-attempt performance.", + "两层模型共有 9 个单边通过配对": "Nine task pairs were one-sided across the two model cohorts", + "14 题含已确认的外部干扰": "Fourteen tasks contain confirmed external interference", + "干扰来自验收合同、站点、共享环境或参考状态。官方分数保留,另用 76 题事后诊断子集观察目标变量。": "The interference comes from acceptance contracts, sites, shared state, or reference state. Official scores remain unchanged; a post-hoc 76-task subset is used only for diagnosis.", + "Astra 在两个浏览器上都是 71/90;Sol 都是 70/90。Sol × Chromium 有 3 个重试任务恢复成功,最终才到 73/90。": "Astra scored 71/90 with both browsers on the first attempt; Sol scored 70/90. Sol × Chromium reached 73/90 only after three failed tasks recovered on retry.", + "Moli 与 Chromium 的总数接近,不代表执行等价": "Similar totals for Moli and Chromium do not mean equivalent execution", + "Astra 层有 69 题共同通过、2 题各自单边通过;Sol 层是 69 题共同通过、4 题仅 Chromium 通过、1 题仅 Moli 通过。": "Under Astra, both browsers passed 69 tasks and each passed two tasks that the other failed. Under Sol, both passed 69, Chromium alone passed four, and Moli alone passed one.", + "官方 90 题成绩不是干净的模型或内核排名": "The official 90-task score is not a clean model or browser-core ranking", + "14 题已证实受验收合同、站点、共享环境或参考状态干扰。官方分数保留,另用 76 题事后诊断子集观察目标变量。": "Fourteen tasks have confirmed interference from acceptance contracts, sites, shared state, or reference state. Their official scores remain unchanged; a post-hoc 76-task subset is shown only for diagnosis.", + "先把分数读对": "Read the score correctly first", + "首次成功与重试恢复": "First-attempt success and retry recovery", + "报告的主成绩按预先固定的规则取最后一次执行,失败后最多重试一次。因此首次通过率回答“第一次能否完成”,最终通过率还包含“失败后能否恢复”。": "The report's official score uses the last execution under the preregistered rule, with at most one retry after failure. First-attempt success answers whether a task worked immediately; final success also reflects recovery after failure.", + "首次 70/90": "First attempt 70/90", + "首次 71/90": "First attempt 71/90", + "首次通过 70/90": "First-attempt passes: 70/90", + "首次通过 71/90": "First-attempt passes: 71/90", + "失败后重试 20 题": "20 failed tasks retried", + "失败后重试 19 题": "19 failed tasks retried", + "恢复 +3 题": "3 tasks recovered", + "恢复 0 题": "No task recovered", + "首次通过": "First-attempt passes", + "重试任务": "Retried tasks", + "重试恢复": "Recovered on retry", + "官方最终通过": "Official final passes", + "为什么看起来 Astra 更低?": "Why does Astra appear lower?", + "只有在 Chromium 的最终分数上,Astra 的 71 低于 Sol 的 73。这 2 题差距不是首次能力差:Sol 首次只有 70,随后恢复 3 题;Astra 首次 71,没有重试恢复。Astra 的 Task 77 还受共享环境污染。": "Astra trails Sol only in the final Chromium score, 71 versus 73. That two-task gap did not exist on the first attempt: Sol started at 70 and recovered three tasks, while Astra started at 71 and recovered none. Astra's Task 77 was also contaminated by shared-state concurrency.", + "两个模型层在不同日期、不同 browser‑eval 提交上运行(GPT-5.6 Sol 51a37526、GPT-6 Astra bed8469e)。任务、CowAgent 提交、浏览器镜像与 evaluator 数据相同,但这仍不是严格的模型 A/B。": "The model cohorts ran on different dates and browser-eval commits (GPT-5.6 Sol 51a37526; GPT-6 Astra bed8469e). They share tasks, CowAgent commit, browser images, and evaluator data, but this is still not a strict model A/B test.", + "总数相近,不是同一批任务相近": "Similar totals can hide different task outcomes", + "按同题比较浏览器": "Compare browsers task by task", + "浏览器对比必须留在同一模型层内。汇总数会隐藏互换的成败任务;下表给出同题对照。": "Browser comparisons must stay within a model cohort. Aggregate totals hide swapped outcomes; the table gives the paired view.", + "浏览器对比必须留在同一模型层内。它们各自的总数会隐藏互换的成败任务;下表才是同题对照。": "Browser comparisons must stay within a model cohort. Aggregate totals hide tasks that swap outcomes; the paired table shows the relevant comparison.", + "每行只比较同一模型下的 90 个配对任务。": "Each row compares the 90 paired tasks within one model cohort.", + "两个模型层合计有 9 个单边通过,可用于定位兼容性问题;样本量和方向都不支持总体优劣判断。": "Across both model cohorts, nine task pairs were one-sided. They help locate compatibility defects, while the sample size and direction do not support an overall ranking.", + "查看 90 题的四配置矩阵 →": "View the four-configuration matrix for all 90 tasks →", + "错误到底发生在哪里": "Where the failures occurred", + "程序判分落点与复核根因": "Programmatic failure location and reviewed root cause", + "程序判分先说明哪类规则未通过;人工复核再沿最终回答、evaluator 断言、HAR 和 Agent 轨迹追到直接原因。这两个层次需要分开。": "Programmatic scoring first identifies which rule class failed. Human review then follows the final answer, evaluator assertions, HAR, and agent trajectory to the direct cause. These are separate layers.", + "程序判分落点": "Programmatic failure location", + "只有网络副作用规则未通过": "Only the network-side-effect rule failed", + "evaluator 没找到合同要求的请求或可观察副作用": "The evaluator did not find the required request or observable side effect", + "只有最终回答规则未通过": "Only the final-answer rule failed", + "最终回答与结构化答案合同不一致": "The final answer disagreed with the structured answer contract", + "两类规则同时未通过": "Both rule classes failed", + "最终回答和网络副作用均未满足合同": "Neither the final answer nor the network side effect satisfied the contract", + "官方 evaluator 的失败规则组合": "Failed rule combinations from the official evaluator", + "“网络副作用未通过”只描述程序判分位置。它可能源于 Agent 没有提交,也可能源于 evaluator、站点或 HAR 采集问题。": "A failed network-side-effect rule identifies the scoring location only. It can result from an agent not submitting, or from the evaluator, site, or HAR capture.", + "复核根因": "Reviewed root cause", + "75 个失败结果的复核原因分布": "Reviewed cause distribution for 75 failed results", + "75 个官方失败结果都读取了最终回答、evaluator 断言、HAR 和 Agent 轨迹。分类统计不是第二套打分,而是回答“这个 0 分由哪一层直接产生”。": "All 75 official failures were reviewed against the final answer, evaluator assertions, HAR, and agent trajectory. The categories do not rescore tasks; they identify the layer that directly produced each zero.", + "检索不全、选错对象、字段遗漏或流程提前结束": "Incomplete retrieval, wrong selection, omitted fields, or premature termination", + "验收合同与页面语义不一致,或 evaluator 自身报错": "Acceptance contracts disagree with page semantics, or the evaluator itself errors", + "搜索、会话或保存依赖故障": "Search, session, or persistence dependency failure", + "HAR 缺少 Referer、响应正文或出现重复记录": "HAR lacks Referer or response bodies, or contains duplicate records", + "共享环境": "Shared environment", + "并发执行改写了任务初始状态": "Concurrent execution changed the task's initial state", + "参考漂移与 Agent 遗漏同时存在": "Reference drift and an agent omission both contributed", + "Agent / 模型行为": "Agent / model behavior", + "保留并修复": "Retain and fix", + "Agent 行为": "Agent behavior", + "例如漏掉商品、选错最优候选、写错字段,或在 40 步前提前结束。这些属于当前 Agent × 模型配置的真实失败。": "Examples include missing products, choosing the wrong candidate, writing the wrong field, or stopping before the 40-step limit. These are genuine failures of the current agent-model configuration.", + "验收规则": "Acceptance rules", + "Task 699 的 evaluator 在比较实际提交前就因单元素 Alternatives 配置抛错;Task 528 要求“不提交”,验收却要求 POST。这些 0 分不能归给模型或浏览器。": "Task 699's evaluator throws on a one-item Alternatives configuration before comparing the submission. Task 528 says not to submit, while its evaluator requires a POST. These zeros cannot be attributed to the model or browser.", + "站点与环境": "Sites and environment", + "Magento 搜索或库存保存服务失效,以及 Task 77 的并发状态污染,都会让正确操作无法通过。": "Magento search or stock-save failures, plus shared-state contamination in Task 77, can prevent otherwise correct work from passing.", + "浏览器证据": "Browser evidence", + "的 evaluator 在比较实际提交前就因单元素 Alternatives 配置抛错;": " fails before comparing the actual submission because the evaluator throws on a one-item Alternatives configuration; ", + "要求“不提交”,验收却要求 POST。这些 0 分不能归给模型或浏览器。": " says not to submit, while its evaluator requires a POST. These zeros cannot be attributed to the model or browser.", + "Magento 搜索或库存保存服务失效,以及": "Magento search or stock-save services failed, and ", + "的并发状态污染,都会让正确操作无法通过。": " suffered shared-state contamination. Either can prevent correct work from passing.", + "Moli 的部分 HAR 没有 Referer 或响应正文,例如": "Some Moli HAR records omit Referer or response bodies, including ", + "。页面可能已经操作成功,但 evaluator 无法从网络记录确认;这是浏览器集成层的有效信号。": ". The UI operation may have succeeded, but the evaluator cannot confirm it from network evidence. This is a valid browser-integration signal.", + "Moli 的部分 HAR 没有 Referer 或响应正文。页面可能已经操作成功,但 evaluator 无法从网络记录确认;这是浏览器集成层的有效信号。": "Some Moli HAR records omit Referer or response bodies. The UI operation may have succeeded, but the evaluator cannot confirm it from network evidence. This is a valid browser-integration signal.", + "官方成绩与诊断子集分开看": "Keep official scores separate from the diagnostic subset", + "历史官方分数不人工改写。对已证实受任务 / evaluator、站点、共享环境或参考状态干扰的 14 题,诊断时按整题排除四种配置,剩余 76 题。": "Historical official scores are never manually rewritten. For diagnosis, 14 tasks with confirmed task/evaluator, site, shared-state, or reference-state interference are removed across all four configurations, leaving 76 tasks.", + "76 题通过": "Passes on 76 tasks", + "诊断通过率": "Diagnostic success rate", + "这是事后诊断口径,用于确定修复顺序,不是新的官方分数。只有修复后重跑完整四配置,才能产生新成绩。": "This post-hoc view prioritizes fixes; it is not a replacement official score. A new score requires rerunning all four configurations after the fixes.", + "这份结果审查到了什么": "What the review established", + "审查覆盖与边界": "Review coverage and boundaries", + "438/438 次物理执行检查了 HAR、Agent trace、最终回答、evaluator 输出和环境准备回执。": "All 438 physical executions were checked for HAR, agent trace, final answer, evaluator output, and environment-preparation receipts.", + "438/438 个证据包由程序核对文件存在性、哈希、JSON 可读性、回答 / 分数 / 状态一致性和运行身份。": "Programs checked all 438 evidence bundles for file presence, hashes, JSON readability, response/score/status consistency, and runtime identity.", + "官方程序验收": "Official programmatic evaluation", + "每题按最终回答和必要网络副作用判分;没有 LLM judge。": "Each task is scored from its final answer and required network side effects; no LLM judge is used.", + "失败根因复核": "Failure root-cause review", + "75/75 个最终失败结果逐条查看错误、断言、轨迹与网络记录,归类并给出处理边界。": "All 75 final failures were reviewed against errors, assertions, trajectories, and network records, then classified with a defined treatment boundary.", + "75/75 个最终失败结果逐条阅读错误、断言、最终回答、HAR 和 Agent 轨迹,归类并给出处理边界。": "All 75 final failures received case-by-case review of errors, assertions, final answers, HAR, and agent trajectories, followed by classification and treatment boundaries.", + "审查分为完整性检查和失败根因复核。": "The review has two layers: integrity checks and failure root-cause review.", + "前者确认“证据包与 evaluator 记录一致”;后者检查 evaluator 语义,并识别出 Task 699 等问题。已通过结果没有再做独立语义盲审,其判定依据仍是官方 evaluator。": "The first confirms that evidence bundles agree with evaluator records. The second checks evaluator semantics and identified issues such as Task 699. Passed results did not receive a separate blinded semantic review and still rely on the official evaluator.", + "查看核验记录 →": "View review records →", + "性能与资源:只在四种配置首次都成功的任务上看": "Performance and resources on tasks all four configurations passed on the first attempt", + "共同成功任务的性能与资源": "Performance and resources on common final successes", + "当前没有四种配置最终都成功的任务,资源差异暂不比较。": "No task was finally passed by all four configurations, so resource differences are unavailable.", + "首次共同成功任务的资源差异": "Resource differences on common first-attempt successes", + "当前没有四种配置首次都成功的任务,资源差异暂不比较。": "No task was passed on the first attempt by all four configurations, so resource differences are unavailable.", + "Agent 链路还包含模型推理和页面等待,因此不能解读为纯内核性能。": "The agent path also includes model inference and page waits, so it is not a pure browser-core measurement.", + "查看口径与四配置明细 →": "View the cohort definition and four-configuration details →", + "Moli 的内存区间更低": "Moli has the lower memory range", + "Chromium 的工具耗时与 CPU 更低": "Chromium has lower tool latency and CPU time", + "下表限定为首次执行共同成功的 64 题,每个配置只取第一次执行并报告 P50。这既排除了重试累计,也避免把“很快失败”错当成速度优势。": "The table uses the 64 tasks that all four configurations passed on the first attempt. It reports P50 from the first execution only, excluding retry accumulation and the false speed advantage of fast failures.", + "峰值内存 P50": "Peak memory P50", + "CPU 时间 P50": "CPU time P50", + "Moli 的优势很明确:内存": "Moli's clear advantage is memory", + "Moli:峰值内存更低": "Moli: lower peak memory", + "这批首次共同成功任务上,Moli 的峰值内存 P50 约为 190 MiB,Chromium 约为 500–567 MiB。这个差距在两个模型层都出现。": "On these common first-attempt successes, Moli's P50 peak memory is about 190 MiB versus roughly 500-567 MiB for Chromium. The gap appears in both model cohorts.", + "代价也很明确:工具耗时和 CPU": "The tradeoff is tool latency and CPU time", + "Chromium:工具耗时与 CPU 更低": "Chromium: lower tool latency and CPU time", + "同一批任务上,Chromium 的 BrowserTool P50 和 CPU 时间均低于 Moli。整条 Agent 链路还包含模型推理和页面等待,不能解读为纯内核性能。": "On the same tasks, Chromium has lower BrowserTool P50 and CPU time than Moli. The full agent path also includes model inference and page waits, so it is not a pure browser-core measurement.", + "同一批任务上,Moli 的 BrowserTool P50 和 CPU 时间都高于 Chromium。整条 Agent 链路还包含模型推理和页面等待,不能解读为纯内核性能。": "On the same tasks, Moli has higher BrowserTool P50 and CPU time than Chromium. The full agent path also includes model inference and page waits, so it is not a pure browser-core measurement.", + "最后怎么读这份报告": "The practical reading", + "工程结论": "Engineering conclusions", + "官方 90 题成绩显示,四种端到端配置都完成了大约八成任务,浏览器之间的单边差异集中在少数任务。75 个零分分布在 Agent、验收规则、站点、浏览器证据接口和共享环境等多层。": "The official 90-task scores show that all four end-to-end configurations complete about four-fifths of the sample, with one-sided browser differences concentrated in a few tasks. The 75 zeros span agent behavior, acceptance rules, sites, browser-evidence interfaces, and shared state.", + "官方 90 题成绩说明,四种端到端配置都完成了大约八成任务,浏览器之间的单边差异集中在少数任务。错误复核又证明,不能把 75 个零分全部归给 Agent 或浏览器。": "The official 90-task scores show that all four end-to-end configurations complete about four-fifths of the sample and that one-sided browser differences concentrate in a few tasks. Failure review also shows that the 75 zeros do not all belong to the agent or browser.", + "对工程选型有用的结论是:Moli 的内存占用明显更低,但当前 HAR 完整性、工具耗时和部分任务兼容性还需修复;Chromium 在 76 题诊断子集中两个模型层都多通过任务,但当前只有一个正式样本,不能外推成总体优势。": "For engineering decisions, Moli uses substantially less memory, while HAR completeness, tool latency, and some task compatibility still need work. Chromium passes more tasks in both model cohorts on the 76-task diagnostic subset, but one formal sample cannot establish a general advantage.", + "下一轮只做两件事": "Two priorities for the next run", + "下一轮验证": "Next validation run", + "Task 77 与 771 可在串行锁下补跑;Task 339、475、600、625、676、699 的合同或 evaluator 修复已提交": "Tasks 77 and 771 can be rerun under the serialization lock. Contract or evaluator fixes for tasks 339, 475, 600, 625, 676, and 699 have been submitted in", + "上游 PR #45": "upstream PR #45", + ",合并后补跑;Task 528 在具备可移植的未提交表单验收前继续排除。站点题先通过读写预检。任何补跑都覆盖同题四种配置,并作为新的完整证据批次发布。": ". They will be rerun after the fix is merged. Task 528 remains excluded until a portable evaluator can verify an unsubmitted filled form. Site-dependent tasks must first pass read/write preflight checks. Every rerun covers all four configurations for the task and is published as a new complete evidence cohort.", + "先修复并重跑 14 题干扰项,再对同一冻结代码的四种配置增加重复次数。在此之前,这份报告用于定位兼容性和证据链问题,不发布模型或浏览器排名。": "First fix and rerun the 14 interference tasks; then add repetitions for all four configurations on one frozen codebase. Until then, this report is for locating compatibility and evidence-chain defects, not for publishing a model or browser ranking.", + "先修复并重跑 14 题干扰项,再对同一冻结代码的四种配置增加重复次数。在此之前,这份报告适合找兼容性和证据链问题,不适合发布模型或浏览器排名。": "First fix and rerun the 14 interference tasks; then add repetitions for all four configurations on one frozen codebase. Until then, this report is useful for locating compatibility and evidence-chain defects, not for publishing a model or browser ranking.", + "查看全部配置的逐题矩阵 →": "View the per-task matrix for all configurations →", + "耗时统计包含什么": "What the latency metrics include", + "Agent 总耗时包含模型推理、页面等待和浏览器操作;BrowserTool 耗时是全部浏览器工具调用的墙钟时间之和,仍包含工具封装与 CDP 往返,不是纯内核 CPU 时间。": "Agent total time includes model inference, page waits, and browser actions. BrowserTool time is the summed wall time of all browser-tool calls; it still includes tool-wrapper and CDP round-trip overhead and is not pure browser-core CPU time.", + "把取得逻辑结果所发生的执行相加,包含登录、模型推理、调度、页面等待、BrowserTool 与收尾,不包含此前的 Docker 站点重置。": " sums the executions used to obtain the logical result, including sign-in, model inference, orchestration, page waits, BrowserTool calls, and cleanup, but excluding the preceding Docker site reset. ", + "是所有 browser 工具调用的墙钟时间之和,包含工具封装、CDP 往返和页面等待。Tokens 同样累计正式结果包含的物理执行。": " is the sum of wall time across all browser-tool calls, including tool-wrapper, CDP round-trip, and page-wait overhead. Tokens likewise sum the physical executions included in the official result.", + "Browser Use 能力评测": "Browser Use capability evaluation", + "首次结果与重试恢复分开统计": "Separate first-attempt results from retry recovery", + "最终通过": "Final passes", + "查看评测集": "View dataset", + "下钻全部结果": "Inspect all results", + "下载论点与证据": "Download claims and evidence", + "Task 77 的并发污染已经由串行锁修复,但旧结果不会人工改分;合同或站点尚不能可靠验收的题也不进入分母。": "A serialization lock fixes the Task 77 concurrency defect, but the old result is not manually rescored. Tasks whose contracts or sites still cannot be evaluated reliably are also excluded from the denominator.", + "正式比较与原始记录分开看": "Separate the formal comparison from raw records", + "Task 77 可在串行锁下补跑;站点题先通过读写预检,合同矛盾题由上游修正或退出样本。任何补跑都覆盖同题四种配置,并作为新的完整证据批次发布。": "Task 77 can be rerun under the serialization lock. Site-dependent tasks must first pass read/write preflight checks, while contradictory contracts must be corrected upstream or excluded. Every rerun covers all four configurations for the task and is published as a new complete evidence cohort.", + "每行是一道审查合格任务,四列分别对应模型×浏览器配置;可筛出结果有差异、全部通过或全部失败的任务。": "Each row is a review-qualified task, with one column per model-browser configuration. Filters isolate divergent, all-pass, or all-fail outcomes.", + "这里保留预注册 90 题的全部原始结果,包括退出正式比较的 14 题。默认显示失败;点击“查看诊断”可核对发生位置、程序报错、复核根因、轨迹、画面和日志。": "This inventory retains raw results for all 90 preregistered tasks, including the 14 excluded from the formal comparison. Failures are shown by default; open a diagnosis to inspect the failure point, program error, reviewed cause, trajectory, frames, and logs.", + } +) + + +def translate_text(value: str) -> str: + leading = value[: len(value) - len(value.lstrip())] + trailing = value[len(value.rstrip()) :] + core = value.strip() + if not core: + return value + translated = TEXT.get(core) + if translated is None: + patterns = ( + (r"首次 (\d+/\d+)", lambda m: f"First attempt {m.group(1)}"), + (r"失败后重试 (\d+) 题", lambda m: f"{m.group(1)} failed tasks retried"), + (r"查看 (\d+) 题的四配置矩阵 →", lambda m: f"View the four-configuration matrix for all {m.group(1)} tasks →"), + ( + r"Sol × Chromium 首次 (\d+/\d+),有 (\d+) 个任务在重试中恢复,最终 (\d+/\d+);Astra × Chromium 首次和最终均为 (\d+/\d+)。", + lambda m: f"Sol × Chromium passed {m.group(1)} on the first attempt, recovered {m.group(2)} tasks on retry, and finished at {m.group(3)}. Astra × Chromium was {m.group(4)} both before and after retries.", + ), + ( + r"(\d+) 题审查合格比较集:Chromium 与 Moli 的结果、代价与失败位置", + lambda m: f"{m.group(1)}-task review-qualified set: Chromium vs. Moli outcomes, cost, and failure points", + ), + ( + r"预注册的 (\d+) 题中,(\d+) 题确认受到验收合同、站点、共享状态或参考状态干扰。首页只用剩余 (\d+) 题比较四种配置;全部原始结果仍可下钻审计。", + lambda m: f"Of {m.group(1)} preregistered tasks, {m.group(2)} were confirmed to be affected by acceptance contracts, sites, shared state, or reference-state drift. The homepage compares the four configurations only on the remaining {m.group(3)} tasks; every raw result remains available for audit.", + ), + ( + r"(\d+) 题退出正式比较", + lambda m: f"{m.group(1)} tasks are excluded from the formal comparison", + ), + ( + r"只有在 Chromium 的最终分数上,Astra 的 (\d+/\d+) 低于 Sol 的 (\d+/\d+)。首次执行时,Sol 为 (\d+/\d+),Astra 为 (\d+/\d+);差值来自 Sol 恢复的 (\d+) 个任务。两批模型运行使用了不同的 browser‑eval 提交,因此这里保留为观察,不作严格模型排名。", + lambda m: f"Astra trails Sol only in the final Chromium result, {m.group(1)} versus {m.group(2)}. On the first attempt Sol scored {m.group(3)} and Astra {m.group(4)}; the difference comes from {m.group(5)} Sol recoveries. The two model cohorts used different browser-eval commits, so this remains an observation rather than a strict model ranking.", + ), + ( + r"对已证实受任务 / evaluator、站点、共享环境或参考状态干扰的 (\d+) 题,按整题排除四种配置,剩余 (\d+) 题。原始 evaluator 输出不改写,但不再用作首页比较分母。", + lambda m: f"For {m.group(1)} tasks confirmed to be affected by task/evaluator, site, shared-state, or reference-state problems, all four configurations are excluded together, leaving {m.group(2)} tasks. Raw evaluator outputs are preserved but no longer serve as the homepage comparison denominator.", + ), + ( + r"这是本报告的正式比较分母。被排除任务及其 (\d+) 个原始失败记录继续保存在结果页和核验记录中;修复并按同一协议补跑四种配置后,任务才可重新进入比较集。", + lambda m: f"This is the report's formal comparison denominator. Excluded tasks and their {m.group(1)} raw failure records remain in the result and review pages. A task can re-enter only after correction and a four-configuration rerun under the same protocol.", + ), + ( + r"(\d+)/(\d+) 个原始最终失败结果逐条阅读错误、断言、最终回答、HAR 和 Agent 轨迹,归类并给出处理边界。", + lambda m: f"All {m.group(1)}/{m.group(2)} raw final failures received case-by-case review of errors, assertions, final answers, HAR, and agent trajectories, followed by classification and treatment boundaries.", + ), + ( + r"(\d+) 题审查合格比较集显示,四种端到端配置都完成了约九成任务,浏览器之间的单边差异集中在少数任务。有效失败主要来自 Agent 行为和浏览器证据接口,不再混入已确认的合同、站点或共享状态问题。", + lambda m: f"On the {m.group(1)}-task review-qualified set, all four end-to-end configurations complete about nine in ten tasks, with one-sided browser differences concentrated in a few tasks. Remaining failures are mainly agent behavior and browser-evidence-interface issues, without confirmed contract, site, or shared-state interference.", + ), + ( + r"Moli 的内存占用明显更低,但 HAR 完整性、工具耗时和部分任务兼容性还需修复;Chromium 在两个模型层都多通过任务,但当前每种配置只有一个正式样本,不能外推成总体优势。", + lambda m: "Moli uses substantially less memory, while HAR completeness, tool latency, and some task compatibility still need work. Chromium passes more tasks in both model cohorts, but one formal sample per configuration cannot establish a general advantage.", + ), + ( + r"四种配置总览:(\d+) 题审查合格比较集", + lambda m: f"Four-configuration overview: {m.group(1)}-task review-qualified set", + ), + ( + r"同题全配置对比:(\d+) 题", + lambda m: f"Same-task comparison across all configurations: {m.group(1)} tasks", + ), + ( + r"Astra 首次通过 (\d+/\d+),Sol 为 (\d+/\d+)", + lambda m: f"Astra passed {m.group(1)} on the first attempt; Sol passed {m.group(2)}", + ), + ( + r"Sol × Chromium 有 (\d+) 个重试任务恢复成功,最终到 (\d+/\d+)。Astra 的 (\d+/\d+) 并非首次表现落后。", + lambda m: f"{m.group(1)} Sol × Chromium tasks recovered on retry, raising its final score to {m.group(2)}. Astra's {m.group(3)} does not reflect weaker first-attempt performance.", + ), + ( + r"两层模型共有 (\d+) 个单边通过配对", + lambda m: f"{m.group(1)} task pairs were one-sided across the two model cohorts", + ), + ( + r"(.+):(\d+) 题共同通过、(\d+) 题仅 (.+) 通过、(\d+) 题仅 (.+) 通过;(.+):(\d+) 题共同通过、(\d+) 题仅 (.+) 通过、(\d+) 题仅 (.+) 通过。", + lambda m: f"{m.group(1)}: both browsers passed {m.group(2)} tasks, only {m.group(4)} passed {m.group(3)}, and only {m.group(6)} passed {m.group(5)}. {m.group(7)}: both browsers passed {m.group(8)} tasks, only {m.group(10)} passed {m.group(9)}, and only {m.group(12)} passed {m.group(11)}.", + ), + ( + r"(\d+) 题含已确认的外部干扰", + lambda m: f"{m.group(1)} tasks contain confirmed external interference", + ), + ( + r"(\d+) 个失败结果已经逐条核验。与任务合同、站点、共享环境或参考状态有关的任务不会进入诊断子集,修复后需完整重跑。", + lambda m: f"All {m.group(1)} failed results have been reviewed individually. Tasks affected by task-contract, site, shared-environment, or reference-state issues are excluded from the diagnostic subset and require a complete rerun after correction.", + ), + ( + r"干扰来自验收合同、站点、共享环境或参考状态。官方分数保留,另用 (\d+) 题事后诊断子集观察目标变量。", + lambda m: f"The interference comes from acceptance contracts, sites, shared state, or reference state. Official scores remain unchanged; a post-hoc {m.group(1)}-task subset is used only for diagnosis.", + ), + ( + r"首次通过 (\d+/\d+),重试恢复 (\d+) 题,最终通过 (\d+/\d+)", + lambda m: f"First-attempt passes: {m.group(1)}; recovered on retry: {m.group(2)}; final passes: {m.group(3)}", + ), + ( + r"只有在 Chromium 的最终分数上,Astra 的 (\d+/\d+) 低于 Sol 的 (\d+/\d+)。首次执行时,Sol 为 (\d+/\d+),Astra 为 (\d+/\d+);差值来自 Sol 恢复的 (\d+) 个任务。Astra 的 Task 77 还受共享环境污染。", + lambda m: f"Astra trails Sol only in the final Chromium score, {m.group(1)} versus {m.group(2)}. On the first attempt Sol scored {m.group(3)} and Astra {m.group(4)}; the difference comes from {m.group(5)} Sol recoveries. Astra's Task 77 was also contaminated by shared-state concurrency.", + ), + ( + r"(\d+) 题、四种配置:相近成绩背后的不同原因", + lambda m: f"{m.group(1)} tasks and four configurations: different causes behind similar scores", + ), + ( + r"四种配置的官方分数很接近;(\d+) 个零分经复核后分散在多个责任层,不能统一归给 Agent 或浏览器。", + lambda m: f"The four official scores are close. Review placed the {m.group(1)} zeros across several owning layers rather than assigning them all to the agent or browser.", + ), + ( + r"两个模型层合计有 (\d+) 个单边通过,可用于定位兼容性问题;样本量和方向都不支持总体优劣判断。", + lambda m: f"Across both model cohorts, {m.group(1)} task pairs were one-sided. They help locate compatibility defects, while the sample size and direction do not support an overall ranking.", + ), + ( + r"(\d+)/(\d+) 次物理执行检查了 HAR、Agent trace、最终回答、evaluator 输出和环境准备回执。", + lambda m: f"All {m.group(1)}/{m.group(2)} physical executions were checked for HAR, agent trace, final answer, evaluator output, and environment-preparation receipts.", + ), + ( + r"(\d+)/(\d+) 个证据包由程序核对文件存在性、哈希、JSON 可读性、回答 / 分数 / 状态一致性和运行身份。", + lambda m: f"Programs checked all {m.group(1)}/{m.group(2)} evidence bundles for file presence, hashes, JSON readability, response/score/status consistency, and runtime identity.", + ), + ( + r"(\d+)/(\d+) 个最终失败结果逐条查看错误、断言、轨迹与网络记录,归类并给出处理边界。", + lambda m: f"All {m.group(1)}/{m.group(2)} final failures were reviewed against errors, assertions, trajectories, and network records, then classified with a defined treatment boundary.", + ), + ( + r"(\d+)/(\d+) 个最终失败结果逐条阅读错误、断言、最终回答、HAR 和 Agent 轨迹,归类并给出处理边界。", + lambda m: f"All {m.group(1)}/{m.group(2)} final failures received case-by-case review of errors, assertions, final answers, HAR, and agent trajectories, followed by classification and treatment boundaries.", + ), + ( + r"(\d+) 个失败结果的复核原因分布", + lambda m: f"Reviewed cause distribution for {m.group(1)} failed results", + ), + ( + r"(\d+) 题通过", + lambda m: f"Passes on {m.group(1)} tasks", + ), + ( + r"每行只比较同一模型下的 (\d+) 个配对任务。", + lambda m: f"Each row compares the {m.group(1)} paired tasks within one model cohort.", + ), + ( + r"下表限定为首次执行共同成功的 (\d+) 题,每个配置只取第一次执行并报告 P50。这既排除了重试累计,也避免把“很快失败”错当成速度优势。", + lambda m: f"The table uses the {m.group(1)} tasks that all four configurations passed on the first attempt. It reports P50 from the first execution only, excluding retry accumulation and the false speed advantage of fast failures.", + ), + ( + r"下表限定为四种配置最终都通过的 (\d+) 题,并以每题最终逻辑结果包含的全部物理执行计算 P50;这样不会把快速失败误当成性能优势。", + lambda m: f"The table uses the {m.group(1)} tasks finally passed by all four configurations. P50 includes every physical execution belonging to each final logical result, so fast failures are not mistaken for a performance advantage.", + ), + ( + r"(\d+) 题首次共同成功样本中,Moli 峰值内存 P50 为 ([0-9.]+)–([0-9.]+) MiB;Chromium 为 ([0-9.]+)–([0-9.]+) MiB。", + lambda m: f"Across {m.group(1)} common first-attempt successes, Moli's P50 peak memory spans {m.group(2)}-{m.group(3)} MiB, versus {m.group(4)}-{m.group(5)} MiB for Chromium.", + ), + ( + r"(\d+) 题最终共同成功样本中,Moli 峰值内存 P50 为 ([0-9.]+)–([0-9.]+) MiB;Chromium 为 ([0-9.]+)–([0-9.]+) MiB。", + lambda m: f"Across {m.group(1)} common final successes, Moli's P50 peak memory spans {m.group(2)}-{m.group(3)} MiB, versus {m.group(4)}-{m.group(5)} MiB for Chromium.", + ), + ( + r"BrowserTool P50:Chromium ([0-9.]+)–([0-9.]+) 秒,Moli ([0-9.]+)–([0-9.]+) 秒;CPU 时间 P50:Chromium ([0-9.]+)–([0-9.]+) 秒,Moli ([0-9.]+)–([0-9.]+) 秒。", + lambda m: f"BrowserTool P50: Chromium {m.group(1)}-{m.group(2)} seconds, Moli {m.group(3)}-{m.group(4)} seconds. CPU-time P50: Chromium {m.group(5)}-{m.group(6)} seconds, Moli {m.group(7)}-{m.group(8)} seconds.", + ), + ( + r"两个模型层在不同日期、不同 browser‑eval 提交上运行((.+))。任务、CowAgent 提交、浏览器镜像与 evaluator 数据相同,但这仍不是严格的模型 A/B。", + lambda m: f"The model cohorts ran on different dates and browser-eval commits ({m.group(1).replace('、', '; ')}). They share tasks, CowAgent commit, browser images, and evaluator data, but this is still not a strict model A/B test.", + ), + ( + r"官方 (\d+) 题成绩显示,四种端到端配置都完成了大约八成任务,浏览器之间的单边差异集中在少数任务。(\d+) 个零分分布在 Agent、验收规则、站点、浏览器证据接口和共享环境等多层。", + lambda m: f"The official {m.group(1)}-task scores show that all four end-to-end configurations complete about four-fifths of the sample, with one-sided browser differences concentrated in a few tasks. The {m.group(2)} zeros span agent behavior, acceptance rules, sites, browser-evidence interfaces, and shared state.", + ), + ( + r"Moli 的内存占用明显更低,但当前 HAR 完整性、工具耗时和部分任务兼容性还需修复;Chromium 在 (\d+) 题诊断子集中两个模型层都多通过任务,但当前只有一个正式样本,不能外推成总体优势。", + lambda m: f"Moli uses substantially less memory, while HAR completeness, tool latency, and some task compatibility still need work. Chromium passes more tasks in both model cohorts on the {m.group(1)}-task diagnostic subset, but one formal sample cannot establish a general advantage.", + ), + ( + r"先修复并重跑 (\d+) 题干扰项,再对同一冻结代码的四种配置增加重复次数。在此之前,这份报告用于定位兼容性和证据链问题,不发布模型或浏览器排名。", + lambda m: f"First fix and rerun the {m.group(1)} interference tasks; then add repetitions for all four configurations on one frozen codebase. Until then, this report is for locating compatibility and evidence-chain defects, not for publishing a model or browser ranking.", + ), + ( + r"(\d+) 个结果与失败诊断", + lambda m: f"{m.group(1)} results and failure diagnoses", + ), + ( + r"失败发生在(.+);下方保留程序报错、复核根因和对应执行证据。", + lambda m: f"The failure occurred at {translate_text(m.group(1))}. The exact program error, reviewed root cause, and corresponding execution evidence are provided below.", + ), + ( + r"四种配置均使用同一组 (\d+) 题。模型和浏览器同时变化时,表格比较的是整套配置,不能把差值只归因于浏览器。", + lambda m: f"All four configurations use the same {m.group(1)} tasks. When both model and browser vary, the table compares complete configurations; differences cannot be attributed to the browser alone.", + ), + ( + r"下表统一比较四种配置。CPU、内存和网络只计浏览器容器,不含 CowAgent 或模型服务;均值、P50、P95 都按 (\d+) 个逻辑任务结果统计。模型会改变动作数量和页面停留时间,因此这里比较的是整套配置,不能单独归因于浏览器内核。", + lambda m: f"The table compares all four configurations. CPU, memory, and network measurements cover only the browser container, excluding CowAgent and the model service; mean, P50, and P95 are computed across {m.group(1)} logical task results. Because the model changes action counts and time spent on pages, these are whole-configuration comparisons and cannot be attributed to the browser core alone.", + ), + ( + r"(.+?) · (Chromium|Moli 1\.1\.3) 结果", + lambda m: f"{m.group(1)} · {m.group(2)} result", + ), + ( + r"(简单|中等|困难) · (\d+) 步起", + lambda m: f"{ {'简单': 'Easy', '中等': 'Medium', '困难': 'Hard'}[m.group(1)] } · {m.group(2)} minimum steps", + ), + ( + r"第 (\d+) 步 BrowserTool 执行异常", + lambda m: f"Step {m.group(1)}: BrowserTool execution error", + ), + ( + r"第 (\d+) 步后达到步骤上限", + lambda m: f"After {m.group(1)} steps: step limit reached", + ), + ( + r"第 (\d+) 步后的最终回答/任务状态", + lambda m: f"After {m.group(1)} steps: final answer/task state", + ), + ( + r"第 (\d+) 步后的站点响应/保存", + lambda m: f"After {m.group(1)} steps: site response/save", + ), + ( + r"(.+?) · 单次调用墙钟时间", + lambda m: f"{m.group(1)} · per-call wall time", + ), + ( + r"判分 ([01]) · Agent 链路 ([\d.]+) 秒 · BrowserTool (\d+) 次 / ([\d.]+) 秒 · 模型 tokens ([\d,]+)", + lambda m: f"Score {m.group(1)} · Agent path {m.group(2)} s · BrowserTool {m.group(3)} calls / {m.group(4)} s · model tokens {m.group(5)}", + ), + ( + r",只保留该题所需站点;env-ctrl 报告 ready 后,再做连续应用健康检查、清浏览器状态、登录测试账号并打开起始页。本轮 (\d+)/(\d+) 个物理执行都保存了环境准备回执,并与对应执行记录绑定。", + lambda m: f", keeping only the sites required by the task. After env-ctrl reports ready, the runner performs repeated application-health checks, clears browser state, signs in with test accounts, and opens the start page. All {m.group(1)}/{m.group(2)} physical executions retain preparation receipts bound to their execution records.", + ), + ( + r"预注册文件只含任务 ID、层级和入选理由,不含点击序列。CowAgent 收到自然语言目标后,在最多 (\d+) 步内循环读取页面观察、请求模型决策并生成下一次 BrowserTool 调用。Chromium 与 Moli 分别连接独立 CDP 浏览器容器;每次物理执行都保留最终 JSON、Agent 事件流、网络 HAR 和 Playwright trace。", + lambda m: f"The preregistration contains task IDs, tiers, and selection rationales, but no click sequence. Given a natural-language objective, CowAgent repeatedly reads the page, requests a model decision, and generates the next BrowserTool call for up to {m.group(1)} steps. Chromium and Moli connect to separate CDP browser containers. Every physical execution retains the final JSON, Agent event stream, network HAR, and Playwright trace.", + ), + ( + r"(\d+)/(\d+) 个物理执行均记录 reset、ready、应用健康检查和登录准备状态;事件流随后证明 CowAgent 自主产生 BrowserTool 调用序列。", + lambda m: f"All {m.group(1)}/{m.group(2)} physical executions record reset, readiness, application-health, and sign-in preparation status. The subsequent event streams show CowAgent generating BrowserTool calls autonomously.", + ), + ( + r"本轮 (\d+)/(\d+) 个物理执行均保存环境准备回执;公开环境摘要保留逐项状态,原始登录态不发布。", + lambda m: f"All {m.group(1)}/{m.group(2)} physical executions retain environment-preparation receipts. The public summary preserves each status while authenticated session state remains private.", + ), + ( + r"每次物理执行的脱敏回答、evaluator、事件和轨迹位于", + lambda _m: "The sanitized answer, evaluator output, events, and trajectory for each physical execution are located under", + ), + ( + r"给出全部 (\d+) 个逻辑结果及其物理执行、下载路径和公开文件哈希。", + lambda m: f"lists all {m.group(1)} logical results, their physical executions, download paths, and public-file hashes.", + ), + ( + r"CowAgent 使用 (.+),在同一组 WebArena‑Verified (\d+) 题上分别操作 (.+)。", + lambda m: f"CowAgent uses {m.group(1)} to operate {m.group(3)} on the same {m.group(2)} WebArena-Verified tasks.", + ), + ( + r"(.+?):(.+?) (\d+)/(\d+)(([\d.]+)%),(.+?) (\d+)/(\d+)(([\d.]+)%);(.+?) 多通过 (\d+) 题。", + lambda m: f"{m.group(1)}: {m.group(2)} passed {m.group(3)}/{m.group(4)} ({m.group(5)}%); {m.group(6)} passed {m.group(7)}/{m.group(8)} ({m.group(9)}%). {m.group(10)} passed {m.group(11)} more task(s).", + ), + ( + r"(.+?):(.+?) (\d+)/(\d+)(([\d.]+)%),(.+?) (\d+)/(\d+)(([\d.]+)%);通过数相同。", + lambda m: f"{m.group(1)}: {m.group(2)} passed {m.group(3)}/{m.group(4)} ({m.group(5)}%); {m.group(6)} passed {m.group(7)}/{m.group(8)} ({m.group(9)}%). The pass counts are tied.", + ), + ( + r":(.+?) (\d+)/(\d+)(([\d.]+)%),(.+?) (\d+)/(\d+)(([\d.]+)%);(.+?) 多通过 (\d+) 题。", + lambda m: f": {m.group(1)} passed {m.group(2)}/{m.group(3)} ({m.group(4)}%); {m.group(5)} passed {m.group(6)}/{m.group(7)} ({m.group(8)}%). {m.group(9)} passed {m.group(10)} more task(s).", + ), + ( + r":(.+?) (\d+)/(\d+)(([\d.]+)%),(.+?) (\d+)/(\d+)(([\d.]+)%);通过数相同。", + lambda m: f": {m.group(1)} passed {m.group(2)}/{m.group(3)} ({m.group(4)}%); {m.group(5)} passed {m.group(6)}/{m.group(7)} ({m.group(8)}%). The pass counts are tied.", + ), + ( + r"这 (\d+) 题是固定选择的样本,结果不能直接外推到 WebArena‑Verified 全集,也不能仅凭总耗时判断浏览器内核速度。", + lambda m: f"These {m.group(1)} tasks are a fixed cohort. Results do not generalize directly to the full WebArena-Verified benchmark, and total elapsed time alone does not measure browser-engine speed.", + ), + ( + r"两组模型合计有 (\d+) 个同题结果不同;不同模型的运行互不配对。", + lambda m: f"Across both model runs, {m.group(1)} same-task results differ between browsers. Runs from different models are not paired.", + ), + ( + r"(\d+) 个失败是怎么造成的", + lambda m: f"What caused the {m.group(1)} failures", + ), + ( + r"(\d+) 个失败结果已经逐条核验。与任务合同、站点、共享环境或参考状态有关的任务不会进入诊断子集,修复后需完整重跑。", + lambda m: f"All {m.group(1)} failed results have been reviewed individually. Tasks affected by task-contract, site, shared-environment, or reference-state issues are excluded from the diagnostic subset and require a complete rerun after correction.", + ), + ( + r"以上共 (\d+) 个失败结果;每个结果只计一次。", + lambda m: f"The table covers {m.group(1)} failed results, each counted once.", + ), + ( + r"(\d+) 个模型 × (\d+) 个浏览器 × (\d+) 题都有最终判定", + lambda m: f"Every result has a final verdict: {m.group(1)} models × {m.group(2)} browsers × {m.group(3)} tasks", + ), + ( + r"另有 (\d+) 题在本轮没有成功轨迹", + lambda m: f"Another {m.group(1)} tasks have no successful trajectory in this run", + ), + (r"查看 (\d+) 个任务", lambda m: f"View {m.group(1)} tasks"), + (r"查看 (\d+) 个结果", lambda m: f"View {m.group(1)} results"), + ( + r"(\d+) 个任务中,Chromium 与 Moli 的差异在哪里", + lambda m: f"Where Chromium and Moli differ across {m.group(1)} tasks", + ), + ( + r"WebArena‑Verified 没有官方 easy、medium、hard 三档。本项目在运行前按最少可见 UI 操作数分为 1–4、5–7、8\+ 三档,各有 (\d+)、(\d+)、(\d+) 题。这个分组描述操作长度,不代表推理难度。", + lambda m: f"WebArena-Verified has no official easy, medium, or hard tiers. Before execution, this project grouped tasks by minimum visible UI actions: 1-4, 5-7, and 8+, containing {m.group(1)}, {m.group(2)}, and {m.group(3)} tasks. These groups describe action length, not reasoning difficulty.", + ), + ( + r"两次运行合计有 (\d+) 个同题结果不同。Sol 和 Astra 的方向并不一致,因此现有结果不足以支持某个浏览器在所有任务上更好。", + lambda m: f"Across the two runs, {m.group(1)} same-task results differ between browsers. The Sol and Astra runs point in different directions, so the current evidence does not show that one browser is better on every task.", + ), + ( + r"本轮固定 (\d+) 题,覆盖四个已验证站点。结果适用于当前 CowAgent、模型、任务和运行协议。它没有证明两个浏览器内核等价,也不能证明 Moli 已经可以全面替代 Chromium。端到端耗时包含模型、调度、页面等待和工具开销。", + lambda m: f"This run fixes {m.group(1)} tasks across four verified sites. Results apply to the current CowAgent, models, tasks, and run protocol. They do not establish browser-engine equivalence or show that Moli can fully replace Chromium. End-to-end time includes model, scheduling, page waits, and tool overhead.", + ), + ( + r"最终数据缺少 (\d+) 项;官方期望 (\d+) 项,匹配 (\d+) 项。", + lambda match: ( + f"The final data is missing {match.group(1)} item(s); " + f"{match.group(3)} of {match.group(2)} expected item(s) matched." + ), + ), + ( + r"Browser Use Capability Evaluation · (.+)", + lambda m: "Browser Use Capability Evaluation · " + + ("Not recorded" if m.group(1) == "未记录" else m.group(1)), + ), + ( + r"最终状态不符:期望 (.+),实际为 (.+)。", + lambda match: ( + f"Final status mismatch: expected {match.group(1)}, observed {match.group(2)}." + ), + ), + ( + r"在同一 CowAgent、同一 (.+?)(reasoning=low)和同一 WebArena‑Verified (\d+) 题样本下,对比两种浏览器完成真实网页任务的能力。", + lambda match: ( + f"Compare how Chromium and Moli complete real browser tasks with the same CowAgent, {match.group(1)} (reasoning=low), and the same {match.group(2)}-task WebArena-Verified sample." + ), + ), + ( + r"(.+?) 任务通过", + lambda match: f"{match.group(1)} tasks passed", + ), + ( + r"本轮观察到 (\d+) 个浏览器间不一致配对;结论只适用于这批受控任务,不证明内核等价或全面替代关系。", + lambda match: f"This run observed {match.group(1)} discordant browser pair(s). The conclusion applies only to this controlled cohort and does not establish browser-core equivalence or comprehensive substitutability.", + ), + ( + r"这是固定选择的 (\d+) 题试验,不是从全部 WebArena‑Verified 任务随机抽样;比例只描述本样本,不外推总体能力。", + lambda match: f"This is a fixed {match.group(1)}-task experiment, not a random sample of all WebArena-Verified tasks. Percentages describe this cohort only and are not population estimates.", + ), + ( + r"从 (\d+)/(\d+) 对 (\d+)/(\d+) 读出下一轮该怎么做", + lambda match: f"What {match.group(1)}/{match.group(2)} versus {match.group(3)}/{match.group(4)} implies for the next run", + ), + ( + r"(\d+)/(\d+) 对 (\d+)/(\d+):(\d+) 个任务出现浏览器分歧", + lambda match: f"{match.group(1)}/{match.group(2)} versus {match.group(3)}/{match.group(4)}: {match.group(5)} task(s) show a browser disagreement", + ), + ( + r"同一 CowAgent、模型、任务、站点状态和 evaluator 下,只替换浏览器后端。总分给出结果,(\d+) 个同题配对用于定位差异;每个失败结果还要沿官方合同、最终回答、网络证据和自主轨迹追到可复核根因。", + lambda match: f"The browser backend is the only changed variable; CowAgent, model, tasks, site state, and evaluator are held constant. Aggregate scores summarize outcomes, while {match.group(1)} paired tasks locate differences. Every failed result is then traced through the official contract, final answer, network evidence, and autonomous trajectory to a reviewable root cause.", + ), + ( + r"Chromium 与 Moli 分别通过 (\d+)/(\d+) 和 (\d+)/(\d+)。逐题配对后,(\d+) 题共同通过、(\d+) 题共同失败、(\d+) 题仅 Chromium 通过、(\d+) 题仅 Moli 通过。", + lambda m: f"Chromium and Moli passed {m.group(1)}/{m.group(2)} and {m.group(3)}/{m.group(4)}, respectively. Across paired tasks, both passed {m.group(5)}, both failed {m.group(6)}, Chromium alone passed {m.group(7)}, and Moli alone passed {m.group(8)}.", + ), + ( + r"表 1|(\d+) 个任务的逐题最终结果配对。", + lambda match: f"Table 1 | Paired final outcomes for {match.group(1)} tasks.", + ), + ( + r"固定样本中观察到 (\d+) 个不一致配对。这个数字用于发现浏览器相关差异,但不能单独证明整个浏览器内核的优劣,也不对 WebArena‑Verified 全集作统计外推。", + lambda match: f"The fixed cohort contains {match.group(1)} discordant pair(s). This identifies browser-related differences, but does not by itself establish overall browser-core superiority or support statistical generalization to the complete WebArena-Verified benchmark.", + ), + ( + r"WebArena‑Verified 没有官方 easy/medium/hard 三档。本项目在运行前按最少可见 UI 操作数注册为 1–4、5–7、8\+ 三档,配额 (\d+)/(\d+)/(\d+)。它控制操作链覆盖,不等同于推理难度。", + lambda m: f"WebArena-Verified has no official easy/medium/hard tiers. Before execution, this project registered tasks by minimum visible UI actions as 1-4, 5-7, and 8+, with quotas of {m.group(1)}/{m.group(2)}/{m.group(3)}. This controls operation-chain coverage and is not equivalent to reasoning difficulty.", + ), + ( + r"本轮固定 (\d+) 题,覆盖四个已验证站点。结果只说明当前 CowAgent、模型、任务与运行协议下的浏览器差异;不支持把总分直接解释为浏览器内核等价、全面替代关系或纯内核速度。端到端耗时包含模型、调度、页面等待和工具开销,动作延迟也包含 CDP 往返。", + lambda match: f"This run fixes {match.group(1)} tasks across four verified sites. Results describe browser differences only under the current CowAgent, model, tasks, and run protocol. Aggregate scores do not establish browser-core equivalence, comprehensive substitutability, or pure engine speed. End-to-end latency includes model, scheduling, page waits, and tool overhead; action latency also includes CDP round trips.", + ), + ( + r"(\d+) 个逻辑结果", + lambda match: f"{match.group(1)} logical results", + ), + ( + r"本轮剔除 (\d+) 题:(.+)。", + lambda match: ( + f"{match.group(1)} tasks excluded in this run: " + f"{match.group(2).replace('、', ', ')}." + ), + ), + ( + r"(\d+) 个任务×浏览器结果均保留最终回答、官方 evaluator 结果、Agent 事件和证据哈希。", + lambda match: f"All {match.group(1)} task-browser results retain the final answer, official evaluator output, agent events, and evidence hashes.", + ), + ( + r"(\d+)/(\d+) 个逻辑结果已覆盖,(\d+) 个未决。它不重新判题,也不是独立语义复核。", + lambda m: f"Evidence-integrity review covers {m.group(1)}/{m.group(2)} logical results, with {m.group(3)} unresolved. It does not rescore tasks or constitute independent semantic review.", + ), + (r"预注册 (\d+) 题", lambda m: f"Preregister {m.group(1)} tasks"), + ( + r"这 (\d+) 题如何部署并由 Agent 执行", + lambda m: f"How the {m.group(1)} tasks were deployed and run by the agent", + ), + ( + r"WebArena‑Verified 当前提供 Shopping、Shopping Admin、Reddit、GitLab、Wikipedia、Map 六类环境。本轮 (\d+) 题只覆盖前四类;Wikipedia 与 Map 需要额外大型数据卷,因此没有部署,也没有任务进入成绩。", + lambda m: f"WebArena-Verified provides six environments: Shopping, Shopping Admin, Reddit, GitLab, Wikipedia, and Map. This {m.group(1)}-task run covers only the first four. Wikipedia and Map require additional large data volumes, so they were not deployed and contribute no scored tasks.", + ), + ( + r"预注册文件只含任务 ID、层级和入选理由,不含点击序列。CowAgent 收到自然语言目标后,在最多 (.+) 步内循环读取页面观察、请求模型决策并生成下一次 BrowserTool 调用。Chromium 与 Moli 分别连接独立 CDP 浏览器容器;每个逻辑结果保留最终 JSON、Agent 事件流、网络 HAR 和 Playwright trace。", + lambda m: f"The preregistration contains task IDs, tiers, and selection rationales, but no click sequence. Given a natural-language objective, CowAgent repeatedly reads the page, requests a model decision, and generates the next BrowserTool call for up to {'an unrecorded number of' if m.group(1) == '未记录' else m.group(1)} steps. Chromium and Moli connect to separate CDP browser containers. Every logical result retains the final JSON, agent event stream, network HAR, and Playwright trace.", + ), + ( + r"未部署;不在 (\d+) 题", + lambda m: f"Not deployed; outside the {m.group(1)} tasks", + ), + ( + r"给出全部 (\d+) 个逻辑结果的页面与下载路径。", + lambda m: f"lists page and download paths for all {m.group(1)} logical results.", + ), + ( + r"同一 (\d+) 题;同一批本地站点镜像", + lambda m: f"Same {m.group(1)} tasks; same local site images", + ), + ( + r"相同 CPU、内存、viewport 与最多 (.+) 步", + lambda m: f"Same CPU, memory, viewport, and up to {'an unrecorded number of' if m.group(1) == '未记录' else m.group(1)} steps", + ), + ( + r"这 (\d+) 题由 CowAgent 自主执行。", + lambda m: f"CowAgent executes all {m.group(1)} tasks autonomously.", + ), + ( + r"仓库中的 Browser Trajectory Replay(轨迹重放)固定动作微基准属于另一条实验轨道,不是本报告的参评选手,也不计入这 (\d+) 题成绩。", + lambda m: f"The repository's fixed-action Browser Trajectory Replay microbenchmark is a separate evaluation track, not a participant in this report, and is excluded from these {m.group(1)} task results.", + ), + ( + r"CowAgent \+ (.+?)(reasoning=low)在预注册的 (\d+) 题 WebArena‑Verified 样本上对比 Chromium 与 Moli。", + lambda match: ( + f"CowAgent + {match.group(1)} (reasoning=low) compares Chromium and Moli on a preregistered {match.group(2)}-task WebArena-Verified sample." + ), + ), + (r"(\d+) 个完整配对", lambda match: f"{match.group(1)} complete pairs"), + ( + r"(\d+) 个共同失败任务没有任何成功轨迹。报告只能确认结果未满足官方合同,尚不能证明任务在当前快照可完成,也不能归因给 Agent 或浏览器。", + lambda match: ( + f"The {match.group(1)} shared-failure tasks have no successful trajectory in this run. The report establishes only that their official contracts were not met; it does not prove current-snapshot solvability or attribute failure to the agent or browser." + ), + ), + ( + r"(\d+) 个试次均保留回答、官方 evaluator 结果、Agent 事件和证据哈希;每题每浏览器首轮一次,只有失败任务再诊断重试一次。", + lambda match: ( + f"All {match.group(1)} attempts retain the answer, official evaluator result, agent events, and evidence hashes. Each task-browser pair has one primary run; only failures receive one diagnostic retry." + ), + ), + ( + r"(\d+)/(\d+) 个试次完成确定性一致性检查,(\d+) 个未决。它不重新判题,也不是独立语义复核。", + lambda match: ( + f"{match.group(1)}/{match.group(2)} attempts completed deterministic consistency checks, with {match.group(3)} unresolved. This does not rescore tasks and is not an independent semantic review." + ), + ), + ( + r"(\d+)/(\d+) 题至少有一条自主成功轨迹;其余 (\d+) 题仅有官方验收合同。", + lambda match: ( + f"{match.group(1)}/{match.group(2)} tasks have at least one autonomous successful trajectory; the remaining {match.group(3)} have only an official acceptance contract." + ), + ), + ( + r"(\d+) 项规则未通过:(.+)。", + lambda match: ( + f"{match.group(1)} rule{'s' if match.group(1) != '1' else ''} failed: " + + match.group(2).replace("规则 ", "Rule ").replace("、", ", ") + + "." + ), + ), + ) + for pattern, replacement in patterns: + match = re.fullmatch(pattern, core) + if match: + translated = replacement(match) + break + if translated is None: + translated = translate_common_text(core) + if translated == core: + for source, target in sorted( + PHRASES.items(), key=lambda item: len(item[0]), reverse=True + ): + translated = translated.replace(source, target) + if translated == core: + translated = None + return leading + (translated if translated is not None else core) + trailing + + +class Localizer(HTMLParser): + def __init__(self) -> None: + super().__init__(convert_charrefs=False) + self.parts: list[str] = [] + self.stack: list[str] = [] + + def handle_decl(self, decl: str) -> None: + self.parts.append(f"") + + def handle_starttag(self, tag: str, attrs) -> None: + self.stack.append(tag) + self.parts.append(self._tag(tag, attrs)) + + def handle_startendtag(self, tag: str, attrs) -> None: + self.parts.append(self._tag(tag, attrs, closed=True)) + + @staticmethod + def _tag(tag: str, attrs, *, closed: bool = False) -> str: + rendered = [] + for key, value in attrs: + if value is None: + rendered.append(key) + continue + if key in {"title", "aria-label", "placeholder", "alt", "data-search"}: + value = translate_text(value) + rendered.append(f'{key}="{html.escape(value, quote=True)}"') + suffix = " /" if closed else "" + return f"<{tag}{(' ' + ' '.join(rendered)) if rendered else ''}{suffix}>" + + def handle_endtag(self, tag: str) -> None: + if tag in self.stack: + self.stack = self.stack[: len(self.stack) - 1 - self.stack[::-1].index(tag)] + self.parts.append(f"") + + def handle_data(self, data: str) -> None: + if any(tag in self.stack for tag in ("pre", "code", "script", "style")): + self.parts.append(data) + else: + self.parts.append(translate_text(data)) + + def handle_entityref(self, name: str) -> None: + self.parts.append(f"&{name};") + + def handle_charref(self, name: str) -> None: + self.parts.append(f"&#{name};") + + def handle_comment(self, data: str) -> None: + self.parts.append(f"") + + +def _relative(source: PurePosixPath, target: PurePosixPath) -> str: + return posixpath.relpath(str(target), str(source.parent)) + + +def _rewrite_url(value: str, source: PurePosixPath, english: PurePosixPath) -> str: + parsed = urlsplit(html.unescape(value)) + if parsed.scheme or parsed.netloc or value.startswith(("mailto:", "data:", "javascript:")): + return value + if not parsed.path: + return value + # Generated report pages declare a root-relative , so local links are + # resolved from the report root even when the source page is nested. + resolved = PurePosixPath(posixpath.normpath(parsed.path.lstrip("/"))) + is_page = resolved.suffix.lower() in {".html", ".htm"} or not resolved.suffix + target = PurePosixPath("en") / resolved if is_page else resolved + return html.escape(urlunsplit(("", "", str(target), parsed.query, parsed.fragment)), quote=True) + + +def _rewrite_links(page: str, source: PurePosixPath, english: PurePosixPath) -> str: + pattern = re.compile(r'(?P\b(?:href|src)=)(?P["\'])(?P.*?)(?P=quote)') + return pattern.sub( + lambda match: ( + match.group("prefix") + + match.group("quote") + + _rewrite_url(match.group("url"), source, english) + + match.group("quote") + ), + page, + ) + + +def _header(source: PurePosixPath, english: PurePosixPath) -> str: + links = "".join( + f'{label}' + for path, label in NAV + ) + chinese = str(source) + return ( + '
    ' + ) + + +def localize_html(page: str, source: PurePosixPath) -> str: + english = PurePosixPath("en") / source + page = page.replace('', '', 1) + page = re.sub( + r"(.*?) · Browser Use 评测", + lambda match: ( + "" + + ( + f"Task {match.group(1)[3:]}" + if match.group(1).startswith("任务 ") + else translate_text(match.group(1)) + ) + + " · Browser Use evaluation" + ), + page, + count=1, + ) + page = re.sub( + r'
    .*?
    ', + "
    ", + page, + count=1, + flags=re.S, + ) + had_base = bool(re.search(r'', page, flags=re.I)) + page = re.sub( + r'', "", page, count=1, flags=re.I + ) + parser = Localizer() + parser.feed(page) + parser.close() + localized = _rewrite_links("".join(parser.parts), source, english) + localized = localized.replace("
    ", _header(source, english), 1) + if had_base: + base = _relative(english, PurePosixPath(".")) + "/" + localized = localized.replace("", f'', 1) + return localized + + +def write_english_report(out: Path) -> int: + """Mirror every Chinese HTML route and return the route count.""" + english_root = out / "en" + if english_root.exists(): + shutil.rmtree(english_root) + sources = sorted( + path for path in out.rglob("*.html") if "en" not in path.relative_to(out).parts + ) + for source_path in sources: + source = PurePosixPath(source_path.relative_to(out).as_posix()) + target = out / "en" / Path(str(source)) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text( + localize_html(source_path.read_text(encoding="utf-8"), source), encoding="utf-8" + ) + return len(sources) diff --git a/harness/browser_use/review.py b/harness/browser_use/review.py new file mode 100644 index 000000000..a8930a1eb --- /dev/null +++ b/harness/browser_use/review.py @@ -0,0 +1,168 @@ +"""Deterministic evidence review for autonomous browser-use attempts. + +This review verifies evidence integrity. It does not replace or reinterpret the +official WebArena-Verified evaluator score. +""" + +from __future__ import annotations + +import hashlib +import json +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +REQUIRED_ARTIFACTS = ("network_har", "agent_trace", "response", "evaluation", "events") + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def _attempts_sha256(path: Path) -> str: + return _sha256(path) + + +def _read_attempts(path: Path) -> list[dict[str, Any]]: + return [ + json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip() + ] + + +def review_results(*, evidence: Path, selection_sha256: str) -> dict[str, Any]: + """Review every recorded attempt and return a result bound to its input bytes.""" + attempts_path = evidence / "attempts.jsonl" + manifest_path = evidence / "manifest.json" + if not attempts_path.is_file() or not manifest_path.is_file(): + raise ValueError("result review requires attempts.jsonl and manifest.json") + + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + if manifest.get("selection_sha256") != selection_sha256: + raise ValueError("manifest does not match the pre-registered selection") + + rows = _read_attempts(attempts_path) + model = str((manifest.get("protocol") or {}).get("model") or "") + if not model: + raise ValueError("manifest protocol must identify the evidence model") + required_artifacts = list(REQUIRED_ARTIFACTS) + if (manifest.get("protocol") or {}).get("environment_prepare_receipt") == "required": + required_artifacts.append("environment_prepare") + reviewed: list[dict[str, Any]] = [] + disagreements: list[dict[str, Any]] = [] + unresolved: list[dict[str, Any]] = [] + expected_provenance = manifest.get("environment", {}).get("official_evaluator", {}) + + for row in rows: + row_model = str(row.get("model") or model) + if row_model != model: + raise ValueError("attempt model differs from the evidence manifest") + identity = { + "model": model, + "engine": str(row.get("engine")), + "task_id": int(row.get("task_id")), + "attempt": int(row.get("attempt")), + } + paths = row.get("paths") or {} + checks: dict[str, bool] = {} + artifacts: dict[str, dict[str, Any]] = {} + for name in required_artifacts: + raw_path = paths.get(name) + path = Path(str(raw_path)) if raw_path else Path() + exists = bool(raw_path) and path.is_file() + checks[f"{name}_exists"] = exists + if exists: + artifacts[name] = {"sha256": _sha256(path), "bytes": path.stat().st_size} + + evaluation: dict[str, Any] = {} + response: Any = None + if checks["evaluation_exists"]: + try: + evaluation = json.loads(Path(str(paths["evaluation"])).read_text(encoding="utf-8")) + checks["evaluation_json"] = isinstance(evaluation, dict) + except (OSError, json.JSONDecodeError): + checks["evaluation_json"] = False + else: + checks["evaluation_json"] = False + if checks["response_exists"]: + try: + response = json.loads(Path(str(paths["response"])).read_text(encoding="utf-8")) + checks["response_json"] = True + except (OSError, json.JSONDecodeError): + checks["response_json"] = False + else: + checks["response_json"] = False + + checks["response_matches_record"] = checks["response_json"] and response == row.get( + "agent_response" + ) + checks["score_matches_evaluator"] = checks["evaluation_json"] and float( + evaluation.get("score") or 0.0 + ) == float(row.get("score") or 0.0) + checks["status_matches_evaluator"] = checks["evaluation_json"] and str( + evaluation.get("status") or "error" + ) == str(row.get("status") or "error") + checks["official_evaluator_version_present"] = bool( + evaluation.get("webarena_verified_version") + ) + checks["official_evaluator_checksum_matches"] = bool( + expected_provenance.get("evaluator_checksum") + ) and evaluation.get("webarena_verified_evaluator_checksum") == expected_provenance.get( + "evaluator_checksum" + ) + checks["official_data_checksum_matches"] = bool( + expected_provenance.get("data_checksum") + ) and evaluation.get("webarena_verified_data_checksum") == expected_provenance.get( + "data_checksum" + ) + usage = row.get("model_usage") or {} + transport_errors = row.get("model_transport_errors") or [] + checks["model_request_attempts_accounted"] = isinstance(transport_errors, list) and int( + usage.get("request_attempt_count") or 0 + ) == int(usage.get("request_count") or 0) + len(transport_errors) + + failed = sorted(name for name, passed in checks.items() if not passed) + item = {**identity, "outcome": "verified" if not failed else "unresolved", "checks": checks} + item["artifacts"] = artifacts + reviewed.append(item) + if not checks["score_matches_evaluator"] or not checks["status_matches_evaluator"]: + disagreements.append({**identity, "failed_checks": failed}) + if failed: + unresolved.append({**identity, "failed_checks": failed}) + + return { + "schema": "lex_browser_use_result_review/1", + "reviewer": { + "name": "Browser Use evidence auditor", + "kind": "deterministic programmatic review", + "version": 1, + }, + "reviewed_at": datetime.now(UTC).isoformat(), + "bindings": { + "attempts_sha256": _attempts_sha256(attempts_path), + "manifest_sha256": _sha256(manifest_path), + "selection_sha256": selection_sha256, + }, + "protocol": { + "scope": "evidence completeness, byte identity, and recorded score/status agreement", + "score_authority": "unmodified WebArena-Verified evaluator", + "manual_score_override": "not permitted", + "adjudication": ( + "any disagreement or missing artifact remains unresolved and blocks publication" + ), + }, + "coverage": {"recorded_attempts": len(rows), "reviewed_attempts": len(reviewed)}, + "disagreements": disagreements, + "unresolved": unresolved, + "attempts": reviewed, + } + + +def write_review(*, evidence: Path, selection_sha256: str) -> Path: + review = review_results(evidence=evidence, selection_sha256=selection_sha256) + path = evidence / "result-review.json" + path.write_text(json.dumps(review, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + return path diff --git a/harness/browser_use/runner.py b/harness/browser_use/runner.py index caf6a0a94..643de58d9 100644 --- a/harness/browser_use/runner.py +++ b/harness/browser_use/runner.py @@ -1,633 +1,1205 @@ -"""Docker-isolated browser use evaluation runner.""" +"""Run CowAgent autonomously against official self-hosted WebArena-Verified sites.""" from __future__ import annotations +import argparse +import fcntl import hashlib +import importlib.metadata +import ipaddress import json +import os import platform -import threading +import subprocess +import sys +import tempfile import time import traceback -import urllib.parse -import urllib.request import uuid -from functools import partial -from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer +from collections.abc import Iterator +from contextlib import contextmanager from pathlib import Path from typing import Any +from urllib.parse import urlparse import docker -import tiktoken -from playwright.sync_api import Browser, Page, sync_playwright - -from harness.container_isolation import ContainerLimits, require_linux_cgroup_v2 +import requests +from dotenv import dotenv_values +from json_repair import repair_json +from webarena_verified import WebArenaVerified +from webarena_verified.environments.container.config import get_container_config +from webarena_verified.environments.container.manager import ContainerManager +from webarena_verified.environments.env_ctrl_client import EnvCtrlDockerClient +from webarena_verified.types.config import ( + ContainerConfig, + ContainerSetupConfig, + ContainerVolumeSpec, + EnvironmentConfig, + WebArenaVerifiedConfig, +) +from webarena_verified.types.task import WebArenaSite -from .diagnostics import diagnose_summary -from .executor import execute_steps, judge_outputs, validate_outputs +from harness.container_isolation import require_linux_cgroup_v2 from harness.resource_accounting import ( DockerStatsResources, ResourceSampler, delta_counters, ) -from .metrics import summarize_attempts -from .report import write_latest_report -from .suite import BrowserTask, load_suite, validate_ground_truth_evidence +from harness.trajectory_replay.runner import ( + ENGINE_IMAGES, + ENGINE_PORTS, + _published_endpoint, + _start_browser_container, + _wait_cdp, +) -ENGINE_IMAGES = { - "chromium": "lex-browser-use-eval-chromium:local", - "moli": "lex-browser-use-eval-moli:local", +from .har import CdpHarRecorder +from .model import OpenAIResponsesModel +from .selection import PilotTask, load_pilot, validate_against_official + +SCHEMA = "lex_browser_use_attempt/1" +SITE_PORTS = { + WebArenaSite.SHOPPING: (7770, 7771), + WebArenaSite.SHOPPING_ADMIN: (7780, 7781), + WebArenaSite.REDDIT: (9999, 9998), + WebArenaSite.GITLAB: (8023, 8024), +} +SITE_CREDENTIALS = { + WebArenaSite.SHOPPING: {"username": "emma.lopez@gmail.com", "password": "Password.123"}, + WebArenaSite.SHOPPING_ADMIN: {"username": "admin", "password": "admin1234"}, + WebArenaSite.REDDIT: {"username": "MarvelsGrantMan136", "password": "test1234"}, + WebArenaSite.GITLAB: {"username": "byteblaze", "password": "hello1234"}, } -ENGINE_PORTS = {"chromium": 9223, "moli": 9222} -ENCODING = tiktoken.get_encoding("o200k_base") +SITE_URL_SUFFIX = {WebArenaSite.SHOPPING_ADMIN: "/admin"} +SITE_AUTH_HEADERS = { + WebArenaSite.SHOPPING: "X-M2-Customer-Auto-Login", + WebArenaSite.SHOPPING_ADMIN: "X-M2-Admin-Auto-Login", + WebArenaSite.REDDIT: "X-Postmill-Auto-Login", +} +# The official all-in-one shopping image restores Magento, MariaDB and +# Elasticsearch before env-ctrl declares the fixture ready. On Colima/QEMU this +# repeatedly exceeded the package example's 180-second default despite becoming +# healthy shortly afterwards. A longer ceiling changes no measured task time; +# it only keeps environment preparation from aborting a formal cohort. +ENV_CTRL_READY_TIMEOUT_SECONDS = 600 +MAGENTO_DEPENDENCY_READY_TIMEOUT_SECONDS = 300 +PHYSICAL_EXECUTION_LOCK_NAMESPACE = "webarena-verified-fixed-site-stack-v1" +AMD64_PHP_SITES = frozenset( + {WebArenaSite.SHOPPING, WebArenaSite.SHOPPING_ADMIN, WebArenaSite.REDDIT} +) +PHP_FPM_COMPAT_CONFIG = Path(__file__).with_name("runtime") / "php-fpm-qemu.ini" +QEMU_X86_64_HOST_PATH = "/usr/bin/qemu-x86_64" +QEMU_X86_64_CONTAINER_PATH = "/usr/local/bin/qemu-x86_64" + + +def _docker_daemon_lock_namespace(docker_info: dict[str, Any]) -> str: + """Bind a serialization lane to one Docker daemon and its fixed site ports.""" + daemon_id = str(docker_info.get("ID") or "").strip() + if not daemon_id: + raise RuntimeError("Docker daemon ID is required for environment-lane isolation") + suffix = hashlib.sha256(daemon_id.encode("utf-8")).hexdigest()[:16] + return f"{PHYSICAL_EXECUTION_LOCK_NAMESPACE}-{suffix}" -def _external_websocket_url(endpoint: str, advertised: str) -> str: - endpoint_parts = urllib.parse.urlsplit(endpoint) - websocket_parts = urllib.parse.urlsplit(advertised) - scheme = "wss" if endpoint_parts.scheme == "https" else "ws" - return urllib.parse.urlunsplit( - (scheme, endpoint_parts.netloc, websocket_parts.path, websocket_parts.query, "") +def _physical_execution_lock_path( + lock_namespace: str = PHYSICAL_EXECUTION_LOCK_NAMESPACE, +) -> Path: + # The WebArena stack uses fixed host ports and is shared by every run owned + # by this OS user. The lock therefore must not follow BROWSER_EVAL_DATA_HOME: + # two cohorts may intentionally write to different evidence directories + # while still mutating the same containers and databases. + return ( + Path(tempfile.gettempdir()) + / f"browser-eval-{os.getuid()}" + / "locks" + / f"{lock_namespace}.lock" ) -def _wait_cdp(url: str, timeout_seconds: float = 60, *, container: Any = None) -> str: +@contextmanager +def _physical_execution_lock(lock_path: Path) -> Iterator[float]: + """Serialize the full attempt lifecycle for the shared fixed-port site stack.""" + lock_path.parent.mkdir(parents=True, exist_ok=True) + with lock_path.open("a+b") as lock_file: + wait_started = time.perf_counter() + fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX) + wait_ms = (time.perf_counter() - wait_started) * 1000 + try: + yield wait_ms + finally: + fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN) + + +def _environment_prepare_receipt( + sites: tuple[WebArenaSite, ...], *, lock_wait_ms: float, lock_namespace: str +) -> dict[str, Any]: + return { + "required_sites": [site.value for site in sites], + "reset_requested": True, + "physical_execution_lock": { + "namespace": lock_namespace, + "wait_ms": round(lock_wait_ms, 3), + }, + "status": "running", + } + + +def _load_repo_env(repo_root: Path) -> dict[str, str]: + """Load repository defaults, with explicit process overrides for automation.""" + env_file = repo_root / ".env" + values = ( + {str(key): str(value) for key, value in dotenv_values(env_file).items() if value} + if env_file.exists() + else {} + ) + for key in ("OPENAI_API_KEY", "OPENAI_BASE_URL", "COWAGENT_REPO"): + if os.environ.get(key): + values[key] = os.environ[key] + return values + + +def _git_head(path: Path) -> str: + return subprocess.run( + ["git", "-C", str(path), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def _require_clean_git_checkout(path: Path) -> str: + commit = _git_head(path) + status = subprocess.run( + ["git", "-C", str(path), "status", "--porcelain"], + check=True, + capture_output=True, + text=True, + ).stdout + if status.strip(): + raise RuntimeError( + "CowAgent checkout must be clean so its recorded commit identifies the code" + ) + return commit + + +def _cdp_metadata(http_endpoint: str) -> dict[str, str]: + response = requests.get(http_endpoint.rstrip("/") + "/json/version", timeout=15) + response.raise_for_status() + payload = response.json() + return { + key: str(payload[key]) + for key in ("Browser", "Protocol-Version", "User-Agent", "V8-Version") + if payload.get(key) + } + + +def _docker_image_metadata(image: Any) -> dict[str, Any]: + attrs = image.attrs + return { + "id": image.id, + "tags": image.tags, + "repo_digests": attrs.get("RepoDigests") or [], + "architecture": attrs.get("Architecture"), + "os": attrs.get("Os"), + "size_bytes": attrs.get("Size"), + } + + +def _container_host_gateway(container: Any) -> str: + """Return the literal host-gateway address visible to the browser process. + + CowAgent validates URLs in the host process before forwarding them over CDP. + On Linux/Colima, ``host.docker.internal`` can exist only in the container's + /etc/hosts, so a literal RFC1918 address is required on both sides. + """ + result = container.exec_run(["getent", "hosts", "host.docker.internal"]) + if int(result.exit_code) != 0: + raise RuntimeError("browser container cannot resolve host.docker.internal") + line = result.output.decode("utf-8", errors="replace").splitlines()[0] + address = line.split()[0] + ipaddress.ip_address(address) + return address + + +def _limit_cowagent_transport_retries(executor_class: Any) -> None: + """Limit CowAgent API errors to the initial request plus one retry.""" + if getattr(executor_class, "_lex_browser_eval_retry_limit", False): + return + original = executor_class._call_llm_stream + + def limited( + executor: Any, + retry_on_empty: bool = True, + retry_count: int = 0, + max_retries: int = 1, + _overflow_stage: int = 0, + ) -> Any: + return original( + executor, + retry_on_empty=retry_on_empty, + retry_count=retry_count, + max_retries=min(int(max_retries), 1), + _overflow_stage=_overflow_stage, + ) + + executor_class._call_llm_stream = limited + executor_class._lex_browser_eval_retry_limit = True + + +def _allow_official_container_hostname(tool: Any) -> None: + """Allow only the hostname emitted by the official self-hosted images.""" + original = tool._validate_url_safe + + def validate(url: str) -> None: + if urlparse(url).hostname == "host.docker.internal": + return + original(url) + + tool._validate_url_safe = validate + + +def _configure_benchmark(hostname: str = "host.docker.internal") -> WebArenaVerified: + def site_url(site: WebArenaSite, host: str) -> str: + return f"http://{host}:{SITE_PORTS[site][0]}{SITE_URL_SUFFIX.get(site, '')}" + + environments = { + site: EnvironmentConfig( + # The gateway IP is active. The official images may redirect to + # host.docker.internal, so retain that as an equivalent URL for the + # package's own derendering and NetworkEventEvaluator logic. + urls=list( + dict.fromkeys([site_url(site, hostname), site_url(site, "host.docker.internal")]) + ), + credentials=SITE_CREDENTIALS[site], + use_header_login=site == WebArenaSite.SHOPPING_ADMIN, + ) + for site in SITE_PORTS + } + return WebArenaVerified(config=WebArenaVerifiedConfig(environments=environments)) + + +def _wait_container_healthy(container: Any, *, timeout_seconds: float = 600) -> None: + """Wait past GitLab's delayed reconfigure/Puma restart before app probing.""" deadline = time.monotonic() + timeout_seconds - last_error: Exception | None = None + last_status = "missing" while time.monotonic() < deadline: - try: - with urllib.request.urlopen(f"{url}/json/version", timeout=2) as response: - if response.status == 200: - payload = json.load(response) - advertised = str(payload.get("webSocketDebuggerUrl") or "") - if advertised: - return _external_websocket_url(url, advertised) - raise RuntimeError("CDP version response omitted webSocketDebuggerUrl") - except Exception as exc: - last_error = exc - time.sleep(0.25) - logs = "" - if container is not None: - try: - container.reload() - logs = container.logs(tail=30).decode("utf-8", errors="replace") - except Exception: - pass - detail = f"; container logs:\n{logs}" if logs else "" - raise RuntimeError(f"CDP endpoint did not become ready: {last_error}{detail}") + container.reload() + health = (container.attrs.get("State") or {}).get("Health") or {} + last_status = str(health.get("Status") or "missing") + if last_status == "healthy": + return + if last_status == "unhealthy": + logs = health.get("Log") or [] + detail = str(logs[-1].get("Output") or "")[-500:] if logs else "" + raise RuntimeError(f"GitLab container became unhealthy: {detail}") + time.sleep(2) + raise RuntimeError(f"GitLab container health timed out in state {last_status!r}") -def _http_origin(url: str) -> str | None: - parts = urllib.parse.urlsplit(url) - if parts.scheme not in {"http", "https"} or not parts.netloc: - return None - return f"{parts.scheme}://{parts.netloc}" +def _site_container_manager(site: WebArenaSite) -> ContainerManager: + """Use QEMU only for PHP-FPM when amd64 site images run on an ARM host. + Rosetta starts the official images' Java services reliably, but PHP-FPM in + both Magento images segfaults under Rosetta. Mounting Colima's static QEMU + interpreter and a one-line supervisor override keeps every other process on + Rosetta while preserving the official image and site data. + """ + config = get_container_config(site=site) + docker_arch = str(docker.from_env().info().get("Architecture") or "") + if site not in AMD64_PHP_SITES or docker_arch not in {"aarch64", "arm64"}: + return ContainerManager(site=site, hostname="host.docker.internal") + if not PHP_FPM_COMPAT_CONFIG.is_file(): + raise RuntimeError(f"missing PHP-FPM compatibility config: {PHP_FPM_COMPAT_CONFIG}") + compat = ContainerConfig( + **config.model_dump(exclude={"setup"}), + setup=ContainerSetupConfig( + volumes=( + ContainerVolumeSpec( + volume_name=QEMU_X86_64_HOST_PATH, + mount_path=QEMU_X86_64_CONTAINER_PATH, + ), + ContainerVolumeSpec( + volume_name=str(PHP_FPM_COMPAT_CONFIG.resolve()), + mount_path="/etc/supervisor.d/php-fpm.ini:ro", + ), + ) + ), + ) + return ContainerManager( + site=site, + config=compat, + hostname="host.docker.internal", + ) -def _reset_state(page: Page, visited_origins: set[str]) -> None: - context = page.context - origins = set(visited_origins) - current_origin = _http_origin(page.url) - if current_origin: - origins.add(current_origin) - try: - page.evaluate( - """async () => { - try { localStorage.clear(); } catch (_) {} - try { sessionStorage.clear(); } catch (_) {} - try { for (const key of await caches.keys()) await caches.delete(key); } catch (_) {} - try { - for (const item of await navigator.serviceWorker.getRegistrations()) - await item.unregister(); - } catch (_) {} - try { - for (const item of await indexedDB.databases()) - indexedDB.deleteDatabase(item.name); - } catch (_) {} - }""" + +def _verify_arm_php_compatibility(container_name: str) -> None: + container = docker.from_env().containers.get(container_name) + processes = container.exec_run(["ps", "-o", "pid,args"]) + process_table = processes.output.decode(errors="replace") + qemu = container.exec_run(["test", "-x", QEMU_X86_64_CONTAINER_PATH]) + if processes.exit_code != 0 or "/mnt/lima-rosetta/rosetta" not in process_table: + raise RuntimeError("ARM Docker host must enable Rosetta for official amd64 site images") + if qemu.exit_code != 0: + raise RuntimeError("PHP-FPM QEMU compatibility interpreter was not mounted") + if f"{QEMU_X86_64_CONTAINER_PATH} /usr/local/sbin/php-fpm" not in process_table: + raise RuntimeError("PHP-FPM did not start through the mounted QEMU interpreter") + + +def _site_runtime_compatibility(docker_arch: str) -> dict[str, Any]: + if docker_arch not in {"aarch64", "arm64"}: + return {"mode": "host-default"} + return { + "mode": "rosetta-with-qemu-php-fpm", + "sites": sorted(site.value for site in AMD64_PHP_SITES), + "qemu_host_path": QEMU_X86_64_HOST_PATH, + "supervisor_config_sha256": hashlib.sha256(PHP_FPM_COMPAT_CONFIG.read_bytes()).hexdigest(), + } + + +def _ensure_site(site: WebArenaSite, *, reset: bool) -> None: + port, ctrl_port = SITE_PORTS[site] + manager = _site_container_manager(site) + if reset or not manager.is_running(): + manager.start(port=port, env_ctrl_port=ctrl_port, wait=False) + ready = EnvCtrlDockerClient.create( + manager.container_name, + timeout=ENV_CTRL_READY_TIMEOUT_SECONDS, + ).wait_until_ready( + timeout=ENV_CTRL_READY_TIMEOUT_SECONDS, + interval=2, + ) + if not ready.success: + raise RuntimeError(f"{site.value} did not become ready: {ready.message}") + container = docker.from_env().containers.get(manager.container_name) + if site == WebArenaSite.GITLAB: + _wait_container_healthy(container) + if site in {WebArenaSite.SHOPPING, WebArenaSite.SHOPPING_ADMIN}: + _wait_magento_dependencies_ready(container) + _wait_application_ready(site, manager.config.health_check_path) + if QEMU_X86_64_HOST_PATH in manager.config.volumes: + _verify_arm_php_compatibility(manager.container_name) + + +def _wait_magento_dependencies_ready( + container: Any, *, timeout_seconds: float = MAGENTO_DEPENDENCY_READY_TIMEOUT_SECONDS +) -> None: + """Wait for Magento's search dependency after fixture restoration. + + The official env-ctrl ready signal can precede Elasticsearch accepting + requests. Magento write paths then fail after committing their primary + record because the synchronous search-index update cannot connect. Waiting + for a non-red cluster removes that infrastructure race from measured tasks. + """ + deadline = time.monotonic() + timeout_seconds + last_detail = "no response" + while time.monotonic() < deadline: + result = container.exec_run( + [ + "curl", + "-fsS", + "http://127.0.0.1:9200/_cluster/health?wait_for_status=yellow&timeout=5s", + ] ) - except Exception: - pass - context.clear_cookies() - context.clear_permissions() - try: - cdp = context.new_cdp_session(page) - cdp.send("Network.enable") - cdp.send("Network.clearBrowserCache") - cdp.send("Network.clearBrowserCookies") - for origin in sorted(origins): + output = result.output.decode("utf-8", errors="replace") + last_detail = output[-500:] or f"exit {result.exit_code}" + if result.exit_code == 0: try: - cdp.send( - "Storage.clearDataForOrigin", - {"origin": origin, "storageTypes": "all"}, - ) - except Exception: - pass - page.goto("about:blank", wait_until="load", timeout=10_000) + status = str(json.loads(output).get("status") or "") + except json.JSONDecodeError: + status = "" + if status in {"yellow", "green"}: + return + time.sleep(2) + raise RuntimeError(f"Magento Elasticsearch was not ready: {last_detail}") + + +def _wait_application_ready( + site: WebArenaSite, + health_path: str, + *, + timeout_seconds: float = 300, + stable_responses: int = 3, +) -> None: + """Require a stable app endpoint after env-ctrl reports ready. + + GitLab's control process can report ready during the Puma handover performed + by container initialization. A single successful request can therefore be + followed by a run of 502 responses. Consecutive app-level responses prevent + that startup race from consuming an Agent attempt. + """ + port = SITE_PORTS[site][0] + url = f"http://127.0.0.1:{port}{health_path}" + deadline = time.monotonic() + timeout_seconds + consecutive = 0 + last_detail = "no response" + while time.monotonic() < deadline: try: - cdp.send("Page.resetNavigationHistory") + # The official shopping images redirect to their configured + # host.docker.internal origin. That name is intentionally scoped to + # browser containers and may not resolve on the macOS host. A stable + # pre-redirect response proves the app port is serving; _login_context + # then validates the rendered authenticated page from inside the + # actual browser before tracing or model calls begin. + response = requests.get(url, timeout=15, allow_redirects=False) + last_detail = f"HTTP {response.status_code}" + consecutive = consecutive + 1 if response.status_code < 500 else 0 + if consecutive >= stable_responses: + return + except requests.RequestException as exc: + consecutive = 0 + last_detail = str(exc) + time.sleep(2) + raise RuntimeError(f"{site.value} application was not stably ready: {last_detail}") + + +def _prepare_sites(sites: tuple[WebArenaSite, ...]) -> None: + """Keep only task-required sites running, then reset those sites from official images.""" + for site in SITE_PORTS: + manager = _site_container_manager(site) + if site not in sites and manager.is_running(): + manager.stop() + for site in sites: + _ensure_site(site, reset=True) + + +def _login_context( + service: Any, + sites: tuple[WebArenaSite, ...], + benchmark: WebArenaVerified, + start_urls: list[str], +) -> int: + def login() -> None: + context = service._context + assert context is not None + primary_page = service._page + assert primary_page is not None + for existing_page in context.pages: + if existing_page != primary_page: + existing_page.close() + context.clear_cookies() + context.set_extra_http_headers( + { + SITE_AUTH_HEADERS[site]: ( + f"{SITE_CREDENTIALS[site]['username']}:{SITE_CREDENTIALS[site]['password']}" + ) + for site in sites + if site in SITE_AUTH_HEADERS + } + ) + for site in sites: + base = benchmark.config.get_environment(site).active_url + creds = SITE_CREDENTIALS[site] + page = context.new_page() try: - cdp.send("HeapProfiler.enable") - cdp.send("HeapProfiler.collectGarbage") - cdp.send("HeapProfiler.disable") - except Exception: - pass - finally: - cdp.detach() + page.set_default_timeout(60_000) + page.set_default_navigation_timeout(90_000) + if site == WebArenaSite.SHOPPING: + page.goto(f"{base}/customer/account", wait_until="commit") + page.evaluate("localStorage.clear(); sessionStorage.clear()") + page.reload(wait_until="domcontentloaded") + page.locator(".block-dashboard-info, .box-information").first.wait_for() + elif site == WebArenaSite.SHOPPING_ADMIN: + page.goto(base, wait_until="commit") + page.evaluate("localStorage.clear(); sessionStorage.clear()") + page.reload(wait_until="domcontentloaded") + page.locator(".page-title").wait_for() + elif site == WebArenaSite.REDDIT: + page.goto(base, wait_until="commit") + page.evaluate("localStorage.clear(); sessionStorage.clear()") + page.reload(wait_until="domcontentloaded") + page.get_by_role("button", name=creds["username"], exact=True).wait_for() + elif site == WebArenaSite.GITLAB: + page.goto(f"{base}/users/sign_in", wait_until="commit") + page.evaluate("localStorage.clear(); sessionStorage.clear()") + page.reload(wait_until="domcontentloaded") + page.get_by_test_id("username-field").fill(creds["username"]) + page.get_by_test_id("password-field").fill(creds["password"]) + page.get_by_test_id("sign-in-button").click() + page.wait_for_url("**/", timeout=90_000) + page.wait_for_timeout(1_000) + finally: + page.close() + primary_page.set_default_timeout(60_000) + primary_page.set_default_navigation_timeout(90_000) + primary_page.goto(start_urls[0], wait_until="domcontentloaded") + primary_page.wait_for_timeout(500) + + readiness_retries = 0 + try: + service._submit(login) + except Exception as exc: + # Moli can expose its CDP endpoint just before the default browser + # context is ready. This is a session precondition failure: no Agent + # step has started and no score-bearing attempt exists yet. Recover it + # once, record the recovery on the eventual attempt, and keep every + # other setup failure fatal. + if "BrowserContextNotLoaded" not in str(exc): + raise + readiness_retries = 1 + time.sleep(1) + service._submit(login) + return readiness_retries + + +def _system_prompt() -> str: + return """You are the autonomous web agent under evaluation. Use only the browser tool. +Complete the objective from the supplied start URLs. Do not ask for help, download files, or +change state unless the objective requires it. When finished, return only one JSON object with +task_type (RETRIEVE, MUTATE, or NAVIGATE), status (SUCCESS, NOT_FOUND_ERROR, +ACTION_NOT_ALLOWED_ERROR, PERMISSION_DENIED_ERROR, DATA_VALIDATION_ERROR, or UNKNOWN_ERROR), +retrieved_data, and error_details. Treat search and list pages as candidate sets, not proof that +every result satisfies the objective: verify every included item against all stated constraints +using its title, category, breadcrumb, detail page, or other visible site evidence before sorting, +aggregating, or mutating. Before returning, check semantic eligibility, completeness, arithmetic, +and any required saved or submitted state. A successful RETRIEVE returns a list. If an exhaustive +RETRIEVE finds no matching item, return NOT_FOUND_ERROR with retrieved_data null. For MUTATE and +NAVIGATE, retrieved_data is null.""" + + +def _exhausted_agent_steps(events: list[dict[str, Any]], max_steps: int) -> bool: + """Return whether CowAgent used its final decision turn for another tool call. + + CowAgent's max-step fallback asks the model for a prose summary without tools. That + fallback is not a benchmark completion and may claim success even though the last + browser action has not been observed or verified. + """ + + turns = [event for event in events if event.get("type") == "turn_end"] + if not turns: + return False + data = turns[-1].get("data") or {} + return len(turns) >= max_steps and bool(data.get("has_tool_calls")) + + +def _max_steps_response(task_type: str, max_steps: int) -> dict[str, Any]: + return { + "task_type": task_type.upper(), + "status": "UNKNOWN_ERROR", + "retrieved_data": None, + "error_details": ( + f"Agent reached the {max_steps}-step decision limit before producing " + "a tool-free, post-action completion." + ), + } + + +def _user_prompt(task: Any, start_urls: list[str]) -> str: + response_eval = next(item for item in task.eval if item.evaluator == "AgentResponseEvaluator") + return ( + "Task objective:\n" + + task.intent + + "\n\nStart URLs:\n" + + "\n".join(f"- {url}" for url in start_urls) + + "\n\nOfficial response contract (schema only; this does not reveal the answer):\n" + + f"- task_type: {str(response_eval.expected.task_type).upper()}\n" + + "- retrieved_data JSON Schema: " + + json.dumps( + response_eval.model_dump(mode="json")["results_schema"], + ensure_ascii=False, + separators=(",", ":"), + ) + ) + + +def _parse_agent_response(text: str) -> dict[str, Any] | str: + try: + repaired = repair_json(text, return_objects=True) except Exception: - page.goto("about:blank", wait_until="load", timeout=10_000) + return text + return repaired if isinstance(repaired, dict) else text -def _resource_result( - source: DockerStatsResources, +def _resource_summary( + resources: DockerStatsResources, sampler: ResourceSampler, before: Any, after: Any, duration_ms: float, -) -> dict[str, int | float]: +) -> dict[str, Any]: result = {**delta_counters(before, after), **sampler.summary()} if duration_ms > 0: result["cpu_average_utilization_pct"] = result["cpu_ms"] / duration_ms * 100 - result["network_rx_bytes_per_second"] = result["network_rx_bytes"] / duration_ms * 1000 - result["network_tx_bytes_per_second"] = result["network_tx_bytes"] / duration_ms * 1000 return result -def _published_endpoint(container: Any, *, engine: str, cdp_port: int) -> str: - ports = (container.attrs.get("NetworkSettings") or {}).get("Ports") or {} - bindings = ports.get(f"{cdp_port}/tcp") - if not bindings or not bindings[0].get("HostPort"): - raise RuntimeError(f"{engine}: Docker did not publish CDP port") - return f"http://127.0.0.1:{bindings[0]['HostPort']}" - - -def _prepare_page(page: Page, visited_origins: set[str], *, settle_ms: int) -> Page: - """Reset task state and replace only the renderer page, keeping the browser/context warm.""" - context = page.context - _reset_state(page, visited_origins) - visited_origins.clear() - page.close(run_before_unload=False) - replacement = context.new_page() - replacement.set_viewport_size({"width": 1280, "height": 900}) - viewport = replacement.evaluate("() => ({width: innerWidth, height: innerHeight})") - if viewport != {"width": 1280, "height": 900}: - raise RuntimeError(f"browser viewport mismatch: {viewport}") - replacement.wait_for_timeout(max(0, settle_ms)) - return replacement +def _run_attempt( # noqa: PLR0913 + *, + lock_path: Path, + lock_namespace: str = PHYSICAL_EXECUTION_LOCK_NAMESPACE, + **kwargs: Any, +) -> dict[str, Any]: + with _physical_execution_lock(lock_path) as lock_wait_ms: + return _run_attempt_locked( + **kwargs, + lock_wait_ms=lock_wait_ms, + lock_namespace=lock_namespace, + ) -def _run_task( +def _run_attempt_locked( # noqa: PLR0913 + *, + benchmark: WebArenaVerified, + selected: PilotTask, engine: str, - page: Page, - visited_origins: set[str], - resources: DockerStatsResources, - task: BrowserTask, attempt: int, + endpoint: str, + container: Any, + cowagent_repo: Path, + output_dir: Path, + model: OpenAIResponsesModel, sample_interval_ms: int, - *, - is_retry: bool = False, + max_agent_steps: int, + lock_wait_ms: float, + lock_namespace: str, ) -> dict[str, Any]: - def record_origin(frame: Any) -> None: - if frame != page.main_frame: - return - origin = _http_origin(str(frame.url)) - if origin: - visited_origins.add(origin) + task = benchmark.get_task(selected.task_id) + sites = tuple(task.sites) + start_urls = list(benchmark.config.render_url(list(task.start_urls), list(task.sites))) + + attempt_dir = output_dir / engine / str(selected.task_id) / f"attempt-{attempt}" + attempt_dir.mkdir(parents=True, exist_ok=True) + prepare_path = attempt_dir / "environment_prepare.json" + prepare_started = time.perf_counter() + prepare_receipt = _environment_prepare_receipt( + sites, lock_wait_ms=lock_wait_ms, lock_namespace=lock_namespace + ) + try: + _prepare_sites(sites) + prepare_receipt["status"] = "ready" + except Exception as exc: + prepare_receipt["status"] = "failed" + prepare_receipt["error"] = str(exc) + raise + finally: + prepare_receipt["duration_ms"] = round((time.perf_counter() - prepare_started) * 1000, 3) + prepare_path.write_text( + json.dumps(prepare_receipt, ensure_ascii=False, indent=2), encoding="utf-8" + ) + + if str(cowagent_repo) not in sys.path: + sys.path.insert(0, str(cowagent_repo)) + from agent.protocol.agent import Agent # noqa: PLC0415 + from agent.protocol.agent_stream import AgentStreamExecutor # noqa: PLC0415 + from agent.tools.browser.browser_tool import BrowserTool # noqa: PLC0415 - page.on("framenavigated", record_origin) + _limit_cowagent_transport_retries(AgentStreamExecutor) + + trace_path = attempt_dir / "agent_trace.zip" + har_path = attempt_dir / "network.har" + response_path = attempt_dir / "agent_response.json" + eval_path = attempt_dir / "eval_result.json" + events_path = attempt_dir / "agent_events.json" + tool = BrowserTool( + config={ + "cdp_endpoint": endpoint, + "headless": True, + "viewport": {"width": 1280, "height": 900}, + "startup_timeout": 90, + } + ) + _allow_official_container_hostname(tool) + events: list[dict[str, Any]] = [] + resources = DockerStatsResources(container) before = resources.counters() sampler = ResourceSampler(resources, interval_ms=sample_interval_ms) - sampler.start() - started = time.perf_counter() - outputs: dict[str, str] = {} - trace: list[dict[str, Any]] = [] - validation: list[dict[str, Any]] = [] + raw_response = "" error = "" + prepared = False + recorder = CdpHarRecorder() + sampler_started = False + browser_session_setup_ms = 0.0 + browser_session_readiness_retries = 0 + agent_run_ms = 0.0 + evidence_finalize_ms = 0.0 + evaluation_ms = 0.0 + started = time.perf_counter() try: - execute_steps(page, task.tool_calls, trace=trace, outputs=outputs) - validation = judge_outputs(outputs, task.assertions) - failures = validate_outputs(outputs, task.assertions) - status = "pass" if not failures else "fail" - error = "; ".join(failures) - trace.append( - { - "index": len(task.tool_calls) + 1, - "action": "validate_assertions", - "arguments": {"assertions": list(task.assertions)}, - "status": status, - "duration_ms": 0.0, - "output": "all assertions passed" if not failures else error, - "checks": validation, - } + session_setup_started = time.perf_counter() + model.reset() + service = tool._get_service() + service._submit(lambda: None) + browser_session_readiness_retries = _login_context(service, sites, benchmark, start_urls) + service._submit( + lambda: ( + recorder.start(service._context), + service._context.tracing.start(screenshots=True, snapshots=True), + ) + ) + prepared = True + browser_session_setup_ms = (time.perf_counter() - session_setup_started) * 1000 + sampler.start() + sampler_started = True + agent = Agent( + system_prompt="", + description="WebArena-Verified CowAgent", + model=model, + tools=[tool], + output_mode="logger", + max_steps=max_agent_steps, + enable_skills=False, + skip_context_files=True, + ) + # CowAgent rebuilds its base prompt at run time; the suffix is the supported + # way to retain benchmark-specific completion and output constraints. + agent.extra_system_suffix = _system_prompt() + agent_started = time.perf_counter() + raw_response = agent.run_stream( + _user_prompt(task, start_urls), on_event=events.append, clear_history=True + ) + if _exhausted_agent_steps(events, max_agent_steps): + response_eval = next( + item for item in task.eval if item.evaluator == "AgentResponseEvaluator" + ) + raw_response = json.dumps( + _max_steps_response(str(response_eval.expected.task_type), max_agent_steps) + ) + agent_run_ms = (time.perf_counter() - agent_started) * 1000 + finalize_started = time.perf_counter() + service._submit( + lambda: ( + service._page.wait_for_timeout(500), + recorder.stop(service._context, har_path), + service._context.tracing.stop(path=str(trace_path)), + ) ) + evidence_finalize_ms = (time.perf_counter() - finalize_started) * 1000 except Exception: - status = "fail" error = traceback.format_exc() - validation = judge_outputs(outputs, task.assertions) + try: + service = tool._get_service() + service._submit( + lambda: ( + recorder.stop(service._context, har_path), + service._context.tracing.stop(path=str(trace_path)), + ) + ) + except Exception: + pass finally: + cleanup_errors: list[str] = [] + if sampler_started: + try: + sampler.stop() + except Exception as exc: + cleanup_errors.append(f"resource sampler stop failed: {exc}") duration_ms = (time.perf_counter() - started) * 1000 - sampler.stop() - page.remove_listener("framenavigated", record_origin) - after = resources.counters() - expected_actions = list(task.ground_truth.get("expected_actions") or []) - tool_trace = [item for item in trace if item.get("action") != "validate_assertions"] - actual_actions = [str(item.get("action") or "") for item in tool_trace] - process_passed = actual_actions == expected_actions and all( - item.get("status") == "pass" for item in tool_trace + try: + after = resources.counters() + except Exception as exc: + after = None + cleanup_errors.append(f"final resource counters failed: {exc}") + try: + tool.close() + except Exception as exc: + cleanup_errors.append(f"browser tool close failed: {exc}") + + if not prepared: + raise RuntimeError( + f"{engine} task {selected.task_id}: environment precondition failed\n{error}" + ) + + parsed_response = _parse_agent_response(raw_response) + response_path.write_text( + json.dumps(parsed_response, ensure_ascii=False, indent=2), encoding="utf-8" + ) + events_path.write_text(json.dumps(events, ensure_ascii=False, indent=2), encoding="utf-8") + if har_path.exists(): + evaluation_started = time.perf_counter() + evaluation = benchmark.evaluate_task( + task_id=selected.task_id, + agent_response=parsed_response, + network_trace=har_path, + ) + eval_data = evaluation.model_dump(mode="json") + evaluation_ms = (time.perf_counter() - evaluation_started) * 1000 + else: + eval_data = {"status": "error", "score": 0.0, "error_msg": error or "HAR missing"} + eval_path.write_text(json.dumps(eval_data, ensure_ascii=False, indent=2), encoding="utf-8") + resource = ( + _resource_summary(resources, sampler, before, after, duration_ms) + if sampler.samples and after is not None + else {} ) - result_passed = bool(validation) and all(item.get("passed") is True for item in validation) return { + "schema": SCHEMA, + "model": model.model, "engine": engine, - "task_id": task.task_id, - "title": task.title, - "complexity": task.complexity, - "tags": list(task.tags), - "step_count": task.step_count, + "task_id": selected.task_id, + "tier": selected.tier, + "estimated_min_steps": selected.estimated_min_steps, "attempt": attempt, - "is_retry": is_retry, - "status": status, - "result_status": "pass" if result_passed else "fail", - "process_status": "pass" if process_passed else "fail", - "combined_status": "pass" if result_passed and process_passed else "fail", + "is_retry": attempt > 1, + "status": eval_data.get("status", "error"), + "score": float(eval_data.get("score") or 0.0), "duration_ms": round(duration_ms, 3), - "tool_output_tokens": sum(len(ENCODING.encode(value)) for value in outputs.values()), - "output_chars": sum(len(value) for value in outputs.values()), + "timings": { + "environment_prepare_ms": prepare_receipt["duration_ms"], + "browser_session_setup_ms": round(browser_session_setup_ms, 3), + "agent_run_ms": round(agent_run_ms, 3), + "evidence_finalize_ms": round(evidence_finalize_ms, 3), + "evaluation_ms": round(evaluation_ms, 3), + "measured_run_wall_ms": round(duration_ms, 3), + }, + "browser_session_readiness_retries": browser_session_readiness_retries, + "agent_response": parsed_response, "error": error, - "trace": trace, - "validation": validation, - "process_validation": { - "expected_actions": expected_actions, - "actual_actions": actual_actions, - "all_calls_passed": all(item.get("status") == "pass" for item in tool_trace), - "passed": process_passed, + "cleanup_errors": cleanup_errors, + "har_capture_errors": recorder.errors, + "resource": resource, + "model_usage": { + **model.usage, + "request_count": model.request_count, + "request_attempt_count": model.request_attempt_count, + }, + "model_transport_errors": model.transport_errors, + "paths": { + "network_har": str(har_path), + "agent_trace": str(trace_path), + "response": str(response_path), + "evaluation": str(eval_path), + "events": str(events_path), + "environment_prepare": str(prepare_path), }, - "resource": _resource_result(resources, sampler, before, after, duration_ms), } -def _start_browser_container( - client: Any, - *, - engine: str, - run_id: str, - cpus: float, - memory: str, - network: str, -) -> Any: - name = f"lex-browser-use-eval-{run_id}-{engine}" - cdp_port = ENGINE_PORTS[engine] - return client.containers.run( - ENGINE_IMAGES[engine], - name=name, - detach=True, - ports={f"{cdp_port}/tcp": ("127.0.0.1", None)}, - labels={"lex.browser-use-eval.run": run_id, "lex.browser-use-eval.engine": engine}, - **ContainerLimits(cpus=cpus, memory=memory).docker_kwargs(), - network=network, - extra_hosts={"host.docker.internal": "host-gateway"}, - ) +def _load_completed_attempts( + path: Path, expected_model: str | None = None +) -> dict[tuple[str, str, int, int], dict[str, Any]]: + """Load durable attempt rows so an interrupted multi-hour pilot can resume safely.""" + if not path.exists(): + return {} + completed: dict[tuple[str, str, int, int], dict[str, Any]] = {} + for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + if not line.strip(): + continue + row = json.loads(line) + model = str(row.get("model") or expected_model or "") + if not model: + raise ValueError(f"attempt does not identify its model at {path}:{line_number}") + if expected_model and model != expected_model: + raise ValueError( + f"attempt model differs from the evidence cohort at {path}:{line_number}" + ) + row["model"] = model + key = (model, str(row["engine"]), int(row["task_id"]), int(row["attempt"])) + if key in completed: + raise ValueError(f"duplicate attempt at {path}:{line_number}: {key}") + completed[key] = row + return completed -class _QuietFixtureHandler(SimpleHTTPRequestHandler): - def log_message(self, format: str, *args: Any) -> None: +def _write_or_validate_run_config(path: Path, planned: dict[str, Any]) -> None: + if path.exists(): + existing = json.loads(path.read_text(encoding="utf-8")) + if existing != planned: + raise ValueError("resume configuration differs from the existing evidence cohort") return + path.write_text(json.dumps(planned, ensure_ascii=False, indent=2), encoding="utf-8") -def _start_fixture_server(root: Path) -> tuple[ThreadingHTTPServer, threading.Thread, str]: - handler = partial(_QuietFixtureHandler, directory=str(root)) - server = ThreadingHTTPServer(("0.0.0.0", 0), handler) - thread = threading.Thread( - target=server.serve_forever, - name="browser-eval-fixtures", - daemon=True, - ) - thread.start() - origin = f"http://host.docker.internal:{server.server_port}" - return server, thread, origin - - -def _materialize_task(task: BrowserTask, fixture_origin: str) -> BrowserTask: - calls = [] - for call in task.tool_calls: - item = dict(call) - for key, value in tuple(item.items()): - if isinstance(value, str): - item[key] = value.replace("{fixture_origin}", fixture_origin) - calls.append(item) - return BrowserTask( - task_id=task.task_id, - title=task.title, - tool_calls=tuple(calls), - assertions=task.assertions, - tags=task.tags, - difficulty=task.difficulty, - ground_truth=task.ground_truth, - source=task.source, - ) - - -def _environment(client: Any, containers: dict[str, Any]) -> dict[str, Any]: - info = client.info() - version_commands = {"chromium": ["chromium", "--version"], "moli": ["moli", "--version"]} - browser_versions: dict[str, str] = {} - for engine, container in containers.items(): - try: - result = container.exec_run(version_commands[engine]) - output = result.output.decode("utf-8", errors="replace").strip() - browser_versions[engine] = output or f"exit {result.exit_code}, no output" - except Exception as exc: - browser_versions[engine] = f"unavailable: {type(exc).__name__}" - return { - "runner_platform": platform.platform(), - "docker_server_version": client.version().get("Version"), - "docker_os": info.get("OperatingSystem"), - "docker_architecture": info.get("Architecture"), - "docker_cpus": info.get("NCPU"), - "docker_memory_bytes": info.get("MemTotal"), - "cgroup_version": info.get("CgroupVersion"), - "browser_versions": browser_versions, - "images": { - engine: { - "id": container.image.id, - "tags": container.image.tags, - } - for engine, container in containers.items() - }, - } - - -def _cleanup_runtime( - browsers: dict[str, Browser], - playwright: Any, - containers: dict[str, Any], - networks: dict[str, Any], -) -> None: - for browser in browsers.values(): - try: - browser.close() - except Exception: - pass - try: - playwright.stop() - except Exception: - pass - for container in containers.values(): - try: - container.remove(force=True) - except Exception: - pass - for network in networks.values(): - try: - network.remove() - except Exception: - pass +def _scheduled_tasks(tasks: tuple[PilotTask, ...], max_tasks: int | None) -> tuple[PilotTask, ...]: + """Limit only this invocation; a later invocation resumes the same pre-registered cohort.""" + if max_tasks is None: + return tasks + if max_tasks < 1 or max_tasks > len(tasks): + raise ValueError(f"--max-tasks must be between 1 and {len(tasks)}") + return tasks[:max_tasks] -def _write_run_artifacts( - output: Path, rows: list[dict[str, Any]], summary: dict[str, Any], manifest: dict[str, Any] -) -> None: - attempts_bytes = "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows).encode( - "utf-8" - ) - summary_bytes = (json.dumps(summary, indent=2, ensure_ascii=False) + "\n").encode("utf-8") - (output / "attempts.jsonl").write_bytes(attempts_bytes) - (output / "summary.json").write_bytes(summary_bytes) - manifest["artifact_provenance"] = { - "artifact_directory": str(output), - "attempts_sha256": hashlib.sha256(attempts_bytes).hexdigest(), - "summary_sha256": hashlib.sha256(summary_bytes).hexdigest(), - } - (output / "manifest.json").write_text( - json.dumps(manifest, indent=2, ensure_ascii=False) + "\n", encoding="utf-8" +def run(args: argparse.Namespace) -> Path: # noqa: PLR0915 + repo_root = Path(args.repo_root).resolve() + repo_env = _load_repo_env(repo_root) + env_file = Path(args.env_file).expanduser().resolve() if args.env_file else None + env = ( + {key: value for key, value in dotenv_values(env_file).items() if value is not None} + if env_file + else repo_env ) + api_key = env.get("OPENAI_API_KEY") + if not api_key: + source = env_file or repo_root / ".env" + raise RuntimeError(f"OPENAI_API_KEY is required in {source}") + cowagent_repo = Path(args.cowagent_repo or repo_env.get("COWAGENT_REPO", "")).resolve() + if not (cowagent_repo / "agent/protocol/agent.py").exists(): + raise RuntimeError("--cowagent-repo must point to a CowAgent checkout") + harness_commit = _require_clean_git_checkout(repo_root) + cowagent_commit = _require_clean_git_checkout(cowagent_repo) - -def run_evaluation( - *, - suite_path: str | Path, - output_dir: str | Path, - reports_dir: str | Path, - ground_truth_evidence_path: str | Path | None = None, - max_retries: int = 1, - sample_interval_ms: int = 50, - settle_ms: int = 1500, - cpus: float = 2.0, - memory: str = "2g", - llm_diagnostics: bool = False, - diagnosis_model: str = "gpt-5.6-sol", - diagnosis_reasoning_effort: str = "low", -) -> dict[str, Any]: - if max_retries < 0: - raise ValueError("max_retries must be non-negative") - suite = load_suite(suite_path) - ground_truth_evidence = ( - validate_ground_truth_evidence(suite_path, suite, ground_truth_evidence_path) - if ground_truth_evidence_path is not None - else None + output_dir = Path(args.output).expanduser().resolve() + output_dir.mkdir(parents=True, exist_ok=True) + selection = load_pilot(args.selection) + benchmark = _configure_benchmark() + resolved = validate_against_official(selection, benchmark) + model = OpenAIResponsesModel( + api_key=api_key, + base_url=env.get("OPENAI_BASE_URL", "https://api.openai.com/v1"), + model=args.model, + reasoning_effort="low", ) - output = Path(output_dir) - output.mkdir(parents=True, exist_ok=True) - run_id = uuid.uuid4().hex[:12] + model.verify_access() + client = docker.from_env() require_linux_cgroup_v2(client) + docker_info = client.info() + physical_execution_lock_namespace = _docker_daemon_lock_namespace(docker_info) + physical_execution_lock_path = _physical_execution_lock_path( + physical_execution_lock_namespace + ) + run_id = uuid.uuid4().hex[:12] containers: dict[str, Any] = {} networks: dict[str, Any] = {} - browsers: dict[str, Browser] = {} - pages: dict[str, Page] = {} - visited_origins: dict[str, set[str]] = {} - resources: dict[str, DockerStatsResources] = {} - rows: list[dict[str, Any]] = [] - playwright = sync_playwright().start() - uses_local_fixtures = any( - "{fixture_origin}" in str(value) - for task in suite.tasks - for call in task.tool_calls - for value in call.values() - ) - fixture_runtime: tuple[ThreadingHTTPServer, threading.Thread, str] | None = None - if uses_local_fixtures: - fixture_runtime = _start_fixture_server(Path(suite_path).resolve().parent / "fixtures") - fixture_origin = fixture_runtime[2] if fixture_runtime else "" + endpoints: dict[str, str] = {} + attempts_file = output_dir / "attempts.jsonl" + run_config_file = output_dir / "run-config.json" + completed = _load_completed_attempts(attempts_file, args.model) + image_metadata: dict[str, dict[str, Any]] = {} + site_image_metadata = { + site.value: { + "reference": ContainerManager( + site=site, hostname="host.docker.internal" + ).config.docker_img, + **_docker_image_metadata( + client.images.get( + ContainerManager(site=site, hostname="host.docker.internal").config.docker_img + ) + ), + } + for site in SITE_PORTS + } + browser_metadata: dict[str, dict[str, str]] = {} + site_access_host = "" + docker_architecture = str(docker_info.get("Architecture") or "") + site_runtime_compatibility = _site_runtime_compatibility(docker_architecture) try: for engine in ENGINE_IMAGES: - network = client.networks.create( - f"lex-browser-use-eval-{run_id}-{engine}", - driver="bridge", - labels={ - "lex.browser-use-eval.run": run_id, - "lex.browser-use-eval.engine": engine, - }, - ) + network = client.networks.create(f"lex-browser-use-{run_id}-{engine}") networks[engine] = network container = _start_browser_container( client, engine=engine, - run_id=run_id, - cpus=cpus, - memory=memory, + run_id=f"agent-{run_id}", + cpus=args.cpus, + memory=args.memory, network=network.name, ) containers[engine] = container container.reload() - cdp_port = ENGINE_PORTS[engine] - endpoint = _published_endpoint(container, engine=engine, cdp_port=cdp_port) - websocket_url = _wait_cdp(endpoint, container=container) - browsers[engine] = playwright.chromium.connect_over_cdp(websocket_url) - resources[engine] = DockerStatsResources(container) - page = browsers[engine].contexts[0].pages[0] - page.goto("https://example.com/", wait_until="domcontentloaded", timeout=30_000) - _reset_state(page, {"https://example.com"}) - pages[engine] = page - visited_origins[engine] = set() - - for task_index, declared_task in enumerate(suite.tasks): - task = _materialize_task(declared_task, fixture_origin) - engine_order = ("chromium", "moli") if task_index % 2 == 0 else ("moli", "chromium") - for engine in engine_order: - for attempt in range(1, max_retries + 2): - pages[engine] = _prepare_page( - pages[engine], - visited_origins[engine], - settle_ms=settle_ms, - ) - row = _run_task( - engine, - pages[engine], - visited_origins[engine], - resources[engine], - task, - attempt, - sample_interval_ms, - is_retry=attempt > 1, - ) - row["run_id"] = run_id - rows.append(row) - print( - json.dumps( - { - "engine": row["engine"], - "task_id": row["task_id"], - "attempt": row["attempt"], - "status": row["status"], - "duration_ms": row["duration_ms"], - "error": next( - ( - line - for line in reversed(row["error"].splitlines()) - if line.strip() - ), - "", - ), - }, - ensure_ascii=False, - ), - flush=True, - ) - if row["status"] == "pass": - break - summary = summarize_attempts(rows, engines=tuple(ENGINE_IMAGES)) - manifest = { - "schema": "lex_browser_use_manifest/2", - "run_id": run_id, - "created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), - "suite_sha256": hashlib.sha256(Path(suite_path).read_bytes()).hexdigest(), - "suite": [ - { - "task_id": task.task_id, - "title": task.title, - "step_count": task.step_count, - "complexity": task.complexity, - "tags": list(task.tags), - "tool_calls": list(task.tool_calls), - "assertions": list(task.assertions), - "ground_truth": task.ground_truth, - "source": task.source, - } - for task in suite.tasks - ], - "ground_truth_validation": ( - { - key: ground_truth_evidence.get(key) - for key in ( - "schema", - "dataset", - "dataset_sha256", - "source_commit", - "validator", - "method", - "started_at", - "finished_at", - "automatic_browser_replay", - "human_review", - ) - } - if ground_truth_evidence is not None - else None - ), - "protocol": { - "browser_reuse": True, - "viewport": {"width": 1280, "height": 900}, - "viewport_verified_before_each_task": True, - "step_definition": ( - "one declared CowAgent BrowserTool-equivalent call; implicit waits excluded" - ), - "llm_scope": ("LLM never judges success; optional LLM only explains failed traces"), - "llm_failure_diagnostics": llm_diagnostics, - "diagnosis_model": diagnosis_model if llm_diagnostics else None, - "diagnosis_reasoning_effort": ( - diagnosis_reasoning_effort if llm_diagnostics else None - ), - "cold_start_excluded": True, - "state_reset_between_tasks": True, - "state_reset_method": ( - "warm browser/context; clear all top-level navigation origins, " - "browser cache, cookies, permissions and history; replace renderer page" - ), - "sample_interval_ms": sample_interval_ms, - "post_reset_settle_ms": settle_ms, - "primary_attempts_per_task": 1, - "max_failure_retries": max_retries, - "engine_order": "balanced by task index", - "target_mode": "public_web" if not uses_local_fixtures else "mixed", - "fixture_origin": ( - "temporary self-hosted deterministic fixture server" - if uses_local_fixtures - else None - ), - "resource_scope": "browser container cgroup/network namespace", - "controller_excluded": True, - "cpus_per_browser": cpus, - "memory_per_browser": memory, - "network_isolation": "one dedicated bridge with no peer containers per engine", - }, - "environment": _environment(client, containers), + image_metadata[engine] = _docker_image_metadata(container.image) + http_endpoint = _published_endpoint( + container, engine=engine, cdp_port=ENGINE_PORTS[engine] + ) + endpoints[engine] = _wait_cdp(http_endpoint, container=container) + browser_metadata[engine] = _cdp_metadata(http_endpoint) + + gateway_addresses = { + _container_host_gateway(container) for container in containers.values() } - smoke_path = Path(suite_path).resolve().parent / "agent_smoke.json" - if smoke_path.exists(): - smoke_bytes = smoke_path.read_bytes() - manifest["agent_smoke"] = { - **json.loads(smoke_bytes.decode("utf-8")), - "provenance": { - "source_file": smoke_path.name, - "source_sha256": hashlib.sha256(smoke_bytes).hexdigest(), - "independent_of_browser_microbenchmark": True, - "reused_not_rerun": True, + if len(gateway_addresses) != 1: + raise RuntimeError(f"browser containers disagree on host gateway: {gateway_addresses}") + site_access_host = gateway_addresses.pop() + benchmark = _configure_benchmark(hostname=site_access_host) + resolved = validate_against_official(selection, benchmark) + + _write_or_validate_run_config( + run_config_file, + { + "schema": "lex_browser_use_run_config/1", + "selection_sha256": selection.sha256, + "browser_eval_commit": harness_commit, + "cowagent_commit": cowagent_commit, + "model": args.model, + "reasoning_effort": "low", + "model_transport_retries": 1, + "max_agent_steps": args.max_agent_steps, + "provider_base_url": env.get("OPENAI_BASE_URL", "https://api.openai.com/v1"), + "browser_images": image_metadata, + "site_images": site_image_metadata, + "browser_versions": browser_metadata, + "site_access_host": site_access_host, + "docker_context": subprocess.run( + ["docker", "context", "show"], + check=True, + capture_output=True, + text=True, + ).stdout.strip(), + "environment_lane": { + "lock_namespace": physical_execution_lock_namespace, + "isolation_unit": "docker-daemon-and-fixed-site-ports", }, - } - # Persist core evidence before optional remote diagnostics can fail or stall. - _write_run_artifacts(output, rows, summary, manifest) - diagnose_summary( - summary, - use_llm=llm_diagnostics, - model=diagnosis_model, - reasoning_effort=diagnosis_reasoning_effort, + "site_runtime_compatibility": site_runtime_compatibility, + }, ) - _write_run_artifacts(output, rows, summary, manifest) - write_latest_report(reports_dir, manifest=manifest, summary=summary) - return summary + + with attempts_file.open("a", encoding="utf-8") as stream: + for task_index, selected in enumerate( + _scheduled_tasks(selection.tasks, args.max_tasks) + ): + order = ("chromium", "moli") if task_index % 2 == 0 else ("moli", "chromium") + for engine in order: + for attempt in (1, 2): + key = (args.model, engine, selected.task_id, attempt) + if key in completed: + row = completed[key] + print( + json.dumps( + { + "engine": engine, + "task_id": selected.task_id, + "attempt": attempt, + "status": "resume-skip", + "score": row["score"], + } + ), + flush=True, + ) + if float(row["score"]) == 1.0: + break + continue + primary = completed.get((args.model, engine, selected.task_id, 1)) + if attempt == 2 and primary is not None and float(primary["score"]) == 1.0: + break + row = _run_attempt( + lock_path=physical_execution_lock_path, + lock_namespace=physical_execution_lock_namespace, + benchmark=benchmark, + selected=selected, + engine=engine, + attempt=attempt, + endpoint=endpoints[engine], + container=containers[engine], + cowagent_repo=cowagent_repo, + output_dir=output_dir, + model=model, + sample_interval_ms=args.sample_interval_ms, + max_agent_steps=args.max_agent_steps, + ) + row["run_id"] = run_id + stream.write(json.dumps(row, ensure_ascii=False) + "\n") + stream.flush() + completed[key] = row + visible = { + key: row[key] + for key in ("engine", "task_id", "attempt", "status", "score") + } + print(json.dumps(visible), flush=True) + if row["score"] == 1.0: + break finally: - if fixture_runtime: - fixture_server, fixture_thread, _ = fixture_runtime - fixture_server.shutdown() - fixture_server.server_close() - fixture_thread.join(timeout=2) - _cleanup_runtime(browsers, playwright, containers, networks) + for container in containers.values(): + try: + container.remove(force=True) + except Exception: + pass + for network in networks.values(): + try: + network.remove() + except Exception: + pass + + evaluator_provenance: dict[str, Any] = {} + for row in completed.values(): + eval_path = Path(str((row.get("paths") or {}).get("evaluation") or "")) + if eval_path.is_file(): + eval_data = json.loads(eval_path.read_text(encoding="utf-8")) + evaluator_provenance = { + "version": eval_data.get("webarena_verified_version"), + "evaluator_checksum": eval_data.get("webarena_verified_evaluator_checksum"), + "data_checksum": eval_data.get("webarena_verified_data_checksum"), + } + break + + manifest = { + "schema": "lex_browser_use_manifest/1", + "run_id": run_id, + "run_ids": sorted({str(row.get("run_id")) for row in completed.values()}), + "created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "selection_sha256": selection.sha256, + "selection": resolved, + "protocol": { + "agent": "CowAgent", + "browser_eval_commit": harness_commit, + "cowagent_commit": cowagent_commit, + "model": args.model, + "reasoning_effort": "low", + "primary_attempts_per_engine_task": 1, + "failure_retries": 1, + "task_failure_retries": 1, + "model_transport_retries": 1, + "max_agent_steps": args.max_agent_steps, + "official_score_uses": "final execution for each browser and task", + "logical_result_cost": "sum of all executions used for each browser and task", + "evaluator": "official webarena-verified package, unmodified", + "environment_reset": "official site container reset before every attempt", + "environment_prepare_receipt": "required", + "environment_lane": { + "lock_namespace": physical_execution_lock_namespace, + "isolation_unit": "docker-daemon-and-fixed-site-ports", + "parallelism_rule": "one active attempt per Docker daemon", + }, + "engine_order": "alternating by task", + "sites": [site.value for site in SITE_PORTS], + "official_container_hostname_allowlist": ["host.docker.internal"], + }, + "environment": { + "platform": platform.platform(), + "docker_server_version": client.version().get("Version"), + "docker_architecture": docker_info.get("Architecture"), + "docker_name": docker_info.get("Name"), + "webarena_verified_version": importlib.metadata.version("webarena-verified"), + "browser_images": image_metadata, + "browser_versions": browser_metadata, + "site_images": site_image_metadata, + "site_access_host": site_access_host, + "official_evaluator": evaluator_provenance, + "site_runtime_compatibility": site_runtime_compatibility, + }, + } + (output_dir / "manifest.json").write_text( + json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8" + ) + return output_dir + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--repo-root", default=".") + parser.add_argument( + "--selection", default="evaluation/browser-use/webarena-verified-pilot.json" + ) + parser.add_argument("--output", required=True) + parser.add_argument( + "--env-file", + help="explicit dotenv file for provider credentials; defaults to the repository .env", + ) + parser.add_argument( + "--cowagent-repo", + help="trusted, clean CowAgent Git checkout; Python code is imported and executed from it", + ) + parser.add_argument("--model", required=True) + parser.add_argument("--cpus", type=float, default=2.0) + parser.add_argument("--memory", default="2g") + parser.add_argument("--sample-interval-ms", type=int, default=100) + parser.add_argument( + "--max-agent-steps", + type=int, + default=40, + help="maximum CowAgent decision turns; exhaustion fails closed instead of summarizing", + ) + parser.add_argument( + "--max-tasks", + type=int, + help=( + "run only the first N pre-registered tasks in this invocation; " + "resume later without this option to finish the unchanged cohort" + ), + ) + return parser.parse_args() + + +def main() -> None: + run(parse_args()) + + +if __name__ == "__main__": + main() diff --git a/harness/browser_use/runtime/php-fpm-qemu.ini b/harness/browser_use/runtime/php-fpm-qemu.ini new file mode 100644 index 000000000..8149b77f7 --- /dev/null +++ b/harness/browser_use/runtime/php-fpm-qemu.ini @@ -0,0 +1,12 @@ +# ARM-host compatibility for the official amd64 Magento images. +[program:php-fpm] +command=/usr/local/bin/qemu-x86_64 /usr/local/sbin/php-fpm --nodaemonize --fpm-config /usr/local/etc/php-fpm.conf +autostart=true +autorestart=true +priority=2 +startretries=3 +stopwaitsecs=10 +stdout_logfile=/dev/stdout +stdout_logfile_maxbytes=0 +stderr_logfile=/dev/stderr +stderr_logfile_maxbytes=0 diff --git a/harness/browser_use/selection.py b/harness/browser_use/selection.py new file mode 100644 index 000000000..b1e90004f --- /dev/null +++ b/harness/browser_use/selection.py @@ -0,0 +1,322 @@ +"""Pre-registered WebArena-Verified pilot selection and validation.""" + +from __future__ import annotations + +import hashlib +import json +from collections import Counter +from dataclasses import dataclass +from functools import lru_cache +from importlib.resources import files +from pathlib import Path +from typing import Any + +PILOT_SCHEMA = "lex_webarena_verified_pilot/1" +ALLOWED_SITES = frozenset({"shopping", "shopping_admin", "reddit", "gitlab"}) +TIER_COUNTS = {"easy": 15, "medium": 30, "hard": 45} +TIER_BOUNDS = {"easy": (1, 4), "medium": (5, 7), "hard": (8, None)} +TASK_EXCLUSIONS_PATH = Path(__file__).parents[2] / "evaluation/browser-use/task-exclusions.json" + + +@dataclass(frozen=True) +class PilotTask: + task_id: int + tier: str + estimated_min_steps: int + rationale: str + official_hard: bool + + +@dataclass(frozen=True) +class PilotSelection: + tasks: tuple[PilotTask, ...] + source_path: Path + sha256: str + metadata: dict[str, Any] + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _json_equal(left: Any, right: Any) -> bool: + """Compare JSON values without treating booleans as numbers.""" + if type(left) is not type(right): + return False + if isinstance(left, list): + return len(left) == len(right) and all( + _json_equal(left_item, right_item) + for left_item, right_item in zip(left, right, strict=True) + ) + if isinstance(left, dict): + return left.keys() == right.keys() and all( + _json_equal(left[key], right[key]) for key in left + ) + return bool(left == right) + + +def _validate_json_schema(value: Any, schema: Any, *, path: str) -> None: + """Validate the JSON Schema subset used by official response contracts. + + The current official dataset uses structural keywords plus ``format`` as a + normalization hint. Unknown validation keywords fail closed so a future + contract cannot silently bypass this preflight. + """ + if not isinstance(schema, dict): + raise ValueError(f"{path}: schema must be an object") + supported = { + "anyOf", + "oneOf", + "const", + "enum", + "type", + "required", + "properties", + "items", + "format", + "title", + } + unsupported = sorted(set(schema) - supported) + if unsupported: + raise ValueError(f"{path}: unsupported schema keyword {unsupported[0]!r}") + if "format" in schema and not isinstance(schema["format"], str): + raise ValueError(f"{path}: format must be a string") + if "title" in schema and not isinstance(schema["title"], str): + raise ValueError(f"{path}: title must be a string") + + for keyword in ("anyOf", "oneOf"): + if keyword not in schema: + continue + branches = schema[keyword] + if not isinstance(branches, list) or not branches: + raise ValueError(f"{path}: {keyword} must be a non-empty array") + failures: list[str] = [] + matches = 0 + for branch in branches: + try: + _validate_json_schema(value, branch, path=path) + except ValueError as exc: + failures.append(str(exc)) + else: + matches += 1 + if keyword == "anyOf" and matches == 0: + raise ValueError(f"{path}: does not match anyOf ({'; '.join(failures)})") + if keyword == "oneOf" and matches != 1: + detail = f"; {'; '.join(failures)}" if failures else "" + raise ValueError( + f"{path}: must match exactly one oneOf branch, matched {matches}{detail}" + ) + + if "const" in schema and not _json_equal(value, schema["const"]): + raise ValueError(f"{path}: expected const {schema['const']!r}, got {value!r}") + if "enum" in schema: + options = schema["enum"] + if not isinstance(options, list): + raise ValueError(f"{path}: enum must be an array") + if not any(_json_equal(value, option) for option in options): + raise ValueError(f"{path}: expected one of {options!r}, got {value!r}") + + declared_types = schema.get("type") + if declared_types is not None: + if isinstance(declared_types, str): + type_names = [declared_types] + elif isinstance(declared_types, list) and all( + isinstance(item, str) for item in declared_types + ): + type_names = declared_types + else: + raise ValueError(f"{path}: type must be a string or array of strings") + checks = { + "object": lambda item: isinstance(item, dict), + "array": lambda item: isinstance(item, list), + "string": lambda item: isinstance(item, str), + "number": lambda item: isinstance(item, (int, float)) and not isinstance(item, bool), + "integer": lambda item: isinstance(item, int) and not isinstance(item, bool), + "boolean": lambda item: isinstance(item, bool), + "null": lambda item: item is None, + } + unknown = [name for name in type_names if name not in checks] + if unknown: + raise ValueError(f"{path}: unsupported schema type {unknown[0]!r}") + if not any(checks[name](value) for name in type_names): + actual = "null" if value is None else type(value).__name__ + expected = " or ".join(type_names) + raise ValueError(f"{path}: expected {expected}, got {actual}") + + if isinstance(value, dict): + required = schema.get("required", []) + if not isinstance(required, list) or not all(isinstance(key, str) for key in required): + raise ValueError(f"{path}: required must be an array of strings") + for key in required: + if key not in value: + raise ValueError(f"{path}: missing required property {key!r}") + properties = schema.get("properties", {}) + if not isinstance(properties, dict): + raise ValueError(f"{path}: properties must be an object") + for key, child_schema in properties.items(): + if key in value: + _validate_json_schema(value[key], child_schema, path=f"{path}.{key}") + + if isinstance(value, list) and "items" in schema: + item_schema = schema["items"] + for index, item in enumerate(value): + _validate_json_schema(item, item_schema, path=f"{path}[{index}]") + + +def _validate_agent_response_contract(task_id: int, task: Any) -> None: + evaluators = [item for item in task.eval if item.evaluator == "AgentResponseEvaluator"] + if not evaluators: + raise ValueError(f"task {task_id}: missing AgentResponseEvaluator") + for evaluator in evaluators: + payload = evaluator.model_dump(mode="json") + expected = payload.get("expected") + schema = payload.get("results_schema") + if not isinstance(expected, dict) or "retrieved_data" not in expected: + raise ValueError( + f"task {task_id}: AgentResponseEvaluator is missing expected.retrieved_data" + ) + if schema is None: + raise ValueError(f"task {task_id}: AgentResponseEvaluator is missing results_schema") + try: + _validate_json_schema( + expected["retrieved_data"], schema, path="expected.retrieved_data" + ) + except ValueError as exc: + raise ValueError( + f"task {task_id}: AgentResponseEvaluator expected.retrieved_data " + f"violates results_schema: {exc}" + ) from exc + + +@lru_cache(maxsize=1) +def _task_exclusions() -> dict[str, dict[str, str]]: + payload = json.loads(TASK_EXCLUSIONS_PATH.read_text(encoding="utf-8")) + if payload.get("schema") != "lex_browser_use_task_exclusions/1": + raise ValueError("unsupported Browser Use task-exclusion schema") + return { + str(task_id): {str(key): str(value) for key, value in item.items()} + for task_id, item in (payload.get("tasks") or {}).items() + } + + +def _task_exclusion(task_id: int) -> dict[str, str] | None: + return _task_exclusions().get(str(task_id)) + + +def load_pilot(path: str | Path) -> PilotSelection: + source = Path(path) + payload = json.loads(source.read_text(encoding="utf-8")) + if payload.get("schema") != PILOT_SCHEMA: + raise ValueError(f"unsupported pilot schema: {payload.get('schema')!r}") + metadata = dict(payload.get("metadata") or {}) + tasks = tuple( + PilotTask( + task_id=int(item["task_id"]), + tier=str(item["tier"]), + estimated_min_steps=int(item["estimated_min_steps"]), + rationale=str(item["rationale"]), + official_hard=bool(item["official_hard"]), + ) + for item in payload.get("tasks", []) + ) + ids = [task.task_id for task in tasks] + if len(ids) != len(set(ids)): + raise ValueError("pilot task IDs must be unique") + counts = {tier: sum(task.tier == tier for task in tasks) for tier in TIER_COUNTS} + if counts != TIER_COUNTS: + raise ValueError(f"pilot tier counts must be {TIER_COUNTS}, got {counts}") + if metadata.get("tier_counts") != TIER_COUNTS: + raise ValueError("selection metadata tier_counts must match the frozen tier counts") + for task in tasks: + low, high = TIER_BOUNDS[task.tier] + if task.estimated_min_steps < low or (high is not None and task.estimated_min_steps > high): + raise ValueError(f"task {task.task_id}: step estimate is outside {task.tier}") + return PilotSelection(tasks, source, _sha256(source), metadata) + + +def validate_against_official( + selection: PilotSelection, + benchmark: Any, + *, + enforce_contract_integrity: bool = True, +) -> list[dict[str, Any]]: + """Resolve official metadata and, for new runs, reject invalid response contracts. + + Historical report reconstruction may disable the prospective integrity gate because + its immutable cohort can contain the very contract defect being documented. It must + still resolve the exact official package and checksum used by that evidence. + """ + hard_payload = json.loads( + files("webarena_verified") + .joinpath("assets/dataset/subsets/webarena-verified-hard.json") + .read_text(encoding="utf-8") + ) + official_hard_ids = {int(task_id) for task_id in hard_payload["task_ids"]} + resolved: list[dict[str, Any]] = [] + for selected in selection.tasks: + task = benchmark.get_task(selected.task_id) + if enforce_contract_integrity: + exclusion = _task_exclusion(selected.task_id) + if exclusion: + raise ValueError( + f"task {selected.task_id}: excluded from new Browser Use runs: " + f"{exclusion['reason']}; {exclusion['disposition']}" + ) + _validate_agent_response_contract(selected.task_id, task) + sites = [getattr(site, "value", str(site)) for site in task.sites] + if not set(sites).issubset(ALLOWED_SITES): + raise ValueError(f"task {selected.task_id}: unsupported pilot sites {sites}") + if len(sites) != 1: + raise ValueError( + f"task {selected.task_id}: pilot requires one resettable site, got {sites}" + ) + if selected.official_hard != (selected.task_id in official_hard_ids): + raise ValueError( + f"task {selected.task_id}: official_hard does not match package subset" + ) + resolved.append( + { + "task_id": selected.task_id, + "tier": selected.tier, + "estimated_min_steps": selected.estimated_min_steps, + "rationale": selected.rationale, + "official_hard": selected.official_hard, + "sites": sites, + "intent": task.intent, + "intent_template_id": task.intent_template_id, + "revision": task.revision, + "start_urls": list(task.start_urls), + } + ) + site_counts = Counter(row["sites"][0] for row in resolved) + expected_site_counts = selection.metadata.get("site_counts") + if expected_site_counts is not None and expected_site_counts != dict(site_counts): + raise ValueError( + f"selection site counts must be {expected_site_counts}, got {dict(site_counts)}" + ) + expected_task_type_counts = selection.metadata.get("task_type_counts") + if expected_task_type_counts is not None: + task_type_counts: Counter[str] = Counter() + for selected in selection.tasks: + task = benchmark.get_task(selected.task_id) + response_evaluator = next( + ( + item + for item in task.eval + if item.evaluator == "AgentResponseEvaluator" + ), + None, + ) + if response_evaluator is None: + raise ValueError( + f"task {selected.task_id}: missing AgentResponseEvaluator " + "for task_type_counts" + ) + task_type = response_evaluator.expected.task_type + task_type_counts[getattr(task_type, "value", str(task_type))] += 1 + if expected_task_type_counts != dict(task_type_counts): + raise ValueError( + "selection task type counts must be " + f"{expected_task_type_counts}, got {dict(task_type_counts)}" + ) + return resolved diff --git a/harness/trajectory_replay/__init__.py b/harness/trajectory_replay/__init__.py new file mode 100644 index 000000000..5d8e0824d --- /dev/null +++ b/harness/trajectory_replay/__init__.py @@ -0,0 +1,5 @@ +"""Reproducible browser-engine evaluation with isolated resource accounting.""" + +from .suite import BrowserTask, TaskSuite, load_suite + +__all__ = ["BrowserTask", "TaskSuite", "load_suite"] diff --git a/harness/trajectory_replay/browser_use_seeds.py b/harness/trajectory_replay/browser_use_seeds.py new file mode 100644 index 000000000..aa1fa5e98 --- /dev/null +++ b/harness/trajectory_replay/browser_use_seeds.py @@ -0,0 +1,367 @@ +"""Derive fixed replay candidates from successful autonomous Browser Use evidence.""" + +from __future__ import annotations + +import hashlib +import json +import re +from collections import defaultdict +from pathlib import Path +from typing import Any + +SCHEMA = "lex_browser_use_replay_seeds/2" + +_REF_ACTIONS = {"click", "fill", "select"} +_NODE_RE = re.compile(r"^(?P\s*)\[(?P\d+)\]\s+(?P[a-zA-Z][\w-]*)(?P.*)$") +_ATTR_RE = re.compile(r'(?P[A-Za-z][\w-]*)="(?P(?:[^"\\]|\\.)*)"') + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _read_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +def _contains_ref(value: Any) -> bool: + if isinstance(value, dict): + return "ref" in value or any(_contains_ref(item) for item in value.values()) + if isinstance(value, list): + return any(_contains_ref(item) for item in value) + return False + + +def _snapshot_nodes(result: str) -> list[dict[str, Any]]: + """Parse the stable, reader-visible element facts emitted by BrowserTool snapshots.""" + + nodes = [] + for line in result.splitlines(): + match = _NODE_RE.match(line) + if match is None: + continue + body = match.group("body") + attributes = { + item.group("name"): json.loads(f'"{item.group("value")}"') + for item in _ATTR_RE.finditer(body) + } + body_without_attributes = _ATTR_RE.sub("", body).strip() + text = ( + body_without_attributes[1:].strip() if body_without_attributes.startswith(":") else "" + ) + nodes.append( + { + "ref": int(match.group("ref")), + "tag": match.group("tag").lower(), + "attributes": attributes, + "text": text, + } + ) + return nodes + + +def _quoted(value: str) -> str: + return json.dumps(value, ensure_ascii=False) + + +def _base_selector(node: dict[str, Any]) -> str: + tag = str(node["tag"]) + attributes = node["attributes"] + for snapshot_name, dom_name in ( + ("name", "name"), + ("href", "href"), + ("ariaLabel", "aria-label"), + ("placeholder", "placeholder"), + ("alt", "alt"), + ("role", "role"), + ): + value = attributes.get(snapshot_name) + if value and not str(value).endswith("..."): + return f"{tag}[{dom_name}={_quoted(str(value))}]:visible" + text = str(node.get("text") or "").strip() + if text and not text.endswith("..."): + return f"{tag}:visible:has-text({_quoted(text)})" + type_value = attributes.get("type") + if type_value: + return f"{tag}[type={_quoted(str(type_value))}]:visible" + return f"{tag}:visible" + + +def _same_selector_population(target: dict[str, Any], other: dict[str, Any]) -> bool: + """Mirror the DOM population matched by the target's generated selector.""" + + if target["tag"] != other["tag"]: + return False + target_attributes = target["attributes"] + other_attributes = other["attributes"] + for name in ("name", "href", "ariaLabel", "placeholder", "alt", "role"): + value = target_attributes.get(name) + if value and not str(value).endswith("..."): + return other_attributes.get(name) == value + text = str(target.get("text") or "").strip() + if text and not text.endswith("..."): + return text in str(other.get("text") or "") + type_value = target_attributes.get("type") + if type_value: + return other_attributes.get("type") == type_value + return True + + +def _selector_for_ref(nodes: list[dict[str, Any]], ref: int, *, context: str) -> str: + target = next((node for node in nodes if node["ref"] == ref), None) + if target is None: + raise ValueError(f"{context}: ref {ref} is absent from the preceding snapshot") + base = _base_selector(target) + matches = [node for node in nodes if _same_selector_population(target, node)] + if len(matches) == 1: + return base + occurrence = next(index for index, node in enumerate(matches, start=1) if node is target) + return f":nth-match({base}, {occurrence})" + + +def compile_replay_calls(events_path: str | Path) -> list[dict[str, Any]]: + """Compile ephemeral refs into fixed selectors while preserving ref action semantics.""" + + calls = extract_browser_calls(events_path) + latest_nodes: list[dict[str, Any]] = [] + compiled: list[dict[str, Any]] = [] + for index, call in enumerate(calls, start=1): + arguments = dict(call.get("arguments") or {}) + action = str(arguments.get("action") or "") + if action in _REF_ACTIONS and "ref" in arguments: + if arguments.get("selector"): + arguments.pop("ref", None) + else: + arguments["selector"] = _selector_for_ref( + latest_nodes, + int(arguments.pop("ref")), + context=f"{events_path}: call {index}", + ) + arguments["replay_mode"] = "snapshot_ref_semantics" + elif "ref" in arguments: + # BrowserTool ignores refs for actions such as ``press``. Keeping that + # inert model-supplied field would make the fixed trajectory appear to + # depend on an ephemeral snapshot identity when it does not. + arguments.pop("ref") + if _contains_ref(arguments): + raise ValueError(f"{events_path}: call {index} contains an unsupported ref") + compiled.append(arguments) + result = str(call.get("result") or "") + snapshot_nodes = _snapshot_nodes(result) + if snapshot_nodes: + latest_nodes = snapshot_nodes + return compiled + + +def extract_browser_calls(events_path: str | Path) -> list[dict[str, Any]]: + """Return complete BrowserTool calls in execution order, including call status.""" + + starts: dict[str, dict[str, Any]] = {} + calls: list[dict[str, Any]] = [] + for event in _read_json(Path(events_path)): + event_type = event.get("type") + data = event.get("data") or {} + call_id = str(data.get("tool_call_id") or "") + if event_type == "tool_execution_start": + if not call_id or call_id in starts: + raise ValueError(f"invalid or duplicate tool call id in {events_path}: {call_id!r}") + starts[call_id] = { + "tool_call_id": call_id, + "tool_name": str(data.get("tool_name") or ""), + "arguments": dict(data.get("arguments") or {}), + "started_at": event.get("timestamp"), + } + elif event_type == "tool_execution_end": + start = starts.pop(call_id, None) + if start is None: + raise ValueError(f"tool end event has no start in {events_path}: {call_id!r}") + start.update( + { + "status": str(data.get("status") or ""), + "duration_seconds": data.get("execution_time"), + "error": data.get("error"), + "result": data.get("result"), + "ended_at": event.get("timestamp"), + } + ) + calls.append(start) + if starts: + raise ValueError(f"tool calls have no end event in {events_path}: {sorted(starts)}") + return calls + + +def _authoritative_rows(root: Path) -> tuple[dict[str, Any], list[dict[str, Any]]]: + manifest = _read_json(root / "manifest.json") + model = str((manifest.get("protocol") or {}).get("model") or "") + if not model: + raise ValueError(f"evidence manifest does not identify a model: {root}") + grouped: dict[tuple[str, int], list[dict[str, Any]]] = defaultdict(list) + for line in (root / "attempts.jsonl").read_text(encoding="utf-8").splitlines(): + if line.strip(): + row = json.loads(line) + grouped[(str(row["engine"]), int(row["task_id"]))].append(row) + rows: list[dict[str, Any]] = [] + for (engine, task_id), executions in sorted(grouped.items()): + executions.sort(key=lambda row: int(row["attempt"])) + expected = list(range(1, len(executions) + 1)) + if [int(row["attempt"]) for row in executions] != expected: + raise ValueError(f"non-contiguous executions for {model}/{engine}/{task_id}") + if len(executions) > 2: + raise ValueError(f"more than one retry for {model}/{engine}/{task_id}") + row = dict(executions[-1]) + row["model"] = model + rows.append(row) + return manifest, rows + + +def build_replay_seed_manifest( + evidence_dirs: list[str | Path], profile: dict[str, Any] +) -> dict[str, Any]: + """Build a private replay seed inventory without using human demonstration traces.""" + + models = [str(value) for value in profile.get("models") or []] + engines = [str(value) for value in profile.get("engines") or []] + method_priority = [(model, engine) for model in models for engine in engines] + if not method_priority: + raise ValueError("profile must register models and engines") + + manifests: dict[str, dict[str, Any]] = {} + authoritative: list[dict[str, Any]] = [] + source_roots: dict[str, Path] = {} + for raw_root in evidence_dirs: + root = Path(raw_root).expanduser().resolve() + manifest, rows = _authoritative_rows(root) + model = str((manifest.get("protocol") or {}).get("model")) + if model in manifests: + raise ValueError(f"duplicate evidence cohort for model: {model}") + manifests[model] = manifest + source_roots[model] = root + authoritative.extend(rows) + if set(manifests) != set(models): + raise ValueError( + f"evidence models differ from profile: expected={models}, actual={sorted(manifests)}" + ) + selection_hashes = {str(item.get("selection_sha256") or "") for item in manifests.values()} + if len(selection_hashes) != 1 or "" in selection_hashes: + raise ValueError("evidence cohorts do not share one frozen task selection") + + task_metadata: dict[int, dict[str, Any]] = {} + for manifest in manifests.values(): + for item in manifest.get("selection") or []: + task_id = int(item["task_id"]) + if task_id in task_metadata and task_metadata[task_id] != item: + raise ValueError(f"task metadata differs between evidence cohorts: {task_id}") + task_metadata[task_id] = dict(item) + + candidates_by_task: dict[int, list[dict[str, Any]]] = defaultdict(list) + passed_methods_by_task: dict[int, list[str]] = defaultdict(list) + for row in authoritative: + task_id = int(row["task_id"]) + if float(row.get("score") or 0) != 1: + continue + model, engine = str(row["model"]), str(row["engine"]) + method_id = f"{model}__{engine}" + passed_methods_by_task[task_id].append(method_id) + events_path = Path(str((row.get("paths") or {}).get("events") or "")) + response_path = Path(str((row.get("paths") or {}).get("response") or "")) + evaluation_path = Path(str((row.get("paths") or {}).get("evaluation") or "")) + calls = extract_browser_calls(events_path) + try: + replay_calls = compile_replay_calls(events_path) + except (TypeError, ValueError): + replay_calls = [] + clean = ( + bool(calls) + and bool(replay_calls) + and all(call["status"] == "success" for call in calls) + ) + candidate = { + "id": f"{method_id}__task-{task_id}", + "method": {"model": model, "engine": engine}, + "source_execution": int(row["attempt"]), + "source_run_id": str(row.get("run_id") or ""), + "official_score": 1.0, + "clean": clean, + "action_count": len(calls), + "actions": [str((call.get("arguments") or {}).get("action") or "") for call in calls], + "tool_calls": replay_calls, + "recorded_response": _read_json(response_path), + "source": { + "events_path": str(events_path), + "events_sha256": _sha256(events_path), + "response_path": str(response_path), + "response_sha256": _sha256(response_path), + "evaluation_path": str(evaluation_path), + "evaluation_sha256": _sha256(evaluation_path), + }, + } + candidates_by_task[task_id].append(candidate) + + priority_index = {method: index for index, method in enumerate(method_priority)} + tasks: list[dict[str, Any]] = [] + for task_id in sorted(task_metadata): + candidates = sorted( + candidates_by_task.get(task_id, []), + key=lambda item: priority_index[(item["method"]["model"], item["method"]["engine"])], + ) + clean_candidates = [item for item in candidates if item["clean"]] + tasks.append( + { + **task_metadata[task_id], + "successful_methods": sorted(passed_methods_by_task.get(task_id, [])), + "candidates": candidates, + "canonical_candidate_id": clean_candidates[0]["id"] if clean_candidates else None, + "replay_status": ( + "ready" + if clean_candidates + else "success_contains_failed_actions" + if candidates + else "all_methods_failed" + ), + } + ) + + return { + "schema": SCHEMA, + "source": { + "kind": "authoritative successful Browser Use executions", + "human_demonstrations_used": False, + "benchmark": str(profile.get("benchmark") or "WebArena-Verified"), + "benchmark_version": str(profile.get("benchmark_version") or ""), + "official_data_checksum": str(profile.get("official_data_checksum") or ""), + "selection_sha256": selection_hashes.pop(), + "evidence": { + model: { + "root": str(source_roots[model]), + "attempts_sha256": _sha256(source_roots[model] / "attempts.jsonl"), + "manifest_sha256": _sha256(source_roots[model] / "manifest.json"), + } + for model in models + }, + }, + "selection_policy": { + "authority": "final execution declared by each frozen Browser Use manifest", + "eligibility": ( + "official score 1.0, every BrowserTool call completed successfully, and " + "every ephemeral snapshot ref compiled to a fixed selector" + ), + "canonical_priority": [f"{model}__{engine}" for model, engine in method_priority], + "performance_used_for_selection": False, + }, + "counts": { + "tasks": len(tasks), + "tasks_with_authoritative_success": sum(bool(item["candidates"]) for item in tasks), + "replay_ready_tasks": sum(item["replay_status"] == "ready" for item in tasks), + "success_with_failed_actions_tasks": sum( + item["replay_status"] == "success_contains_failed_actions" for item in tasks + ), + "all_methods_failed_tasks": sum( + item["replay_status"] == "all_methods_failed" for item in tasks + ), + "successful_method_trajectories": sum(len(item["candidates"]) for item in tasks), + "clean_method_trajectories": sum( + sum(bool(candidate["clean"]) for candidate in item["candidates"]) for item in tasks + ), + }, + "tasks": tasks, + } diff --git a/harness/browser_use/cli.py b/harness/trajectory_replay/cli.py similarity index 95% rename from harness/browser_use/cli.py rename to harness/trajectory_replay/cli.py index 22e12ec6d..f9923c8e8 100644 --- a/harness/browser_use/cli.py +++ b/harness/trajectory_replay/cli.py @@ -1,4 +1,4 @@ -"""Command-line entrypoint for isolated browser use evaluation.""" +"""Command-line entrypoint for isolated browser trajectory replay.""" from __future__ import annotations diff --git a/harness/browser_use/diagnostics.py b/harness/trajectory_replay/diagnostics.py similarity index 100% rename from harness/browser_use/diagnostics.py rename to harness/trajectory_replay/diagnostics.py diff --git a/harness/browser_use/executor.py b/harness/trajectory_replay/executor.py similarity index 100% rename from harness/browser_use/executor.py rename to harness/trajectory_replay/executor.py diff --git a/harness/trajectory_replay/expert_seeds.py b/harness/trajectory_replay/expert_seeds.py new file mode 100644 index 000000000..3e3568378 --- /dev/null +++ b/harness/trajectory_replay/expert_seeds.py @@ -0,0 +1,199 @@ +"""Merge explicitly reviewed expert repairs into Browser Use replay seeds.""" + +from __future__ import annotations + +import copy +import hashlib +import json +from pathlib import Path +from typing import Any + +BASE_SCHEMA = "lex_browser_use_replay_seeds/2" +REPAIR_SCHEMA = "lex_webarena_expert_seed_repairs/1" + + +def _canonical_bytes(value: Any) -> bytes: + return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode() + + +def _sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def _sha256(path: Path) -> str: + return _sha256_bytes(path.read_bytes()) + + +def _read_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +def _contains_ref(value: Any) -> bool: + if isinstance(value, dict): + return "ref" in value or any(_contains_ref(item) for item in value.values()) + if isinstance(value, list): + return any(_contains_ref(item) for item in value) + return False + + +def _validate_tool_calls(task_id: int, tool_calls: Any) -> list[dict[str, Any]]: + if not isinstance(tool_calls, list) or not tool_calls: + raise ValueError(f"expert repair for task {task_id} must contain tool_calls") + validated: list[dict[str, Any]] = [] + for index, raw_call in enumerate(tool_calls, start=1): + if not isinstance(raw_call, dict): + raise ValueError(f"expert repair for task {task_id} call {index} must be an object") + call = copy.deepcopy(raw_call) + if not isinstance(call.get("action"), str) or not call["action"].strip(): + raise ValueError(f"expert repair for task {task_id} call {index} has no action") + if _contains_ref(call): + raise ValueError( + f"expert repair for task {task_id} call {index} contains forbidden ref" + ) + if call["action"] == "wait": + seconds = call.get("seconds") + if not isinstance(seconds, (int, float)) or isinstance(seconds, bool) or seconds <= 0: + raise ValueError( + f"expert repair for task {task_id} call {index} wait requires " + "a positive seconds value" + ) + unsupported = {"timeout", "selector"} & call.keys() + if unsupported: + raise ValueError( + f"expert repair for task {task_id} call {index} wait contains " + f"unsupported fields: {sorted(unsupported)}" + ) + validated.append(call) + return validated + + +def _required_record(repair: dict[str, Any], task_id: int, field: str) -> Any: + value = repair.get(field) + if value is None or value == "" or value == {} or value == []: + raise ValueError(f"expert repair for task {task_id} has no {field}") + return copy.deepcopy(value) + + +def _refresh_counts(manifest: dict[str, Any]) -> None: + tasks = manifest.get("tasks") or [] + counts = manifest.setdefault("counts", {}) + counts["replay_ready_tasks"] = sum(task.get("replay_status") == "ready" for task in tasks) + counts["success_with_failed_actions_tasks"] = sum( + task.get("replay_status") == "success_contains_failed_actions" for task in tasks + ) + counts["all_methods_failed_tasks"] = sum( + task.get("replay_status") == "all_methods_failed" for task in tasks + ) + counts["expert_repair_candidates"] = sum( + candidate.get("candidate_kind") == "expert_repair" + for task in tasks + for candidate in task.get("candidates") or [] + ) + + +def merge_expert_seed_manifest( + base_manifest: dict[str, Any], + repair_spec: dict[str, Any], + *, + base_manifest_sha256: str | None = None, + repair_spec_sha256: str | None = None, +) -> dict[str, Any]: + """Append expert candidates, replacing a ready canonical only when explicit.""" + + if base_manifest.get("schema") != BASE_SCHEMA: + raise ValueError("unsupported base replay seed manifest") + if repair_spec.get("schema") != REPAIR_SCHEMA: + raise ValueError("unsupported expert repair specification") + + merged = copy.deepcopy(base_manifest) + tasks = {int(task["task_id"]): task for task in merged.get("tasks") or []} + repairs = repair_spec.get("repairs") + if not isinstance(repairs, list) or not repairs: + raise ValueError("expert repair specification must contain repairs") + + seen_tasks: set[int] = set() + seen_candidate_ids = { + str(candidate.get("id")) + for task in tasks.values() + for candidate in task.get("candidates") or [] + } + spec_sha256 = repair_spec_sha256 or _sha256_bytes(_canonical_bytes(repair_spec)) + + for repair in repairs: + if not isinstance(repair, dict) or "task_id" not in repair: + raise ValueError("every expert repair must be an object with task_id") + task_id = int(repair["task_id"]) + if task_id in seen_tasks: + raise ValueError(f"duplicate expert repair for task {task_id}") + seen_tasks.add(task_id) + task = tasks.get(task_id) + if task is None: + raise ValueError(f"expert repair references task missing from base seeds: {task_id}") + replaces_canonical = repair.get("replaces_canonical") is True + if ( + task.get("replay_status") == "ready" or task.get("canonical_candidate_id") + ) and not replaces_canonical: + raise ValueError(f"expert repair cannot replace ready task {task_id}") + + candidate_id = str(repair.get("candidate_id") or f"expert__task-{task_id}") + if not candidate_id.strip() or candidate_id in seen_candidate_ids: + raise ValueError(f"duplicate or empty expert candidate id: {candidate_id!r}") + seen_candidate_ids.add(candidate_id) + tool_calls = _validate_tool_calls(task_id, repair.get("tool_calls")) + expected_response = _required_record(repair, task_id, "expected_response") + if not isinstance(expected_response, dict): + raise ValueError( + f"expert repair for task {task_id} expected_response must be an object" + ) + + candidate = { + "id": candidate_id, + "candidate_kind": "expert_repair", + "action_count": len(tool_calls), + "actions": [str(call["action"]) for call in tool_calls], + "tool_calls": tool_calls, + "expected_response": expected_response, + # The replay runner currently consumes this field as evaluator input. It is an + # authored expectation, not a response observed from an autonomous Agent run. + "recorded_response": copy.deepcopy(expected_response), + "source": { + "kind": "expert_repair_spec", + "details": _required_record(repair, task_id, "source"), + "repair_spec_sha256": spec_sha256, + }, + "repair_basis": _required_record(repair, task_id, "repair_basis"), + "replaces_canonical": replaces_canonical, + } + task.setdefault("candidates", []).append(candidate) + task["canonical_candidate_id"] = candidate_id + task["replay_status"] = "ready" + + _refresh_counts(merged) + merged["expert_repairs"] = { + "schema": REPAIR_SCHEMA, + "base_manifest_sha256": base_manifest_sha256 + or _sha256_bytes(_canonical_bytes(base_manifest)), + "repair_spec_sha256": spec_sha256, + "candidate_count": len(repairs), + "policy": { + "existing_ready_candidates_immutable_unless_explicitly_replaced": True, + "browser_tool_refs_forbidden": True, + "status_meaning": "ready for replay; not yet replay-verified", + }, + } + return merged + + +def build_expert_seed_manifest( + base_manifest_path: str | Path, repair_spec_path: str | Path +) -> dict[str, Any]: + """Read a base manifest and explicit repair spec, then return their bound merge.""" + + base_path = Path(base_manifest_path).expanduser().resolve() + repair_path = Path(repair_spec_path).expanduser().resolve() + return merge_expert_seed_manifest( + _read_json(base_path), + _read_json(repair_path), + base_manifest_sha256=_sha256(base_path), + repair_spec_sha256=_sha256(repair_path), + ) diff --git a/harness/browser_use/metrics.py b/harness/trajectory_replay/metrics.py similarity index 100% rename from harness/browser_use/metrics.py rename to harness/trajectory_replay/metrics.py diff --git a/harness/trajectory_replay/report.py b/harness/trajectory_replay/report.py new file mode 100644 index 000000000..4b01b24fb --- /dev/null +++ b/harness/trajectory_replay/report.py @@ -0,0 +1,1318 @@ +"""Chinese paginated evidence-first browser benchmark report.""" + +from __future__ import annotations + +import html +import json +import re +from collections import Counter +from dataclasses import dataclass +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + +REPORT_STEM = "trajectory-replay-evaluation-latest" +PAGES = tuple( + (slug, label, f"{REPORT_STEM}{suffix}.html") + for slug, label, suffix in ( + ("index", "结论摘要", ""), + ("suite", "评测集", "-suite"), + ("method", "环境与方法", "-method"), + ("analysis", "逐任务分析", "-analysis"), + ("appendix", "复现与附录", "-appendix"), + ) +) + + +def _e(value: Any) -> str: + return html.escape(str(value if value is not None else "")) + + +def _trace_text(value: Any) -> str: + """Remove machine-identifying IPs and diff-breaking line-end whitespace.""" + text = str(value if value is not None else "") + text = re.sub(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", "[IP 已脱敏]", text) + return "\n".join(line.rstrip() for line in text.splitlines()) + + +def _show(value: Any, divisor: float = 1.0, suffix: str = "") -> str: + return f"{value / divisor:,.2f}{suffix}" if isinstance(value, (int, float)) else "无数据" + + +def _stat(metric: dict[str, Any], key: str, divisor: float = 1.0) -> str: + value = metric.get(key, metric.get("median") if key == "p50" else None) + return _show(value, divisor) + + +def _rate(values: dict[str, Any]) -> str: + value = values.get("pass_rate") + return _show(value * 100 if isinstance(value, (int, float)) else None, suffix="%") + + +@dataclass(frozen=True) +class ComparedValue: + text: str + css_class: str + + +def _comparison_class(value: Any, other: Any, *, higher: bool = False) -> str: + if not isinstance(value, (int, float)) or not isinstance(other, (int, float)): + return "" + if value == other: + return "" + return "better" if (value > other if higher else value < other) else "worse" + + +def _table( + headers: list[str], rows: list[list[Any]], caption: str, *, source_marks: bool = False +) -> str: + head = "".join(f"{_e(item)}" for item in headers) + body = "".join( + "" + + "".join( + f'" + + ( + _annotate_source_marks(_e(v.text if isinstance(v, ComparedValue) else v)) + if source_marks + else _e(v.text if isinstance(v, ComparedValue) else v) + ) + + "" + for i, v in enumerate(row) + ) + + "" + for row in rows + ) + return f'
    {head}{body}
    {_e(caption)}
    ' + + +def _numeric_stat(metric: dict[str, Any], key: str) -> float | int | None: + value = metric.get(key, metric.get("median") if key == "p50" else None) + return value if isinstance(value, (int, float)) else None + + +def _display_number(value: float | int | None, divisor: float = 1.0) -> float | None: + """Return the exact two-decimal number shown in an absolute metric cell.""" + if not isinstance(value, (int, float)): + return None + return float(f"{value / divisor:.2f}") + + +def _absolute_metric_cell( + value: float | int | None, + other: float | int | None, + divisor: float, +) -> str: + css_class = "" + if ( + _comparison_class(_display_number(value, divisor), _display_number(other, divisor)) + == "better" + ): + css_class = ' class="better"' + return f"{_e(_show(value, divisor))}" + + +def _ratio_metric_cell(value: float | int | None) -> str: + css_class = _comparison_class(_display_number(value), 1) + class_attr = f' class="{css_class}"' if css_class else "" + return f"{_e(_show(value, suffix='×'))}" + + +def _grouped_resource_table( + summary: dict[str, Any], + metrics: tuple[tuple[str, str, float, str], ...], + caption: str, +) -> str: + engines = (summary.get("resource_comparison") or {}).get("by_engine") or {} + body = [] + for key, label, divisor, unit in metrics: + chromium = (engines.get("chromium") or {}).get(key) or {} + moli = (engines.get("moli") or {}).get(key) or {} + cells = [f'{_e(f"{label} ({unit})")}'] + for stat in ("p50", "p95", "mean"): + c_value = _numeric_stat(chromium, stat) + m_value = _numeric_stat(moli, stat) + cells.append(_absolute_metric_cell(c_value, m_value, divisor)) + cells.append(_absolute_metric_cell(m_value, c_value, divisor)) + body.append("" + "".join(cells) + "") + return ( + f'
    ' + '' + "" + "" + + "".join(body) + + "
    {_e(caption)}
    指标P50P95平均值
    ChromiumMoliChromiumMoliChromiumMoli
    " + ) + + +def _paired_ratio_table( + summary: dict[str, Any], + metrics: tuple[tuple[str, str, float, str], ...], + caption: str, +) -> str: + paired = (summary.get("resource_comparison") or {}).get("paired_ratios") or {} + values = paired.get("metrics") or {} + rows = [] + for key, label, _divisor, _unit in metrics: + metric = values.get(key) or {} + rows.append( + [ + label, + *[ + ComparedValue( + _show(_numeric_stat(metric, stat), suffix="×"), + _comparison_class(_display_number(_numeric_stat(metric, stat)), 1), + ) + for stat in ("p50", "p95", "mean") + ], + ] + ) + return _table( + ["指标", "逐任务比值 P50", "逐任务比值 P95", "逐任务比值平均值"], + rows, + caption, + ) + + +def _duration_ratio_audit(summary: dict[str, Any]) -> str: + paired = (summary.get("resource_comparison") or {}).get("paired_ratios") or {} + metric = (paired.get("metrics") or {}).get("duration_ms") or {} + values = metric.get("values") or [] + if not values: + return "" + p95_rank = max(1, (95 * len(values) + 99) // 100) + rows = [] + for rank, item in enumerate(values, 1): + task_id = str(item.get("task_id") or "") + title = str(((summary.get("by_task") or {}).get(task_id) or {}).get("title") or task_id) + rows.append( + [ + str(rank), + title, + *[ + ComparedValue( + _show(item.get(key), suffix=" ms"), + "better" + if _comparison_class( + _display_number(item.get(key)), _display_number(item.get(other)) + ) + == "better" + else "", + ) + for key, other in (("denominator", "numerator"), ("numerator", "denominator")) + ], + ComparedValue( + _show(item.get("ratio"), suffix="×"), + _comparison_class(_display_number(item.get("ratio")), 1), + ), + "P95 取值" if rank == p95_rank else "", + ] + ) + return ( + "
    查看逐任务耗时比值的完整计算审计" + f"

    脚本先对每个计入成功交集的同任务计算 Moli 耗时 / Chromium 耗时,再排序。" + f"本轮 n={len(values)},nearest-rank P95 的序号为 " + f"ceil(0.95 × {len(values)}) = {p95_rank};因此表中的 P95 是排序后第 " + f"{p95_rank} 个配对比值。

    " + + _table( + ["排序", "任务", "Chromium", "Moli", "逐任务比值", "统计位置"], + rows, + "耗时配对比值原始序列", + ) + + "
    " + ) + + +def _nav(active: str) -> str: + return ( + '" + ) + + +def _shell( + manifest: dict[str, Any], + active: str, + body: str, + *, + detail_title: str = "", + back_anchor: str = "", +) -> str: + label = next(p[1] for p in PAGES if p[0] == active) + index = next(i for i, p in enumerate(PAGES) if p[0] == active) + prev_link = ( + f'← {PAGES[index - 1][1]}' if index else "" + ) + next_link = ( + f'{PAGES[index + 1][1]} →' + if index + 1 < len(PAGES) + else "" + ) + header = ( + f'
    ← 返回逐任务分析' + f'Chromium / Moli 评测运行 {_e(manifest.get("run_id", "unknown"))}
    ' + if detail_title + else f'
    Lexmount Browser Provider Evaluation

    Chromium 与 Moli 浏览器工具调用评测

    运行编号:{_e(manifest.get("run_id", "unknown"))} · 第 {index + 1}/{len(PAGES)} 页
    {_nav(active)}' + ) + pager = "" if detail_title else f'
    {prev_link}{next_link}
    ' + return f"""{_e(detail_title) or label}|Chromium 与 Moli 评测
    {header}
    {body}{pager}
    Lexmount Browser Provider Evaluation
    """ + + +def _functional_rows(summary: dict[str, Any]) -> list[list[str]]: + rows = [] + correctness_by_engine = {row[0]: row for row in _correctness_rows(summary)} + for engine in ("chromium", "moli"): + values = (summary.get("functional") or {}).get(engine) or {} + duration = values.get("duration_ms") or {} + rows.append( + correctness_by_engine[engine] + + [ + _stat(duration, "p50"), + _stat(duration, "mean"), + _stat(duration, "p95"), + ] + ) + return rows + + +def _correctness_rows(summary: dict[str, Any]) -> list[list[str]]: + rows: list[list[str]] = [] + for engine in ("chromium", "moli"): + values = (summary.get("functional") or {}).get(engine) or {} + tasks = int(values.get("tasks") or 0) + + def score( + key: str, + fallback: str, + engine_values: dict[str, Any] = values, + task_total: int = tasks, + ) -> str: + passed = int(engine_values.get(key, engine_values.get(fallback, 0)) or 0) + rate = passed / task_total * 100 if task_total else 0 + return f"{passed}/{task_total}({rate:.2f}%)" + + rows.append( + [ + engine, + score("result_final_passes", "final_passes"), + score("combined_final_passes", "final_passes"), + ] + ) + return rows + + +def _correctness_table(summary: dict[str, Any]) -> str: + functional = summary.get("functional") or {} + scores = { + engine: ( + int((functional.get(engine) or {}).get("result_final_passes", 0) or 0), + int((functional.get(engine) or {}).get("combined_final_passes", 0) or 0), + int((functional.get(engine) or {}).get("tasks", 0) or 0), + ) + for engine in ("chromium", "moli") + } + rows = [] + for engine, (result, combined, tasks) in scores.items(): + result_rate = result / tasks * 100 if tasks else 0 + combined_rate = combined / tasks * 100 if tasks else 0 + other = scores["moli" if engine == "chromium" else "chromium"] + result_winner = ( + tasks + and other[2] + and _comparison_class(result / tasks, other[0] / other[2], higher=True) == "better" + ) + combined_winner = ( + tasks + and other[2] + and _comparison_class(combined / tasks, other[1] / other[2], higher=True) == "better" + ) + rows.append( + "" + f"{_e(engine)}" + f'{result}/{tasks}({result_rate:.2f}%)' + f'{combined}/{tasks}({combined_rate:.2f}%)' + f'{max(0, tasks - result)} 个失败 ' + "" + ) + return ( + '
    ' + "" + "" + "".join(rows) + "
    任务正确率
    Provider结果正确结果与参考过程均正确任务分析
    " + ) + + +def _aggregate_ratio(summary: dict[str, Any], metric_name: str, stat: str) -> float | None: + engines = (summary.get("resource_comparison") or {}).get("by_engine") or {} + chromium = _numeric_stat(((engines.get("chromium") or {}).get(metric_name) or {}), stat) + moli = _numeric_stat(((engines.get("moli") or {}).get(metric_name) or {}), stat) + if ( + not isinstance(chromium, (int, float)) + or chromium == 0 + or not isinstance(moli, (int, float)) + ): + return None + return float(moli) / float(chromium) + + +def _paired_ratio(summary: dict[str, Any], metric_name: str, stat: str) -> float | None: + metric = ( + ((summary.get("resource_comparison") or {}).get("paired_ratios") or {}).get("metrics") or {} + ).get(metric_name) or {} + return _numeric_stat(metric, stat) + + +def _detail_filename(task_id: str, engine: str) -> str: + return f"{REPORT_STEM}-task-{task_id.encode().hex()}-{engine.encode().hex()}.html" + + +def _summary_page(manifest: dict[str, Any], summary: dict[str, Any]) -> str: + task_count = len(manifest.get("suite") or []) + f = summary.get("functional") or {} + c = f.get("chromium") or {} + m = f.get("moli") or {} + paired = (summary.get("resource_comparison") or {}).get("intersection_attempts", 0) + + agent_smoke = manifest.get("agent_smoke") or {} + smoke_html = "" + if agent_smoke: + provenance = agent_smoke.get("provenance") or {} + smoke_html = f"""

    CowAgent 端到端冒烟测试

    该测试与资源微基准分开,不参与上表统计,也没有在运行编号 {_e(manifest.get("run_id"))} 中重新执行。模型 {_e(agent_smoke.get("model"))},思考强度 {_e(agent_smoke.get("reasoning_effort"))};独立执行日期 {_e(agent_smoke.get("executed_at"))},来源文件内容哈希 {_e(str(provenance.get("source_sha256") or "")[:12])}。同一任务每个浏览器方案只执行 1 次。

    {_table(["浏览器方案", "结果", "BrowserTool 调用次数", "最终答案"], [[str(item.get("provider")), "通过" if item.get("status") == "pass" else str(item.get("status")), str(item.get("browser_tool_calls")), str(item.get("answer"))] for item in agent_smoke.get("results") or []], "独立端到端接受性测试")}
    """ + c_passes = int(c.get("result_final_passes", c.get("final_passes", 0))) + m_passes = int(m.get("result_final_passes", m.get("final_passes", 0))) + + if c_passes > m_passes: + recommendation = ( + f"本轮建议默认使用 Chromium:其结果正确任务数为 {c_passes},高于 Moli 的 {m_passes};" + "Moli 在双方均成功的任务上若表现出更低耗时或资源占用,可作为任务已验证兼容时的备选。" + ) + elif m_passes > c_passes: + recommendation = ( + f"本轮建议默认使用 Moli:其结果正确任务数为 {m_passes},高于 Chromium 的 {c_passes};" + "仍需结合双方成功任务上的资源指标和失败类型确认适用边界。" + ) + else: + recommendation = ( + f"本轮两个 Provider 的结果正确任务数相同(均为 {c_passes});" + "仅凭正确性无法确定默认方案,应结合双方成功任务上的资源指标和失败类型决策。" + ) + correctness = _correctness_table(summary) + return f"""

    结论与产品建议

    {recommendation}

    这是浏览器工具调用层的公网确定性微基准,不是 LLM Agent 排名。评测集包含 {task_count} 个结果唯一的任务。

    +

    任务正确率

    {correctness}
    +

    关键技术结论

      +
    • 速度与资源:双方成功任务的逐任务 Moli / Chromium 比值中,耗时比平均值为 {_show(_paired_ratio(summary, "duration_ms", "mean"), suffix="×")},时间加权平均内存比平均值为 {_show(_paired_ratio(summary, "memory_mean_bytes", "mean"), suffix="×")},网络接收量比平均值为 {_show(_paired_ratio(summary, "network_rx_bytes", "mean"), suffix="×")}。
    • +
    • 时序边界:Moli 的 1 Hz 是 CDP screencast 下采样,不等于 JavaScript 或 requestAnimationFrame 只有 1 fps;源码中的 rAF 目标间隔为 16 ms。
    • +
    • 根因边界:布局与 actionability 证据只能定位现象,尚不能确认 screencast、渲染器或适配层中的具体根因。查看逐任务证据与验证方案。
    +

    双方成功任务的速度与资源

    统计范围:双方都成功的 {paired} 个同任务,不含失败与重试样本。绝对值优胜项为绿色粗体;配对比值低于 1 为绿色、高于 1 为红色,显示值相同保持中性。

    {_metric_tables(summary)}{_duration_ratio_audit(summary)}
    {smoke_html}""" + + +def _difficulty(value: str) -> str: + named = {"simple": "简单", "medium": "中等", "complex": "复杂"} + if value in named: + return named[value] + match = re.fullmatch(r"step_(\d+)_(\d+)", value) + if match: + return f"{int(match.group(1))}–{int(match.group(2))} 步" + return value or "未标注" + + +def _action_text(call: dict[str, Any]) -> str: + action = str(call.get("action") or "") + return { + "navigate": f"打开 {call.get('url', '')} 并返回页面观察快照", + "get_text": f"读取 {call.get('selector', '')} 文本", + "get_attribute": f"读取 {call.get('selector', '')} 的 {call.get('name', '')} 属性", + "get_property": f"读取 {call.get('selector', '')} 的 {call.get('name', '')} DOM 属性", + "click": f"点击 {call.get('selector', '')}", + "fill": f"向 {call.get('selector', '')} 填入 {call.get('text', '')}", + "select": f"在 {call.get('selector', '')} 选择 {call.get('value', '')}", + "snapshot": f"读取 {call.get('selector') or 'body'} 当前快照", + "scroll": f"向 {call.get('direction', 'down')} 滚动 {call.get('amount', 700)} px", + "back": "浏览器后退", + "forward": "浏览器前进", + "press": f"按键 {call.get('key', '')}", + }.get(action, action) + + +def _source_references(source: dict[str, Any]) -> str: + links = [] + for url in source.get("design_basis") or []: + parsed = urlparse(str(url)) + if parsed.scheme in {"https", "http"}: + label = parsed.path.rstrip("/").rsplit("/", 1)[-1] or parsed.hostname + links.append(f'{_e(label)}') + return '

    设计依据:' + " · ".join(links) + "

    " if links else "" + + +def _annotate_source_marks(escaped_text: str) -> str: + """Annotate the literal heading permalink character without changing the oracle.""" + return escaped_text.replace( + "¶", '' + ) + + +def _source_mark_note(values: Any) -> str: + return ( + '

    ¶ 为原网页标题的锚点字符,参与逐字判定。

    ' + if "¶" in str(values) + else "" + ) + + +def _suite_page(manifest: dict[str, Any]) -> str: + suite = manifest.get("suite") or [] + validation = manifest.get("ground_truth_validation") or {} + replay = validation.get("automatic_browser_replay") or {} + difficulty_values = [str(task.get("complexity") or "") for task in suite] + ordered_difficulties = [ + difficulty + for difficulty in ("simple", "medium", "complex") + if difficulty in difficulty_values + ] + sorted( + { + difficulty + for difficulty in difficulty_values + if difficulty not in {"simple", "medium", "complex"} + } + ) + difficulty_counts = { + difficulty: difficulty_values.count(difficulty) for difficulty in ordered_difficulties + } + difficulty_summary = ( + "、".join( + f"{_difficulty(difficulty)} {count} 个" + for difficulty, count in difficulty_counts.items() + if count + ) + or "未标注难度" + ) + site_counts = Counter( + urlparse(str((task.get("source") or {}).get("url") or "")).hostname or "未标注" + for task in suite + ) + action_counts = Counter( + str(call.get("action") or "") for task in suite for call in task.get("tool_calls") or [] + ) + distribution = ( + '
    ' + f'
    {len(suite)}公网任务
    ' + f'
    {len(site_counts)}独立网站/域名
    ' + f'
    {sum(action_counts.values())}参考 BrowserTool 调用
    ' + "
    " + + _table( + ["维度", "分布"], + [ + ["难度", difficulty_summary], + [ + "网站", + ";".join(f"{host} {count}" for host, count in sorted(site_counts.items())), + ], + [ + "能力", + ";".join( + f"{action} {count}" for action, count in sorted(action_counts.items()) + ), + ], + ], + "评测集分布", + ) + ) + cards = [] + for index, task in enumerate(suite, 1): + title = str(task.get("title") or task.get("task_id")) + calls = task.get("tool_calls") or [] + call_html = "".join( + f'
    调用 {i} · {_e(c.get("action"))}
    {_e(_action_text(c))}' + + ( + f'
    输出:{_e(c.get("save_as"))}' + if c.get("save_as") + else "" + ) + + "
    " + for i, c in enumerate(calls, 1) + ) + truth = task.get("ground_truth") or {} + + expected_inline = ";".join( + f"{name} = {value}" for name, value in (truth.get("expected_outputs") or {}).items() + ) + expected_rows = ( + '
    ' + + "".join( + f"
    {_e(name)}
    {_annotate_source_marks(_e(value))}
    " + for name, value in (truth.get("expected_outputs") or {}).items() + ) + + "
    " + + _source_mark_note(truth.get("expected_outputs")) + ) + reference_path = " → ".join(truth.get("expected_actions") or []) + source = task.get("source") or {} + target_url = str(source.get("url") or "") + source_html = ( + f'{_e(target_url)}' + ) + shared_tags = {"simple", "medium", "complex", "public-web", "exact-oracle"} + chips = "".join( + f'{_e(tag)}' + for tag in task.get("tags") or [] + if tag not in shared_tags + ) + search_text = ( + f"任务{index:02d} 任务 {index:02d} {task.get('task_id')} {title} {target_url} " + + " ".join(task.get("tags") or []) + + " " + + expected_inline + ) + cards.append( + f'
    任务 {index:02d}

    {_e(title)}

    {_difficulty(str(task.get("complexity")))}{len(calls)} 次工具调用{chips}' + '
    ' + f"
    公网测试页面{source_html}
    " + f"
    参考路径{_e(reference_path)}
    " + f"
    唯一预期结果{expected_rows}
    " + "
    " + f"
    逐步操作与答案依据{call_html}

    答案依据:{_e(truth.get('derivation'))}

    {_source_references(source)}
    " + ) + filters = ( + '

    查找任务

    ' + f'显示 {len(suite)} / {len(suite)} 个任务
    ' + '
    ' + '
    ' + '' + ) + script = """""" + return ( + f'

    评测集概览

    {distribution}

    答案核验:{_e(validation.get("validator") or "未记录")},{_e(replay.get("passed"))}/{_e(replay.get("total"))} 已核验。数据集 SHA-256:{_e(str(validation.get("dataset_sha256") or "")[:16])}。

    标注来源与版本

    {_e(validation.get("method") or "未记录")}

    ' + + filters + + "".join(cards) + + script + ) + + +def _method_page(manifest: dict[str, Any]) -> str: + p = manifest.get("protocol") or {} + env = manifest.get("environment") or {} + versions = env.get("browser_versions") or {} + rows = [ + ["运行平台", str(env.get("runner_platform", "未记录"))], + [ + "容器", + f"Docker {env.get('docker_server_version', '未记录')} / {env.get('docker_os', '未记录')}", + ], + ["Chromium", str(versions.get("chromium", "未记录"))], + ["Moli", str(versions.get("moli", "未记录"))], + [ + "单浏览器配额", + f"{p.get('cpus_per_browser', '?')} CPU / {p.get('memory_per_browser', '?')} / 独立网络", + ], + ["执行次数", "每任务每个 Provider 形成 1 个正式结果"], + ["视口", json.dumps(p.get("viewport") or {}, ensure_ascii=False)], + [ + "失败复核", + f"失败时最多补充 {p.get('max_failure_retries', '?')} 条同任务证据,不另计为新任务", + ], + ["采样", f"每 {p.get('sample_interval_ms', '?')} ms 采样内存、PID、CPU、网络计数"], + ] + return f"""

    步骤的严格定义

    一步 = 一次 CowAgent BrowserTool 等价调用,例如 navigate、click、get_text、fill、select、scroll、back 或 forward。等待页面就绪和目标值出现属于该工具调用内部,不单列、不计步。每次调用的参数、耗时、输出和状态写入执行轨迹。

    成功判定:两个独立程序指标

    1. 结果正确(主指标):每个标注输出与 expected_outputs 做严格字符串相等;不使用模糊匹配或模型评分;Codex 只参与评测前的标注核对。
    2. 结果与参考过程均正确(辅指标):在结果正确之外,本轮被重放的参考动作序列还必须与 expected_actions 一致,且全部 BrowserTool 调用成功。

    边界:expected_actions 是工具层微基准的固定参考路径,不代表任务只有一种解法。若评测自由规划 CowAgent,应把过程判定改成必要里程碑、禁止动作和步数预算,不能要求整条序列逐步相等。

    环境

    {_table(["项目", "记录值"], rows, "本轮环境")}

    执行流程

    1. 分别启动并预热两个浏览器,浏览器进程和上下文在整轮评测中保持运行。
    2. 任务前清 Cookie、权限、HTTP 缓存、Cache Storage、Service Worker、IndexedDB、来源存储和历史,并创建新页面。
    3. 计时外静置 {p.get("post_reset_settle_ms", "?")} ms,按任务交替两个 Provider 的执行顺序。
    4. 执行完整任务并采样;失败复核归入同一任务详情。
    5. 资源基线只比较双方都成功的同任务,失败与复核样本不混入正常路径。

    统计口径

    • P50 与中位数是同一个统计量;总体跨任务报告 P50、平均值和 nearest-rank P95。
    • 配对比值的定义和逐项计算过程见首页性能表与计算审计。
    • 每个任务仅有一个正式样本,因此逐任务不计算分位数;当前跨任务点不足以证明普遍性能优势。
    • 内存平均值是时间加权均值;任务内存 P95 来自单次任务时序采样。
    • RX = Receive(接收),TX = Transmit(发送);总量含页面资源与 CDP,速率来自相邻采样点差分。
    """ + + +METRICS = ( + ( + "时间与计算", + ( + ("duration_ms", "总耗时", 1, "ms"), + ("cpu_ms", "CPU 时间", 1, "ms"), + ("cpu_average_utilization_pct", "平均 CPU 占单核", 1, "%"), + ("cpu_sampled_utilization_p95_pct", "任务内 CPU P95", 1, "%"), + ), + ), + ( + "内存", + ( + ("memory_mean_bytes", "时间加权平均内存", 1048576, "MiB"), + ("memory_p95_bytes", "任务内存 P95", 1048576, "MiB"), + ("memory_peak_bytes", "峰值内存", 1048576, "MiB"), + ("memory_retained_bytes", "结束保留增量", 1048576, "MiB"), + ("memory_mib_seconds", "内存占用积分", 1, "MiB·s"), + ), + ), + ( + "网络", + ( + ("network_rx_bytes", "接收总量 RX", 1024, "KiB"), + ("network_tx_bytes", "发送总量 TX", 1024, "KiB"), + ("network_rx_bytes_per_second", "平均接收速率", 1024, "KiB/s"), + ("network_rx_rate_p95_bytes_per_second", "任务内接收速率 P95", 1024, "KiB/s"), + ("network_tx_rate_p95_bytes_per_second", "任务内发送速率 P95", 1024, "KiB/s"), + ), + ), +) + + +def _metric_tables(summary: dict[str, Any]) -> str: + absolute = "".join( + _grouped_resource_table(summary, metrics, f"{group}绝对值(均为越低越好)") + for group, metrics in METRICS + ) + paired = "".join( + _paired_ratio_table( + summary, + metrics, + f"{group}逐任务 Moli / Chromium 配对比值(均为越低越好)", + ) + for group, metrics in METRICS + ) + return ( + absolute + + "

    逐任务配对比值分布

    先对每个同任务计算 Moli / Chromium,再汇总这些比值;P50、P95 与平均值都来自同一批逐任务比值,不是两个 Provider 边际分位数相除。

    " + + paired + ) + + +def _task_detail_body( + task_id: str, + title: str, + engine: str, + spec: dict[str, Any], + attempts: list[dict[str, Any]], + diagnostic: dict[str, Any] | None, +) -> str: + """One definition, one comparison matrix, one copy of each raw payload.""" + evidence: dict[str, tuple[str, list[str]]] = {} + + def evidence_link(value: Any, label: str, display: str = "") -> str: + if value is None or value == "": + return "" + text = _trace_text( + json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + if isinstance(value, (dict, list)) + else value + ) + if text not in evidence: + evidence[text] = (f"evidence-{len(evidence) + 1}", []) + anchor, labels = evidence[text] + if label not in labels: + labels.append(label) + return f'{_e(display or label)}' + + def status_cell(value: Any) -> str: + return ( + "通过" + if value == "pass" or value is True + else "失败" + if value == "fail" or value is False + else "未记录" + ) + + final = attempts[-1] + source = spec.get("source") or {} + url = str(source.get("url") or "") + safe_url = url if urlparse(url).scheme in {"http", "https"} else "#" + labels = [f"执行记录 {a.get('attempt', i)}" for i, a in enumerate(attempts, 1)] + total = sum(float(a.get("duration_ms") or 0) for a in attempts) + result = final.get("result_status") or final.get("status") + process = final.get("process_status") or final.get("status") + overview = ( + '
    ' + f'

    {_e(title)}

    {_e(engine)}
    ' + f'

    {_e(task_id)}

    ' + '
    ' + f'
    结果{status_cell(result)}
    ' + f'
    参考过程{status_cell(process)}
    ' + f"
    累计耗时 · {len(attempts)} 次执行{_show(total, suffix=' ms')}
    " + ) + if diagnostic: + overview += ( + '
    ' + f"{_e(diagnostic.get('category') or '其他')}" + f"第 {_e(diagnostic.get('failed_step'))} 步 · {_e(diagnostic.get('failed_action'))}" + f" · {_e(diagnostic.get('selector') or '—')}" + + ( + f"

    {_e(diagnostic['observed_mechanism'])}

    " + if diagnostic.get("observed_mechanism") + else "" + ) + + evidence_link(diagnostic.get("actionability_probe"), "Actionability probe") + + "
    " + ) + overview += f'

    起始页面 {_e(url)}

    ' + + expected = (spec.get("ground_truth") or {}).get("expected_outputs") or {} + validations = [{str(v.get("output")): v for v in a.get("validation") or []} for a in attempts] + outputs = list(dict.fromkeys([*expected, *(k for values in validations for k in values)])) + answer_rows = [] + for output in outputs: + expected_value = expected.get( + output, next((v[output].get("expected") for v in validations if output in v), "未声明") + ) + cells = [output, str(expected_value)] + for values in validations: + item = values.get(output) + cells.append( + ComparedValue( + f"{'(空字符串)' if item.get('actual') == '' else item.get('actual')} · {status_cell(item.get('passed'))}", + "ok" + if item.get("passed") is True + else "bad" + if item.get("passed") is False + else "", + ) + if item + else "未产生(过程失败) · 未判定" + ) + answer_rows.append(cells) + answers = ( + _table(["输出", "标准答案", *labels], answer_rows, "答案对照", source_marks=True) + if outputs + else '

    未声明标准答案。

    ' + ) + answers += _source_mark_note(expected) + derivation = (spec.get("ground_truth") or {}).get("derivation") + answers += f'

    答案依据:{_e(derivation)}

    ' if derivation else "" + + calls = spec.get("tool_calls") or [] + traces = [a.get("trace") or [] for a in attempts] + steps = [] + for i in range(max([len(calls), *(len(t) for t in traces)], default=0)): + call = calls[i] if i < len(calls) else {} + reference = ( + f'{_e(call.get("action"))}{_e(_action_text(call))}' + if call + else "额外调用" + ) + parameters = {k: v for k, v in call.items() if k != "save_as"} + if call: + reference += evidence_link(parameters, f"步骤 {i + 1} 参数", "参数") + if call.get("save_as"): + reference += f"保存为 {_e(call['save_as'])}" + cells = [f"{i + 1}", f"{reference}"] + for label, trace, attempt in zip(labels, traces, attempts, strict=True): + if i >= len(trace): + cells.append('未执行') + continue + actual = trace[i] + passed = actual.get("status") == "pass" + cell = f'{_e(actual.get("action"))}{status_cell(actual.get("status"))} {_show(actual.get("duration_ms"), suffix=" ms")}' + if actual.get("action") != call.get("action"): + cell += '与参考动作不同' + if (actual.get("arguments") or {}) != parameters: + cell += evidence_link( + actual.get("arguments") or {}, f"{label} · 步骤 {i + 1} 参数", "实际参数" + ) + cell += evidence_link(actual.get("output"), f"{label} · 步骤 {i + 1} 输出", "输出") + if actual.get("error"): + # The full exception contains the call error; keep the full payload once. + error = attempt.get("error") + if not error or _trace_text(actual["error"]) not in _trace_text(error): + cell += evidence_link(actual["error"], f"{label} · 步骤 {i + 1} 错误", "异常") + else: + cell += evidence_link(error, f"{label} · 完整异常", "完整异常") + cells.append(f"{cell}") + steps.append("" + "".join(cells) + "") + process_cells = [] + for label, attempt in zip(labels, attempts, strict=True): + pv = attempt.get("process_validation") or {} + process_cells.append( + "" + + status_cell(pv.get("passed")) + + evidence_link(pv, f"{label} · 过程判定", "判定详情") + + "" + ) + steps.append("—过程判定" + "".join(process_cells) + "") + step_table = ( + '
    ' + + "".join(f"" for label in labels) + + "" + + "".join(steps) + + "
    参考操作与执行对照
    步骤浏览器参考动作{_e(label)}
    " + ) + + execution_rows = [] + for label, attempt in zip(labels, attempts, strict=True): + execution_rows.append( + "" + + _e(label) + + "" + + _show(attempt.get("duration_ms"), suffix=" ms") + + "" + + (evidence_link(attempt.get("error"), f"{label} · 完整异常") or "无异常") + + "" + ) + execution_table = ( + '
    ' + + "".join(execution_rows) + + "
    执行记录
    记录耗时异常证据
    " + ) + resource_rows = [] + for _, metrics in METRICS: + for key, label, divisor, unit in metrics: + if key == "duration_ms" or not any(key in (a.get("resource") or {}) for a in attempts): + continue + resource_rows.append( + [ + f"{label} ({unit})", + *[_show((a.get("resource") or {}).get(key), divisor) for a in attempts], + ] + ) + resources = ( + _table(["指标", *labels], resource_rows, "资源对照") + if resource_rows + else '

    未记录资源采样。

    ' + ) + raw = "".join( + f'

    {_e(anchor.replace("evidence-", "证据 "))}

    回到步骤

    {_e(" / ".join(refs))}

    {_e(text)}
    ' + for text, (anchor, refs) in evidence.items() + ) + evidence_index = ( + '
    ' + + "".join( + f'{_e(anchor.replace("evidence-", "证据 "))}{_e(" / ".join(refs))}查看' + for _, (anchor, refs) in evidence.items() + ) + + "
    " + ) + dialog = ( + '' + '

    原始证据

    ' + f'

    {_e(engine)} · {_e(title)}

    ' + '
    ' + '

    ' + ) + script = """""" + return ( + '
    ' + + overview + + '' + + '

    操作与答案

    ' + + answers + + step_table + + _source_references(source) + + "
    " + + '

    执行资源

    ' + + execution_table + + resources + + "
    " + + '

    原始证据

    ' + + (evidence_index + raw if raw else "

    未记录执行轨迹。

    ") + + "
    " + + dialog + + script + + "
    " + ) + + +def _execution_details( + manifest: dict[str, Any], summary: dict[str, Any], detail_pages: dict[str, str] +) -> str: + task_specs = {str(task.get("task_id")): task for task in manifest.get("suite") or []} + diagnostic_groups = ((summary.get("failure_diagnostics") or {}).get("rule_engine") or {}).get( + "groups" + ) or [] + diagnostic_by_attempt = { + ( + str(group.get("task_id")), + str(group.get("engine")), + int(attempt_ref.get("attempt") or 1), + ): str(group.get("category") or "其他") + for group in diagnostic_groups + for attempt_ref in (group.get("attempts") or []) + } + diagnostic_by_execution = { + (str(group.get("task_id")), str(group.get("engine"))): group for group in diagnostic_groups + } + blocks = [] + for task_id, task in (summary.get("by_task") or {}).items(): + title = str(task.get("title") or task_id) + executions = task.get("attempts") or task.get("failures") or {} + for engine, attempts in executions.items(): + if not attempts: + continue + anchor = f"execution-{task_id}-{engine}" + # Hex encoding is injective and keeps arbitrary task IDs out of paths. + detail_name = _detail_filename(str(task_id), str(engine)) + final_attempt = attempts[-1] + final_status = str(final_attempt.get("status") or "fail") + final_result_status = str(final_attempt.get("result_status") or final_status) + final_process_status = str(final_attempt.get("process_status") or final_status) + final_combined_status = str(final_attempt.get("combined_status") or final_status) + categories = sorted( + { + diagnostic_by_attempt.get( + (str(task_id), str(engine), int(attempt.get("attempt") or 1)), + _classify(str(attempt.get("error") or "")), + ) + for attempt in attempts + if str(attempt.get("status") or "fail") == "fail" + } + ) + error_category = ",".join(categories) if categories else "无错误" + actions = sorted( + { + str(call.get("action")) + for attempt in attempts + for call in (attempt.get("trace") or []) + if call.get("action") != "validate_assertions" + } + ) + tags = task.get("tags") or [] + total_duration = sum(float(attempt.get("duration_ms") or 0) for attempt in attempts) + diagnostic = diagnostic_by_execution.get((str(task_id), str(engine))) + detail_body = _task_detail_body( + str(task_id), + title, + str(engine), + task_specs.get(str(task_id)) or {}, + attempts, + diagnostic, + ) + # Artifact Hub injects a site-root base URL; fragment links need the file. + detail_body = detail_body.replace('href="#', f'href="{detail_name}#') + detail_pages[detail_name] = _shell( + manifest, + "analysis", + detail_body, + detail_title=title, + back_anchor=anchor, + ) + search_text = " ".join( + [ + title, + str(engine), + error_category, + *[ + str(a.get("error") or "") + + " " + + json.dumps(a.get("validation") or [], ensure_ascii=False) + + " " + + json.dumps(a.get("trace") or [], ensure_ascii=False) + for a in attempts + ], + ] + ) + blocks.append( + f'' + ) + return "".join(blocks) or "

    没有可展示的执行样本。

    " + + +def _execution_explorer( + manifest: dict[str, Any], summary: dict[str, Any], detail_pages: dict[str, str] +) -> str: + physical_attempts = sum( + (values or {}).get("physical_attempts", 0) + for values in (summary.get("functional") or {}).values() + ) + logical_executions = sum( + 1 + for task in (summary.get("by_task") or {}).values() + for attempts in ((task.get("attempts") or task.get("failures") or {}).values()) + if attempts + ) + task_options = "".join( + f'' + for task_id, task in (summary.get("by_task") or {}).items() + ) + action_values = sorted( + { + str(c.get("action")) + for task in (summary.get("by_task") or {}).values() + for attempts in (task.get("attempts") or {}).values() + for attempt in attempts + for c in attempt.get("trace") or [] + if c.get("action") != "validate_assertions" + } + ) + action_options = "".join( + f'' for action in action_values + ) + tag_values = sorted( + { + str(tag) + for task in (summary.get("by_task") or {}).values() + for tag in task.get("tags") or [] + } + ) + tag_options = "".join(f'' for tag in tag_values) + diagnostic_categories = sorted( + { + str(group.get("category") or "其他") + for group in ( + ((summary.get("failure_diagnostics") or {}).get("rule_engine") or {}).get("groups") + or [] + ) + } + ) + diagnostic_groups = ((summary.get("failure_diagnostics") or {}).get("rule_engine") or {}).get( + "groups" + ) or [] + category_counts: dict[str, list[int]] = {} + for group in diagnostic_groups: + category = str(group.get("category") or "其他") + category_counts.setdefault(category, [0, 0]) + category_counts[category][0] += 1 + category_counts[category][1] += len(group.get("attempts") or []) + category_rows = [ + [name, str(values[0]), str(values[1])] for name, values in category_counts.items() + ] or [["无失败", "0", "0"]] + error_options = '' + "".join( + f'' + for category in diagnostic_categories + ) + filters = f"""
    + + + + + + + + + + + +
    """ + script = """""" + grid_head = '' + return f"""

    逐任务分析与原始证据

    {logical_executions} 条任务 × Provider 记录,{physical_attempts} 次执行。点击任务查看完整操作、答案、判定和执行轨迹。

    {_table(["规则类别", "去重失败组", "实际失败次数"], category_rows, "程序化失败分布")}{filters}
    {grid_head}{_execution_details(manifest, summary, detail_pages)}
    {script}
    """ + + +def _classify(error: str) -> str: + text = error.lower() + if "timeout" in text and ("visible" in text or "click" in text): + return "交互目标超时/不可见" + if "timeout" in text: + return "超时" + if "exact mismatch" in text: + return "结果不匹配" + if "net::" in text or "navigation" in text: + return "导航/网络" + if "selector" in text or "locator" in text: + return "定位器" + return "其他" + + +def _analysis_page( + manifest: dict[str, Any], summary: dict[str, Any], detail_pages: dict[str, str] +) -> str: + return f"""

    时序机制研究

    研究假设低频 screencast 或其触发的布局快照,可能放大动态 DOM 更新后几何、可见性或命中测试状态的滞后。
    对照设计固定页面、选择器和调用,比较立即显现、双 rAF、定时器、DOM 替换、动画与视口进入;延迟覆盖 16 ms 到 1 s。
    验证指标记录 DOM/rAF 完成时间与 actionability 恢复时间,检验其差值是否随 screencast 条件系统性变化。
    {_execution_explorer(manifest, summary, detail_pages)}""" + + +def _appendix_page(manifest: dict[str, Any]) -> str: + p = manifest.get("protocol") or {} + provenance = manifest.get("artifact_provenance") or {} + fields = [ + ["task", "ground_truth / source / expected_actions", "唯一结果、固定参考路径与来源"], + [ + "attempt", + "result_status / process_status / combined_status", + "结果主指标与结果+过程辅指标", + ], + ["attempt", "validation / trace / error", "同一任务的判定与完整执行证据"], + ["resource", "memory mean/p95/peak/retained/MiB·s", "内存水平、尾部、峰值、保留量和积分"], + ["resource", "CPU / network total/average/p95/peak", "计算与网络时序压力"], + ["summary", "paired intersection / paired ratios", "双方成功任务与逐任务配对比值审计"], + ["diagnostics", "rule_engine", "程序化失败分类、证据与原始执行记录入口"], + ] + provenance_rows = [ + ["运行编号", str(manifest.get("run_id") or "未记录")], + ["生成时间", str(manifest.get("created_at") or "未记录")], + ["评测集 SHA-256", str(manifest.get("suite_sha256") or "未记录")], + ["attempts.jsonl SHA-256", str(provenance.get("attempts_sha256") or "未记录")], + ["summary.json SHA-256", str(provenance.get("summary_sha256") or "未记录")], + ] + return f"""

    复现步骤

    1. 准备 Docker Engine、Python 3.11+、至少 4 CPU/6 GiB 内存。
    2. 运行 ./examples/trajectory_replay/run.sh;脚本执行完整基准,并依次生成原始记录、汇总数据和分页 HTML。
    3. 如需为失败任务补充一次同任务证据,设置 BROWSER_USE_EVAL_FAILURE_RETRIES=1;该记录不会另计为新任务。
    4. 如只需从已有 attempts.jsonl 重算全部指标和报告,运行 uv run python examples/trajectory_replay/rebuild_report.py。
    5. 检查 attempts.jsonl、manifest.json、summary.json 与分页 HTML;所有比值、分位数和均值均由仓库脚本计算,Codex 不参与统计。

    下载本轮机器可读审计摘要

    运行与原始数据绑定

    {_table(["项目", "值"], provenance_rows, "可复核来源")}

    数据字段

    {_table(["对象", "关键字段", "用途"], fields, "数据结构")}

    复现限制

    • 全部任务来自公网,页面内容、网络状态或站点策略变化可能导致数据集漂移;运行前需复核数据集 SHA 与标注证据。
    • 采样间隔 {p.get("sample_interval_ms", "?")} ms,持续时间短于采样间隔的尖峰可能遗漏。
    • 浏览器进程保持运行且任务间清理状态;分配器保留内存不等于缓存污染。

    原始环境 JSON

    展开
    {_e(json.dumps(manifest.get("environment") or {}, ensure_ascii=False, indent=2, sort_keys=True))}
    """ + + +def _markdown(manifest: dict[str, Any], summary: dict[str, Any]) -> str: + paired_engines = (summary.get("resource_comparison") or {}).get("by_engine") or {} + + def paired_duration(engine: str, stat: str) -> str: + return _stat(((paired_engines.get(engine) or {}).get("duration_ms") or {}), stat) + + lines = [ + "# Chromium 与 Moli 浏览器工具调用评测", + "", + f"运行编号:`{manifest.get('run_id', 'unknown')}`", + "", + "这是确定性的 BrowserTool 等价调用微基准;成功判定和失败分类均由程序完成。", + "", + "| Provider | 结果正确 | 结果与参考过程均正确 | P50 ms | 平均值 ms | P95 ms |", + "|---|---:|---:|---:|---:|---:|", + ] + for row in _correctness_rows(summary): + engine = row[0] + lines.append( + "| " + + " | ".join( + row + + [ + paired_duration(engine, "p50"), + paired_duration(engine, "mean"), + paired_duration(engine, "p95"), + ] + ) + + " |" + ) + lines.extend( + [ + "", + "```bash", + "BROWSER_USE_EVAL_FAILURE_RETRIES=1 ./examples/trajectory_replay/run.sh", + "```", + "", + ] + ) + return "\n".join(lines) + + +def write_latest_report( + reports_dir: str | Path, *, manifest: dict[str, Any], summary: dict[str, Any] +) -> Path: + root = Path(reports_dir) + root.mkdir(parents=True, exist_ok=True) + audit_name = f"{REPORT_STEM}-data.json" + detail_pages: dict[str, str] = {} + analysis = _analysis_page(manifest, summary, detail_pages) + keep = {f"{REPORT_STEM}.md", audit_name, *(filename for _, _, filename in PAGES), *detail_pages} + for pattern in ( + "trajectory-replay-evaluation-*.md", + "trajectory-replay-evaluation-*.html", + "trajectory-replay-evaluation-*.json", + ): + for old in root.glob(pattern): + if old.name not in keep: + old.unlink() + md_path = root / f"{REPORT_STEM}.md" + md_path.write_text(_markdown(manifest, summary), encoding="utf-8") + audit = { + "schema": "lex_browser_use_report_audit/1", + "run_id": manifest.get("run_id"), + "created_at": manifest.get("created_at"), + "suite_sha256": manifest.get("suite_sha256"), + "artifact_provenance": manifest.get("artifact_provenance") or {}, + "ground_truth_validation": { + key: value + for key, value in (manifest.get("ground_truth_validation") or {}).items() + if key != "human_review" + }, + "functional": summary.get("functional") or {}, + "resource_comparison": summary.get("resource_comparison") or {}, + } + (root / audit_name).write_text( + json.dumps(audit, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" + ) + renderers = { + "index": lambda: _summary_page(manifest, summary), + "suite": lambda: _suite_page(manifest), + "method": lambda: _method_page(manifest), + "analysis": lambda: analysis, + "appendix": lambda: _appendix_page(manifest), + } + for slug, _, filename in PAGES: + (root / filename).write_text(_shell(manifest, slug, renderers[slug]()), encoding="utf-8") + for filename, document in detail_pages.items(): + (root / filename).write_text(document, encoding="utf-8") + return md_path diff --git a/harness/trajectory_replay/runner.py b/harness/trajectory_replay/runner.py new file mode 100644 index 000000000..5c58b7248 --- /dev/null +++ b/harness/trajectory_replay/runner.py @@ -0,0 +1,634 @@ +"""Docker-isolated browser trajectory replay runner.""" + +from __future__ import annotations + +import hashlib +import json +import platform +import threading +import time +import traceback +import urllib.parse +import urllib.request +import uuid +from functools import partial +from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Any + +import docker +import tiktoken +from playwright.sync_api import Browser, Page, sync_playwright + +from .diagnostics import diagnose_summary +from .executor import execute_steps, judge_outputs, validate_outputs +from .metrics import ( + DockerStatsResources, + ResourceSampler, + delta_counters, + summarize_attempts, +) +from .report import write_latest_report +from .suite import BrowserTask, load_suite, validate_ground_truth_evidence + +ENGINE_IMAGES = { + "chromium": "lex-browser-use-eval-chromium:local", + "moli": "lex-browser-use-eval-moli:local", +} +ENGINE_PORTS = {"chromium": 9223, "moli": 9222} +ENCODING = tiktoken.get_encoding("o200k_base") + + +def _external_websocket_url(endpoint: str, advertised: str) -> str: + endpoint_parts = urllib.parse.urlsplit(endpoint) + websocket_parts = urllib.parse.urlsplit(advertised) + scheme = "wss" if endpoint_parts.scheme == "https" else "ws" + return urllib.parse.urlunsplit( + (scheme, endpoint_parts.netloc, websocket_parts.path, websocket_parts.query, "") + ) + + +def _wait_cdp(url: str, timeout_seconds: float = 60, *, container: Any = None) -> str: + deadline = time.monotonic() + timeout_seconds + last_error: Exception | None = None + while time.monotonic() < deadline: + try: + with urllib.request.urlopen(f"{url}/json/version", timeout=2) as response: + if response.status == 200: + payload = json.load(response) + advertised = str(payload.get("webSocketDebuggerUrl") or "") + if advertised: + return _external_websocket_url(url, advertised) + raise RuntimeError("CDP version response omitted webSocketDebuggerUrl") + except Exception as exc: + last_error = exc + time.sleep(0.25) + logs = "" + if container is not None: + try: + container.reload() + logs = container.logs(tail=30).decode("utf-8", errors="replace") + except Exception: + pass + detail = f"; container logs:\n{logs}" if logs else "" + raise RuntimeError(f"CDP endpoint did not become ready: {last_error}{detail}") + + +def _http_origin(url: str) -> str | None: + parts = urllib.parse.urlsplit(url) + if parts.scheme not in {"http", "https"} or not parts.netloc: + return None + return f"{parts.scheme}://{parts.netloc}" + + +def _reset_state(page: Page, visited_origins: set[str]) -> None: + context = page.context + origins = set(visited_origins) + current_origin = _http_origin(page.url) + if current_origin: + origins.add(current_origin) + try: + page.evaluate( + """async () => { + try { localStorage.clear(); } catch (_) {} + try { sessionStorage.clear(); } catch (_) {} + try { for (const key of await caches.keys()) await caches.delete(key); } catch (_) {} + try { + for (const item of await navigator.serviceWorker.getRegistrations()) + await item.unregister(); + } catch (_) {} + try { + for (const item of await indexedDB.databases()) + indexedDB.deleteDatabase(item.name); + } catch (_) {} + }""" + ) + except Exception: + pass + context.clear_cookies() + context.clear_permissions() + try: + cdp = context.new_cdp_session(page) + cdp.send("Network.enable") + cdp.send("Network.clearBrowserCache") + cdp.send("Network.clearBrowserCookies") + for origin in sorted(origins): + try: + cdp.send( + "Storage.clearDataForOrigin", + {"origin": origin, "storageTypes": "all"}, + ) + except Exception: + pass + page.goto("about:blank", wait_until="load", timeout=10_000) + try: + cdp.send("Page.resetNavigationHistory") + try: + cdp.send("HeapProfiler.enable") + cdp.send("HeapProfiler.collectGarbage") + cdp.send("HeapProfiler.disable") + except Exception: + pass + finally: + cdp.detach() + except Exception: + page.goto("about:blank", wait_until="load", timeout=10_000) + + +def _resource_result( + source: DockerStatsResources, + sampler: ResourceSampler, + before: Any, + after: Any, + duration_ms: float, +) -> dict[str, int | float]: + result = {**delta_counters(before, after), **sampler.summary()} + if duration_ms > 0: + result["cpu_average_utilization_pct"] = result["cpu_ms"] / duration_ms * 100 + result["network_rx_bytes_per_second"] = result["network_rx_bytes"] / duration_ms * 1000 + result["network_tx_bytes_per_second"] = result["network_tx_bytes"] / duration_ms * 1000 + return result + + +def _published_endpoint(container: Any, *, engine: str, cdp_port: int) -> str: + ports = (container.attrs.get("NetworkSettings") or {}).get("Ports") or {} + bindings = ports.get(f"{cdp_port}/tcp") + if not bindings or not bindings[0].get("HostPort"): + raise RuntimeError(f"{engine}: Docker did not publish CDP port") + return f"http://127.0.0.1:{bindings[0]['HostPort']}" + + +def _prepare_page(page: Page, visited_origins: set[str], *, settle_ms: int) -> Page: + """Reset task state and replace only the renderer page, keeping the browser/context warm.""" + context = page.context + _reset_state(page, visited_origins) + visited_origins.clear() + page.close(run_before_unload=False) + replacement = context.new_page() + replacement.set_viewport_size({"width": 1280, "height": 900}) + viewport = replacement.evaluate("() => ({width: innerWidth, height: innerHeight})") + if viewport != {"width": 1280, "height": 900}: + raise RuntimeError(f"browser viewport mismatch: {viewport}") + replacement.wait_for_timeout(max(0, settle_ms)) + return replacement + + +def _run_task( + engine: str, + page: Page, + visited_origins: set[str], + resources: DockerStatsResources, + task: BrowserTask, + attempt: int, + sample_interval_ms: int, + *, + is_retry: bool = False, +) -> dict[str, Any]: + def record_origin(frame: Any) -> None: + if frame != page.main_frame: + return + origin = _http_origin(str(frame.url)) + if origin: + visited_origins.add(origin) + + page.on("framenavigated", record_origin) + before = resources.counters() + sampler = ResourceSampler(resources, interval_ms=sample_interval_ms) + sampler.start() + started = time.perf_counter() + outputs: dict[str, str] = {} + trace: list[dict[str, Any]] = [] + validation: list[dict[str, Any]] = [] + error = "" + try: + execute_steps(page, task.tool_calls, trace=trace, outputs=outputs) + validation = judge_outputs(outputs, task.assertions) + failures = validate_outputs(outputs, task.assertions) + status = "pass" if not failures else "fail" + error = "; ".join(failures) + trace.append( + { + "index": len(task.tool_calls) + 1, + "action": "validate_assertions", + "arguments": {"assertions": list(task.assertions)}, + "status": status, + "duration_ms": 0.0, + "output": "all assertions passed" if not failures else error, + "checks": validation, + } + ) + except Exception: + status = "fail" + error = traceback.format_exc() + validation = judge_outputs(outputs, task.assertions) + finally: + duration_ms = (time.perf_counter() - started) * 1000 + sampler.stop() + page.remove_listener("framenavigated", record_origin) + after = resources.counters() + expected_actions = list(task.ground_truth.get("expected_actions") or []) + tool_trace = [item for item in trace if item.get("action") != "validate_assertions"] + actual_actions = [str(item.get("action") or "") for item in tool_trace] + process_passed = actual_actions == expected_actions and all( + item.get("status") == "pass" for item in tool_trace + ) + result_passed = bool(validation) and all(item.get("passed") is True for item in validation) + return { + "engine": engine, + "task_id": task.task_id, + "title": task.title, + "complexity": task.complexity, + "tags": list(task.tags), + "step_count": task.step_count, + "attempt": attempt, + "is_retry": is_retry, + "status": status, + "result_status": "pass" if result_passed else "fail", + "process_status": "pass" if process_passed else "fail", + "combined_status": "pass" if result_passed and process_passed else "fail", + "duration_ms": round(duration_ms, 3), + "tool_output_tokens": sum(len(ENCODING.encode(value)) for value in outputs.values()), + "output_chars": sum(len(value) for value in outputs.values()), + "error": error, + "trace": trace, + "validation": validation, + "process_validation": { + "expected_actions": expected_actions, + "actual_actions": actual_actions, + "all_calls_passed": all(item.get("status") == "pass" for item in tool_trace), + "passed": process_passed, + }, + "resource": _resource_result(resources, sampler, before, after, duration_ms), + } + + +def _start_browser_container( + client: Any, + *, + engine: str, + run_id: str, + cpus: float, + memory: str, + network: str, +) -> Any: + name = f"lex-browser-use-eval-{run_id}-{engine}" + cdp_port = ENGINE_PORTS[engine] + return client.containers.run( + ENGINE_IMAGES[engine], + name=name, + detach=True, + ports={f"{cdp_port}/tcp": ("127.0.0.1", None)}, + labels={"lex.browser-use-eval.run": run_id, "lex.browser-use-eval.engine": engine}, + nano_cpus=int(cpus * 1_000_000_000), + mem_limit=memory, + pids_limit=512, + shm_size="512m", + security_opt=["no-new-privileges"], + network=network, + extra_hosts={"host.docker.internal": "host-gateway"}, + ) + + +class _QuietFixtureHandler(SimpleHTTPRequestHandler): + def log_message(self, format: str, *args: Any) -> None: + return + + +def _start_fixture_server(root: Path) -> tuple[ThreadingHTTPServer, threading.Thread, str]: + handler = partial(_QuietFixtureHandler, directory=str(root)) + server = ThreadingHTTPServer(("0.0.0.0", 0), handler) + thread = threading.Thread( + target=server.serve_forever, + name="browser-eval-fixtures", + daemon=True, + ) + thread.start() + origin = f"http://host.docker.internal:{server.server_port}" + return server, thread, origin + + +def _materialize_task(task: BrowserTask, fixture_origin: str) -> BrowserTask: + calls = [] + for call in task.tool_calls: + item = dict(call) + for key, value in tuple(item.items()): + if isinstance(value, str): + item[key] = value.replace("{fixture_origin}", fixture_origin) + calls.append(item) + return BrowserTask( + task_id=task.task_id, + title=task.title, + tool_calls=tuple(calls), + assertions=task.assertions, + tags=task.tags, + difficulty=task.difficulty, + ground_truth=task.ground_truth, + source=task.source, + ) + + +def _environment(client: Any, containers: dict[str, Any]) -> dict[str, Any]: + info = client.info() + version_commands = {"chromium": ["chromium", "--version"], "moli": ["moli", "--version"]} + browser_versions: dict[str, str] = {} + for engine, container in containers.items(): + try: + result = container.exec_run(version_commands[engine]) + output = result.output.decode("utf-8", errors="replace").strip() + browser_versions[engine] = output or f"exit {result.exit_code}, no output" + except Exception as exc: + browser_versions[engine] = f"unavailable: {type(exc).__name__}" + return { + "runner_platform": platform.platform(), + "docker_server_version": client.version().get("Version"), + "docker_os": info.get("OperatingSystem"), + "docker_architecture": info.get("Architecture"), + "docker_cpus": info.get("NCPU"), + "docker_memory_bytes": info.get("MemTotal"), + "cgroup_version": info.get("CgroupVersion"), + "browser_versions": browser_versions, + "images": { + engine: { + "id": container.image.id, + "tags": container.image.tags, + } + for engine, container in containers.items() + }, + } + + +def _cleanup_runtime( + browsers: dict[str, Browser], + playwright: Any, + containers: dict[str, Any], + networks: dict[str, Any], +) -> None: + for browser in browsers.values(): + try: + browser.close() + except Exception: + pass + try: + playwright.stop() + except Exception: + pass + for container in containers.values(): + try: + container.remove(force=True) + except Exception: + pass + for network in networks.values(): + try: + network.remove() + except Exception: + pass + + +def _write_run_artifacts( + output: Path, rows: list[dict[str, Any]], summary: dict[str, Any], manifest: dict[str, Any] +) -> None: + attempts_bytes = "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows).encode( + "utf-8" + ) + summary_bytes = (json.dumps(summary, indent=2, ensure_ascii=False) + "\n").encode("utf-8") + (output / "attempts.jsonl").write_bytes(attempts_bytes) + (output / "summary.json").write_bytes(summary_bytes) + manifest["artifact_provenance"] = { + "artifact_directory": str(output), + "attempts_sha256": hashlib.sha256(attempts_bytes).hexdigest(), + "summary_sha256": hashlib.sha256(summary_bytes).hexdigest(), + } + (output / "manifest.json").write_text( + json.dumps(manifest, indent=2, ensure_ascii=False) + "\n", encoding="utf-8" + ) + + +def run_evaluation( + *, + suite_path: str | Path, + output_dir: str | Path, + reports_dir: str | Path, + ground_truth_evidence_path: str | Path | None = None, + max_retries: int = 1, + sample_interval_ms: int = 50, + settle_ms: int = 1500, + cpus: float = 2.0, + memory: str = "2g", + llm_diagnostics: bool = False, + diagnosis_model: str = "gpt-5.6-sol", + diagnosis_reasoning_effort: str = "low", +) -> dict[str, Any]: + if max_retries < 0: + raise ValueError("max_retries must be non-negative") + suite = load_suite(suite_path) + ground_truth_evidence = ( + validate_ground_truth_evidence(suite_path, suite, ground_truth_evidence_path) + if ground_truth_evidence_path is not None + else None + ) + output = Path(output_dir) + output.mkdir(parents=True, exist_ok=True) + run_id = uuid.uuid4().hex[:12] + client = docker.from_env() + containers: dict[str, Any] = {} + networks: dict[str, Any] = {} + browsers: dict[str, Browser] = {} + pages: dict[str, Page] = {} + visited_origins: dict[str, set[str]] = {} + resources: dict[str, DockerStatsResources] = {} + rows: list[dict[str, Any]] = [] + playwright = sync_playwright().start() + uses_local_fixtures = any( + "{fixture_origin}" in str(value) + for task in suite.tasks + for call in task.tool_calls + for value in call.values() + ) + fixture_runtime: tuple[ThreadingHTTPServer, threading.Thread, str] | None = None + if uses_local_fixtures: + fixture_runtime = _start_fixture_server(Path(suite_path).resolve().parent / "fixtures") + fixture_origin = fixture_runtime[2] if fixture_runtime else "" + try: + for engine in ENGINE_IMAGES: + network = client.networks.create( + f"lex-browser-use-eval-{run_id}-{engine}", + driver="bridge", + labels={ + "lex.browser-use-eval.run": run_id, + "lex.browser-use-eval.engine": engine, + }, + ) + networks[engine] = network + container = _start_browser_container( + client, + engine=engine, + run_id=run_id, + cpus=cpus, + memory=memory, + network=network.name, + ) + containers[engine] = container + container.reload() + cdp_port = ENGINE_PORTS[engine] + endpoint = _published_endpoint(container, engine=engine, cdp_port=cdp_port) + websocket_url = _wait_cdp(endpoint, container=container) + browsers[engine] = playwright.chromium.connect_over_cdp(websocket_url) + resources[engine] = DockerStatsResources(container) + page = browsers[engine].contexts[0].pages[0] + page.goto("https://example.com/", wait_until="domcontentloaded", timeout=30_000) + _reset_state(page, {"https://example.com"}) + pages[engine] = page + visited_origins[engine] = set() + + for task_index, declared_task in enumerate(suite.tasks): + task = _materialize_task(declared_task, fixture_origin) + engine_order = ("chromium", "moli") if task_index % 2 == 0 else ("moli", "chromium") + for engine in engine_order: + for attempt in range(1, max_retries + 2): + pages[engine] = _prepare_page( + pages[engine], + visited_origins[engine], + settle_ms=settle_ms, + ) + row = _run_task( + engine, + pages[engine], + visited_origins[engine], + resources[engine], + task, + attempt, + sample_interval_ms, + is_retry=attempt > 1, + ) + row["run_id"] = run_id + rows.append(row) + print( + json.dumps( + { + "engine": row["engine"], + "task_id": row["task_id"], + "attempt": row["attempt"], + "status": row["status"], + "duration_ms": row["duration_ms"], + "error": next( + ( + line + for line in reversed(row["error"].splitlines()) + if line.strip() + ), + "", + ), + }, + ensure_ascii=False, + ), + flush=True, + ) + if row["status"] == "pass": + break + summary = summarize_attempts(rows, engines=tuple(ENGINE_IMAGES)) + manifest = { + "schema": "lex_browser_use_manifest/2", + "run_id": run_id, + "created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "suite_sha256": hashlib.sha256(Path(suite_path).read_bytes()).hexdigest(), + "suite": [ + { + "task_id": task.task_id, + "title": task.title, + "step_count": task.step_count, + "complexity": task.complexity, + "tags": list(task.tags), + "tool_calls": list(task.tool_calls), + "assertions": list(task.assertions), + "ground_truth": task.ground_truth, + "source": task.source, + } + for task in suite.tasks + ], + "ground_truth_validation": ( + { + key: ground_truth_evidence.get(key) + for key in ( + "schema", + "dataset", + "dataset_sha256", + "source_commit", + "validator", + "method", + "started_at", + "finished_at", + "automatic_browser_replay", + "human_review", + ) + } + if ground_truth_evidence is not None + else None + ), + "protocol": { + "browser_reuse": True, + "viewport": {"width": 1280, "height": 900}, + "viewport_verified_before_each_task": True, + "step_definition": ( + "one declared CowAgent BrowserTool-equivalent call; implicit waits excluded" + ), + "llm_scope": ("LLM never judges success; optional LLM only explains failed traces"), + "llm_failure_diagnostics": llm_diagnostics, + "diagnosis_model": diagnosis_model if llm_diagnostics else None, + "diagnosis_reasoning_effort": ( + diagnosis_reasoning_effort if llm_diagnostics else None + ), + "cold_start_excluded": True, + "state_reset_between_tasks": True, + "state_reset_method": ( + "warm browser/context; clear all top-level navigation origins, " + "browser cache, cookies, permissions and history; replace renderer page" + ), + "sample_interval_ms": sample_interval_ms, + "post_reset_settle_ms": settle_ms, + "primary_attempts_per_task": 1, + "max_failure_retries": max_retries, + "engine_order": "balanced by task index", + "target_mode": "public_web" if not uses_local_fixtures else "mixed", + "fixture_origin": ( + "temporary self-hosted deterministic fixture server" + if uses_local_fixtures + else None + ), + "resource_scope": "browser container cgroup/network namespace", + "controller_excluded": True, + "cpus_per_browser": cpus, + "memory_per_browser": memory, + "network_isolation": "one dedicated bridge with no peer containers per engine", + }, + "environment": _environment(client, containers), + } + smoke_path = Path(suite_path).resolve().parent / "agent_smoke.json" + if smoke_path.exists(): + smoke_bytes = smoke_path.read_bytes() + manifest["agent_smoke"] = { + **json.loads(smoke_bytes.decode("utf-8")), + "provenance": { + "source_file": smoke_path.name, + "source_sha256": hashlib.sha256(smoke_bytes).hexdigest(), + "independent_of_browser_microbenchmark": True, + "reused_not_rerun": True, + }, + } + # Persist core evidence before optional remote diagnostics can fail or stall. + _write_run_artifacts(output, rows, summary, manifest) + diagnose_summary( + summary, + use_llm=llm_diagnostics, + model=diagnosis_model, + reasoning_effort=diagnosis_reasoning_effort, + ) + _write_run_artifacts(output, rows, summary, manifest) + write_latest_report(reports_dir, manifest=manifest, summary=summary) + return summary + finally: + if fixture_runtime: + fixture_server, fixture_thread, _ = fixture_runtime + fixture_server.shutdown() + fixture_server.server_close() + fixture_thread.join(timeout=2) + _cleanup_runtime(browsers, playwright, containers, networks) diff --git a/harness/browser_use/suite.py b/harness/trajectory_replay/suite.py similarity index 99% rename from harness/browser_use/suite.py rename to harness/trajectory_replay/suite.py index f61fe6fb4..8afe17663 100644 --- a/harness/browser_use/suite.py +++ b/harness/trajectory_replay/suite.py @@ -1,4 +1,4 @@ -"""Task-suite schema and validation for browser use evaluation.""" +"""Task-suite schema and validation for browser trajectory replay.""" from __future__ import annotations diff --git a/harness/trajectory_replay/webarena_corrections.py b/harness/trajectory_replay/webarena_corrections.py new file mode 100644 index 000000000..208c97b6c --- /dev/null +++ b/harness/trajectory_replay/webarena_corrections.py @@ -0,0 +1,353 @@ +"""Traceable WebArena contract corrections used only by trajectory replay. + +The official task objects are immutable and remain the scoring authority for +historical Browser Use runs. This module copies a task, applies a small JSON +Patch subset, and returns replay-only assertions that the official evaluator +model cannot express (currently final DOM state). +""" + +from __future__ import annotations + +import copy +import hashlib +import json +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path +from types import MappingProxyType +from typing import Any + +from webarena_verified.core.evaluation.evaluators import EVALUATOR_REGISTRY +from webarena_verified.core.utils.checksum import compute_data_file_checksum +from webarena_verified.types.eval import TaskEvalContext, TaskEvalResult +from webarena_verified.types.task import WebArenaVerifiedTask + +SCHEMA = "lex_webarena_replay_task_corrections/1" +DEFAULT_CORRECTIONS_PATH = ( + Path(__file__).resolve().parents[2] + / "evaluation" + / "trajectory-replay" + / "webarena-verified" + / "task-corrections.json" +) + + +def _canonical_json(value: Any) -> bytes: + return json.dumps( + value, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + + +def _digest(value: Any) -> str: + return hashlib.sha256(_canonical_json(value)).hexdigest() + + +def _decode_pointer_token(value: str) -> str: + return value.replace("~1", "/").replace("~0", "~") + + +def _pointer_parts(pointer: str) -> list[str]: + if pointer == "": + return [] + if not pointer.startswith("/"): + raise ValueError(f"invalid JSON pointer: {pointer!r}") + return [_decode_pointer_token(item) for item in pointer[1:].split("/")] + + +def _resolve(document: Any, pointer: str) -> Any: + current = document + for part in _pointer_parts(pointer): + if isinstance(current, list): + current = current[int(part)] + elif isinstance(current, dict): + current = current[part] + else: + raise ValueError(f"JSON pointer crosses a scalar: {pointer!r}") + return current + + +def _parent(document: Any, pointer: str) -> tuple[Any, str]: + parts = _pointer_parts(pointer) + if not parts: + raise ValueError("root replacement is not supported") + parent_pointer = "/" + "/".join( + item.replace("~", "~0").replace("/", "~1") for item in parts[:-1] + ) + return (_resolve(document, parent_pointer) if parts[:-1] else document, parts[-1]) + + +def _apply_operation(document: dict[str, Any], operation: Mapping[str, Any]) -> None: + kind = operation.get("op") + pointer = operation.get("path") + if kind not in {"add", "remove", "replace"} or not isinstance(pointer, str): + raise ValueError(f"unsupported correction operation: {operation!r}") + parent, key = _parent(document, pointer) + if isinstance(parent, list): + index = int(key) + if kind == "add": + parent.insert(index, copy.deepcopy(operation.get("value"))) + elif kind == "remove": + parent.pop(index) + else: + parent[index] = copy.deepcopy(operation.get("value")) + return + if not isinstance(parent, dict): + raise ValueError(f"operation parent is not a container: {pointer!r}") + if kind == "remove": + del parent[key] + elif kind == "replace": + if key not in parent: + raise ValueError(f"replace target does not exist: {pointer!r}") + parent[key] = copy.deepcopy(operation.get("value")) + else: + parent[key] = copy.deepcopy(operation.get("value")) + + +@dataclass(frozen=True) +class CorrectionCatalog: + """Validated correction records and their independent identities.""" + + records: Mapping[int, Mapping[str, Any]] + record_sha256s: Mapping[int, str] + upstream: Mapping[str, Any] + source_path: Path + source_sha256: str + + +@dataclass(frozen=True) +class CorrectedReplayContract: + """A corrected task and replay-only checks, with before/after identities.""" + + task: WebArenaVerifiedTask + correction_id: str + correction_source_sha256: str + original_task_sha256: str + corrected_task_sha256: str + rationale: str + replay_assertions: tuple[Mapping[str, Any], ...] + + +def load_corrections(path: str | Path = DEFAULT_CORRECTIONS_PATH) -> CorrectionCatalog: + """Load and validate the replay-only correction catalog.""" + + source_path = Path(path).expanduser().resolve() + raw_bytes = source_path.read_bytes() + payload = json.loads(raw_bytes) + if payload.get("schema") != SCHEMA: + raise ValueError(f"unsupported WebArena correction schema: {payload.get('schema')!r}") + upstream = payload.get("upstream") + if not isinstance(upstream, dict) or not all( + upstream.get(key) for key in ("benchmark", "package_version", "dataset_sha256") + ): + raise ValueError("correction catalog must identify its upstream benchmark and dataset") + records: dict[int, Mapping[str, Any]] = {} + record_sha256s: dict[int, str] = {} + for record in payload.get("corrections") or []: + task_id = int(record["task_id"]) + if task_id in records: + raise ValueError(f"duplicate WebArena correction for task {task_id}") + if not record.get("correction_id") or not record.get("rationale"): + raise ValueError(f"task {task_id}: correction id and rationale are required") + if int(record.get("official_revision") or 0) < 1: + raise ValueError(f"task {task_id}: official_revision must be positive") + if not record.get("preconditions") or not record.get("operations"): + raise ValueError(f"task {task_id}: preconditions and operations are required") + records[task_id] = MappingProxyType(copy.deepcopy(record)) + record_sha256s[task_id] = _digest(record) + return CorrectionCatalog( + records=MappingProxyType(records), + record_sha256s=MappingProxyType(record_sha256s), + upstream=MappingProxyType(copy.deepcopy(upstream)), + source_path=source_path, + source_sha256=hashlib.sha256(raw_bytes).hexdigest(), + ) + + +def apply_correction( + task: WebArenaVerifiedTask, + catalog: CorrectionCatalog, +) -> CorrectedReplayContract: + """Return a corrected copy of ``task`` without mutating the official object.""" + + record = catalog.records.get(task.task_id) + if record is None: + raise KeyError(f"no replay correction registered for task {task.task_id}") + if task.revision != int(record["official_revision"]): + raise ValueError( + f"task {task.task_id}: expected official revision " + f"{record['official_revision']}, got {task.revision}" + ) + + original = task.model_dump(mode="json") + corrected = copy.deepcopy(original) + for precondition in record["preconditions"]: + actual = _resolve(original, str(precondition["path"])) + expected = precondition.get("equals") + if actual != expected: + raise ValueError( + f"task {task.task_id}: correction precondition failed at " + f"{precondition['path']}: expected {expected!r}, got {actual!r}" + ) + for operation in record["operations"]: + _apply_operation(corrected, operation) + + corrected_task = type(task).model_validate(corrected) + assertions = tuple( + MappingProxyType(copy.deepcopy(item)) for item in record.get("replay_assertions") or [] + ) + return CorrectedReplayContract( + task=corrected_task, + correction_id=str(record["correction_id"]), + correction_source_sha256=catalog.record_sha256s[task.task_id], + original_task_sha256=_digest(original), + corrected_task_sha256=_digest(corrected_task.model_dump(mode="json")), + rationale=str(record["rationale"]), + replay_assertions=assertions, + ) + + +def apply_registered_correction( + task: WebArenaVerifiedTask, + *, + path: str | Path = DEFAULT_CORRECTIONS_PATH, +) -> CorrectedReplayContract: + """Convenience wrapper for applying the repository correction catalog.""" + + return apply_correction(task, load_corrections(path)) + + +def corrected_task_ids(catalog: CorrectionCatalog) -> tuple[int, ...]: + """Return the stable sorted correction scope.""" + + return tuple(sorted(catalog.records)) + + +def normalize_agent_response_for_evaluator(parsed_response: Any) -> Any: + """Return the exact response shape accepted by the 1.2.3 evaluator.""" + if not isinstance(parsed_response, dict): + return parsed_response + return { + key: ("" if value is None and key != "retrieved_data" else value) + for key, value in parsed_response.items() + if key in {"task_type", "status", "retrieved_data"} + } + + +def evaluate_corrected_contract( + benchmark: Any, + contract: CorrectedReplayContract, + *, + agent_response: Any, + network_trace: str | Path, +) -> TaskEvalResult: + """Evaluate a replay-only corrected task without mutating the benchmark reader. + + Task 676 exposes an upstream evaluator bug: navigate tasks always select the + last document navigation before considering the expected XHR URL. A replay + assertion explicitly opts into ordinary URL/method/query filtering; only + that network evaluator receives a non-navigate view of the same task. + """ + task = contract.task + parsed_response = benchmark._evaluator._parse_agent_response(agent_response) + # webarena-verified 1.2.3 calls ``.strip()`` on scalar response fields. + # Its documented dict format commonly serializes absent fields as JSON null, + # so normalize only those top-level scalars while preserving retrieved_data. + parsed_response = normalize_agent_response_for_evaluator(parsed_response) + trace = benchmark._evaluator._parse_network_trace(network_trace) + context = TaskEvalContext( + task=task, + agent_response_raw=parsed_response, + network_trace=trace, + config=benchmark.config, + ) + ordinary_network_filtering = any( + item.get("kind") == "network_event_selection" + and item.get("policy") == "match_expected_url_method_and_query_before_recency" + for item in contract.replay_assertions + ) + evaluator_results = [] + for config in task.eval: + evaluator_context = context + if ordinary_network_filtering and config.evaluator == "NetworkEventEvaluator": + task_data = task.model_dump(mode="json") + evaluations = task_data.get("eval") or [] + if not evaluations or not isinstance(evaluations[0].get("expected"), dict): + raise ValueError( + "ordinary network filtering requires eval[0].expected" + ) + # webarena-verified 1.2.3 reads the task type used by + # NetworkEventEvaluator from eval[0].expected, regardless of the + # network evaluator's own position in the task configuration. + evaluations[0]["expected"]["task_type"] = "MUTATE" + filtering_task = type(task).model_validate(task_data) + evaluator_context = context.model_copy(update={"task": filtering_task}) + evaluator_results.append( + EVALUATOR_REGISTRY[config.evaluator]().evaluate( + context=evaluator_context, + config=config, + ) + ) + return TaskEvalResult.create( + task_id=task.task_id, + intent_template_id=task.intent_template_id, + sites=task.sites, + task_revision=task.revision, + evaluators_results=evaluator_results, + data_checksum=compute_data_file_checksum(benchmark.config.test_data_file), + ) + + +def check_replay_assertions( + contract: CorrectedReplayContract, + *, + page_url: str, + page_text: str, + field_values: Mapping[str, Any], +) -> tuple[dict[str, Any], ...]: + """Check replay-only final-state assertions captured from the browser page.""" + results: list[dict[str, Any]] = [] + for assertion in contract.replay_assertions: + kind = str(assertion.get("kind") or "") + if kind == "network_event_selection": + results.append({"kind": kind, "status": "pass", "detail": assertion["target"]}) + continue + if kind == "final_dom_form_state": + expected = dict(assertion.get("fields") or {}) + actual = {selector: field_values.get(selector) for selector in expected} + passed = actual == expected and "/contact" in page_url + results.append( + { + "kind": kind, + "status": "pass" if passed else "fail", + "expected": expected, + "actual": actual, + } + ) + continue + if kind == "gitlab_project_state": + project_path = str(assertion["project_path"]) + passed = project_path in page_url and ( + "repository for this project is empty" in page_text.lower() + or "empty repository" in page_text.lower() + ) + results.append( + {"kind": kind, "status": "pass" if passed else "fail", "project_path": project_path} + ) + continue + if kind == "magento_configurable_variation": + required = assertion.get("required_attributes") or {} + values = (required.get("size"), required.get("color")) + passed = all(str(value).lower() in page_text.lower() for value in values) + results.append( + { + "kind": kind, + "status": "pass" if passed else "fail", + "required_attributes": dict(required), + } + ) + continue + raise ValueError(f"unsupported replay assertion kind: {kind!r}") + return tuple(results) diff --git a/harness/trajectory_replay/webarena_dataset.py b/harness/trajectory_replay/webarena_dataset.py new file mode 100644 index 000000000..8e6e82b1f --- /dev/null +++ b/harness/trajectory_replay/webarena_dataset.py @@ -0,0 +1,567 @@ +"""Promote replay-proven Browser Use trajectories into the shared dataset.""" + +from __future__ import annotations + +import hashlib +import json +from collections import defaultdict +from functools import lru_cache +from importlib.metadata import version +from importlib.resources import files +from pathlib import Path +from typing import Any + +from webarena_verified.core.utils.checksum import compute_evaluator_checksum + +from harness.trajectory_replay.webarena_plan import ( + load_replay_plan, + replay_calls_binding_sha256, + replay_site_hosts, +) + +SCHEMA = "lex_webarena_verified_trajectory_dataset/1" +RECEIPT_SCHEMA = "lex_webarena_trajectory_replay_suite/1" + + +@lru_cache(maxsize=1) +def _official_evaluator_names() -> dict[int, tuple[str, ...]]: + """Return evaluator sequences after applying registered replay corrections.""" + + dataset = json.loads( + files("webarena_verified") + .joinpath("assets/dataset/webarena-verified.json") + .read_text(encoding="utf-8") + ) + names = { + int(task["task_id"]): tuple(str(item["evaluator"]) for item in task["eval"]) + for task in dataset + } + for task_id, binding in _effective_correction_bindings().items(): + names[task_id] = binding[3] + return names + + +@lru_cache(maxsize=1) +def _effective_correction_bindings() -> dict[ + int, tuple[str, str, str, tuple[str, ...]] +]: + from harness.browser_use.runner import _configure_benchmark # noqa: PLC0415 + from harness.trajectory_replay.webarena_corrections import ( # noqa: PLC0415 + apply_correction, + load_corrections, + ) + + benchmark = _configure_benchmark() + catalog = load_corrections() + bindings = {} + for task_id in catalog.records: + contract = apply_correction(benchmark.get_task(task_id), catalog) + evaluators = tuple( + str(item["evaluator"]) + for item in contract.task.model_dump(mode="json")["eval"] + ) + bindings[task_id] = ( + contract.correction_id, + contract.correction_source_sha256, + contract.corrected_task_sha256, + evaluators, + ) + return bindings + + +def _validate_qualification_receipt( + receipt: dict[str, Any], + *, + receipt_path: Path, + seed_task_ids: set[int], + required_engines: tuple[str, ...], + expected_resources: dict[str, Any], +) -> None: + def required_int(value: Any, field: str) -> int: + try: + return int(value) + except (TypeError, ValueError) as exc: + raise ValueError( + f"replay receipt has invalid {field}: {receipt_path}" + ) from exc + + if receipt.get("schema") != RECEIPT_SCHEMA: + raise ValueError(f"unsupported replay receipt: {receipt_path}") + if ( + receipt.get("execution_scope") not in {"qualification", "qualification_shard"} + or receipt.get("status") != "complete" + ): + raise ValueError(f"replay receipt is not a completed qualification proof: {receipt_path}") + if not receipt.get("source_seed_manifest_sha256") or not receipt.get("profile_sha256"): + raise ValueError(f"replay receipt is missing source bindings: {receipt_path}") + runtime = receipt.get("runtime") or {} + if runtime.get("resources") != expected_resources: + raise ValueError(f"replay receipt has different runtime resources: {receipt_path}") + if not runtime.get("browser_eval_commit") or not runtime.get("cowagent_commit"): + raise ValueError(f"replay receipt is missing code identity: {receipt_path}") + browser_images = runtime.get("browser_images") or {} + if set(browser_images) != set(required_engines) or any( + not (browser_images.get(engine) or {}).get("id") for engine in required_engines + ): + raise ValueError(f"replay receipt is missing browser image identity: {receipt_path}") + if not runtime.get("site_images") or any( + not image.get("id") for image in runtime["site_images"].values() + ): + raise ValueError(f"replay receipt is missing site image identity: {receipt_path}") + receipt_task_ids = [ + required_int(task_id, "task_id") for task_id in receipt.get("task_ids") or [] + ] + if ( + not receipt_task_ids + or len(receipt_task_ids) != len(set(receipt_task_ids)) + or not set(receipt_task_ids) <= seed_task_ids + ): + raise ValueError(f"replay receipt has an invalid qualification task scope: {receipt_path}") + isolation = receipt.get("isolation") or {} + required_isolation = { + "environment_pool_count": 1, + "method_parallelism": 1, + "task_parallelism": 1, + "reset_before_every_task": True, + "teardown_between_methods": True, + } + if any(isolation.get(key) != value for key, value in required_isolation.items()): + raise ValueError(f"replay receipt does not prove the required isolation: {receipt_path}") + engines = [str(engine) for engine in receipt.get("engines") or []] + if len(engines) != len(set(engines)) or set(engines) != set(required_engines): + raise ValueError(f"replay receipt does not cover the required methods: {receipt_path}") + method_teardown = receipt.get("method_teardown") or {} + if set(method_teardown) != set(required_engines) or not all(method_teardown.values()): + raise ValueError(f"replay receipt does not prove completed method teardown: {receipt_path}") + sessions = receipt.get("sessions") or [] + session_ids = [str(session.get("id") or "") for session in sessions] + if ( + not sessions + or any(not session_id for session_id in session_ids) + or any(not session.get("site_access_host") for session in sessions) + or len(session_ids) != len(set(session_ids)) + ): + raise ValueError(f"replay receipt has invalid browser sessions: {receipt_path}") + sessions_by_id = {str(session["id"]): session for session in sessions} + attempt_keys = [] + for attempt in receipt.get("attempts") or []: + if attempt.get("schema") != "lex_webarena_trajectory_replay_attempt/1": + raise ValueError(f"replay receipt has an invalid attempt schema: {receipt_path}") + attempt_task_id = required_int(attempt.get("task_id"), "attempt task_id") + session = sessions_by_id.get(str(attempt.get("browser_session_id") or "")) + if ( + session is None + or session.get("engine") != attempt.get("engine") + or attempt_task_id not in session.get("task_ids", []) + ): + raise ValueError(f"replay attempt is not bound to a browser session: {receipt_path}") + attempt_keys.append((str(attempt.get("engine")), attempt_task_id)) + expected_keys = { + (engine, task_id) for engine in required_engines for task_id in receipt_task_ids + } + if len(attempt_keys) != len(set(attempt_keys)) or set(attempt_keys) != expected_keys: + raise ValueError( + f"replay receipt does not contain the complete method-task matrix: {receipt_path}" + ) + physical_executions = receipt.get("physical_executions") or [] + physical_keys = [] + for item in physical_executions: + physical_keys.append( + ( + str(item.get("engine")), + required_int(item.get("task_id"), "physical task_id"), + required_int( + item.get("physical_execution"), "physical execution number" + ), + ) + ) + if ( + not physical_executions + or any(item.get("status") != "complete" for item in physical_executions) + or len(physical_keys) != len(set(physical_keys)) + ): + raise ValueError(f"replay receipt has invalid physical execution history: {receipt_path}") + physical_by_attempt: dict[tuple[str, int], list[dict[str, Any]]] = defaultdict(list) + for item in physical_executions: + key = (str(item["engine"]), int(item["task_id"])) + if key not in expected_keys: + raise ValueError( + f"replay receipt has a physical execution outside scope: {receipt_path}" + ) + physical_by_attempt[key].append(item) + attempts_by_key = { + (str(item["engine"]), int(item["task_id"])): item + for item in receipt.get("attempts") or [] + } + if set(physical_by_attempt) != expected_keys: + raise ValueError(f"replay receipt has unbound physical executions: {receipt_path}") + for key, attempt in attempts_by_key.items(): + physical = physical_by_attempt[key] + if ( + len(physical) != int(attempt.get("physical_execution_count") or 1) + or max(int(item["physical_execution"]) for item in physical) + != int(attempt.get("physical_execution") or 0) + ): + raise ValueError( + f"replay receipt has inconsistent physical executions: {receipt_path}" + ) + + +def _validate_passing_attempt_evidence( + attempt: dict[str, Any], + candidate: dict[str, Any], + runtime_site_host: str, + expected_data_checksum: str, + trusted_source_site_hosts: set[str], +) -> None: + if attempt.get("process_status") != "pass" or attempt.get("evaluator_status") != "pass": + return + evidence = attempt.get("evidence") or {} + paths = {} + for kind in ("trace", "network_har", "evaluation"): + item = evidence.get(kind) or {} + path = Path(str(item.get("path") or "")).expanduser().resolve() + if not path.is_file() or _sha256(path) != item.get("sha256"): + raise ValueError( + f"passing replay attempt has missing or changed {kind} evidence: " + f"{attempt.get('engine')}/task-{attempt.get('task_id')}" + ) + paths[kind] = path + + if attempt.get("source_tool_calls_sha256") != _json_sha256(candidate["tool_calls"]): + raise ValueError("passing replay attempt is not bound to the canonical tool calls") + source_site_hosts = replay_site_hosts(candidate["tool_calls"]) + if ( + not source_site_hosts <= trusted_source_site_hosts + or set(attempt.get("source_site_hosts") or []) != source_site_hosts + or attempt.get("runtime_site_host") != runtime_site_host + or attempt.get("replay_calls_binding_sha256") + != replay_calls_binding_sha256( + candidate["tool_calls"], allowed_hosts=trusted_source_site_hosts + ) + ): + raise ValueError("passing replay attempt differs from the canonical replay actions") + if attempt.get("evaluation_input_sha256") != _json_sha256( + candidate["recorded_response"] + ): + raise ValueError("passing replay attempt is not bound to the canonical evaluator input") + + trace = _read_json(paths["trace"]) + trace_arguments = [item.get("arguments") for item in trace] if isinstance(trace, list) else [] + if ( + len(trace_arguments) != int(candidate["action_count"]) + or any(item.get("status") != "success" for item in trace) + or attempt.get("executed_tool_calls_sha256") != _json_sha256(trace_arguments) + or attempt.get("replay_calls_binding_sha256") + != replay_calls_binding_sha256( + trace_arguments, allowed_hosts={runtime_site_host} + ) + ): + raise ValueError("passing replay trace does not prove every executed action") + + network_har = _read_json(paths["network_har"]) + if not isinstance(network_har, dict) or not isinstance( + (network_har.get("log") or {}).get("entries"), list + ): + raise ValueError("passing replay attempt has an invalid network HAR") + + evaluation = _read_json(paths["evaluation"]) + task_id = int(attempt["task_id"]) + expected_evaluators = _official_evaluator_names().get(task_id) + correction = _effective_correction_bindings().get(task_id) + actual_correction = ( + str(attempt.get("contract_correction_id") or ""), + str(attempt.get("contract_correction_source_sha256") or ""), + str(attempt.get("corrected_task_sha256") or ""), + ) + if correction is None: + if any(actual_correction): + raise ValueError(f"task {task_id} has an unexpected replay correction binding") + elif actual_correction != correction[:3]: + raise ValueError(f"task {task_id} replay correction binding is stale or incomplete") + if ( + int(evaluation.get("task_id") or -1) != int(attempt["task_id"]) + or float(evaluation.get("score") or 0) != 1.0 + or str(evaluation.get("status") or "").lower() not in {"pass", "success"} + or evaluation.get("webarena_verified_version") != version("webarena-verified") + or evaluation.get("webarena_verified_evaluator_checksum") + != compute_evaluator_checksum() + or not expected_data_checksum + or evaluation.get("webarena_verified_data_checksum") != expected_data_checksum + ): + raise ValueError("passing replay evaluation evidence does not prove a passing score") + evaluator_rows = evaluation.get("evaluators_results") or [] + if ( + expected_evaluators is None + or tuple(str(row.get("evaluator_name") or "") for row in evaluator_rows) + != expected_evaluators + or any( + float(row.get("score") or 0) != 1.0 + or str(row.get("status") or "").lower() not in {"pass", "success"} + for row in evaluator_rows + ) + ): + actual_evaluators = tuple( + str(row.get("evaluator_name") or "") for row in evaluator_rows + ) + raise ValueError( + "passing replay evaluator evidence does not match the canonical input: " + f"task {attempt['task_id']}, expected evaluators {expected_evaluators}, " + f"actual evaluators {actual_evaluators}" + ) + if any(item.get("status") != "pass" for item in evaluation.get("replay_assertions") or []): + raise ValueError("passing replay evaluation contains a failed replay assertion") + + +def _has_stale_correction_binding(attempt: dict[str, Any], task_id: int) -> bool: + """Return whether a historical proof predates the task's current correction.""" + + correction = _effective_correction_bindings().get(task_id) + actual = ( + str(attempt.get("contract_correction_id") or ""), + str(attempt.get("contract_correction_source_sha256") or ""), + str(attempt.get("corrected_task_sha256") or ""), + ) + return correction is not None and actual != correction[:3] + + +def _read_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _json_sha256(value: Any) -> str: + payload = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(payload.encode()).hexdigest() + + +def _candidate_source(candidate: dict[str, Any]) -> dict[str, Any]: + source = candidate.get("source") or {} + if candidate.get("candidate_kind") == "expert_repair": + return { + "kind": "expert_repair_spec", + "candidate_id": candidate["id"], + "repair_spec_sha256": source["repair_spec_sha256"], + "details": source["details"], + "repair_basis": candidate["repair_basis"], + } + method = candidate["method"] + return { + "kind": "authoritative_successful_browser_use_execution", + "candidate_id": candidate["id"], + "model": method["model"], + "browser": method["engine"], + "execution": candidate["source_execution"], + "events_sha256": source["events_sha256"], + "response_sha256": source["response_sha256"], + "evaluation_sha256": source["evaluation_sha256"], + } + + +def build_verified_trajectory_dataset( + seed_manifest_path: str | Path, + replay_receipt_paths: list[str | Path], + *, + profile_path: str | Path, +) -> dict[str, Any]: + """Return trajectories qualified before the measured browser comparison.""" + + seed_path = Path(seed_manifest_path).expanduser().resolve() + seed_sha256 = _sha256(seed_path) + seeds = _read_json(seed_path) + if seeds.get("schema") != "lex_browser_use_replay_seeds/2": + raise ValueError("unsupported replay seed manifest") + seed_rows = seeds.get("tasks") or [] + seed_tasks = {int(item["task_id"]): item for item in seed_rows} + if len(seed_tasks) != len(seed_rows): + raise ValueError("replay seed manifest contains duplicate task IDs") + seed_task_ids = set(seed_tasks) + plan = load_replay_plan(profile_path, seed_path) + profile = plan["profile"] + qualification_engine = str(plan["qualification_engine"]) + required_engines = (qualification_engine,) + profile_sha256 = plan["profile_sha256"] + + attempts_by_key: dict[tuple[int, str], list[dict[str, Any]]] = defaultdict(list) + receipt_runs_by_key: dict[tuple[int, str], set[str]] = defaultdict(set) + receipts: list[dict[str, str]] = [] + for raw_path in replay_receipt_paths: + receipt_path = Path(raw_path).expanduser().resolve() + receipt = _read_json(receipt_path) + _validate_qualification_receipt( + receipt, + receipt_path=receipt_path, + seed_task_ids=seed_task_ids, + required_engines=required_engines, + expected_resources=profile["resources"], + ) + receipts.append( + { + "sha256": _sha256(receipt_path), + "source_seed_manifest_sha256": str( + receipt.get("source_seed_manifest_sha256") or "" + ), + "profile_sha256": str(receipt.get("profile_sha256") or ""), + } + ) + for attempt in receipt.get("attempts") or []: + task_id = int(attempt["task_id"]) + candidate_id = str(attempt["source_candidate_id"]) + if candidate_id != seed_tasks[task_id].get("canonical_candidate_id"): + if receipt.get("source_seed_manifest_sha256") == seed_sha256: + raise ValueError(f"receipt uses a non-canonical trajectory for task {task_id}") + continue + candidate = next( + ( + item + for item in seed_tasks[task_id].get("candidates") or [] + if item.get("id") == candidate_id + ), + None, + ) + if candidate is None: + if receipt.get("source_seed_manifest_sha256") == seed_sha256: + raise ValueError( + f"receipt references candidate missing from seeds: {candidate_id}" + ) + continue + if ( + receipt.get("source_seed_manifest_sha256") != seed_sha256 + and _has_stale_correction_binding(attempt, task_id) + ): + continue + session = { + str(item["id"]): item for item in receipt.get("sessions") or [] + }[str(attempt["browser_session_id"])] + _validate_passing_attempt_evidence( + attempt, + candidate, + str(session.get("site_access_host") or ""), + str((seeds.get("source") or {}).get("official_data_checksum") or ""), + plan["trusted_source_site_hosts"], + ) + if ( + attempt.get("process_status") != "pass" + or attempt.get("evaluator_status") != "pass" + ): + continue + key = (task_id, candidate_id) + attempts_by_key[key].append(attempt) + receipt_runs_by_key[key].add(str(receipt.get("run_id") or "")) + + trajectories: list[dict[str, Any]] = [] + for (task_id, candidate_id), attempts in sorted(attempts_by_key.items()): + task = seed_tasks.get(task_id) + if task is None: + raise ValueError(f"receipt references task missing from seeds: {task_id}") + candidate = next( + (item for item in task.get("candidates") or [] if item.get("id") == candidate_id), + None, + ) + if candidate is None: + raise ValueError(f"receipt references candidate missing from seeds: {candidate_id}") + by_engine: dict[str, dict[str, Any]] = {} + for item in attempts: + engine = str(item["engine"]) + if engine in by_engine: + raise ValueError(f"duplicate replay proof for {candidate_id}/{engine}") + by_engine[engine] = item + if set(by_engine) != set(required_engines): + continue + if any( + item.get("process_status") != "pass" + or item.get("evaluator_status") != "pass" + or int(item.get("completed_action_count") or 0) != int(candidate["action_count"]) + for item in by_engine.values() + ): + continue + correction_proofs = { + ( + str(item["contract_correction_id"]), + str(item.get("contract_correction_source_sha256") or ""), + str(item.get("corrected_task_sha256") or ""), + ) + for item in by_engine.values() + if item.get("contract_correction_id") + } + if correction_proofs and ( + sum(bool(item.get("contract_correction_id")) for item in by_engine.values()) + != len(required_engines) + or len(correction_proofs) > 1 + or any(not value for value in next(iter(correction_proofs))) + ): + raise ValueError(f"inconsistent replay contracts for {candidate_id}") + correction_proof = next(iter(correction_proofs), None) + trajectories.append( + { + "id": f"webarena-verified-{task_id}-trajectory-1", + "task_id": task_id, + "task": { + key: task[key] + for key in ( + "intent", + "intent_template_id", + "revision", + "sites", + "start_urls", + ) + if key in task + }, + "tool_calls": candidate["tool_calls"], + "evaluation_input": candidate["recorded_response"], + "source": _candidate_source(candidate), + "verification": { + "browsers": list(required_engines), + "process_status": "pass", + "evaluator_status": "pass", + "contract": ( + { + "kind": "corrected_replay_contract", + "correction_id": correction_proof[0], + "correction_source_sha256": correction_proof[1], + "corrected_task_sha256": correction_proof[2], + } + if correction_proof + else {"kind": "official_task_contract"} + ), + "model_calls": 0, + "receipt_run_ids": sorted(receipt_runs_by_key[(task_id, candidate_id)] - {""}), + }, + } + ) + + return { + "schema": SCHEMA, + "benchmark": seeds["source"].get("benchmark") or "WebArena-Verified", + "benchmark_version": seeds["source"].get("benchmark_version") or None, + "official_data_checksum": seeds["source"].get("official_data_checksum") or None, + "selection_sha256": seeds["source"]["selection_sha256"], + "semantics": { + "trajectory_is_one_verified_feasible_path": True, + "trajectory_is_the_only_correct_path": False, + "browser_use_scoring_authority": "official task outcome contract", + "browser_use_agent_visibility": "hidden", + "trajectory_replay_role": "fixed BrowserTool action input", + "evaluation_input_role": "official evaluator input only; never an Agent prompt", + }, + "promotion_policy": { + "qualification_browser": qualification_engine, + "measured_browsers_are_not_promotion_gates": True, + "process_status": "pass", + "evaluator_status": "pass", + "model_calls": 0, + }, + "provenance": { + "seed_manifest_sha256": seed_sha256, + "profile_sha256": profile_sha256, + "replay_receipts": receipts, + }, + "counts": { + "verified_tasks": len({item["task_id"] for item in trajectories}), + "verified_trajectories": len(trajectories), + }, + "trajectories": trajectories, + } diff --git a/harness/trajectory_replay/webarena_plan.py b/harness/trajectory_replay/webarena_plan.py new file mode 100644 index 000000000..7e7dea906 --- /dev/null +++ b/harness/trajectory_replay/webarena_plan.py @@ -0,0 +1,333 @@ +"""Validate the frozen WebArena trajectory-replay run contract.""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from typing import Any +from urllib.parse import urlsplit, urlunsplit + +from harness.browser_use.runner import SITE_PORTS +from harness.trajectory_replay.browser_use_seeds import ( + _authoritative_rows, + _contains_ref, + compile_replay_calls, + extract_browser_calls, +) +from harness.trajectory_replay.runner import ENGINE_IMAGES + +PROFILE_SCHEMA = "lex_webarena_trajectory_replay_profile/1" +SEED_SCHEMA = "lex_browser_use_replay_seeds/2" + + +def _read_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def replay_site_hosts(calls: list[dict[str, Any]]) -> set[str]: + site_ports = {port for pair in SITE_PORTS.values() for port in pair} + return { + str(parsed.hostname) + for call in calls + if isinstance(call.get("url"), str) + for parsed in [urlsplit(call["url"])] + if parsed.scheme in {"http", "https"} + and parsed.port in site_ports + and parsed.hostname + } + + +def replay_calls_binding_sha256( + calls: list[dict[str, Any]], *, allowed_hosts: set[str] +) -> str: + """Hash calls while ignoring only the deployment-specific WebArena gateway host.""" + + site_ports = {port for pair in SITE_PORTS.values() for port in pair} + normalized = [] + for raw_call in calls: + call = dict(raw_call) + url = call.get("url") + if isinstance(url, str): + parsed = urlsplit(url) + if ( + parsed.scheme in {"http", "https"} + and parsed.port in site_ports + and parsed.hostname in allowed_hosts + ): + call["url"] = urlunsplit( + ( + parsed.scheme, + f"__WEBARENA_GATEWAY__:{parsed.port}", + parsed.path, + parsed.query, + parsed.fragment, + ) + ) + normalized.append(call) + payload = json.dumps( + normalized, ensure_ascii=False, sort_keys=True, separators=(",", ":") + ) + return hashlib.sha256(payload.encode()).hexdigest() + + +def _validate_canonical_candidate( + task: dict[str, Any], + candidate: dict[str, Any], + repair_sha256: str, + repair_record: dict[str, Any] | None, + authoritative_row: dict[str, Any] | None, +) -> None: + task_id = int(task["task_id"]) + calls = candidate.get("tool_calls") + if ( + not isinstance(calls, list) + or not calls + or int(candidate.get("action_count") or 0) != len(calls) + or not isinstance(candidate.get("recorded_response"), dict) + ): + raise ValueError(f"task {task_id} has an invalid canonical trajectory payload") + source = candidate.get("source") or {} + if candidate.get("candidate_kind") == "expert_repair": + if ( + repair_record is None + or source.get("kind") != "expert_repair_spec" + or source.get("repair_spec_sha256") != repair_sha256 + or candidate.get("id") + != str( + repair_record.get("candidate_id") or f"expert__task-{task_id}" + ) + or calls != repair_record.get("tool_calls") + or candidate["recorded_response"] != repair_record.get("expected_response") + or source.get("details") != repair_record.get("source") + or candidate.get("repair_basis") != repair_record.get("repair_basis") + ): + raise ValueError(f"task {task_id} expert trajectory is not bound to the repair spec") + return + + if candidate.get("official_score") != 1.0 or candidate.get("clean") is not True: + raise ValueError(f"task {task_id} autonomous trajectory is not a clean official success") + method = candidate.get("method") or {} + if ( + authoritative_row is None + or int(candidate.get("source_execution") or 0) + != int(authoritative_row.get("attempt") or 0) + or str(candidate.get("source_run_id") or "") + != str(authoritative_row.get("run_id") or "") + or method.get("model") != authoritative_row.get("model") + or method.get("engine") != authoritative_row.get("engine") + or float(authoritative_row.get("score") or 0) != 1.0 + ): + raise ValueError( + f"task {task_id} autonomous trajectory is not the authoritative final execution" + ) + paths = { + kind: Path(str(source.get(f"{kind}_path") or "")).expanduser().resolve() + for kind in ("events", "response", "evaluation") + } + for kind, path in paths.items(): + if not path.is_file() or _sha256(path) != source.get(f"{kind}_sha256"): + raise ValueError(f"task {task_id} autonomous {kind} evidence is missing or changed") + if path != Path( + str((authoritative_row.get("paths") or {}).get(kind) or "") + ).expanduser().resolve(): + raise ValueError( + f"task {task_id} autonomous {kind} evidence is not from the final execution" + ) + source_calls = extract_browser_calls(paths["events"]) + if ( + compile_replay_calls(paths["events"]) != calls + or any(item["status"] != "success" for item in source_calls) + or _read_json(paths["response"]) != candidate["recorded_response"] + ): + raise ValueError(f"task {task_id} autonomous trajectory differs from source evidence") + evaluation = _read_json(paths["evaluation"]) + if int(evaluation.get("task_id") or -1) != task_id or float( + evaluation.get("score") or 0 + ) != 1.0: + raise ValueError(f"task {task_id} autonomous source evaluation is not passing") + + +def load_replay_plan( + profile_path: str | Path, seed_manifest_path: str | Path +) -> dict[str, Any]: + """Return a validated, exact method and task plan for one formal replay suite.""" + + profile_file = Path(profile_path).expanduser().resolve() + repo_root = profile_file.parents[3] + profile = _read_json(profile_file) + if profile.get("schema") != PROFILE_SCHEMA: + raise ValueError("unsupported trajectory replay profile") + + selection = profile.get("task_selection") or {} + selection_file = (repo_root / str(selection.get("path") or "")).resolve() + if repo_root not in selection_file.parents: + raise ValueError("task selection must stay inside the repository") + if not selection_file.is_file() or _sha256(selection_file) != selection.get("sha256"): + raise ValueError("trajectory replay task selection does not match the frozen profile") + selection_payload = _read_json(selection_file) + task_ids = [int(item["task_id"]) for item in selection_payload.get("tasks") or []] + if len(task_ids) != int(selection.get("task_count") or 0) or len(task_ids) != len( + set(task_ids) + ): + raise ValueError("trajectory replay task selection cardinality is invalid") + + repair = profile.get("seed_inventory") or {} + repair_file = (repo_root / str(repair.get("expert_repairs_path") or "")).resolve() + if repo_root not in repair_file.parents: + raise ValueError("expert repairs must stay inside the repository") + if not repair_file.is_file() or _sha256(repair_file) != repair.get( + "expert_repairs_sha256" + ): + raise ValueError("expert repairs do not match the frozen replay profile") + repair_payload = _read_json(repair_file) + repair_records = { + int(item["task_id"]): item for item in repair_payload.get("repairs") or [] + } + if len(repair_records) != len(repair_payload.get("repairs") or []): + raise ValueError("expert repairs contain duplicate task IDs") + + seed_file = Path(seed_manifest_path).expanduser().resolve() + seeds = _read_json(seed_file) + if seeds.get("schema") != SEED_SCHEMA: + raise ValueError("unsupported replay seed manifest") + seed_source = seeds.get("source") or {} + if seed_source.get("selection_sha256") != selection.get("sha256"): + raise ValueError("replay seeds are not bound to the frozen task selection") + seed_repairs = seeds.get("expert_repairs") or {} + if seed_repairs.get("repair_spec_sha256") != repair.get("expert_repairs_sha256"): + raise ValueError("replay seeds are not bound to the frozen expert repairs") + authoritative: dict[tuple[str, str, int], dict[str, Any]] = {} + for declared_model, declared in (seed_source.get("evidence") or {}).items(): + root = Path(str((declared or {}).get("root") or "")).expanduser().resolve() + attempts_file = root / "attempts.jsonl" + manifest_file = root / "manifest.json" + if ( + not attempts_file.is_file() + or not manifest_file.is_file() + or _sha256(attempts_file) != declared.get("attempts_sha256") + or _sha256(manifest_file) != declared.get("manifest_sha256") + ): + raise ValueError("replay seed authority manifest is missing or changed") + manifest, rows = _authoritative_rows(root) + model = str((manifest.get("protocol") or {}).get("model") or "") + if model != declared_model or manifest.get("selection_sha256") != selection.get( + "sha256" + ): + raise ValueError("replay seed authority differs from the frozen selection") + for row in rows: + key = (model, str(row["engine"]), int(row["task_id"])) + if key in authoritative: + raise ValueError("replay seed authority contains duplicate final executions") + authoritative[key] = row + seed_tasks = seeds.get("tasks") or [] + seed_task_ids = [int(item["task_id"]) for item in seed_tasks] + if len(seed_task_ids) != len(set(seed_task_ids)) or set(seed_task_ids) != set(task_ids): + raise ValueError("replay seeds do not match the frozen task selection") + ready = [item for item in seed_tasks if item.get("replay_status") == "ready"] + if len(ready) != int(repair.get("required_ready_tasks") or 0): + raise ValueError("replay seeds do not provide the required ready-task coverage") + for item in seed_tasks: + candidate_ids = [ + str(candidate.get("id") or "") for candidate in item.get("candidates") or [] + ] + canonical_id = str(item.get("canonical_candidate_id") or "") + if ( + not canonical_id + or candidate_ids.count(canonical_id) != 1 + or len(candidate_ids) != len(set(candidate_ids)) + ): + raise ValueError("every replay task must identify one unique canonical trajectory") + canonical = next( + candidate + for candidate in item.get("candidates") or [] + if candidate.get("id") == canonical_id + ) + if _contains_ref(canonical["tool_calls"]): + raise ValueError( + f"task {item['task_id']} canonical trajectory contains an ephemeral ref" + ) + _validate_canonical_candidate( + item, + canonical, + str(repair["expert_repairs_sha256"]), + repair_records.get(int(item["task_id"])), + authoritative.get( + ( + str((canonical.get("method") or {}).get("model") or ""), + str((canonical.get("method") or {}).get("engine") or ""), + int(item["task_id"]), + ) + ), + ) + + methods = profile.get("methods") or [] + method_ids = [str(item.get("id") or "") for item in methods] + engines = [str(item.get("engine") or "") for item in methods] + if ( + not methods + or len(method_ids) != len(set(method_ids)) + or len(engines) != len(set(engines)) + or any(engine not in ENGINE_IMAGES for engine in engines) + ): + raise ValueError("trajectory replay methods must be unique registered engines") + + qualification = profile.get("qualification") or {} + qualification_engine = str(qualification.get("reference_engine") or "") + if qualification_engine not in engines: + raise ValueError("trajectory qualification requires one registered reference engine") + if int(qualification.get("required_task_count") or 0) != len(task_ids): + raise ValueError("trajectory qualification must cover the frozen task set") + + execution = profile.get("execution") or {} + if execution.get("environment_pool_count") != 1: + raise ValueError("the fixed-port WebArena stack supports exactly one environment pool") + if execution.get("method_parallelism") != 1 or execution.get("task_parallelism") != 1: + raise ValueError("one environment pool requires serial methods and serial tasks") + if not execution.get("reset_before_every_task") or not execution.get( + "teardown_between_methods" + ): + raise ValueError("formal replay requires task resets and method teardown") + trusted_source_site_hosts = { + str(host) for host in execution.get("trusted_source_site_hosts") or [] + } + if not trusted_source_site_hosts or any( + not host or ":" in host or "/" in host for host in trusted_source_site_hosts + ): + raise ValueError("formal replay requires explicit trusted source site hosts") + for task in seed_tasks: + canonical = next( + candidate + for candidate in task.get("candidates") or [] + if candidate.get("id") == task.get("canonical_candidate_id") + ) + untrusted = replay_site_hosts(canonical["tool_calls"]) - trusted_source_site_hosts + if untrusted: + raise ValueError( + f"task {task['task_id']} canonical trajectory uses untrusted site hosts: " + f"{sorted(untrusted)}" + ) + if profile.get("runs_per_task") != 1: + raise ValueError("formal replay requires exactly one logical run per task and method") + if repair.get("canonical_trajectories_per_task") != 1: + raise ValueError("formal replay requires exactly one canonical trajectory per task") + if (profile.get("scoring") or {}).get("model_calls") != 0: + raise ValueError("trajectory replay must not call a model") + + return { + "profile": profile, + "profile_path": str(profile_file), + "profile_sha256": _sha256(profile_file), + "seed_manifest": seeds, + "seed_manifest_path": str(seed_file), + "seed_manifest_sha256": _sha256(seed_file), + "task_ids": task_ids, + "methods": methods, + "engines": engines, + "qualification_engine": qualification_engine, + "trusted_source_site_hosts": trusted_source_site_hosts, + } diff --git a/harness/trajectory_replay/webarena_replay.py b/harness/trajectory_replay/webarena_replay.py new file mode 100644 index 000000000..c226a2e83 --- /dev/null +++ b/harness/trajectory_replay/webarena_replay.py @@ -0,0 +1,1135 @@ +"""Model-free replay of fixed Browser Use trajectories on WebArena-Verified.""" + +from __future__ import annotations + +import hashlib +import json +import sys +import time +import traceback +import uuid +from pathlib import Path +from typing import Any +from urllib.parse import urlsplit, urlunsplit + +import docker + +from harness.browser_use.har import CdpHarRecorder +from harness.browser_use.runner import ( + SITE_PORTS, + _allow_official_container_hostname, + _cdp_metadata, + _configure_benchmark, + _container_host_gateway, + _docker_daemon_lock_namespace, + _docker_image_metadata, + _login_context, + _physical_execution_lock, + _physical_execution_lock_path, + _prepare_sites, + _require_clean_git_checkout, + _resource_summary, + _site_container_manager, +) +from harness.trajectory_replay.metrics import DockerStatsResources, ResourceSampler +from harness.trajectory_replay.runner import ( + ENGINE_IMAGES, + ENGINE_PORTS, + _published_endpoint, + _start_browser_container, + _wait_cdp, +) +from harness.trajectory_replay.webarena_corrections import ( + apply_correction, + check_replay_assertions, + evaluate_corrected_contract, + load_corrections, +) +from harness.trajectory_replay.webarena_plan import ( + load_replay_plan, + replay_calls_binding_sha256, + replay_site_hosts, +) + +RECEIPT_SCHEMA = "lex_webarena_trajectory_replay_suite/1" + +_CLICK_DISPATCHED_NAVIGATION_TIMEOUT_MARKERS = ( + "Click failed: Page.click: Timeout", + "performing click action", + "click action done", + "waiting for scheduled navigations to finish", +) + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _json_sha256(value: Any) -> str: + payload = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(payload.encode()).hexdigest() + + +def _origin(url: str) -> str: + parsed = urlsplit(url) + if parsed.scheme not in {"http", "https"} or not parsed.netloc: + raise ValueError(f"invalid WebArena environment URL: {url}") + return urlunsplit((parsed.scheme, parsed.netloc, "", "", "")) + + +def _evidence_file(path: Path) -> dict[str, str] | None: + return {"path": str(path), "sha256": _sha256(path)} if path.is_file() else None + + +def _click_was_dispatched_before_navigation_timeout( + arguments: dict[str, Any], result_status: str, result_text: str +) -> bool: + """Separate a completed click from Playwright's later navigation wait timeout. + + Replaying the click would risk applying a mutation twice. The following + trajectory step and official evaluator remain responsible for proving that + the resulting page state is usable and correct. + """ + + return ( + arguments.get("action") == "click" + and result_status != "success" + and all(marker in result_text for marker in _CLICK_DISPATCHED_NAVIGATION_TIMEOUT_MARKERS) + ) + + +def _execute_replay_call(tool: Any, arguments: dict[str, Any]) -> tuple[str, str]: + """Execute one call without changing the semantics of a compiled snapshot ref.""" + + replay_mode = arguments.get("replay_mode") + public_arguments = {key: value for key, value in arguments.items() if key != "replay_mode"} + if public_arguments.get("action") == "wait" and "seconds" in public_arguments: + if "timeout" in public_arguments: + return "error", "Wait call cannot declare both seconds and timeout" + seconds = public_arguments.pop("seconds") + if not isinstance(seconds, (int, float)) or isinstance(seconds, bool) or seconds <= 0: + return "error", "Wait seconds must be a positive number" + public_arguments["timeout"] = round(seconds * 1000) + if replay_mode != "snapshot_ref_semantics": + result = tool.execute(public_arguments) + return str(result.status), str(result.result) + + action = str(public_arguments.get("action") or "") + selector = str(public_arguments.get("selector") or "") + if action not in {"click", "fill", "select"} or not selector: + return "error", "Invalid compiled snapshot-ref replay call" + service = tool._get_service() + + try: + def execute() -> None: + page = service._page + locator = page.locator(selector).first + if action == "click": + locator.evaluate("element => element.click()") + page.wait_for_timeout(500) + elif action == "fill": + locator.evaluate("element => { element.focus(); element.value = ''; }") + page.keyboard.type(str(public_arguments.get("text", ""))) + else: + value = str(public_arguments.get("value", "")) + locator.evaluate( + "(element, value) => { if (element.tagName.toLowerCase() !== 'select') " + "throw new Error('target is not a select element'); element.value = value; " + "element.dispatchEvent(new Event('change', {bubbles: true})); }", + value, + ) + + service._submit(execute) + except Exception as exc: + return "error", f"{action.capitalize()} failed: {exc}" + return "success", f"{action.capitalize()} completed with snapshot-ref semantics." + + +def _candidate(task: dict[str, Any]) -> dict[str, Any]: + candidate_id = task.get("canonical_candidate_id") + for item in task.get("candidates") or []: + if item.get("id") == candidate_id: + return item + raise ValueError(f"task {task.get('task_id')} has no canonical replay candidate") + + +def select_tasks(seed_manifest: dict[str, Any], task_ids: list[int]) -> list[dict[str, Any]]: + by_id = {int(task["task_id"]): task for task in seed_manifest.get("tasks") or []} + missing = sorted(set(task_ids) - set(by_id)) + if missing: + raise ValueError(f"tasks are not present in the replay seed manifest: {missing}") + selected = [by_id[task_id] for task_id in task_ids] + blocked = [int(task["task_id"]) for task in selected if task.get("replay_status") != "ready"] + if blocked: + raise ValueError(f"tasks do not have clean replay candidates: {blocked}") + return selected + + +def select_methods(plan: dict[str, Any], engines: list[str]) -> list[dict[str, Any]]: + """Return only methods registered for this formal or diagnostic engine scope.""" + + selected = [method for method in plan["methods"] if str(method["engine"]) in engines] + selected_engines = [str(method["engine"]) for method in selected] + if selected_engines != engines: + raise ValueError("replay methods do not match the selected engine scope") + return selected + + +def validate_qualification_dataset(plan: dict[str, Any], path: str | Path) -> str: + """Bind a formal comparison to the complete pre-qualified trajectory set.""" + + dataset_path = Path(path).expanduser().resolve() + dataset = json.loads(dataset_path.read_text(encoding="utf-8")) + if dataset.get("schema") != "lex_webarena_verified_trajectory_dataset/1": + raise ValueError("formal replay requires the current qualified trajectory dataset") + provenance = dataset.get("provenance") or {} + if provenance.get("seed_manifest_sha256") != plan["seed_manifest_sha256"]: + raise ValueError("qualified trajectories differ from the frozen replay seeds") + if provenance.get("profile_sha256") != plan["profile_sha256"]: + raise ValueError("qualified trajectories differ from the frozen replay profile") + policy = dataset.get("promotion_policy") or {} + if policy.get("qualification_browser") != plan["qualification_engine"]: + raise ValueError("qualified trajectories use a different reference browser") + trajectories = dataset.get("trajectories") or [] + by_task = {int(item["task_id"]): item for item in trajectories} + if len(by_task) != len(trajectories) or set(by_task) != set(plan["task_ids"]): + raise ValueError("qualified trajectories do not cover the frozen task set") + seed_tasks = { + int(item["task_id"]): item for item in plan["seed_manifest"].get("tasks") or [] + } + for task_id, trajectory in by_task.items(): + if trajectory.get("tool_calls") != _candidate(seed_tasks[task_id]).get("tool_calls"): + raise ValueError(f"qualified trajectory differs from task {task_id} canonical input") + return _sha256(dataset_path) + + +def _rebind_url( + value: str, site_access_host: str, trusted_source_site_hosts: set[str] +) -> str: + parsed = urlsplit(value) + if ( + parsed.scheme not in {"http", "https"} + or parsed.port not in {port for pair in SITE_PORTS.values() for port in pair} + or parsed.hostname not in trusted_source_site_hosts + ): + return value + host = f"[{site_access_host}]" if ":" in site_access_host else site_access_host + netloc = f"{host}:{parsed.port}" + return urlunsplit((parsed.scheme, netloc, parsed.path, parsed.query, parsed.fragment)) + + +def _rebind_calls( + calls: list[dict[str, Any]], + site_access_host: str, + trusted_source_site_hosts: set[str], +) -> list[dict[str, Any]]: + rebound = [] + for call in calls: + item = dict(call) + if isinstance(item.get("url"), str): + item["url"] = _rebind_url( + item["url"], site_access_host, trusted_source_site_hosts + ) + rebound.append(item) + return rebound + + +def _clear_browser_state(service: Any, origins: set[str]) -> None: + """Fail closed unless all persistent browser state is cleared before a task.""" + + def clear() -> None: + context = service._context + page = service._page + if context is None or page is None: + raise RuntimeError("browser context is not ready for state reset") + context.clear_cookies() + context.clear_permissions() + cdp = context.new_cdp_session(page) + try: + cdp.send("Network.enable") + cdp.send("Network.clearBrowserCache") + cdp.send("Network.clearBrowserCookies") + for origin in sorted(origins): + cdp.send( + "Storage.clearDataForOrigin", + {"origin": origin, "storageTypes": "all"}, + ) + page.goto("about:blank", wait_until="load", timeout=10_000) + cdp.send("Page.resetNavigationHistory") + finally: + cdp.detach() + + service._submit(clear) + + +def _run_one( # noqa: PLR0913 + *, + task_seed: dict[str, Any], + engine: str, + endpoint: str, + container: Any, + benchmark: Any, + cowagent_repo: Path, + output_dir: Path, + sample_interval_ms: int, + physical_execution: int, + trusted_source_site_hosts: set[str], +) -> dict[str, Any]: + task_id = int(task_seed["task_id"]) + candidate = _candidate(task_seed) + official_task = benchmark.get_task(task_id) + correction_catalog = load_corrections() + corrected_contract = ( + apply_correction(official_task, correction_catalog) + if task_id in correction_catalog.records + else None + ) + sites = tuple(official_task.sites) + start_urls = list(benchmark.config.render_url(list(official_task.start_urls), list(sites))) + site_access_host = _container_host_gateway(container) + calls = _rebind_calls( + candidate["tool_calls"], site_access_host, trusted_source_site_hosts + ) + source_site_hosts = replay_site_hosts(candidate["tool_calls"]) + attempt_dir = output_dir / engine / str(task_id) / f"physical-{physical_execution}" + attempt_dir.mkdir(parents=True, exist_ok=True) + har_path = attempt_dir / "network.har" + trace_path = attempt_dir / "trace.json" + evaluation_path = attempt_dir / "evaluation.json" + screenshot_path = attempt_dir / "failure.png" + + if str(cowagent_repo) not in sys.path: + sys.path.insert(0, str(cowagent_repo)) + from agent.tools.browser.browser_tool import BrowserTool # noqa: PLC0415 + + tool = BrowserTool( + config={ + "cdp_endpoint": endpoint, + "headless": True, + "viewport": {"width": 1280, "height": 900}, + "startup_timeout": 90, + } + ) + _allow_official_container_hostname(tool) + resources = DockerStatsResources(container) + sampler = ResourceSampler(resources, interval_ms=sample_interval_ms) + before = None + resource_started = None + recorder = CdpHarRecorder() + records: list[dict[str, Any]] = [] + timings: dict[str, float] = {} + process_status = "fail" + evaluator_status = "not_run" + replay_assertions: tuple[dict[str, Any], ...] = () + error = "" + started = time.perf_counter() + sampler_started = False + try: + prepare_started = time.perf_counter() + _prepare_sites(sites) + timings["environment_prepare_ms"] = (time.perf_counter() - prepare_started) * 1000 + + setup_started = time.perf_counter() + service = tool._get_service() + service._submit(lambda: None) + origins = { + _origin(benchmark.config.get_environment(site).active_url) for site in SITE_PORTS + } + _clear_browser_state(service, origins) + readiness_retries = _login_context(service, sites, benchmark, start_urls) + service._submit(lambda: recorder.start(service._context)) + timings["browser_session_setup_ms"] = (time.perf_counter() - setup_started) * 1000 + + before = resources.counters() + resource_started = time.perf_counter() + sampler.start() + sampler_started = True + replay_started = time.perf_counter() + for index, arguments in enumerate(calls, start=1): + call_started = time.perf_counter() + result_status, result_text = _execute_replay_call(tool, arguments) + dispatched_before_timeout = _click_was_dispatched_before_navigation_timeout( + arguments, result_status, result_text + ) + record = { + "index": index, + "arguments": arguments, + "status": "success" if dispatched_before_timeout else result_status, + "tool_status": result_status, + "duration_ms": round((time.perf_counter() - call_started) * 1000, 3), + "result": result_text, + } + observation_error = _embedded_observation_error(arguments, result_status, result_text) + if observation_error is not None: + record["observation_status"] = "error" + record["observation_error"] = observation_error + if dispatched_before_timeout: + record["recovery"] = { + "classification": "click_dispatched_navigation_wait_timeout", + "action_retried": False, + } + records.append(record) + if result_status != "success" and not dispatched_before_timeout: + raise RuntimeError(f"step {index} failed: {result_text}") + timings["trajectory_replay_ms"] = (time.perf_counter() - replay_started) * 1000 + process_status = "pass" + + if corrected_contract is not None and corrected_contract.replay_assertions: + selectors = { + selector + for assertion in corrected_contract.replay_assertions + for selector in (assertion.get("fields") or {}) + } + page_state = service._submit( + lambda: { + "url": service._page.url, + "text": service._page.locator("body").inner_text(), + "fields": service._page.evaluate( + "selectors => Object.fromEntries(selectors.map(selector => " + "[selector, document.querySelector(selector)?.value ?? null]))", + sorted(selectors), + ), + } + ) + replay_assertions = check_replay_assertions( + corrected_contract, + page_url=page_state["url"], + page_text=page_state["text"], + field_values=page_state["fields"], + ) + + finalize_started = time.perf_counter() + service._submit( + lambda: ( + service._page.wait_for_timeout(500), + recorder.stop(service._context, har_path), + ) + ) + timings["evidence_finalize_ms"] = (time.perf_counter() - finalize_started) * 1000 + + evaluation_started = time.perf_counter() + evaluation_result = ( + evaluate_corrected_contract( + benchmark, + corrected_contract, + agent_response=candidate["recorded_response"], + network_trace=har_path, + ) + if corrected_contract is not None + else benchmark.evaluate_task( + task_id=task_id, + agent_response=candidate["recorded_response"], + network_trace=har_path, + ) + ) + evaluation = evaluation_result.model_dump(mode="json") + if corrected_contract is not None: + evaluation["replay_contract_correction"] = { + "correction_id": corrected_contract.correction_id, + "source_sha256": corrected_contract.correction_source_sha256, + "original_task_sha256": corrected_contract.original_task_sha256, + "corrected_task_sha256": corrected_contract.corrected_task_sha256, + "rationale": corrected_contract.rationale, + } + evaluation["replay_assertions"] = list(replay_assertions) + timings["evaluation_ms"] = (time.perf_counter() - evaluation_started) * 1000 + evaluator_status = ( + "pass" + if float(evaluation.get("score") or 0) == 1 + and all(item.get("status") == "pass" for item in replay_assertions) + else "fail" + ) + evaluation_path.write_text( + json.dumps(evaluation, ensure_ascii=False, indent=2), encoding="utf-8" + ) + except Exception: + error = traceback.format_exc() + try: + service = tool._get_service() + service._submit(lambda: recorder.stop(service._context, har_path)) + service._submit(lambda: service._page.screenshot(path=str(screenshot_path))) + except Exception: + pass + readiness_retries = locals().get("readiness_retries", 0) + finally: + cleanup_errors: list[str] = [] + duration_ms = (time.perf_counter() - started) * 1000 + if sampler_started: + try: + sampler.stop() + except Exception as exc: + cleanup_errors.append(f"resource sampler stop failed: {exc}") + try: + after = resources.counters() if before is not None else None + except Exception as exc: + after = None + cleanup_errors.append(f"final resource counters failed: {exc}") + resource_duration_ms = ( + (time.perf_counter() - resource_started) * 1000 if resource_started is not None else 0.0 + ) + try: + tool.close() + except Exception as exc: + cleanup_errors.append(f"browser tool close failed: {exc}") + + timings["browser_resource_measurement_ms"] = resource_duration_ms + + trace_path.write_text(json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8") + return { + "schema": "lex_webarena_trajectory_replay_attempt/1", + "engine": engine, + "task_id": task_id, + "physical_execution": physical_execution, + "source_candidate_id": candidate["id"], + "source_tool_calls_sha256": _json_sha256(candidate["tool_calls"]), + "executed_tool_calls_sha256": _json_sha256(calls), + "source_site_hosts": sorted(source_site_hosts), + "runtime_site_host": site_access_host, + "replay_calls_binding_sha256": replay_calls_binding_sha256( + calls, allowed_hosts={site_access_host} + ), + "evaluation_input_sha256": _json_sha256(candidate["recorded_response"]), + "source_model": (candidate.get("method") or {}).get("model"), + "source_engine": (candidate.get("method") or {}).get("engine"), + "source_kind": (candidate.get("source") or {}).get("kind", "browser_use_execution"), + "action_count": len(calls), + "completed_action_count": sum(record["status"] == "success" for record in records), + "process_status": process_status, + "evaluator_status": evaluator_status, + "replay_assertions": list(replay_assertions), + "contract_correction_id": ( + corrected_contract.correction_id if corrected_contract is not None else None + ), + "contract_correction_source_sha256": ( + corrected_contract.correction_source_sha256 if corrected_contract is not None else None + ), + "corrected_task_sha256": ( + corrected_contract.corrected_task_sha256 if corrected_contract is not None else None + ), + "status": "pass" if process_status == evaluator_status == "pass" else "fail", + "duration_ms": round(duration_ms, 3), + "timings": {key: round(value, 3) for key, value in timings.items()}, + "browser_session_readiness_retries": readiness_retries, + "resource": ( + _resource_summary(resources, sampler, before, after, resource_duration_ms) + if before is not None and after is not None and sampler.samples + else {} + ), + "error": error, + "cleanup_errors": cleanup_errors, + "evidence": { + "trace": _evidence_file(trace_path), + "network_har": _evidence_file(har_path), + "evaluation": _evidence_file(evaluation_path), + "failure_screenshot": _evidence_file(screenshot_path), + }, + "paths": { + "trace": str(trace_path), + "network_har": str(har_path), + "evaluation": str(evaluation_path), + "failure_screenshot": str(screenshot_path) if screenshot_path.is_file() else None, + }, + } + + +def _attempt_needs_retry(attempt: dict[str, Any]) -> bool: + """Give every failed logical replay exactly one clean physical retry.""" + return attempt.get("status") != "pass" + + +def _embedded_observation_error( + arguments: dict[str, Any], result_status: str, result_text: str +) -> str | None: + """Expose BrowserTool snapshot degradation without rewriting task outcome. + + BrowserTool currently returns a successful tool status with a bracketed error + payload when the page execution context changes during a snapshot. The action + remains completed, but the observation is unavailable and must be explicit in + evidence so diagnosis can identify the first degraded step. + """ + + if arguments.get("action") != "snapshot" or result_status != "success": + return None + prefix = "[Snapshot error:" + if not result_text.startswith(prefix): + return None + return result_text[len(prefix) :].removesuffix("]").strip() + + +def _stop_all_sites() -> None: + """Stop the fixed-port site stack before another method can acquire it.""" + + for site in SITE_PORTS: + manager = _site_container_manager(site) + if manager.is_running(): + manager.stop() + + +def _teardown_method(client: Any, run_id: str, engine: str) -> None: + """Remove every runtime object for a method and fail if isolation is incomplete.""" + + errors = [] + labels = [ + f"lex.browser-use-eval.run=replay-{run_id}", + f"lex.browser-use-eval.engine={engine}", + ] + for container in client.containers.list(all=True, filters={"label": labels}): + try: + container.remove(force=True) + except Exception as exc: + errors.append(f"browser container {container.name}: {exc}") + network_name = f"lex-webarena-replay-{run_id}-{engine}" + for network in client.networks.list(names=[network_name]): + try: + network.remove() + except Exception as exc: + errors.append(f"browser network {network.name}: {exc}") + try: + _stop_all_sites() + except Exception as exc: + errors.append(f"site stack: {exc}") + if errors: + raise RuntimeError("method teardown failed: " + "; ".join(errors)) + + +def _finalize_method_teardown( + client: Any, + run_id: str, + engine: str, + method_teardown: dict[str, bool], + receipt: dict[str, Any], + receipt_path: Path, + primary_error: BaseException | None, +) -> None: + """Persist cleanup outcome without replacing an in-flight task failure.""" + + cleanup_errors = [] + try: + _teardown_method(client, run_id, engine) + except Exception as exc: + cleanup_errors.append(str(exc)) + method_teardown[engine] = False + receipt.setdefault("method_teardown_errors", {})[engine] = str(exc) + else: + method_teardown[engine] = True + (receipt.get("method_teardown_errors") or {}).pop(engine, None) + receipt["method_teardown"] = method_teardown + try: + _write_receipt(receipt_path, receipt) + except Exception as exc: + cleanup_errors.append(f"receipt update failed: {exc}") + if not cleanup_errors: + return + message = "method finalization failed: " + "; ".join(cleanup_errors) + if primary_error is not None: + primary_error.add_note(message) + return + raise RuntimeError(message) + + +def _remove_orphan_replay_runtimes(client: Any, current_run_id: str) -> None: + """Remove stale replay containers and networks while the global stack lock is held.""" + + errors = [] + for container in client.containers.list( + all=True, filters={"label": "lex.browser-use-eval.run"} + ): + run_label = str((container.labels or {}).get("lex.browser-use-eval.run") or "") + if run_label.startswith("replay-") and run_label != f"replay-{current_run_id}": + try: + container.remove(force=True) + except Exception as exc: + errors.append(f"orphan container {container.name}: {exc}") + for network in client.networks.list(): + if network.name.startswith("lex-webarena-replay-") and current_run_id not in network.name: + try: + network.remove() + except Exception as exc: + errors.append(f"orphan network {network.name}: {exc}") + if errors: + raise RuntimeError("orphan replay cleanup failed: " + "; ".join(errors)) + + +def _write_receipt(path: Path, receipt: dict[str, Any]) -> None: + temporary = path.with_suffix(".tmp") + temporary.write_text(json.dumps(receipt, ensure_ascii=False, indent=2), encoding="utf-8") + temporary.replace(path) + + +def _next_physical_execution(output: Path, engine: str, task_id: int) -> int: + task_dir = output / engine / str(task_id) + existing = { + int(path.name.removeprefix("physical-")) + for path in task_dir.glob("physical-*") + if path.is_dir() and path.name.removeprefix("physical-").isdigit() + } + return max(existing, default=0) + 1 + + +def _start_physical_execution( + receipt: dict[str, Any], *, engine: str, task_id: int, physical_execution: int +) -> dict[str, Any]: + entry = { + "engine": engine, + "task_id": task_id, + "physical_execution": physical_execution, + "status": "running", + "started_at_unix_ms": round(time.time() * 1000), + } + receipt.setdefault("physical_executions", []).append(entry) + return entry + + +def _finish_physical_execution(entry: dict[str, Any], attempt: dict[str, Any]) -> None: + entry.update( + { + "status": "complete", + "finished_at_unix_ms": round(time.time() * 1000), + "attempt_status": attempt["status"], + "duration_ms": attempt["duration_ms"], + "evidence": attempt["evidence"], + } + ) + + +def _validated_resume( + receipt_path: Path, + *, + plan: dict[str, Any], + task_ids: list[int], + engines: list[str], + scope: str, + runtime: dict[str, Any], + qualification_dataset_sha256: str | None = None, +) -> tuple[ + str, + list[dict[str, Any]], + dict[str, bool], + list[dict[str, Any]], + list[dict[str, Any]], +]: + if not receipt_path.is_file(): + return uuid.uuid4().hex[:12], [], {engine: False for engine in engines}, [], [] + receipt = json.loads(receipt_path.read_text(encoding="utf-8")) + expected = { + "schema": RECEIPT_SCHEMA, + "execution_scope": scope, + "profile_sha256": plan["profile_sha256"], + "source_seed_manifest_sha256": plan["seed_manifest_sha256"], + "qualification_dataset_sha256": qualification_dataset_sha256, + "task_ids": task_ids, + "engines": engines, + "runtime": runtime, + } + for key, value in expected.items(): + if receipt.get(key) != value: + raise ValueError(f"existing replay receipt differs on {key}") + attempts = list(receipt.get("attempts") or []) + canonical_ids = { + int(task["task_id"]): str(task["canonical_candidate_id"]) + for task in plan["seed_manifest"].get("tasks") or [] + } + allowed_keys = {(engine, task_id) for engine in engines for task_id in task_ids} + + def required_receipt_int(item: object, field: str, record: str) -> int: + if not isinstance(item, dict) or field not in item: + raise ValueError(f"existing replay receipt {record} omitted {field}") + value = item[field] + if isinstance(value, bool): + raise ValueError(f"existing replay receipt {record} has invalid {field}") + try: + return int(value) + except (TypeError, ValueError) as exc: + raise ValueError( + f"existing replay receipt {record} has invalid {field}" + ) from exc + + keys = [] + for item in attempts: + task_id = required_receipt_int(item, "task_id", "attempt") + if item.get("schema") != "lex_webarena_trajectory_replay_attempt/1": + raise ValueError("existing replay receipt contains an invalid attempt schema") + key = (str(item.get("engine")), task_id) + if key not in allowed_keys: + raise ValueError("existing replay receipt contains an attempt outside the frozen plan") + if item.get("source_candidate_id") != canonical_ids[key[1]]: + raise ValueError("existing replay receipt contains a non-canonical trajectory") + keys.append(key) + if len(keys) != len(set(keys)): + raise ValueError("existing replay receipt contains duplicate method/task attempts") + method_teardown = receipt.get("method_teardown") or {} + if set(method_teardown) != set(engines) or any( + not isinstance(value, bool) for value in method_teardown.values() + ): + raise ValueError("existing replay receipt has invalid method teardown state") + physical_executions = list(receipt.get("physical_executions") or []) + if any(item.get("status") != "complete" for item in physical_executions): + raise ValueError( + "existing replay receipt contains an interrupted physical execution; " + "restart the complete formal scope in a new output directory" + ) + physical_keys = [ + ( + str(item.get("engine")), + required_receipt_int(item, "task_id", "physical execution"), + required_receipt_int(item, "physical_execution", "physical execution"), + ) + for item in physical_executions + ] + if len(physical_keys) != len(set(physical_keys)): + raise ValueError("existing replay receipt contains duplicate physical executions") + physical_by_logical: dict[tuple[str, int], list[dict[str, Any]]] = {} + for item in physical_executions: + key = ( + str(item.get("engine")), + required_receipt_int(item, "task_id", "physical execution"), + ) + if key not in allowed_keys: + raise ValueError( + "existing replay receipt contains a physical execution outside the plan" + ) + physical_by_logical.setdefault(key, []).append(item) + attempts_by_key = { + (str(item["engine"]), int(item["task_id"])): item for item in attempts + } + if set(physical_by_logical) != set(attempts_by_key): + raise ValueError("existing replay receipt has an unbound physical execution") + for key, attempt in attempts_by_key.items(): + physical = physical_by_logical[key] + if ( + len(physical) != int(attempt.get("physical_execution_count") or 1) + or max(int(item["physical_execution"]) for item in physical) + != int(attempt["physical_execution"]) + ): + raise ValueError("existing replay receipt has inconsistent physical execution history") + sessions = list(receipt.get("sessions") or []) + session_ids = [str(session.get("id") or "") for session in sessions] + if len(session_ids) != len(set(session_ids)) or any(not item for item in session_ids): + raise ValueError("existing replay receipt has invalid browser sessions") + sessions_by_id = {str(session["id"]): session for session in sessions} + for attempt in attempts: + session = sessions_by_id.get(str(attempt.get("browser_session_id") or "")) + if ( + session is None + or session.get("engine") != attempt.get("engine") + or int(attempt["task_id"]) not in session.get("task_ids", []) + ): + raise ValueError("existing replay attempt is not bound to a browser session") + return ( + str(receipt["run_id"]), + attempts, + dict(method_teardown), + sessions, + physical_executions, + ) + + +def run_replay_suite( # noqa: PLR0913 + *, + profile_path: str | Path, + seed_manifest_path: str | Path, + output_dir: str | Path, + cowagent_repo: str | Path, + diagnostic_task_ids: list[int] | None = None, + diagnostic_engines: list[str] | None = None, + qualification_engine: str | None = None, + qualification_task_ids: list[int] | None = None, + correction_task_ids: list[int] | None = None, + qualified_dataset_path: str | Path | None = None, +) -> dict[str, Any]: + plan = load_replay_plan(profile_path, seed_manifest_path) + seeds = plan["seed_manifest"] + selected_scopes = sum( + scope is not None + for scope in (diagnostic_task_ids, qualification_task_ids, correction_task_ids) + ) + if selected_scopes > 1: + raise ValueError("diagnostic, qualification and correction task scopes are exclusive") + selected_task_ids = ( + qualification_task_ids + if qualification_task_ids is not None + else correction_task_ids + if correction_task_ids is not None + else diagnostic_task_ids + ) + task_ids = list(plan["task_ids"] if selected_task_ids is None else selected_task_ids) + tasks = select_tasks(seeds, task_ids) + engines = list(plan["engines"]) + if qualification_engine is not None: + if ( + diagnostic_task_ids is not None + or diagnostic_engines is not None + or correction_task_ids is not None + ): + raise ValueError("qualification and diagnostic scopes cannot be combined") + if qualification_engine != plan["qualification_engine"]: + raise ValueError("qualification engine differs from the frozen reference engine") + engines = [qualification_engine] + scope = "qualification_shard" if qualification_task_ids is not None else "qualification" + elif qualification_task_ids is not None: + raise ValueError("qualification-task requires qualification-engine") + elif correction_task_ids is not None: + if diagnostic_engines is not None: + raise ValueError("formal correction shards must cover every registered browser") + scope = "formal_correction_shard" + elif diagnostic_engines is not None: + unknown_engines = sorted(set(diagnostic_engines) - set(engines)) + if unknown_engines: + raise ValueError(f"diagnostic engines are not registered: {unknown_engines}") + engines = [engine for engine in engines if engine in set(diagnostic_engines)] + if not engines: + raise ValueError("diagnostic engine selection is empty") + scope = "diagnostic" + else: + scope = "formal" if diagnostic_task_ids is None else "diagnostic" + qualification_dataset_sha256 = None + if scope in {"formal", "formal_correction_shard"}: + if qualified_dataset_path is None: + raise ValueError("formal replay requires a complete qualified trajectory dataset") + qualification_dataset_sha256 = validate_qualification_dataset( + plan, qualified_dataset_path + ) + elif qualified_dataset_path is not None: + raise ValueError("a qualified trajectory dataset is only accepted for formal replay") + resources = plan["profile"].get("resources") or {} + cpus = float(resources["cpus"]) + memory = str(resources["memory"]) + sample_interval_ms = int(resources["sample_interval_ms"]) + output = Path(output_dir).expanduser().resolve() + output.mkdir(parents=True, exist_ok=True) + receipt_path = output / "receipt.json" + if any(output.iterdir()) and not receipt_path.is_file(): + raise ValueError("replay output directory is not empty and has no resumable receipt") + cowagent = Path(cowagent_repo).expanduser().resolve() + if not (cowagent / "agent/tools/browser/browser_tool.py").is_file(): + raise ValueError("cowagent_repo does not contain BrowserTool") + + repo_root = Path(__file__).resolve().parents[2] + browser_eval_commit = _require_clean_git_checkout(repo_root) + cowagent_commit = _require_clean_git_checkout(cowagent) + client = docker.from_env() + docker_info = client.info() + browser_images = { + engine: _docker_image_metadata(client.images.get(ENGINE_IMAGES[engine])) + for engine in engines + } + site_images = {} + for site in SITE_PORTS: + reference = _site_container_manager(site).config.docker_img + site_images[site.value] = { + "reference": reference, + **_docker_image_metadata(client.images.get(reference)), + } + runtime = { + "browser_eval_commit": browser_eval_commit, + "cowagent_commit": cowagent_commit, + "resources": { + "cpus": cpus, + "memory": memory, + "sample_interval_ms": sample_interval_ms, + }, + "browser_images": browser_images, + "site_images": site_images, + } + run_id, attempts, method_teardown, sessions, physical_executions = _validated_resume( + receipt_path, + plan=plan, + task_ids=task_ids, + engines=engines, + scope=scope, + runtime=runtime, + qualification_dataset_sha256=qualification_dataset_sha256, + ) + completed = {(str(item["engine"]), int(item["task_id"])) for item in attempts} + lock_path = _physical_execution_lock_path( + _docker_daemon_lock_namespace(docker_info) + ) + receipt = { + "schema": RECEIPT_SCHEMA, + "run_id": run_id, + "execution_scope": scope, + "status": "running", + "profile": plan["profile_path"], + "profile_sha256": plan["profile_sha256"], + "source_seed_manifest": plan["seed_manifest_path"], + "source_seed_manifest_sha256": plan["seed_manifest_sha256"], + "qualification_dataset_sha256": qualification_dataset_sha256, + "task_ids": task_ids, + "engines": engines, + "runtime": runtime, + "method_teardown": method_teardown, + "sessions": sessions, + "physical_executions": physical_executions, + "isolation": { + **plan["profile"]["execution"], + "physical_execution_lock_namespace": lock_path.stem, + }, + "attempts": attempts, + } + _write_receipt(receipt_path, receipt) + with _physical_execution_lock(lock_path) as lock_wait_ms: + _remove_orphan_replay_runtimes(client, run_id) + for engine in engines: + _teardown_method(client, run_id, engine) + method_teardown[engine] = True + receipt["method_teardown"] = method_teardown + receipt["isolation"]["lock_wait_ms"] = round(lock_wait_ms, 3) + _write_receipt(receipt_path, receipt) + for method in select_methods(plan, engines): + engine = str(method["engine"]) + pending = [task for task in tasks if (engine, int(task["task_id"])) not in completed] + if not pending: + continue + network = None + container = None + primary_error: BaseException | None = None + try: + _stop_all_sites() + method_teardown[engine] = False + receipt["method_teardown"] = method_teardown + _write_receipt(receipt_path, receipt) + network = client.networks.create(f"lex-webarena-replay-{run_id}-{engine}") + container = _start_browser_container( + client, + engine=engine, + run_id=f"replay-{run_id}", + cpus=cpus, + memory=memory, + network=network.name, + ) + container.reload() + http_endpoint = _published_endpoint( + container, engine=engine, cdp_port=ENGINE_PORTS[engine] + ) + endpoint = _wait_cdp(http_endpoint, container=container) + benchmark = _configure_benchmark(hostname=_container_host_gateway(container)) + session = { + "id": uuid.uuid4().hex[:12], + "engine": engine, + "container_id": container.id, + "browser_image": _docker_image_metadata(container.image), + "browser": _cdp_metadata(http_endpoint), + "site_access_host": _container_host_gateway(container), + "task_ids": [], + } + sessions.append(session) + receipt["sessions"] = sessions + _write_receipt(receipt_path, receipt) + for task in pending: + physical_execution = _next_physical_execution( + output, engine, int(task["task_id"]) + ) + physical_entry = _start_physical_execution( + receipt, + engine=engine, + task_id=int(task["task_id"]), + physical_execution=physical_execution, + ) + _write_receipt(receipt_path, receipt) + attempt = _run_one( + task_seed=task, + engine=engine, + endpoint=endpoint, + container=container, + benchmark=benchmark, + cowagent_repo=cowagent, + output_dir=output, + sample_interval_ms=sample_interval_ms, + physical_execution=physical_execution, + trusted_source_site_hosts=plan["trusted_source_site_hosts"], + ) + attempt["browser_session_id"] = session["id"] + _finish_physical_execution(physical_entry, attempt) + _write_receipt(receipt_path, receipt) + if _attempt_needs_retry(attempt): + # A declared replay gets one clean retry regardless of where the first + # physical execution failed. Restarting the browser and rerunning + # _run_one also repeats site preparation, so failed page/session state + # cannot leak into the final physical execution. + container.restart() + container.reload() + http_endpoint = _published_endpoint( + container, engine=engine, cdp_port=ENGINE_PORTS[engine] + ) + endpoint = _wait_cdp(http_endpoint, container=container) + prior_attempt = attempt + session = { + "id": uuid.uuid4().hex[:12], + "engine": engine, + "container_id": container.id, + "browser_image": _docker_image_metadata(container.image), + "browser": _cdp_metadata(http_endpoint), + "site_access_host": _container_host_gateway(container), + "task_ids": [], + } + sessions.append(session) + receipt["sessions"] = sessions + retry_entry = _start_physical_execution( + receipt, + engine=engine, + task_id=int(task["task_id"]), + physical_execution=physical_execution + 1, + ) + _write_receipt(receipt_path, receipt) + attempt = _run_one( + task_seed=task, + engine=engine, + endpoint=endpoint, + container=container, + benchmark=benchmark, + cowagent_repo=cowagent, + output_dir=output, + sample_interval_ms=sample_interval_ms, + physical_execution=physical_execution + 1, + trusted_source_site_hosts=plan["trusted_source_site_hosts"], + ) + attempt["browser_session_id"] = session["id"] + attempt["prior_attempts"] = [prior_attempt] + attempt["physical_execution_count"] = 2 + attempt["duration_ms"] = round( + float(prior_attempt["duration_ms"]) + + float(attempt["duration_ms"]), + 3, + ) + _finish_physical_execution(retry_entry, attempt) + _write_receipt(receipt_path, receipt) + attempts.append(attempt) + session["task_ids"].append(int(task["task_id"])) + completed.add((engine, int(task["task_id"]))) + receipt["attempts"] = attempts + _write_receipt(receipt_path, receipt) + print( + json.dumps( + { + "task_id": attempt["task_id"], + "engine": attempt["engine"], + "process_status": attempt["process_status"], + "evaluator_status": attempt["evaluator_status"], + } + ), + flush=True, + ) + except BaseException as exc: + primary_error = exc + raise + finally: + _finalize_method_teardown( + client, + run_id, + engine, + method_teardown, + receipt, + receipt_path, + primary_error, + ) + + expected = {(engine, task_id) for engine in engines for task_id in task_ids} + actual = {(str(item["engine"]), int(item["task_id"])) for item in attempts} + if actual != expected or len(attempts) != len(expected): + raise RuntimeError("formal replay did not produce the complete method-task matrix") + if not all(method_teardown.values()): + raise RuntimeError("formal replay did not complete method teardown") + receipt["status"] = "complete" + receipt["attempts"] = attempts + _write_receipt(receipt_path, receipt) + return receipt diff --git a/harness/trajectory_replay/webarena_report.py b/harness/trajectory_replay/webarena_report.py new file mode 100644 index 000000000..dfa691f4d --- /dev/null +++ b/harness/trajectory_replay/webarena_report.py @@ -0,0 +1,1049 @@ +"""Build the evidence-backed WebArena Browser Trajectory Replay report.""" + +from __future__ import annotations + +import hashlib +import html +import json +import math +import re +import shutil +import statistics +from collections import defaultdict +from pathlib import Path +from typing import Any + +from harness.trajectory_replay.webarena_review import ( + load_validated_failure_review, + programmatic_failure, +) + +LOCAL_PATH_RE = re.compile(r"/(?:Users|home|private|tmp|var|opt)/[^\s\"']+") + + +def _e(value: Any) -> str: + return html.escape(str(value if value is not None else "")) + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _stat(values: list[float]) -> dict[str, float | int | None]: + if not values: + return {"n": 0, "mean": None, "p50": None, "p95": None} + ordered = sorted(values) + + def rank(quantile: float) -> float: + return ordered[min(len(ordered) - 1, math.ceil(quantile * len(ordered)) - 1)] + return { + "n": len(values), + "mean": statistics.fmean(values), + "p50": rank(0.5), + "p95": rank(0.95), + } + + +def _clean(value: Any) -> Any: + if isinstance(value, dict): + return { + key: _clean(item) + for key, item in value.items() + if key not in {"path", "paths"} + } + if isinstance(value, list): + return [_clean(item) for item in value] + if isinstance(value, str): + value = LOCAL_PATH_RE.sub("[local path removed]", value) + value = re.sub(r"sk-[A-Za-z0-9_-]{12,}", "[secret removed]", value) + return value + + +def _write_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps(_clean(value), ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + + +def _load_artifact(ref: dict[str, Any] | None) -> Any: + if not ref or not ref.get("path"): + return None + path = Path(str(ref["path"])).expanduser().resolve() + if not path.is_file() or _sha256(path) != ref.get("sha256"): + raise ValueError(f"evidence artifact is missing or changed: {path.name}") + return json.loads(path.read_text(encoding="utf-8")) + + +def _format_ms(value: float | None) -> str: + return "—" if value is None else f"{value / 1000:.1f} 秒" + + +def _format_mib(value: float | None) -> str: + return "—" if value is None else f"{value / 1024 / 1024:.1f} MiB" + + +def _metric_lines( + metric: dict[str, Any], + formatter: Any, + method_id: str, + winners: dict[str, set[str]], + *, + lang: str, +) -> str: + labels = (("mean", "均值"), ("p50", "P50"), ("p95", "P95")) + if lang == "en": + labels = (("mean", "Mean"), ("p50", "P50"), ("p95", "P95")) + return "
    ".join( + f'{label} ' + + _winner(formatter(metric.get(key)), method_id in winners[key], lang=lang) + for key, label in labels + ) + + +def _unique_winners( + rows: list[dict[str, Any]], key: str, *, higher: bool +) -> set[str]: + numeric = [(row["id"], row.get(key)) for row in rows if isinstance(row.get(key), (int, float))] + if not numeric: + return set() + best = (max if higher else min)(value for _, value in numeric) + winners = {name for name, value in numeric if value == best} + return winners if len(winners) == 1 else set() + + +def _nav(lang: str, active: str, *, depth: int, current_route: str = "index.html") -> str: + labels = { + "zh": (("index", "结论"), ("dataset", "评测集"), ("results", "结果分析"), ("environment", "环境与方法"), ("review", "核验记录")), + "en": (("index", "Summary"), ("dataset", "Dataset"), ("results", "Results analysis"), ("environment", "Environment & method"), ("review", "Review history")), + }[lang] + local_prefix = "../" * depth + switch = ("../../" if lang == "en" and depth else "../" if lang == "en" else "../en/" if depth else "en/") + current_route + switch_label = "中文" if lang == "en" else "English" + brand = "Trajectory Replay" if lang == "en" else "轨迹重放" + theme_label = "Toggle theme" if lang == "en" else "切换主题" + links = "".join( + f'{label}" + for slug, label in labels + ) + return f'
    {brand}{switch_label}
    ' + + +def _shell(lang: str, active: str, title: str, body: str, *, depth: int = 0, current_route: str = "index.html") -> str: + prefix = "../" * depth + asset_prefix = prefix if lang == "zh" else "../" + prefix + footer = ( + "WebArena-Verified 1.2.3 · 浏览器轨迹重放" + if lang == "zh" + else "WebArena-Verified 1.2.3 · Browser Trajectory Replay" + ) + preview_bootstrap = r'''''' + return f'''{_e(title)}{preview_bootstrap}{_nav(lang, active, depth=depth, current_route=current_route)}
    {body}
    {footer}
    ''' + + +def _winner(value: str, won: bool, *, lang: str) -> str: + badge = "优" if lang == "zh" else "Best" + return f'{value}{badge}' if won else value + + +def _metrics_table(methods: list[dict[str, Any]], lang: str) -> str: + pass_winners = _unique_winners(methods, "pass_rate", higher=True) + metric_winners = { + metric: { + stat: _unique_winners( + [{"id": row["id"], "value": row[metric][stat]} for row in methods], + "value", + higher=False, + ) + for stat in ("mean", "p50", "p95") + } + for metric in ("duration", "action_latency", "cpu", "memory") + } + headers = ( + ("浏览器", "通过", "通过率", "成功交集端到端耗时", "单次动作耗时", "CPU 时间", "平均内存") + if lang == "zh" + else ("Browser", "Passed", "Pass rate", "End-to-end time on shared passes", "Per-action latency", "CPU time", "Mean memory") + ) + body = [] + + def time_value(value: float | None) -> str: + if lang == "zh": + return _format_ms(value) + return "—" if value is None else f"{value / 1000:.1f} s" + + for row in methods: + key = row["id"] + cells = [ + row["label"], + _winner( + f'{row["passed"]}/{row["total"]}', + key in pass_winners, + lang=lang, + ), + _winner(f'{row["pass_rate"] * 100:.1f}%', key in pass_winners, lang=lang), + _metric_lines( + row["duration"], time_value, key, metric_winners["duration"], lang=lang + ), + _metric_lines( + row["action_latency"], + time_value, + key, + metric_winners["action_latency"], + lang=lang, + ), + _metric_lines(row["cpu"], time_value, key, metric_winners["cpu"], lang=lang), + _metric_lines( + row["memory"], _format_mib, key, metric_winners["memory"], lang=lang + ), + ] + body.append("" + "".join(f"{cell}" for cell in cells) + "") + return f'
    {"".join(f"" for h in headers)}{"".join(body)}
    {_e(h)}
    ' + + +def _resource_table(methods: list[dict[str, Any]], lang: str) -> str: + zh = lang == "zh" + metrics = ( + ("memory_peak", "峰值内存", "Peak memory", _format_mib), + ("network_rx", "接收数据", "Network received", _format_mib), + ("network_tx", "发送数据", "Network sent", _format_mib), + ("cpu_utilization", "CPU 平均使用率", "Mean CPU utilization", lambda value: "—" if value is None else f"{value:.1f}%"), + ) + rows = [] + for key, label_zh, label_en, formatter in metrics: + cells = [_e(label_zh if zh else label_en)] + for method in methods: + cells.append(_metric_lines(method[key], formatter, method["id"], {stat: set() for stat in ("mean", "p50", "p95")}, lang=lang)) + rows.append("" + "".join(f"{cell}" for cell in cells) + "") + headers = ("资源指标" if zh else "Resource metric", *(method["label"] for method in methods)) + return f'
    {"".join(f"" for h in headers)}{"".join(rows)}
    {_e(h)}
    ' + + +def _copy_failure_screenshot(attempt: dict[str, Any], output: Path, key: str) -> str | None: + ref = (attempt.get("evidence") or {}).get("failure_screenshot") + if not ref or not ref.get("path"): + return None + source = Path(str(ref["path"])).expanduser().resolve() + if not source.is_file() or _sha256(source) != ref.get("sha256"): + raise ValueError(f"failure screenshot is missing or changed: {key}") + target = output / "assets/screenshots" / f"{key}.png" + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, target) + return f"assets/screenshots/{key}.png" + + +def _public_evidence( + attempt: dict[str, Any], review: dict[str, Any] | None, output: Path +) -> dict[str, Any]: + key = f'{attempt["engine"]}-{attempt["task_id"]}' + trace = _load_artifact((attempt.get("evidence") or {}).get("trace")) or [] + evaluation = _load_artifact((attempt.get("evidence") or {}).get("evaluation")) + failed = programmatic_failure(attempt) + if failed["stage"] == "official_evaluation" and evaluation: + missing_events = [ + result for result in evaluation.get("evaluators_results") or [] + if result.get("evaluator_name") == "NetworkEventEvaluator" + and any( + assertion.get("assertion_name") == "missing_navigation_event" + for assertion in result.get("assertions") or [] + ) + ] + if missing_events: + assertion_messages = [ + str(message) + for result in missing_events + for assertion in result.get("assertions") or [] + if assertion.get("assertion_name") == "missing_navigation_event" + for message in assertion.get("assertion_msgs") or [] + ] + trigger = next( + ( + step for step in reversed(trace) + if (step.get("arguments") or {}).get("action") in {"click", "press", "evaluate"} + ), + None, + ) + failed = { + "stage": "side_effect_not_observed", + "step": None, + "trigger_step": trigger.get("index") if trigger else None, + "message": "NetworkEventEvaluator: missing_navigation_event" + (" — " + " | ".join(assertion_messages) if assertion_messages else ""), + "expected_network": [result.get("expected") for result in missing_events], + } + screenshot = _copy_failure_screenshot(attempt, output, key) + public = { + "result": { + "engine": attempt["engine"], + "task_id": attempt["task_id"], + "status": attempt["status"], + "action_count": attempt["action_count"], + "completed_action_count": attempt["completed_action_count"], + "duration_ms": attempt["duration_ms"], + "timings": attempt.get("timings") or {}, + "resource": attempt.get("resource") or {}, + }, + "programmatic_failure": failed, + "review": ( + { + "cause_code": review["cause_group"]["code"], + "label_zh": review["cause_group"]["label_zh"], + "label_en": review["cause_group"]["label_en"], + "reason_zh": review["reason_zh"], + "reason_en": review["reason_en"], + "responsibility": review["cause_group"]["responsibility"], + "remediation_owner": review["cause_group"]["remediation_owner"], + "remediation_status": review["cause_group"]["remediation_status"], + } + if review + else None + ), + "trajectory": trace, + "official_evaluation": evaluation, + "screenshot": screenshot, + } + _write_json(output / "data/evidence" / f"{key}.json", public) + return public + + +def _detail_page( + lang: str, + attempt: dict[str, Any], + task: dict[str, Any], + method: dict[str, Any], + evidence: dict[str, Any], + peers: list[dict[str, Any]], +) -> str: + failed = evidence["programmatic_failure"] + review = evidence.get("review") + zh = lang == "zh" + status = ("通过" if attempt["status"] == "pass" else "失败") if zh else attempt["status"].title() + reason = (review.get("reason_zh") if zh else review.get("reason_en")) if review else "" + label = (review.get("label_zh") if zh else review.get("label_en")) if review else "" + key = f'{attempt["engine"]}-{attempt["task_id"]}' + report_root = "../" if zh else "../../" + stage_labels = { + "browser_tool_execution": ("浏览器动作执行", "Browser action execution"), + "side_effect_not_observed": ("提交请求未出现", "Submission request not observed"), + "official_evaluation": ("官方验收", "Official evaluation"), + "pass": ("无", "None"), + } + stage = stage_labels.get(failed["stage"], (failed["stage"], failed["stage"]))[0 if zh else 1] + failed_index = failed.get("step") + trigger_index = failed.get("trigger_step") + failed_action = next((str((step.get("arguments") or {}).get("action") or "") for step in evidence["trajectory"] if step.get("index") == failed_index), "") + peer_cells = [] + for peer in peers: + peer_key = f'{peer["engine"]}-{peer["task_id"]}' + peer_status = ("通过" if peer["status"] == "pass" else "失败") if zh else peer["status"].title() + peer_cells.append( + f'{_e(peer["browser"])}{peer_status}' + ) + actual_by_index = {int(step["index"]): step for step in evidence["trajectory"]} + steps = [] + for number, planned in enumerate(task.get("tool_calls") or [], 1): + actual = actual_by_index.get(number) + planned_target = planned.get("selector") or planned.get("url") or planned.get("key") or "" + actual_arguments = (actual or {}).get("arguments") or {} + actual_target = actual_arguments.get("selector") or actual_arguments.get("url") or actual_arguments.get("key") or "" + actual_result = str((actual or {}).get("result") or "") + preview = actual_result.splitlines()[0][:160] if actual_result else "" + failed_step = failed_index == number + actual_status = str((actual or {}).get("status") or "") + status_text = ( + ("未执行", "Not executed") if actual is None + else ("报错", "Error") if actual_status == "error" + else ("完成", "Completed") + )[0 if zh else 1] + status_class = "status-muted" if actual is None else "status-fail" if actual_status == "error" else "status-pass" + duration = f'{float(actual.get("duration_ms") or 0) / 1000:.2f} s' if actual else "—" + steps.append( + f'
    ' + f'
    {number}
    ' + f'
    {"标准轨迹" if zh else "Qualified input"}{_e(planned.get("action") or "")}' + f'{_e(planned_target)}
    ' + f'
    {"实际执行" if zh else "Actual execution"}
    {_e(actual_arguments.get("action") or "—")}' + f'{status_text}{_e(duration)}
    ' + f'{_e(actual_target) if actual else "—"}' + f'

    {_e(preview) if preview else ("没有工具返回" if zh else "No tool response")}

    ' + f'
    ' + ) + screenshot = ( + f'
    {
    {"失败后保存的最后页面;不等同于错误发生瞬间。点击查看原图。" if zh else "Last page saved after failure, not necessarily the error instant. Open the full image."}
    ' + if evidence.get("screenshot") + else "" + ) + failure_panel = "" + if attempt["status"] != "pass": + step_link = f'{"跳到失败动作" if zh else "Go to failed action"} #{failed_index}' if failed_index else f'{"查看触发动作" if zh else "Inspect trigger action"} #{trigger_index}' if trigger_index else f'{"查看已完成动作" if zh else "View completed actions"}' + action_locator = f'{failed_action} #{failed_index}' if failed_index else ((f'{("触发动作" if zh else "Trigger action")} #{trigger_index}; {("请求未出现" if zh else "request absent")}') if trigger_index else ("所有动作执行后" if zh else "After all actions")) + expected_network = f'

    {"未出现的请求" if zh else "Missing required request"}

    {_e(json.dumps(failed["expected_network"], ensure_ascii=False, indent=2))}
    ' if failed.get("expected_network") else "" + failure_panel = f'''

    {"失败定位与处理" if zh else "Failure diagnosis and treatment"}

    {"失败环节" if zh else "Failure stage"}{_e(stage)}
    {"失败位置" if zh else "Failure location"}{_e(action_locator)}
    {"复核分类" if zh else "Reviewed category"}{_e(label)}

    {"程序原始判定" if zh else "Exact programmatic finding"}

    {_e(failed["message"])}
    {expected_network}

    {"复核判断与依据" if zh else "Reviewed cause and basis"}

    {_e(reason)}

    {"处理方" if zh else "Owner"}: {_e(review.get("remediation_owner") if review else "—")} · {"状态" if zh else "Status"}: {_e(review.get("remediation_status") if review else "—")}

    {step_link}
    ''' + facts = [ + (("完成动作", "Actions completed"), f'{attempt["completed_action_count"]}/{attempt["action_count"]}'), + (("端到端耗时", "End-to-end time"), _format_ms(attempt.get("duration_ms")) if zh else f'{attempt.get("duration_ms", 0) / 1000:.1f} s'), + (("CPU 时间", "CPU time"), _format_ms((attempt.get("resource") or {}).get("cpu_ms")) if zh else f'{((attempt.get("resource") or {}).get("cpu_ms") or 0) / 1000:.1f} s'), + (("平均内存", "Mean memory"), _format_mib((attempt.get("resource") or {}).get("memory_mean_bytes"))), + ] + fact_html = "".join(f'
    {_e(label[0 if zh else 1])}{_e(value)}
    ' for label, value in facts) + viewer_file = ("data/evidence/" if zh else "../data/evidence/") + key + ".json" + body = f''' +

    {_e(key)}

    {_e(method["label"])} · {"任务" if zh else "Task"} #{attempt["task_id"]}

    {_e(task["task"]["intent"])}

    +

    {"同题浏览器结果" if zh else "All browser results for this task"}

    {''.join(peer_cells)}
    +
    {status}{_e(stage) if attempt["status"] != "pass" else ("固定轨迹及官方验收均通过" if zh else "Qualified trajectory and official evaluation passed")}
    {fact_html}
    + {failure_panel} +

    {"标准轨迹与实际执行" if zh else "Qualified trajectory and actual execution"}

    {"左侧是所有浏览器共用的已验证固定输入,右侧是本浏览器的实际操作。未执行不等于动作报错;点击任一步读取完整参数、工具返回、日志和原始报错。" if zh else "The left column is the qualified fixed input shared by all browsers; the right column is this browser's execution. Not executed does not mean an action error. Open a step for exact arguments, tool response, logs and errors."}

    #{"标准轨迹" if zh else "Qualified input"}{"本浏览器实际执行" if zh else "Actual execution in this browser"}
    {''.join(steps)}
    +

    {"标准动作参数" if zh else "Qualified action arguments"}

    {"实际动作参数" if zh else "Actual action arguments"}

    {"工具返回、日志与报错" if zh else "Tool response, logs and error"}

    {"查看完整执行证据" if zh else "Inspect complete execution evidence"}

    +

    {"验收、资源与页面证据" if zh else "Evaluation, resources, and page evidence"}

    {screenshot}
    ''' + title = ( + f"{method['label']} · 任务 #{attempt['task_id']}" + if zh + else f"{method['label']} · Task #{attempt['task_id']}" + ) + return _shell(lang, "results", title, body, depth=1, current_route=f"result/{key}.html") + + +def _task_page(lang: str, task: dict[str, Any], peers: list[dict[str, Any]]) -> str: + zh = lang == "zh" + task_id = int(task["task_id"]) + source_kind = str((task.get("source") or {}).get("kind") or "") + source_label = ( + ("来自已通过的自主浏览器执行", "From a successful autonomous browser run") + if source_kind == "authoritative_successful_browser_use_execution" + else ("修复后在基准浏览器重新验证", "Repaired and re-qualified on the reference browser") + )[0 if zh else 1] + peer_rows = "".join( + f'{_e(peer["browser"])}' + f'{("通过" if peer["status"] == "pass" else "失败") if zh else peer["status"].title()}' + f'{peer["completed_actions"]}/{peer["action_count"]}' + f'{"查看诊断与执行轨迹" if zh else "Diagnosis and executed actions"}' + for peer in peers + ) + planned_rows = [] + for index, call in enumerate(task.get("tool_calls") or [], 1): + action = str(call.get("action") or "") + target = str(call.get("selector") or call.get("url") or call.get("key") or "") + planned_rows.append( + f'{index}{_e(action)}{_e(target)}' + f'
    {"查看完整参数" if zh else "Full arguments"}
    {_e(json.dumps(_clean(call), ensure_ascii=False, indent=2))}
    ' + ) + contract = task.get("evaluation_input") or {} + contract_type = str(contract.get("task_type") or "") + contract_summary = ({"RETRIEVE": "信息提取", "NAVIGATE": "页面导航", "MUTATE": "页面操作"}.get(contract_type, contract_type) if zh else contract_type) or "—" + body = f'''

    WebArena-Verified · #{task_id}

    {"任务定义与标准轨迹" if zh else "Task contract and qualified trajectory"}

    {_e(task["task"]["intent"])}

    +
    {"站点" if zh else "Sites"}{_e(", ".join(task["task"].get("sites") or []))}
    {"轨迹来源" if zh else "Trajectory source"}{source_label}
    {"计划动作" if zh else "Planned actions"}{len(task.get("tool_calls") or [])}
    {"任务类型" if zh else "Task type"}{contract_summary}
    +

    {"同题所有浏览器结果" if zh else "All browser outcomes on this task"}

    {peer_rows}
    {"浏览器" if zh else "Browser"}{"结果" if zh else "Verdict"}{"完成动作" if zh else "Completed actions"}{"证据" if zh else "Evidence"}
    +

    {"冻结的操作轨迹" if zh else "Frozen action trajectory"}

    {"这是所有浏览器共用的输入;下方记录的是预定动作,不是任一浏览器的实测执行结果。" if zh else "The same input is used for every browser. These are planned actions, not any browser's measured execution."}

    {''.join(planned_rows)}
    #{"动作" if zh else "Action"}{"目标" if zh else "Target"}{"参数" if zh else "Arguments"}
    +

    {"验收输入" if zh else "Evaluation input"}

    {"供官方 evaluator 判分;不是 Agent 提示词,也不是唯一正确路径。" if zh else "Used by the official evaluator, not an Agent prompt or the only correct path."}

    {"查看答案与验收字段" if zh else "Inspect answer and acceptance fields"}
    {_e(json.dumps(_clean(contract), ensure_ascii=False, indent=2))}
    ''' + title = f"任务 #{task_id} · 评测集" if zh else f"Task #{task_id} · Dataset" + return _shell(lang, "dataset", title, body, depth=1, current_route=f"task/{task_id}.html") + + +def _assets(output: Path) -> None: + css = r''':root{--bg:#f4f7fb;--panel:#fff;--ink:#102235;--muted:#60748a;--line:#d4e0ec;--accent:#0969da;--good:#087f4f;--bad:#c52c45;--wash:#eaf3ff}html[data-theme=dark]{--bg:#0c1520;--panel:#121f2d;--ink:#eef5fb;--muted:#9db0c2;--line:#2b4054;--accent:#62a8ff;--good:#4ed29b;--bad:#ff7d91;--wash:#142d49}*{box-sizing:border-box}[hidden]{display:none!important}body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.6 Inter,ui-sans-serif,system-ui,-apple-system,"PingFang SC","Microsoft YaHei",sans-serif}.top{height:70px;display:flex;align-items:center;gap:28px;padding:0 max(24px,calc((100% - 1440px)/2));background:var(--panel);border-bottom:1px solid var(--line);position:sticky;top:0;z-index:10}.brand{font-weight:900;color:var(--ink);text-decoration:none;white-space:nowrap}.top nav{display:flex;gap:22px;overflow:auto;flex:1}.top nav a,.language{color:var(--accent);text-decoration:none;white-space:nowrap}.top nav a[aria-current=page]{color:var(--ink);font-weight:800;border-bottom:3px solid var(--ink)}.theme{border:1px solid var(--line);background:var(--panel);color:var(--ink);border-radius:50%;width:38px;height:38px}main,footer{width:min(1440px,calc(100% - 48px));margin:auto}footer{padding:36px 0;color:var(--muted);border-top:1px solid var(--line)}section{padding:34px 0}.hero{padding:46px 0 28px}.hero h1,.detail-hero h1{font-size:clamp(36px,5vw,68px);line-height:1.08;margin:8px 0 14px;letter-spacing:-.04em}.hero p,.detail-hero p{max-width:880px;color:var(--muted);font-size:18px}.eyebrow{text-transform:uppercase;letter-spacing:.14em;color:var(--accent)!important;font-size:13px!important;font-weight:800}h2{font-size:30px;margin:0 0 16px}.table-scroll{overflow:auto;border:1px solid var(--line);border-radius:16px;background:var(--panel)}table{border-collapse:collapse;width:100%;min-width:900px}th,td{text-align:left;vertical-align:top;padding:14px 16px;border-bottom:1px solid var(--line)}th{background:color-mix(in srgb,var(--wash) 70%,var(--panel));font-size:14px;position:sticky;top:0}td{overflow-wrap:anywhere}.metrics td{min-width:150px}.metric-label{display:inline-block;width:42px;color:var(--muted);font-size:13px}.winner{color:var(--good)}.winner>span{display:inline-grid;place-items:center;margin-left:7px;min-width:24px;height:24px;padding:0 6px;border-radius:12px;background:var(--good);color:white;font-size:12px}.callout{padding:16px 18px;border-left:4px solid var(--accent);background:var(--panel);border-radius:0 12px 12px 0}.summary-grid,.diagnosis-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(220px,1fr));gap:14px}.summary-grid article,.diagnosis-grid article{background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:18px;min-height:150px}.summary-grid strong,.diagnosis-grid strong{display:block;font-size:24px;margin:8px 0}.summary-grid span,.diagnosis-grid span{color:var(--muted);font-size:13px}.verdict.fail{border-left:5px solid var(--bad)}.verdict.pass{border-left:5px solid var(--good)}.filters{display:flex;gap:10px;flex-wrap:wrap;margin:16px 0}.filters input,.filters select{padding:10px 12px;border:1px solid var(--line);background:var(--panel);color:var(--ink);border-radius:9px;min-width:170px}.inventory{height:620px;overflow:auto}.inventory table{min-width:1500px}.inventory tbody tr{cursor:pointer}.inventory tbody tr:hover{background:var(--wash)}.status-pass{color:var(--good);font-weight:800}.status-fail{color:var(--bad);font-weight:800}.failed-step{background:color-mix(in srgb,var(--bad) 10%,var(--panel))}.failed-step td:first-child{border-left:4px solid var(--bad)}pre{white-space:pre-wrap;overflow-wrap:anywhere;max-height:420px;overflow:auto;background:#071923;color:#dff4ff;padding:14px;border-radius:10px;font:13px/1.55 ui-monospace,SFMono-Regular,Menlo,monospace}figure{margin:22px 0}figure img{display:block;max-width:100%;border:1px solid var(--line);border-radius:12px}figcaption{color:var(--muted);font-size:13px}.evidence-actions{display:flex;gap:12px;flex-wrap:wrap}.evidence-actions a{padding:10px 14px;border:1px solid var(--line);border-radius:9px;background:var(--panel);color:var(--accent);text-decoration:none}.detail-hero{padding-top:40px}.viewer{height:75vh}.viewer pre{height:100%;max-height:none}dialog{width:min(850px,calc(100% - 32px));border:1px solid var(--line);border-radius:16px;background:var(--panel);color:var(--ink);padding:0}dialog::backdrop{background:#07111bbf}.dialog-head{display:flex;justify-content:space-between;padding:18px;border-bottom:1px solid var(--line)}.dialog-body{padding:18px}.dialog-head button{border:0;background:none;color:var(--ink);font-size:24px}.mermaid{width:100%;max-width:100%;background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:24px;overflow:auto}.flow-fallback{display:flex;align-items:stretch;gap:10px;overflow:auto;padding:18px;background:var(--panel);border:1px solid var(--line);border-radius:14px}.flow-node{min-width:150px;padding:14px;border-radius:10px;background:var(--wash)}.flow-node strong{display:block;color:var(--accent);margin-bottom:6px}.flow-arrow{display:grid;place-items:center;color:var(--muted);font-size:24px}@media(max-width:720px){.top{padding:0 16px;gap:14px}.top nav{gap:14px}main,footer{width:calc(100% - 28px)}section{padding:26px 0}.hero h1,.detail-hero h1{font-size:36px}.inventory{height:560px}.language{display:none}}''' + css += r''' +html{scroll-padding-top:88px}a{color:var(--accent)}button,summary{cursor:pointer} +.top nav a{padding:20px 0 15px}.top nav a:hover{text-decoration:underline} +.hero{padding:54px 0 38px;border-bottom:1px solid var(--line)}.hero h1{max-width:1100px} +.compact-section{padding:22px 0}.detail-hero{padding:40px 0 28px;border-bottom:1px solid var(--line)} +.detail-hero h1{font-size:clamp(34px,4vw,56px)}.task-intent{max-width:1040px!important;color:var(--ink)!important;line-height:1.55} +.detail-links,.evidence-actions{display:flex;gap:12px;flex-wrap:wrap;margin-top:16px} +.detail-links a,.evidence-actions a,.dialog-body .evidence-actions a{display:inline-flex;align-items:center;min-height:42px;padding:9px 13px;border:1px solid var(--line);background:var(--panel);border-radius:9px;text-decoration:none;font-weight:700} +.detail-links a:hover,.evidence-actions a:hover{border-color:var(--accent);background:var(--wash)} +.peer-list{display:flex;gap:12px;overflow:auto;padding:3px 1px}.peer{display:flex;flex-direction:column;gap:3px;min-width:170px;padding:12px 16px;border:1px solid var(--line);border-radius:12px;background:var(--panel);text-decoration:none;color:var(--ink)} +.peer strong{color:var(--good)}.peer.fail strong{color:var(--bad)}.peer[aria-current=page]{border:2px solid var(--accent);background:var(--wash)} +.verdict-line{display:flex;align-items:center;gap:14px;padding:15px 18px;border-radius:12px;background:color-mix(in srgb,var(--good) 8%,var(--panel));border-left:5px solid var(--good)} +.verdict-line.fail{background:color-mix(in srgb,var(--bad) 9%,var(--panel));border-color:var(--bad)}.verdict-line strong{font-size:25px;color:var(--good)}.verdict-line.fail strong{color:var(--bad)} +.fact-strip{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:10px;margin-top:12px}.fact-strip>div{padding:13px 16px;background:var(--panel);border:1px solid var(--line);border-radius:11px;min-height:84px} +.fact-strip span,.diagnosis-facts span{display:block;color:var(--muted);font-size:13px}.fact-strip strong,.diagnosis-facts strong{display:block;margin-top:7px;font-size:16px;line-height:1.35;overflow-wrap:anywhere} +.diagnosis-panel{background:var(--panel);border:1px solid var(--line);border-left:5px solid var(--bad);border-radius:14px;padding:20px 24px;max-width:1120px} +.diagnosis-panel h3{margin:20px 0 8px}.diagnosis-panel p{max-width:1000px}.diagnosis-panel pre,.dialog-body pre{max-height:360px;white-space:pre-wrap} +.diagnosis-facts{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:10px}.diagnosis-facts>div{padding:12px;background:var(--wash);border-radius:10px} +.trace-list{display:grid;grid-template-columns:minmax(0,1fr);gap:9px;max-width:1220px}.trace-step{min-width:0;padding:13px 18px;background:var(--panel);border:1px solid var(--line);border-radius:12px} +.trace-step.failed-step{border:2px solid var(--bad);background:color-mix(in srgb,var(--bad) 6%,var(--panel))}.step-heading{display:flex;align-items:center;gap:12px;flex-wrap:wrap} +.step-number{display:grid;place-items:center;width:29px;height:29px;background:var(--wash);border-radius:50%;font-weight:800}.step-heading strong{min-width:90px}.step-heading>span:last-child{margin-left:auto;color:var(--muted)} +.step-target{margin:7px 0 0 41px;color:var(--muted);font:13px/1.55 ui-monospace,SFMono-Regular,monospace;overflow-wrap:anywhere}.step-preview{margin:6px 0 0 41px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap} +.trace-step details{margin:10px 0 0 41px}.trace-step summary,.matrix-disclosure summary{color:var(--accent);font-weight:700}.step-data{display:grid;grid-template-columns:minmax(260px,1fr) minmax(340px,2fr);gap:12px;min-width:0}.step-data>div{min-width:0}.step-data pre{max-width:100%;max-height:380px} +.dataset-table table{min-width:600px}.dataset-table td,.dataset-table th{overflow-wrap:normal;white-space:nowrap}.task-peer-table{min-width:650px}.planned-table{min-width:900px}.planned-table td:nth-child(3){min-width:320px;max-width:500px;overflow-wrap:anywhere} +.matrix-disclosure{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:14px 18px}.matrix-disclosure[open] summary{margin-bottom:13px}.matrix-table{max-height:570px}.matrix-table table{min-width:700px} +.filters{align-items:end;gap:12px}.filters label{display:flex;flex-direction:column;gap:5px;font-size:13px;font-weight:700;color:var(--muted)}.filters input,.filters select{min-width:140px;max-width:260px;font-size:15px} +.inventory{height:610px}.inventory table{min-width:1060px}.inventory th,.inventory td{white-space:nowrap;overflow-wrap:normal;vertical-align:middle}.inventory th:nth-child(7),.inventory td:nth-child(7){min-width:230px;max-width:280px;white-space:normal;overflow-wrap:anywhere}.inventory tbody tr:hover{background:var(--wash)} +.inventory button{border:1px solid var(--accent);color:var(--accent);background:var(--panel);padding:6px 10px;border-radius:7px;font-weight:700;margin-left:10px}.inventory button:hover{background:var(--wash)}.result-count{color:var(--muted);font-size:14px} +dialog{max-height:90vh;overflow:auto}.dialog-head{position:sticky;top:0;background:var(--panel);z-index:1}.dialog-body{padding:20px 24px}.dialog-body h3{margin:18px 0 8px}.dialog-body .evidence-actions{margin-bottom:4px} +.viewer-tabs{display:flex;gap:8px;flex-wrap:wrap;margin:14px 0}.viewer-tabs button{padding:9px 13px;border:1px solid var(--line);background:var(--panel);color:var(--accent);border-radius:8px}.viewer-tabs button[aria-selected=true]{background:var(--accent);color:white}.viewer{height:auto;min-height:50vh}.viewer pre{min-height:45vh;max-height:65vh} +@media(max-width:720px){.hero{padding:30px 0 24px}.top{height:60px}.top nav a{padding:15px 0}.detail-hero h1{font-size:34px}.step-data{grid-template-columns:minmax(0,1fr)}.step-heading>span:last-child{margin-left:0}.trace-step details,.step-target,.step-preview{margin-left:0}.filters label{flex:1 1 145px}.filters input,.filters select{min-width:0;width:100%}.diagnosis-panel{padding:16px}.fact-strip{grid-template-columns:repeat(2,minmax(0,1fr))}} +:root{--bg:#f6f8fb;--ink:#17212b;--muted:#5d6b78;--line:#d9e2ea;--accent:#1268d8;--good:#117a47;--bad:#c22e43;--wash:#e9f2ff;--panel2:#eef3f8;--shadow:0 8px 28px rgba(30,55,80,.08)}html[data-theme=dark]{--bg:#071015;--panel:#0d1a21;--ink:#edf6f7;--muted:#9aadb4;--line:#263c46;--accent:#49d6ed;--good:#67df9f;--bad:#ff6b7a;--wash:#10303a;--panel2:#122630;--shadow:none} +body{background:linear-gradient(180deg,var(--wash) 0,transparent 240px),var(--bg);font-size:15px}.top{height:58px;gap:17px;padding:0 max(20px,calc((100% - 1180px)/2));background:color-mix(in srgb,var(--panel) 92%,transparent);backdrop-filter:blur(14px)}.top nav{gap:17px}.top nav a,.language{font-size:14px}.theme{border-radius:999px;width:auto;height:auto;padding:6px 12px}main,footer{width:min(1180px,calc(100% - 40px))}section{padding:38px 0;border-top:1px solid var(--line)}.hero{padding:38px 0 30px;border:0}.hero h1{font-size:clamp(34px,5vw,58px)}h2{font-size:clamp(25px,3vw,36px);line-height:1.16}.table-scroll{border-radius:13px;box-shadow:var(--shadow)}th{background:var(--panel2);font-size:12px;white-space:nowrap}tbody tr:hover{background:var(--wash)} +.method-comparison{padding:24px;border:1px solid var(--line);border-radius:18px;background:var(--panel);box-shadow:var(--shadow)}.comparison-heading{display:flex;justify-content:space-between;align-items:start;gap:24px;margin-bottom:20px}.comparison-heading h2{margin-bottom:6px;font-size:25px}.comparison-heading p{margin:0;color:var(--muted)}.comparison-count{white-space:nowrap;padding:7px 11px;border-radius:999px;background:var(--wash);color:var(--accent);font-size:13px}.comparison-controls{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:14px}.comparison-controls label{display:grid;gap:5px;font-size:13px;font-weight:700}.comparison-controls select{width:100%;padding:10px 12px;border:1px solid var(--line);background:var(--bg);color:var(--ink);border-radius:9px;font:inherit}.comparison-methods{border:0;padding:0;margin:18px 0}.comparison-methods legend{font-weight:700}.method-checks,.outcome-list{display:flex;gap:7px;flex-wrap:wrap}.compare-chip{display:flex;align-items:center;gap:6px;padding:7px 11px;border:1px solid var(--line);border-radius:999px;background:var(--bg);cursor:pointer}.compare-chip.disabled{opacity:.45;cursor:not-allowed}.compare-chip input{accent-color:var(--accent)}.outcome-pill{display:inline-block;padding:5px 9px;border-radius:8px;font-size:13px;white-space:nowrap;text-decoration:none}.outcome-pill.pass{background:color-mix(in srgb,var(--good) 12%,var(--panel));color:var(--good)}.outcome-pill.fail{background:color-mix(in srgb,var(--bad) 12%,var(--panel));color:var(--bad)}.matrix-table table{min-width:760px}.matrix-table th:nth-child(n+3),.matrix-table td:nth-child(n+3){min-width:270px}.matrix-table td{vertical-align:middle;overflow-wrap:normal} +.dataset-browser{background:var(--panel);border:1px solid var(--line);border-radius:18px;padding:24px;box-shadow:var(--shadow)}.dataset-browser h2{margin-bottom:4px}.dataset-browser>p{margin:0;color:var(--muted)}.dataset-controls{display:grid;grid-template-columns:minmax(220px,2fr) repeat(3,minmax(145px,1fr));gap:12px;margin:22px 0 10px}.dataset-controls label{display:grid;gap:5px;color:var(--muted);font-size:13px;font-weight:700}.dataset-controls input,.dataset-controls select{width:100%;padding:10px 12px;border:1px solid var(--line);background:var(--bg);color:var(--ink);border-radius:9px;font:inherit}.dataset-count{color:var(--accent);font-weight:700;margin:10px 0}.dataset-table{height:590px}.dataset-table table{min-width:950px}.dataset-table td,.dataset-table th{vertical-align:middle;white-space:normal;overflow-wrap:normal}.dataset-table td:nth-child(2){min-width:190px}.dataset-table td:nth-child(3){min-width:330px;max-width:460px}.intent-preview{display:-webkit-box;-webkit-line-clamp:2;-webkit-box-orient:vertical;overflow:hidden;line-height:1.45}.dataset-table td:last-child{white-space:nowrap}.dataset-table a{font-weight:700}.dataset-actions{display:flex;gap:10px;flex-wrap:wrap}.dataset-actions a{white-space:nowrap}.dataset-table tbody tr:hover{background:var(--wash)}.dataset-empty{color:var(--muted);padding:20px 0} +.mermaid svg{min-width:850px;height:auto!important}.flow-fallback{font-size:14px}.flow-node{min-width:170px} +.review-table table{min-width:1120px}.review-table td:first-child{min-width:145px;font-weight:700}.review-table td:nth-child(n+2){min-width:230px;max-width:320px}.review-table td{overflow-wrap:anywhere} +.trajectory-columns,.aligned-step{display:grid;grid-template-columns:48px minmax(0,1fr) minmax(0,1fr);gap:12px;align-items:start} +.trajectory-columns{padding:0 18px 10px;color:var(--muted);font-size:13px}.aligned-step{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:15px 18px} +.aligned-step.failed-step{border:2px solid var(--bad);background:color-mix(in srgb,var(--bad) 5%,var(--panel))}.aligned-standard,.aligned-actual{min-width:0;display:grid;gap:7px}.aligned-standard{border-right:1px solid var(--line);padding-right:12px}.aligned-step code{color:var(--muted);font:13px/1.45 ui-monospace,SFMono-Regular,monospace;overflow-wrap:anywhere}.aligned-step button{justify-self:start;border:0;background:transparent;color:var(--accent);font:inherit;font-weight:700;padding:4px 0;text-align:left}.aligned-actual-head{display:flex;align-items:center;gap:10px;flex-wrap:wrap}.aligned-actual-head span:last-child{margin-left:auto;color:var(--muted)}.aligned-actual p{margin:0;color:var(--muted);overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.status-muted{color:var(--muted)} +.mobile-step-label{display:none;color:var(--muted);font-size:12px}.step-dialog-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:14px}.step-dialog-grid>div{min-width:0}.step-dialog-grid pre{max-height:250px}.step-dialog-grid h3{margin-top:8px}.step-dialog-grid pre,[data-step-result],[data-step-error]{max-width:100%} +@media(max-width:760px){.top{padding:0 14px}.comparison-heading{display:block}.comparison-count{display:inline-block;margin-top:10px}.comparison-controls{grid-template-columns:1fr}.method-comparison{padding:16px}main,footer{width:calc(100% - 28px)}} +@media(max-width:760px){.dataset-browser{padding:16px}.dataset-controls{grid-template-columns:repeat(2,minmax(0,1fr))}.dataset-controls label:first-child{grid-column:1/-1}.dataset-table{height:510px}} +@media(max-width:760px){.trajectory-columns{display:none}.aligned-step{grid-template-columns:34px minmax(0,1fr);padding:12px}.aligned-index{grid-row:1/3}.aligned-standard{border-right:0;border-bottom:1px solid var(--line);padding:0 0 10px}.mobile-step-label{display:block}.step-dialog-grid{grid-template-columns:1fr}} +''' + js = r'''const lang=document.documentElement.lang; +const zh=lang==='zh'; +const base=zh?'':'../'; +document.documentElement.dataset.theme=localStorage.getItem('report-theme')||'light'; +document.querySelector('.theme')?.addEventListener('click',()=>{ + const root=document.documentElement; + root.dataset.theme=root.dataset.theme==='dark'?'light':'dark'; + localStorage.setItem('report-theme',root.dataset.theme); +}); +const fetchJson=async path=>{const response=await fetch(path);if(!response.ok)throw Error(`HTTP ${response.status}`);return response.json()}; +const datasetTable=document.querySelector('[data-dataset-table]'); +if(datasetTable){ + const rows=[...datasetTable.querySelectorAll('tbody tr')]; + const controls=[...document.querySelectorAll('[data-dataset-filter]')]; + const params=new URLSearchParams(location.search); + controls.forEach(control=>{const value=params.get(control.dataset.datasetFilter);if(value!==null)control.value=value}); + const applyDataset=()=>{ + const query=(document.querySelector('[data-dataset-filter=q]')?.value||'').trim().toLocaleLowerCase(); + let shown=0; + rows.forEach(row=>{ + const matches=controls.every(control=>!control.value||control.dataset.datasetFilter==='q'||(control.dataset.datasetFilter==='site'?(row.dataset.sites||'').split(' ').includes(control.value):row.dataset[control.dataset.datasetFilter]===control.value))&&(!query||(row.dataset.search||row.textContent).toLocaleLowerCase().includes(query)); + row.hidden=!matches;if(matches)shown++; + }); + document.querySelector('[data-dataset-count]').textContent=zh?`显示 ${shown} / ${rows.length} 个任务`:`Showing ${shown} / ${rows.length} tasks`; + document.querySelector('[data-dataset-empty]').hidden=shown!==0; + const next=new URLSearchParams();controls.forEach(control=>{if(control.value)next.set(control.dataset.datasetFilter,control.value)}); + history.replaceState(null,'',location.pathname+(next.size?'?'+next:'')); + }; + controls.forEach(control=>control.addEventListener(control.tagName==='INPUT'?'input':'change',applyDataset)); + applyDataset(); +} +const matrix=document.querySelector('[data-task-matrix]'); +if(matrix){ + const matrixRows=[...matrix.querySelectorAll('tbody tr')]; + const relation=document.querySelector('[data-matrix-relation]'); + const site=document.querySelector('[data-matrix-site]'); + const baseline=document.querySelector('[data-matrix-baseline]'); + const boxes=[...document.querySelectorAll('[data-matrix-method]')]; + const count=document.querySelector('[data-matrix-count]'); + const applyMatrix=()=>{ + boxes.forEach(box=>{box.disabled=box.value===baseline.value;box.closest('label')?.classList.toggle('disabled',box.disabled);if(box.disabled)box.checked=false}); + const others=boxes.filter(box=>box.checked&&!box.disabled).map(box=>box.value); + const selected=[baseline.value,...others]; + let shown=0; + matrixRows.forEach(row=>{ + const pass=new Set((row.dataset.pass||'').split(' ').filter(Boolean)); + const fail=new Set((row.dataset.fail||'').split(' ').filter(Boolean)); + row.querySelectorAll('a[data-engine]').forEach(link=>{link.hidden=!selected.includes(link.dataset.engine)}); + const matchRelation=others.length>0&&(!relation.value||(relation.value==='different'&&selected.some(engine=>pass.has(engine))&&selected.some(engine=>fail.has(engine)))||(relation.value==='baseline-pass-others-fail'&&pass.has(baseline.value)&&others.every(engine=>fail.has(engine)))||(relation.value==='baseline-fail-others-pass'&&fail.has(baseline.value)&&others.every(engine=>pass.has(engine)))||(relation.value==='all-pass'&&selected.every(engine=>pass.has(engine)))||(relation.value==='all-fail'&&selected.every(engine=>fail.has(engine)))||(relation.value==='exact-pass'&&selected.every(engine=>pass.has(engine))&&pass.size===selected.length)); + const matchSite=!site.value||row.dataset.site===site.value; + row.hidden=!(matchRelation&&matchSite); + if(!row.hidden)shown++; + }); + count.textContent=others.length?(zh?`找到 ${shown} / ${matrixRows.length} 个任务`:`Showing ${shown} / ${matrixRows.length} tasks`):(zh?'请至少选择一个对比浏览器':'Select at least one comparison browser'); + }; + baseline.addEventListener('change',()=>{boxes.forEach(box=>box.checked=box.value!==baseline.value);applyMatrix()}); + [...boxes,relation,site].forEach(control=>control.addEventListener('change',applyMatrix)); + applyMatrix(); +} +const inventory=document.querySelector('[data-results]'); +if(inventory){ + const rows=[...inventory.querySelectorAll('tbody tr')]; + const controls=[...document.querySelectorAll('[data-filter]')]; + const dialog=document.querySelector('#diagnosis-dialog'); + const params=new URLSearchParams(location.search); + let byKey={}; + controls.forEach(control=>{ + const selected=params.get(control.dataset.filter); + if(selected!==null)control.value=selected; + }); + function applyFilters(){ + const query=(document.querySelector('[data-filter=q]')?.value||'').toLowerCase(); + const task=params.get('task'); + let shown=0; + rows.forEach(row=>{ + const result=byKey[row.dataset.key]; + const haystack=(row.textContent+' '+(result?.intent||'')+' '+(result?.program_error||'')+' '+(result?.reason_zh||'')+' '+(result?.reason_en||'')).toLowerCase(); + const matches=controls.every(control=>!control.value||control.dataset.filter==='q'||row.dataset[control.dataset.filter]===control.value)&&(!task||row.dataset.task===task)&&(!query||haystack.includes(query)); + row.hidden=!matches; + if(matches)shown++; + }); + const counter=document.querySelector('[data-result-count]'); + if(counter)counter.textContent=zh?`显示 ${shown} / ${rows.length} 条结果`:`Showing ${shown} / ${rows.length} results`; + } + controls.forEach(control=>control.addEventListener('input',applyFilters)); + fetchJson(base+'data/results.json').then(data=>{byKey=Object.fromEntries(data.map(item=>[item.key,item]));inventory.dataset.resultsReady='true';applyFilters()}).catch(()=>applyFilters()); + inventory.addEventListener('click',async event=>{ + if(event.target.closest('a'))return; + const row=event.target.closest('tr[data-key]'); + if(!row||!dialog)return; + const result=byKey[row.dataset.key]; + if(!result)return; + const detail='result/'+result.key+'.html'; + dialog.querySelector('[data-dialog-title]').textContent=result.browser+' · '+(zh?'任务':'Task')+' #'+result.task_id; + dialog.querySelector('[data-dialog-stage]').textContent=result.failure_stage==='browser_tool_execution'?(zh?'浏览器动作':'Browser action'):result.failure_stage==='side_effect_not_observed'?(zh?'提交请求未出现':'Submission request absent'):result.failure_stage==='official_evaluation'?(zh?'官方验收':'Official evaluation'):(zh?'无':'None'); + dialog.querySelector('[data-dialog-error]').textContent=zh?'正在读取完整证据…':'Loading complete evidence…'; + delete dialog.dataset.loaded; + dialog.querySelector('[data-dialog-link]').href=detail; + dialog.querySelector('[data-dialog-step-link]').href=detail+'#trajectory'; + dialog.showModal(); + try{ + const evidence=await fetchJson(base+'data/evidence/'+result.key+'.json'); + const step=evidence.programmatic_failure?.step; + const trigger=evidence.programmatic_failure?.trigger_step; + const failedAction=(evidence.trajectory||[]).find(action=>action.index===step); + dialog.querySelector('[data-dialog-step]').textContent=step?(failedAction?.arguments?.action||'')+' #'+step:trigger?(zh?'触发动作':'Trigger action')+' #'+trigger:(zh?'动作完成后':'After actions completed'); + dialog.querySelector('[data-dialog-step-link]').href=detail+(step?'#step-'+step:trigger?'#step-'+trigger:'#trajectory'); + dialog.querySelector('[data-dialog-category]').textContent=(zh?evidence.review?.label_zh:evidence.review?.label_en)||(zh?'未复核':'Not reviewed'); + const exact=step?(failedAction?.result||evidence.programmatic_failure?.message||''):evidence.programmatic_failure?.stage==='side_effect_not_observed'?(evidence.programmatic_failure.message+'\n\nExpected network event:\n'+JSON.stringify(evidence.programmatic_failure.expected_network,null,2)):evidence.official_evaluation?JSON.stringify(evidence.official_evaluation,null,2):(evidence.programmatic_failure?.message||''); + dialog.querySelector('[data-dialog-error]').textContent=exact; + dialog.querySelector('[data-dialog-cause]').textContent=(zh?evidence.review?.reason_zh:evidence.review?.reason_en)||(zh?'尚无复核判断':'No reviewed cause'); + dialog.querySelector('[data-dialog-remediation]').textContent=(zh?'处理方:':'Owner: ')+(evidence.review?.remediation_owner||'—')+' · '+(zh?'状态:':'Status: ')+(evidence.review?.remediation_status||'—'); + dialog.dataset.loaded='true'; + }catch(error){dialog.querySelector('[data-dialog-error]').textContent=(zh?'证据读取失败:':'Evidence loading failed: ')+error.message;dialog.dataset.loaded='error'} + }); + dialog?.querySelector('.dialog-head button')?.addEventListener('click',()=>dialog.close()); +} +const stepDialog=document.querySelector('#step-dialog'); +if(stepDialog){ + let requestId=0; + let evidencePromise; + let standardPromise; + const evidence=()=>evidencePromise??=fetchJson(stepDialog.dataset.evidenceUrl); + const standard=()=>standardPromise??=fetchJson(stepDialog.dataset.standardUrl); + document.querySelector('#trajectory')?.addEventListener('click',async event=>{ + const button=event.target.closest('button[data-open-step]'); + if(!button)return; + const index=Number(button.dataset.openStep); + const current=++requestId; + stepDialog.querySelector('[data-step-title]').textContent=(zh?'第 ':'Step ')+index+(zh?' 步':''); + stepDialog.querySelector('[data-step-status]').textContent=zh?'正在读取逐步证据…':'Loading step evidence…'; + for(const key of ['planned','actual','result','error'])stepDialog.querySelector('[data-step-'+key+']').textContent=''; + stepDialog.querySelector('[data-step-finding]').hidden=true; + stepDialog.querySelector('[data-step-image-wrap]').hidden=true; + if(!stepDialog.open)stepDialog.showModal(); + try{ + const [frozen,measured]=await Promise.all([standard(),evidence()]); + if(current!==requestId||!stepDialog.open)return; + const planned=frozen.tool_calls[index-1]; + const actual=(measured.trajectory||[]).find(item=>item.index===index); + if(!planned)throw Error(zh?'标准轨迹缺少该步':'Qualified trajectory is missing this step'); + stepDialog.querySelector('[data-step-planned]').textContent=JSON.stringify(planned,null,2); + stepDialog.querySelector('[data-step-actual]').textContent=actual?JSON.stringify(actual.arguments||{},null,2):(zh?'未执行':'Not executed'); + stepDialog.querySelector('[data-step-result]').textContent=actual?(actual.result||''):(zh?'该步没有工具调用或返回。':'No tool call or response for this step.'); + stepDialog.querySelector('[data-step-status]').textContent=actual + ?(zh?'实际状态:':'Actual status: ')+(actual.status||'—')+' · '+((Number(actual.duration_ms)||0)/1000).toFixed(2)+' s' + :(zh?'实际状态:未执行':'Actual status: not executed'); + const finding=measured.programmatic_failure; + if(finding&&(finding.step===index||finding.trigger_step===index)){ + stepDialog.querySelector('[data-step-finding]').hidden=false; + stepDialog.querySelector('[data-step-error]').textContent=finding.message||''; + if(finding.expected_network)stepDialog.querySelector('[data-step-error]').textContent+='\n\n'+JSON.stringify(finding.expected_network,null,2); + } + if(measured.screenshot){ + stepDialog.querySelector('[data-step-image-wrap]').hidden=false; + stepDialog.querySelector('[data-step-image]').href=(zh?'../':'../../')+measured.screenshot; + } + }catch(error){ + if(current!==requestId||!stepDialog.open)return; + stepDialog.querySelector('[data-step-status]').textContent=(zh?'逐步证据读取失败:':'Could not load step evidence: ')+error.message; + evidencePromise=undefined;standardPromise=undefined; + } + }); + stepDialog.querySelector('[data-close-step]')?.addEventListener('click',()=>stepDialog.close()); + stepDialog.addEventListener('close',()=>{requestId++}); +} +const viewer=document.querySelector('[data-evidence-viewer]'); +if(viewer){ + const selected=new URLSearchParams(location.search).get('file'); + const safe=/^(?:\.\.\/)?data\/evidence\/[a-z0-9.-]+\.json$/.test(selected||''); + const tabs=[...document.querySelectorAll('[data-evidence-tab]')]; + const heading=document.querySelector('[data-evidence-heading]'); + const raw=document.querySelector('[data-evidence-raw]'); + const image=document.querySelector('[data-evidence-image]'); + const render=(evidence,section)=>{ + const labels={result:zh?'结果与资源':'Result and resources',programmatic_failure:zh?'程序定位':'Programmatic finding',review:zh?'复核原因':'Reviewed cause',trajectory:zh?'执行轨迹与完整日志':'Trajectory and full logs',official_evaluation:zh?'官方验收':'Official evaluation'}; + heading.textContent=labels[section]||section; + viewer.textContent=JSON.stringify(evidence[section]??null,null,2); + tabs.forEach(tab=>tab.setAttribute('aria-selected',String(tab.dataset.evidenceTab===section))); + }; + if(!safe){viewer.textContent=zh?'没有选中证据文件。':'No evidence file selected.'} + else fetchJson(selected).then(evidence=>{ + raw.href=selected; + if(evidence.screenshot){image.href=base+evidence.screenshot;image.textContent=zh?'查看最后页面原图':'Open last observed page';image.hidden=false} + tabs.forEach(tab=>tab.addEventListener('click',()=>render(evidence,tab.dataset.evidenceTab))); + render(evidence,'result'); + }).catch(error=>viewer.textContent=(zh?'证据读取失败:':'Evidence loading failed: ')+error.message); +}''' + (output / "assets").mkdir(parents=True, exist_ok=True) + (output / "assets/report.css").write_text(css, encoding="utf-8") + (output / "assets/app.js").write_text(js, encoding="utf-8") + + +def build_webarena_replay_report( + *, + results_path: str | Path, + dataset_path: str | Path, + review_path: str | Path, + profile_path: str | Path, + output_dir: str | Path, +) -> dict[str, Any]: + validated = load_validated_failure_review(results_path, review_path) + results = validated["results"] + reviews = validated["by_result"] + dataset = json.loads(Path(dataset_path).read_text(encoding="utf-8")) + profile = json.loads(Path(profile_path).read_text(encoding="utf-8")) + output = Path(output_dir).resolve() + if output.exists(): + shutil.rmtree(output) + output.mkdir(parents=True) + _assets(output) + + tasks = {int(item["task_id"]): item for item in dataset["trajectories"]} + for task_id, task in tasks.items(): + _write_json(output / "data/trajectory" / f"{task_id}.json", { + "task_id": task_id, + "tool_calls": task.get("tool_calls") or [], + }) + methods = {item["engine"]: item for item in profile["methods"]} + attempts = results["attempts"] + if {(item["engine"], int(item["task_id"])) for item in attempts} != { + (engine, task_id) for engine in methods for task_id in tasks + }: + raise ValueError("report requires the complete participant-task matrix") + + action_by_engine: dict[str, list[float]] = defaultdict(list) + action_by_name: dict[str, dict[str, list[float]]] = defaultdict(lambda: defaultdict(list)) + for item in results["physical_executions"]: + key = (str(item["engine"]), int(item["task_id"])) + trace = _load_artifact((item.get("evidence") or {}).get("trace")) or [] + for step in trace: + value = step.get("duration_ms") + if isinstance(value, (int, float)): + action_by_engine[key[0]].append(float(value)) + action = str((step.get("arguments") or {}).get("action") or "unknown") + action_by_name[action][key[0]].append(float(value)) + + passing_tasks = { + task_id + for task_id in tasks + if all( + next( + item for item in attempts if item["engine"] == engine and item["task_id"] == task_id + )["status"] + == "pass" + for engine in methods + ) + } + method_rows = [] + for engine, method in methods.items(): + rows = [item for item in attempts if item["engine"] == engine] + passes = sum(item["status"] == "pass" for item in rows) + duration = _stat( + [float(item["duration_ms"]) for item in rows if item["task_id"] in passing_tasks] + ) + action_latency = _stat(action_by_engine[engine]) + comparable_rows = [item for item in rows if item["task_id"] in passing_tasks] + cpu = _stat( + [float((item.get("resource") or {}).get("cpu_ms") or 0) for item in comparable_rows] + ) + memory = _stat( + [ + float((item.get("resource") or {}).get("memory_mean_bytes") or 0) + for item in comparable_rows + ] + ) + memory_peak = _stat([float((item.get("resource") or {}).get("memory_peak_bytes") or 0) for item in comparable_rows]) + network_rx = _stat([float((item.get("resource") or {}).get("network_rx_bytes") or 0) for item in comparable_rows]) + network_tx = _stat([float((item.get("resource") or {}).get("network_tx_bytes") or 0) for item in comparable_rows]) + cpu_utilization = _stat([float((item.get("resource") or {}).get("cpu_average_utilization_pct") or 0) for item in comparable_rows]) + method_rows.append( + { + "id": method["id"], + "engine": engine, + "label": method["label"], + "passed": passes, + "total": len(rows), + "pass_rate": passes / len(rows), + "duration": duration, + "duration_mean": duration["mean"], + "action_latency": action_latency, + "action_mean": action_latency["mean"], + "cpu": cpu, + "cpu_mean": cpu["mean"], + "memory": memory, + "memory_mean": memory["mean"], + "memory_peak": memory_peak, + "network_rx": network_rx, + "network_tx": network_tx, + "cpu_utilization": cpu_utilization, + "physical_executions": len( + [item for item in results["physical_executions"] if item["engine"] == engine] + ), + } + ) + + public_rows = [] + public_evidence: dict[tuple[str, int], dict[str, Any]] = {} + for attempt in attempts: + key = (attempt["engine"], int(attempt["task_id"])) + task = tasks[key[1]] + review = reviews.get(key) + evidence = _public_evidence(attempt, review, output) + public_evidence[key] = evidence + failure = evidence["programmatic_failure"] + cause = review["cause_group"] if review else {} + row = { + "key": f"{key[0]}-{key[1]}", + "engine": key[0], + "browser": methods[key[0]]["label"], + "task_id": key[1], + "intent": task["task"]["intent"], + "sites": task["task"].get("sites") or [], + "status": attempt["status"], + "completed_actions": attempt["completed_action_count"], + "action_count": attempt["action_count"], + "failure_stage": failure["stage"], + "program_error": failure["message"], + "cause_zh": cause.get("label_zh", ""), + "cause_en": cause.get("label_en", ""), + "reason_zh": review.get("reason_zh", "") if review else "", + "reason_en": review.get("reason_en", "") if review else "", + } + public_rows.append(row) + rows_by_task = defaultdict(list) + for row in public_rows: + rows_by_task[int(row["task_id"])].append(row) + for attempt in attempts: + key = (str(attempt["engine"]), int(attempt["task_id"])) + row = next(item for item in rows_by_task[key[1]] if item["engine"] == key[0]) + for lang in ("zh", "en"): + target = output / ("result" if lang == "zh" else "en/result") / f"{row['key']}.html" + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text( + _detail_page(lang, attempt, tasks[key[1]], methods[key[0]], public_evidence[key], rows_by_task[key[1]]), encoding="utf-8" + ) + for task_id, task in tasks.items(): + for lang in ("zh", "en"): + target = output / ("task" if lang == "zh" else "en/task") / f"{task_id}.html" + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(_task_page(lang, task, rows_by_task[task_id]), encoding="utf-8") + + _write_json(output / "data/results.json", public_rows) + _write_json(output / "data/metrics.json", {"methods": method_rows, "passing_intersection": len(passing_tasks)}) + + source_counts: dict[str, int] = defaultdict(int) + for task in tasks.values(): + source_counts[str((task.get("source") or {}).get("kind") or "unknown")] += 1 + browser_use_source_count = source_counts["authoritative_successful_browser_use_execution"] + repaired_source_count = source_counts["expert_repair_spec"] + reference_engine = str((profile.get("qualification") or {}).get("reference_engine") or "") + reference_method = methods.get(reference_engine) + reference_row = next( + (item for item in method_rows if item["engine"] == reference_engine), + None, + ) + if reference_method is None or reference_row is None: + raise ValueError("profile qualification reference engine must be a registered participant") + reference_label = _e(reference_method["label"]) + reviewed_failure_count = len(reviews) + + for lang in ("zh", "en"): + zh = lang == "zh" + root = output if zh else output / "en" + root.mkdir(exist_ok=True) + action_headers = ["动作" if zh else "Action", *[item["label"] for item in methods.values()]] + action_rows = [] + for action in sorted(action_by_name): + action_metrics = { + engine: _stat(action_by_name[action][engine]) for engine in methods + } + action_winners = { + stat: _unique_winners( + [ + {"id": methods[engine]["id"], "value": metric[stat]} + for engine, metric in action_metrics.items() + ], + "value", + higher=False, + ) + for stat in ("mean", "p50", "p95") + } + cells = [] + for engine in methods: + metric = action_metrics[engine] + text = _metric_lines( + metric, + ( + lambda value, is_zh=zh: _format_ms(value) + if is_zh + else ("—" if value is None else f"{value / 1000:.1f} s") + ), + methods[engine]["id"], + action_winners, + lang=lang, + ) + calls_label = "次调用" if zh else "calls" + cells.append(f"{text}
    {metric['n']} {calls_label}") + action_rows.append(f'{_e(action)}{"".join(cells)}') + action_table = f'
    {"".join(f"" for h in action_headers)}{"".join(action_rows)}
    {_e(h)}
    ' + task_count = len(tasks) + shared_count = len(passing_tasks) + eyebrow = ( + f"WebArena-Verified · {task_count} 个任务 · 模型调用 0 次" + if zh + else f"WebArena-Verified · {task_count} tasks · model calls 0" + ) + stage_rows = [] + for engine, method in methods.items(): + for code, label_pair in (("browser_tool_execution", ("浏览器动作", "Browser action")), ("side_effect_not_observed", ("提交请求未出现", "Submission request absent")), ("official_evaluation", ("官方验收", "Official evaluation"))): + count = sum(row["engine"] == engine and row["status"] != "pass" and row["failure_stage"] == code for row in public_rows) + if count: + stage_rows.append(f'{_e(method["label"])}{label_pair[0 if zh else 1]}{count}{"下钻到失败结果" if zh else "Inspect failures"} →') + stage_table = f'
    {"".join(stage_rows)}
    {"浏览器" if zh else "Browser"}{"失败环节" if zh else "Failure stage"}{"结果数" if zh else "Outcomes"}{"逐题证据" if zh else "Per-task evidence"}
    ' + summary_body = f'''

    {eyebrow}

    {"浏览器轨迹重放评测" if zh else "Browser Trajectory Replay"}

    {"90 条经过资格验证的固定操作轨迹依次在全部注册浏览器上执行。Agent 与模型不参与;每题重置站点,浏览器方法之间完全销毁环境。" if zh else "The same 90 qualified fixed action trajectories ran serially on every registered browser. No Agent or model participated. Sites were reset per task and torn down between browser methods."}

    +

    {"核心指标" if zh else "Core metrics"}

    {f"通过率覆盖全部 {task_count} 题;端到端耗时、CPU 和内存只比较所有浏览器共同通过的 {shared_count} 题,避免提前失败缩短工作量。单次动作耗时含实际工具调用,调用次数不判优。" if zh else f"Pass rate covers all {task_count} tasks. End-to-end time, CPU, and memory use the {shared_count} tasks passed by every browser so early failure cannot shorten workload. Per-action latency covers actual tool calls; call counts are not ranked."}

    {_metrics_table(method_rows, lang)}
    +

    {"资源与网络" if zh else "Resources and network"}

    {f"下表也只统计共同通过的 {shared_count} 题;每个数值按单题采样。网络传输量和 CPU 使用率不直接判优,低数值可能来自缓存或等待,而非浏览器更快。" if zh else f"These per-task samples also cover the {shared_count} shared passes. Network bytes and CPU utilization are not ranked: lower values can reflect caching or waiting, not a faster browser."}

    {"展开资源与网络统计" if zh else "Open resource and network statistics"}{_resource_table(method_rows, lang)}
    +

    {"失败发生在哪个环节" if zh else "Where failures occurred"}

    {"动作报错与提交请求未出现分别统计。后者表示工具调用返回成功,但验收要求的网络请求没有发生;不能据此改判通过。逐题详情给出触发动作、验收规则和复核原因。" if zh else "Action errors and absent submission requests are counted separately. In the latter case the tool call returned success, but the required network request did not occur; this does not justify changing the verdict to pass. Task detail shows the trigger action, evaluator rule, and reviewed cause."}

    {stage_table}
    +

    {"浏览器动作耗时" if zh else "Browser action latency"}

    {"均值、P50、P95 按单次调用统计;调用次数只说明工作量。下钻到结果详情可查看具体动作耗时和工具返回。" if zh else "Mean, P50, and P95 are per-call statistics; call counts only describe workload. Open a task result for the individual action timing and full tool response."}

    {"展开各类动作耗时" if zh else "Open latency by action type"}{action_table}
    ''' + (root / "index.html").write_text(_shell(lang, "index", "浏览器轨迹重放" if zh else "Browser Trajectory Replay", summary_body, current_route="index.html"), encoding="utf-8") + + headers = ["任务" if zh else "Task", "站点" if zh else "Site", "任务目标" if zh else "Task goal", "类型" if zh else "Type", "轨迹来源" if zh else "Trajectory source", "标准动作" if zh else "Qualified actions", "下钻" if zh else "Drill down"] + matrix_rows = [] + for task_id, task in tasks.items(): + source = str((task.get("source") or {}).get("kind") or "") + source_code = "autonomous" if source == "authoritative_successful_browser_use_execution" else "repaired" + source_text = ("自主执行成功" if zh else "Autonomous success") if source_code == "autonomous" else ("修复后验证" if zh else "Repaired and qualified") + task_type = str((task.get("evaluation_input") or {}).get("task_type") or "") + type_text = ({"RETRIEVE": "信息提取", "NAVIGATE": "页面导航", "MUTATE": "页面操作"}.get(task_type, task_type) if zh else task_type) or "—" + sites = task["task"].get("sites") or [] + intent = str(task["task"].get("intent") or "") + action_count = len(task.get("tool_calls") or []) + search = _e(f"{task_id} {', '.join(sites)} {intent} {task_type} {source_text}") + matrix_rows.append(f'#{task_id}{_e(", ".join(sites))}
    {_e(intent)}
    {_e(type_text)}{source_text}{action_count}') + dataset_sites = sorted({site for task in tasks.values() for site in (task["task"].get("sites") or [])}) + dataset_site_options = "".join(f'' for site in dataset_sites) + dataset_type_options = "".join(f'' for code, label_zh, label_en in (("RETRIEVE", "信息提取", "Retrieve"), ("NAVIGATE", "页面导航", "Navigate"), ("MUTATE", "页面操作", "Mutate"))) + dataset_body = f'''

    WebArena-Verified · {task_count}

    {"评测集" if zh else "Dataset"}

    {f"{task_count} 个任务,每题一条经过资格验证的固定轨迹。这里展示任务目标和验收类型;实测成败在结果分析中查看。" if zh else f"{task_count} tasks, each with one qualified fixed trajectory. This page contains goals and acceptance types; measured verdicts appear in Results analysis."}

    {"任务与标准轨迹" if zh else "Tasks and qualified trajectories"}

    {f"{browser_use_source_count} 条来自成功的自主执行,{repaired_source_count} 条经修复后在 {reference_label} 验证。点开任务可查看验收输入、完整轨迹和各浏览器实测证据。" if zh else f"{browser_use_source_count} came from successful autonomous runs; {repaired_source_count} were repaired and qualified on {reference_label}. Open a task for its acceptance input, full trajectory, and evidence from every browser."}

    {"".join(f"" for h in headers)}{"".join(matrix_rows)}
    {_e(h)}
    ''' + (root / "dataset.html").write_text(_shell(lang, "dataset", "评测集" if zh else "Dataset", dataset_body, current_route="dataset.html"), encoding="utf-8") + + result_rows = [] + for row in public_rows: + cause = row["cause_zh"] if zh else row["cause_en"] + status = ("通过" if row["status"] == "pass" else "失败") if zh else row["status"].title() + stage_label = {"browser_tool_execution": "浏览器动作" if zh else "Browser action", "side_effect_not_observed": "提交请求未出现" if zh else "Submission request absent", "official_evaluation": "官方验收" if zh else "Official evaluation", "pass": "—"}.get(row["failure_stage"], row["failure_stage"]) + result_rows.append(f'#{row["task_id"]}{_e(row["browser"])}{_e(row["sites"][0] if row["sites"] else "")}{status}{row["completed_actions"]}/{row["action_count"]}{_e(stage_label)}{_e(cause) if cause else "—"}{"证据页" if zh else "Evidence page"}') + browser_options = "".join(f'' for key, item in methods.items()) + site_options = "".join(f'' for site in sorted({s for row in public_rows for s in row["sites"]})) + stage_labels = {"browser_tool_execution": ("浏览器动作", "Browser action"), "side_effect_not_observed": ("提交请求未出现", "Submission request absent"), "official_evaluation": ("官方验收", "Official evaluation")} + present_stages = {row["failure_stage"] for row in public_rows if row["status"] == "fail"} + stage_options = "".join(f'' for code in stage_labels if code in present_stages) + causes = {row["cause_zh"]: row["cause_zh"] if zh else row["cause_en"] for row in public_rows if row["cause_zh"]} + cause_options = "".join(f'' for code, label in sorted(causes.items())) + result_matrix_rows = [] + for task_id, task in tasks.items(): + pass_links = [] + fail_links = [] + passed_engines = [] + failed_engines = [] + for engine in methods: + row = next(item for item in rows_by_task[task_id] if item["engine"] == engine) + label = ("通过" if row["status"] == "pass" else "失败") if zh else row["status"].title() + link = f'{_e(row["browser"])} · {label}' + if row["status"] == "pass": + pass_links.append(link) + passed_engines.append(engine) + else: + fail_links.append(link) + failed_engines.append(engine) + site = (task["task"].get("sites") or [""])[0] + result_matrix_rows.append(f'#{task_id}{_e(", ".join(task["task"].get("sites") or []))}
    {"".join(pass_links) or "—"}
    {"".join(fail_links) or "—"}
    ') + matrix_sites = "".join(f'' for site in sorted({(task["task"].get("sites") or [""])[0] for task in tasks.values()})) + matrix_baselines = "".join(f'' for engine, item in methods.items()) + matrix_checks = "".join(f'' for engine, item in methods.items()) + result_matrix = f'

    {"同题浏览器对比" if zh else "Same-task browser comparison"}

    {"按通过与失败分组展示所有参评浏览器;点击浏览器名称查看完整动作和验收证据。" if zh else "All registered browsers are grouped by pass and fail. Open a browser outcome for its actions and evaluator evidence."}

    {"参与比较的浏览器(至少选一个)" if zh else "Comparison browsers (select at least one)"}
    {matrix_checks}
    {"".join(result_matrix_rows)}
    {"任务" if zh else "Task"}{"站点" if zh else "Site"}{"通过的浏览器" if zh else "Passing browsers"}{"失败的浏览器" if zh else "Failing browsers"}
    ' + results_body = f'''

    {len(public_rows)} {"条逻辑结果" if zh else "logical outcomes"}

    {"结果与失败诊断" if zh else "Results and failure diagnosis"}

    {"同题对比支持选择基准和多个浏览器;下方筛选表默认只显示失败,点击“诊断”查看原始判定、复核原因与执行证据。" if zh else "Choose a baseline and multiple browsers for same-task comparison. The inventory starts with failures; open Diagnose for the exact finding, reviewed cause, and execution evidence."}

    {result_matrix}

    {''.join(result_rows)}
    {"任务" if zh else "Task"}{"浏览器" if zh else "Browser"}{"站点" if zh else "Site"}{"判定" if zh else "Verdict"}{"动作" if zh else "Actions"}{"失败环节" if zh else "Failure stage"}{"复核分类" if zh else "Reviewed category"}{"详情" if zh else "Detail"}
    {"失败环节" if zh else "Failure stage"}
    {"动作定位" if zh else "Action locator"}
    {"复核分类" if zh else "Reviewed category"}

    {"程序判定与验收规则" if zh else "Programmatic finding and evaluator rule"}

    {"复核原因与处理" if zh else "Reviewed cause and treatment"}

    ''' + results_body = results_body.replace("结果与失败诊断", "结果分析").replace("Results and failure diagnosis", "Results analysis") + (root / "results.html").write_text(_shell(lang, "results", "结果分析" if zh else "Results analysis", results_body, current_route="results.html"), encoding="utf-8") + + flow_labels = ( + ("重放器", "浏览器容器", "本地站点", "逐题重置", "证据采集", "官方验收", "完全销毁") + if zh + else ("Replay runner", "Browser container", "Local sites", "Per-task reset", "Evidence", "Official evaluator", "Full teardown") + ) + fallback = '
    ' + "".join( + f'
    {index}{_e(label)}
    ' + + ('' if index < len(flow_labels) else "") + for index, label in enumerate(flow_labels, 1) + ) + "
    " + topology = f'''''' + environment_body = f'''

    {"环境与评测流程" if zh else "Environment and evaluation flow"}

    {"192.168.5.2 是 Colima 虚拟机中的宿主网关:浏览器容器通过它访问运行在同一台 Mac 虚拟化环境里的 WebArena 站点。它不是公网测试网站。" if zh else "192.168.5.2 is the host gateway inside the Colima VM. Browser containers use it to reach the WebArena sites running in the same Mac virtualization environment; it is not a public test website."}

    {"执行拓扑" if zh else "Execution topology"}

    {fallback}{topology}

    {"隔离规则" if zh else "Isolation rules"}

    1

    {"一套固定端口站点环境,因此只允许一个方法运行。" if zh else "One fixed-port site stack permits only one running method."}

    {task_count}

    {"一个浏览器完整跑完全部任务,每题前恢复官方夹具。" if zh else "A browser completes all tasks; official fixtures reset before each task."}

    0

    {"没有 Agent 决策和模型请求;输入是冻结的 BrowserTool 动作。" if zh else "No Agent decisions or model calls; inputs are frozen BrowserTool actions."}

    ''' + environment_body = environment_body.replace("环境与评测流程", "环境与方法").replace("Environment and evaluation flow", "Environment and method") + (root / "environment.html").write_text(_shell(lang, "environment", "环境与方法" if zh else "Environment and method", environment_body, current_route="environment.html"), encoding="utf-8") + + review_rows = ( + ( + "轨迹资格核验" if zh else "Trajectory qualification", + f"{task_count} 题;{browser_use_source_count} 条来自自主执行,{repaired_source_count} 条经修复。" if zh else f"{task_count} tasks: {browser_use_source_count} autonomous successes and {repaired_source_count} repaired trajectories.", + f"{repaired_source_count} 条候选轨迹需要修复后才能作为固定输入。" if zh else f"{repaired_source_count} candidate trajectories required repair before use as frozen inputs.", + f"逐题资格验证后,{reference_label} 通过 {reference_row['passed']}/{reference_row['total']} 题;固定轨迹不包含 Agent 或模型决策。" if zh else f"Per-task qualification yielded {reference_row['passed']}/{reference_row['total']} passes on {reference_label}. Frozen trajectories contain no Agent or model decisions.", + "资格验证无未决项;实测浏览器差异不在此处改分。" if zh else "No qualification items remain unresolved. Measured browser differences are not rescored here.", + ), + ( + "失败证据复核" if zh else "Failure evidence review", + f"{reviewed_failure_count} 条最终失败结果。" if zh else f"{reviewed_failure_count} final failed outcomes.", + f"{sum(row['failure_stage'] == 'browser_tool_execution' for row in public_rows)} 条动作执行失败;{sum(row['failure_stage'] == 'side_effect_not_observed' for row in public_rows)} 条验收要求的提交请求未出现。" if zh else f"{sum(row['failure_stage'] == 'browser_tool_execution' for row in public_rows)} browser-action failures; {sum(row['failure_stage'] == 'side_effect_not_observed' for row in public_rows)} missing required submission requests.", + f"{reviewed_failure_count}/{reviewed_failure_count} 条绑定当前证据哈希;分别展示程序报错、动作位置、复核原因和处理状态,原始分数未改。" if zh else f"{reviewed_failure_count}/{reviewed_failure_count} reviews are bound to the current evidence hash. Program errors, action locators, reviewed causes and treatment states are separate; official scores did not change.", + "失败根因的后续修复和重测状态见逐题结果详情。" if zh else "Follow-up remediation and rerun status appear on each failed result page.", + ), + ) + review_table_rows = "".join( + "" + "".join(f"{_e(cell)}" for cell in (title, scope, finding, correction, remaining)) + "" + for title, scope, finding, correction, remaining in review_rows + ) + review_headers = ("轮次", "范围", "发现", "纠正", "未完成") if zh else ("Round", "Scope", "Finding", "Correction", "Open") + review_body = f'''

    {"核验记录" if zh else "Review history"}

    {"只记录影响当前评测集和结果解读的核验轮次;每个失败的完整证据与处理状态在逐题结果页。" if zh else "Rounds that affected the current dataset or result interpretation. Full evidence and treatment status for each failure are on result pages."}

    {"当前状态" if zh else "Current status"}

    {f"标准轨迹 {task_count}/{task_count} 条已资格验证;最终失败 {reviewed_failure_count}/{reviewed_failure_count} 条已完成证据绑定复核,未复核 0 条。" if zh else f"{task_count}/{task_count} qualified trajectories and {reviewed_failure_count}/{reviewed_failure_count} evidence-bound failure reviews; zero unreviewed failures."}

    {"核验轮次" if zh else "Review rounds"}

    {''.join(f'' for label in review_headers)}{review_table_rows}
    {_e(label)}
    ''' + (root / "review.html").write_text(_shell(lang, "review", "核验记录" if zh else "Review history", review_body, current_route="review.html"), encoding="utf-8") + + viewer_tabs = (("result", "结果与资源", "Result and resources"), ("programmatic_failure", "程序定位", "Programmatic finding"), ("review", "复核原因", "Reviewed cause"), ("trajectory", "动作与日志", "Actions and logs"), ("official_evaluation", "官方验收", "Official evaluation")) + tab_buttons = "".join(f'' for key, label_zh, label_en in viewer_tabs) + viewer_body = f'

    {"证据查看器" if zh else "Evidence viewer"}

    {"按结果、动作、程序判定和复核原因查看完整净化证据;原始 JSON 仍可下载。" if zh else "Inspect full sanitized evidence by outcome, action, programmatic finding, and review. The raw JSON remains downloadable."}

    {tab_buttons}

    ' + (root / "evidence-viewer.html").write_text(_shell(lang, "results", "证据查看器" if zh else "Evidence viewer", viewer_body, current_route="evidence-viewer.html"), encoding="utf-8") + + manifest = { + "schema": "lex_webarena_trajectory_replay_report/1", + "results_sha256": _sha256(Path(results_path)), + "dataset_sha256": _sha256(Path(dataset_path)), + "review_sha256": _sha256(Path(review_path)), + "task_count": len(tasks), + "participant_count": len(methods), + "participants": list(methods), + "logical_results": len(public_rows), + "failed_results": sum(row["status"] != "pass" for row in public_rows), + "reviewed_failures": len(reviews), + "routes": 12 + 2 * len(public_rows) + 2 * len(tasks), + } + _write_json(output / "report-manifest.json", manifest) + return manifest diff --git a/harness/trajectory_replay/webarena_results.py b/harness/trajectory_replay/webarena_results.py new file mode 100644 index 000000000..9cd4203a1 --- /dev/null +++ b/harness/trajectory_replay/webarena_results.py @@ -0,0 +1,158 @@ +"""Build one current logical replay matrix from formal runs and correction shards.""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from typing import Any + +from .webarena_plan import load_replay_plan +from .webarena_replay import RECEIPT_SCHEMA, validate_qualification_dataset + +EFFECTIVE_SCHEMA = "lex_webarena_trajectory_replay_effective_suite/1" + + +def _read(path: str | Path) -> tuple[Path, dict[str, Any]]: + resolved = Path(path).expanduser().resolve() + return resolved, json.loads(resolved.read_text(encoding="utf-8")) + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _attempt_map(receipt: dict[str, Any]) -> dict[tuple[str, int], dict[str, Any]]: + attempts = { + (str(row["engine"]), int(row["task_id"])): row + for row in receipt.get("attempts") or [] + } + expected = { + (str(engine), int(task_id)) + for engine in receipt.get("engines") or [] + for task_id in receipt.get("task_ids") or [] + } + if len(attempts) != len(receipt.get("attempts") or []) or set(attempts) != expected: + raise ValueError("replay receipt does not contain its complete unique result matrix") + return attempts + + +def _validate_receipt( + receipt: dict[str, Any], *, scope: str, engines: list[str] +) -> dict[tuple[str, int], dict[str, Any]]: + if ( + receipt.get("schema") != RECEIPT_SCHEMA + or receipt.get("execution_scope") != scope + or receipt.get("status") != "complete" + ): + raise ValueError(f"expected a completed {scope} replay receipt") + if list(receipt.get("engines") or []) != engines: + raise ValueError("replay receipt browser registry differs from the current profile") + if not all((receipt.get("method_teardown") or {}).get(engine) for engine in engines): + raise ValueError("replay receipt does not prove method teardown") + return _attempt_map(receipt) + + +def build_effective_results( + *, + profile_path: str | Path, + seed_manifest_path: str | Path, + qualified_dataset_path: str | Path, + base_receipt_path: str | Path, + correction_receipt_paths: list[str | Path], +) -> dict[str, Any]: + """Overlay declared correction shards onto one complete formal base matrix.""" + + plan = load_replay_plan(profile_path, seed_manifest_path) + qualified_sha256 = validate_qualification_dataset(plan, qualified_dataset_path) + engines = list(plan["engines"]) + base_path, base = _read(base_receipt_path) + effective = _validate_receipt(base, scope="formal", engines=engines) + expected = {(engine, task_id) for engine in engines for task_id in plan["task_ids"]} + if set(effective) != expected: + raise ValueError("base formal receipt does not cover the complete current task selection") + + source_receipts = [ + { + "role": "base", + "path": str(base_path), + "sha256": _sha256(base_path), + "run_id": str(base.get("run_id") or ""), + "task_ids": list(base.get("task_ids") or []), + } + ] + physical_by_key: dict[tuple[str, int], list[dict[str, Any]]] = {} + for row in base.get("physical_executions") or []: + physical_by_key.setdefault((str(row["engine"]), int(row["task_id"])), []).append(row) + receipts = [base] + + for raw_path in correction_receipt_paths: + path, receipt = _read(raw_path) + correction = _validate_receipt( + receipt, scope="formal_correction_shard", engines=engines + ) + if ( + receipt.get("profile_sha256") != plan["profile_sha256"] + or receipt.get("source_seed_manifest_sha256") != plan["seed_manifest_sha256"] + or receipt.get("qualification_dataset_sha256") != qualified_sha256 + ): + raise ValueError("formal correction shard is not bound to the current replay inputs") + effective.update(correction) + for key in correction: + physical_by_key[key] = [] + for row in receipt.get("physical_executions") or []: + physical_by_key[(str(row["engine"]), int(row["task_id"]))].append(row) + source_receipts.append( + { + "role": "correction", + "path": str(path), + "sha256": _sha256(path), + "run_id": str(receipt.get("run_id") or ""), + "task_ids": list(receipt.get("task_ids") or []), + } + ) + receipts.append(receipt) + + if set(effective) != expected or set(physical_by_key) != expected: + raise ValueError("effective replay matrix is incomplete") + candidates = { + int(task["task_id"]): str(task["canonical_candidate_id"]) + for task in plan["seed_manifest"]["tasks"] + } + stale = [ + key + for key, row in effective.items() + if str(row.get("source_candidate_id") or "") != candidates[key[1]] + ] + if stale: + raise ValueError(f"effective replay matrix contains stale trajectories: {stale}") + + bindings = json.dumps(source_receipts, sort_keys=True, separators=(",", ":")) + return { + "schema": EFFECTIVE_SCHEMA, + "run_id": f"effective-{hashlib.sha256(bindings.encode()).hexdigest()[:12]}", + "status": "complete", + "execution_scope": "formal_effective", + "profile": plan["profile_path"], + "profile_sha256": plan["profile_sha256"], + "source_seed_manifest": plan["seed_manifest_path"], + "source_seed_manifest_sha256": plan["seed_manifest_sha256"], + "qualification_dataset_sha256": qualified_sha256, + "task_ids": list(plan["task_ids"]), + "engines": engines, + "isolation": dict(plan["profile"]["execution"]), + "method_teardown": {engine: True for engine in engines}, + "source_receipts": source_receipts, + "runtime_receipts": [receipt.get("runtime") or {} for receipt in receipts], + "attempts": [effective[key] for key in sorted(effective)], + "physical_executions": [ + row for key in sorted(physical_by_key) for row in physical_by_key[key] + ], + "counts": { + "logical_results": len(effective), + "physical_executions": sum(len(rows) for rows in physical_by_key.values()), + "passes": sum(row.get("status") == "pass" for row in effective.values()), + "failures": sum(row.get("status") != "pass" for row in effective.values()), + }, + } + diff --git a/harness/trajectory_replay/webarena_review.py b/harness/trajectory_replay/webarena_review.py new file mode 100644 index 000000000..be6e2142b --- /dev/null +++ b/harness/trajectory_replay/webarena_review.py @@ -0,0 +1,90 @@ +"""Validate evidence-bound root-cause review for trajectory replay results.""" + +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import Path +from typing import Any + +REVIEW_SCHEMA = "lex_webarena_trajectory_replay_failure_review/1" + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def programmatic_failure(attempt: dict[str, Any]) -> dict[str, Any]: + """Return the automatic failure location without asserting a root cause.""" + + if attempt.get("process_status") != "pass": + message = str(attempt.get("error") or "BrowserTool execution failed") + match = re.search(r"RuntimeError: step (\d+) failed: ([^\n]+)", message) + return { + "stage": "browser_tool_execution", + "step": int(match.group(1)) if match else None, + "message": match.group(2) if match else message.strip(), + } + if attempt.get("evaluator_status") != "pass": + return { + "stage": "official_evaluation", + "step": None, + "message": "Official evaluator rejected the completed trajectory", + } + return {"stage": "pass", "step": None, "message": ""} + + +def load_validated_failure_review( + results_path: str | Path, review_path: str | Path +) -> dict[str, Any]: + """Load a review only when it exactly covers the bound final failures.""" + + results_file = Path(results_path).expanduser().resolve() + review_file = Path(review_path).expanduser().resolve() + results = json.loads(results_file.read_text(encoding="utf-8")) + review = json.loads(review_file.read_text(encoding="utf-8")) + if review.get("schema") != REVIEW_SCHEMA: + raise ValueError("unsupported trajectory replay failure review") + if (review.get("bindings") or {}).get("effective_results_sha256") != _sha256( + results_file + ): + raise ValueError("failure review is not bound to the effective result bytes") + + attempts = results.get("attempts") or [] + failed = { + (str(item.get("engine") or ""), int(item["task_id"])) + for item in attempts + if item.get("status") != "pass" + } + reviewed: dict[tuple[str, int], dict[str, Any]] = {} + for group in review.get("cause_groups") or []: + required_group_fields = { + "code", + "label_zh", + "label_en", + "responsibility", + "remediation_owner", + "remediation_status", + "results", + } + if required_group_fields - set(group): + raise ValueError("failure review cause group is incomplete") + for item in group["results"]: + key = (str(item.get("engine") or ""), int(item["task_id"])) + if key in reviewed: + raise ValueError(f"duplicate failure review result: {key}") + if not str(item.get("reason_zh") or "").strip() or not str( + item.get("reason_en") or "" + ).strip(): + raise ValueError(f"failure review reason is incomplete: {key}") + reviewed[key] = {**item, "cause_group": group} + + if set(reviewed) != failed: + missing = sorted(failed - set(reviewed)) + extra = sorted(set(reviewed) - failed) + raise ValueError(f"failure review coverage mismatch: missing={missing}, extra={extra}") + coverage = review.get("coverage") or {} + if coverage != {"failed_results": len(failed), "verified_results": len(reviewed)}: + raise ValueError("failure review coverage summary is incorrect") + return {"review": review, "results": results, "by_result": reviewed} diff --git a/harness/web_fetch/README.md b/harness/web_fetch/README.md index 61b0beb64..9b845e6af 100644 --- a/harness/web_fetch/README.md +++ b/harness/web_fetch/README.md @@ -158,7 +158,9 @@ evaluation records, task decisions and `word_*` metrics are unchanged. Full-body metrics require a separately scoped, comparable complete reference. Web Fetch providers, execution, reference checks, scoring and report generation -live in this package. Browser Use remains independent in `harness/browser_use`. +live in this package. The fixed-action kernel benchmark remains independent in +`harness/trajectory_replay`; autonomous browser-use evaluation lives in +`harness/browser_use`. From the repository root: diff --git a/harness/web_fetch/artifact_auth.py b/harness/web_fetch/artifact_auth.py index 478a3e15a..dcaff90a2 100644 --- a/harness/web_fetch/artifact_auth.py +++ b/harness/web_fetch/artifact_auth.py @@ -1,4 +1,4 @@ -"""Artifact Hub's user-scoped publishing credential, separate from eval secrets.""" +"""Artifact Site's user-scoped publishing credential, separate from eval secrets.""" import os from pathlib import Path import stat @@ -8,29 +8,44 @@ def token_path(base_url): - root = Path.home() / '.config/artifact-hub' + root = Path.home() / '.config/artifact-site' parsed = urlsplit(base_url) host = parsed.netloc.lower() if (parsed.scheme not in {'http', 'https'} or not parsed.hostname or parsed.username is not None or parsed.password is not None or host in {'.', '..'} or any(c.isspace() or c in '/\\' for c in host)): - raise ValueError('Artifact Hub base URL requires an HTTP(S) host without credentials') + raise ValueError('Artifact Site base URL requires an HTTP(S) host without credentials') # Include an explicit port, matching the server manual's host key. return root / 'tokens' / host def validate_token(token): if not token.startswith('ahp_') or any(c.isspace() for c in token): - raise ValueError('Invalid Artifact Hub token format') + raise ValueError('Invalid Artifact Site token format') return token def load_token(*, base_url): - # Explicit process environment, then the target host. Never .env. - path = token_path(base_url) - token = os.environ.get('ARTIFACT_HUB_TOKEN', '').strip() + # Explicit process environment, then the target host. Never .env. Previous + # product-name locations remain read-only migration fallbacks. + host = urlsplit(base_url).netloc.lower() + token = os.environ.get('ARTIFACT_SITE_TOKEN', '').strip() if not token: - token = path.read_text().strip() + token = os.environ.get('ARTIFACT_HUB_TOKEN', '').strip() + candidates = ( + token_path(base_url), + Path.home() / '.config/artifact-hub' / 'tokens' / host, + Path.home() / '.config/artifact-site' / 'token', + Path.home() / '.config/artifact-hub' / 'token', + ) + if not token: + for path in candidates: + try: + token = path.read_text().strip() + except FileNotFoundError: + continue + if token: + break return validate_token(token) diff --git a/pyproject.toml b/pyproject.toml index f37ac575b..1c83b99bb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,8 +21,15 @@ reference-audit = [ ] browser-use-eval = [ "docker>=7,<8", + "json-repair>=0.54,<1", + "pillow>=11,<13", "playwright>=1.52,<2", + "python-dotenv>=1,<2", + "pyyaml>=6,<7", + "regex>=2025.7,<2027", + "requests>=2.32,<3", "tiktoken>=0.7,<1", + "webarena-verified==1.2.3", ] test = [ "pytest>=8,<10", @@ -47,7 +54,9 @@ markers = [ line-length = 100 include = [ "harness/browser_use/**/*.py", + "harness/trajectory_replay/**/*.py", "tests/browser_use/**/*.py", + "tests/trajectory_replay/**/*.py", ] target-version = "py311" @@ -55,7 +64,10 @@ target-version = "py311" select = ["B", "E", "F", "I", "UP"] [tool.ruff.lint.per-file-ignores] +"harness/trajectory_replay/report.py" = ["E501"] +"harness/trajectory_replay/webarena_report.py" = ["E501"] "harness/browser_use/report.py" = ["E501"] +"harness/browser_use/report_i18n.py" = ["E501"] [build-system] requires = ["setuptools>=64"] diff --git a/research/browser-use-benchmark-adoption.json b/research/browser-use-benchmark-adoption.json new file mode 100644 index 000000000..e3e866b4a --- /dev/null +++ b/research/browser-use-benchmark-adoption.json @@ -0,0 +1,104 @@ +{ + "schema_version": 1, + "verified_at": "2026-09-08", + "scope": "Public first-party model, system-card, paper, and official benchmark evidence. Search/research benchmarks are separated from browser-interaction benchmarks.", + "organizations": [ + { + "organization": "OpenAI", + "browser_interaction": ["WebArena", "WebVoyager", "Mind2Web"], + "computer_use": ["OSWorld 2.0", "ScreenSpot-Pro", "Agents' Last Exam"], + "agentic_search": ["BrowseComp"], + "notes": "GPT-6 Astra publishes computer-use scores and a Mind2Web latency comparison, but no WebArena-Verified score on its launch page.", + "sources": [ + "https://openai.com/index/gpt-6-astra/", + "https://openai.com/index/computer-using-agent/" + ] + }, + { + "organization": "Anthropic", + "browser_interaction": ["WebArena", "WebArena-Verified", "Online-Mind2Web"], + "computer_use": ["OSWorld 2.0"], + "agentic_search": [], + "notes": "The Claude Sonnet 4.6 system card documents official WebArena-Verified prompt/grader use and Average@5 reporting.", + "sources": [ + "https://www-cdn.anthropic.com/bbd8ef16d70b7a1665f14f306ee88b53f686aa75/Claude%20Sonnet%204.6%20System%20Card.pdf" + ] + }, + { + "organization": "Google DeepMind", + "browser_interaction": ["Online-Mind2Web", "WebVoyager"], + "computer_use": ["AndroidWorld"], + "agentic_search": [], + "notes": "Gemini 2.5 Computer Use reports both official-leaderboard and Browserbase-harness Online-Mind2Web results, illustrating harness sensitivity.", + "sources": [ + "https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-2-5-Computer-Use-Model-Card.pdf" + ] + }, + { + "organization": "THUDM / ChatGLM", + "browser_interaction": ["Mind2Web", "WebArena", "MiniWob++", "AutoWebBench"], + "computer_use": [], + "agentic_search": [], + "notes": "AutoWebGLM publishes adapters/evaluation assets, but modifies WebArena for its interaction system, so numbers are not automatically comparable with an unmodified harness.", + "sources": ["https://github.com/THUDM/AutoWebGLM"] + }, + { + "organization": "Moonshot AI / Kimi", + "browser_interaction": [], + "computer_use": [], + "agentic_search": ["BrowseComp", "DeepSearchQA", "ResearchRubrics"], + "notes": "The inspected official Kimi releases emphasize search/research benchmarks; no first-party browser-interaction evaluation was found in this audit. Do not count BrowseComp as UI browser use.", + "sources": ["https://github.com/MoonshotAI/Kimi-K3"] + }, + { + "organization": "DeepSeek", + "browser_interaction": [], + "computer_use": [], + "agentic_search": [], + "notes": "No first-party browser-interaction benchmark disclosure was found in the inspected official model materials. Absence of public evidence is not evidence of no internal evaluation.", + "sources": ["https://github.com/deepseek-ai"] + }, + { + "organization": "Browser Use", + "browser_interaction": ["BU Bench V1", "Online-Mind2Web", "Odysseys"], + "browser_infrastructure": ["Stealth Bench V1"], + "computer_use": [], + "agentic_search": [], + "notes": "The main product README highlights the open 100-task BU Bench and an Odysseys leaderboard result. The benchmark repository currently puts Stealth Bench V1 first, but that measures anti-bot browser infrastructure rather than end-to-end agent task completion. BU Bench, Online-Mind2Web, and Odysseys all use model-based judging in their documented workflows.", + "sources": [ + "https://github.com/browser-use/benchmark", + "https://github.com/browser-use/online-mind2web", + "https://github.com/browser-use/browser-use", + "https://github.com/ljang0/Odysseys" + ] + } + ], + "benchmark_families": [ + { + "family": "WebArena", + "adopters": ["OpenAI", "Anthropic", "THUDM / ChatGLM"], + "interaction": true + }, + { + "family": "Mind2Web", + "adopters": ["OpenAI", "Anthropic", "Google DeepMind", "THUDM / ChatGLM", "Browser Use"], + "interaction": true, + "warning": "Original Mind2Web, Online-Mind2Web, and Mind2Web 2 have different tasks and graders and must not share a score table." + }, + { + "family": "WebVoyager", + "adopters": ["OpenAI", "Google DeepMind"], + "interaction": true + }, + { + "family": "BU Bench V1", + "adopters": ["Browser Use"], + "interaction": true + }, + { + "family": "Odysseys", + "adopters": ["Browser Use"], + "interaction": true + } + ] +} diff --git a/research/browser-use-failure-analysis.json b/research/browser-use-failure-analysis.json new file mode 100644 index 000000000..6e3b6f396 --- /dev/null +++ b/research/browser-use-failure-analysis.json @@ -0,0 +1,52 @@ +{ + "schema": "lex_browser_use_failure_analysis/3", + "scope": { + "benchmark": "WebArena-Verified 1.2.3", + "task_ids": [68, 124, 142, 168, 506], + "engines": ["chromium", "moli"], + "evidence": "score-bearing final response, official evaluator output, HAR, and autonomous BrowserTool event trajectory", + "score_effect": "historical official scores are unchanged; corrected protocols require new executions" + }, + "tasks": { + "68": { + "observed_result": "Both results returned only The Hobbit; the official expected payload also requires A Christmas Carol with author=null.", + "root_cause": {"zh": "官方响应合同内部矛盾:expected 中 author 为 null,而公开 results_schema 只允许 string。", "en": "The official response contract is internally inconsistent: the expected payload contains a null author while the published results_schema permits only string for author."}, + "owner": {"zh": "评测集任务合同", "en": "Benchmark task contract"}, + "proof": {"zh": "程序化扫描 812 个官方任务,发现 8 个 expected 不符合自身 schema(2、3、6、21、68、69、96、319);本轮 30 题中只有 #68。两条轨迹都打开过 A Christmas Carol 帖子,页面访问不是阻断点。", "en": "A scan of all 812 official tasks found eight expected payloads that violate their own published schema (2, 3, 6, 21, 68, 69, 96, 319). Task 68 is the only one in this 30-task cohort. Both trajectories opened the A Christmas Carol post, so page access was not the stopping point."}, + "corrective_action": {"zh": "新运行在调用模型前拒绝合同不一致任务。#68 退出干净浏览器 A/B,除非官方修正 schema,或实验预注册明确的 benchmark 补丁。", "en": "Reject contract-inconsistent tasks during preflight. Exclude task 68 from a clean browser A/B unless the benchmark publishes a corrected schema or the experiment preregisters an explicit benchmark patch."}, + "verification": {"zh": "准入检查必须在任何模型或浏览器调用前失败,并指出违反 schema 的 expected 字段路径。", "en": "The preflight must fail before any model or browser call and identify the expected-value path that violates the schema."} + }, + "124": { + "observed_result": "Both results used 17774.32 as the maximum instead of the official 298.00.", + "root_cause": {"zh": "Agent 把 Magento 全文搜索命中直接当成合格的无线耳机集合,未逐项验证任务语义便聚合页面极值。", "en": "The Agent treated Magento full-text search hits as an already-qualified wireless-earphone set and aggregated page extrema without validating each candidate against the task noun phrase."}, + "owner": {"zh": "Agent 完成策略", "en": "Agent completion policy"}, + "proof": {"zh": "两条轨迹中的 17,774.32 都属于 QFWCJ 4K Short Throw Projector,并与最终错误最大值完全一致;两条轨迹都没有打开或以其他方式验证该候选是无线耳机。", "en": "The 17774.32 value in both trajectories belongs to QFWCJ 4K Short Throw Projector; it is exactly the incorrect maximum in both final answers. Neither trajectory opened or otherwise validated that candidate as a wireless earphone."}, + "corrective_action": {"zh": "把候选语义核验和提交前的完整性检查写入统一 Agent 协议,不增加针对 #124 标准答案的特例。", "en": "Make candidate validation and a pre-submit semantic/completeness check part of the common Agent protocol; do not add a task-specific expected-answer rule."}, + "verification": {"zh": "按修订协议在两个浏览器重跑 #124,并要求证据证明纳入极值的商品满足全部任务约束。", "en": "Rerun task 124 on both browsers under the revised protocol and require evidence that included extrema satisfy every task constraint."} + }, + "142": { + "observed_result": "Both results returned 80.50; the official expected value is 68.51.", + "root_cause": {"zh": "任务没有可操作地定义“hair care and hair style”的归类边界。Agent 按商品名归类并纳入 11.99 的 Crystal Headband,官方答案却在未给规则的情况下排除了它。", "en": "The task does not operationally define membership in 'hair care and hair style'. The Agent classified by product name and included a 11.99 Crystal Headband, while the official answer silently excludes it."}, + "owner": {"zh": "评测集验收边界,叠加 Agent 类别核验不足", "en": "Benchmark acceptance boundary, amplified by Agent category verification"}, + "proof": {"zh": "轨迹列出 50.52 的 conditioner、11.99 的 headband 和 17.99 的 haircolor,三者合计 80.50;排除 headband 恰为官方 68.51。任务合同只给最终数字,没有类别 taxonomy 或纳入规则。", "en": "The trajectories list 50.52 conditioner, 11.99 headband, and 17.99 haircolor. Their sum is 80.50; excluding the headband yields the expected 68.51. The task contract contains only the final number and no category taxonomy or inclusion rubric."}, + "corrective_action": {"zh": "#142 退出浏览器 A/B,直至类别归属绑定到站点 taxonomy 等可观察规则;统一 Agent 协议同时要求用站点证据核验类别。", "en": "Exclude task 142 from browser A/B until category membership is bound to an observable rule such as site taxonomy. The generic Agent protocol should also verify category claims against site evidence."}, + "verification": {"zh": "替代任务必须具有无需查看 expected total、两名独立复核者也能一致执行的客观类别谓词。", "en": "A replacement task must have an objective category predicate that two independent reviewers can apply without seeing the expected total."} + }, + "168": { + "observed_result": "Both results returned SUCCESS with an empty list; the official contract requires NOT_FOUND_ERROR with null.", + "root_cause": {"zh": "本项目 Agent 适配协议错误地规定所有 RETRIEVE 都必须返回列表,并遗漏零结果状态映射,直接违反该题官方 null schema。", "en": "The benchmark adapter's system protocol incorrectly stated that every RETRIEVE result must be a list and omitted the required zero-result status mapping, directly contradicting this task's official null schema."}, + "owner": {"zh": "browser-eval Agent 适配层", "en": "browser-eval Agent adapter"}, + "proof": {"zh": "官方 evaluator 唯一失败断言是状态不一致。该题 schema 为 null,而旧 runner 协议写明“For RETRIEVE, retrieved_data is always a list”。两条轨迹均完成搜索且没有匹配项目。", "en": "The official evaluator's only failed assertion is the status mismatch. The task schema is null, while the runner's system protocol said 'For RETRIEVE, retrieved_data is always a list.' Both trajectories completed the search and found no matching project."}, + "corrective_action": {"zh": "只在成功检索时返回 SUCCESS 加列表;穷尽搜索仍无结果时返回 NOT_FOUND_ERROR 加 null。", "en": "Specify SUCCESS plus list only for successful retrieval and NOT_FOUND_ERROR plus null after an exhaustive zero-result search."}, + "verification": {"zh": "单测锁定协议,并在两个浏览器重跑 #168;导航行为可以不变,但最终状态合同必须通过。", "en": "Unit-test the protocol and rerun task 168 on both browsers; navigation behavior need not change, but the final status contract must pass."} + }, + "506": { + "observed_result": "Both official scores are zero, but the score-bearing executions failed for different reasons.", + "root_cause": {"zh": "两端根因不同。Chromium 是测量假阴性:结账已完成,但 HAR 在运行结束才取响应体,CDP 已淘汰付款响应。Moli 是执行与完成确认失败:第 30 个决策回合才点击 Place order,没有后续观察,CowAgent 超限总结仍生成了成功形态的回答。", "en": "Chromium is a measurement false negative: checkout completed, but the HAR collector deferred Network.getResponseBody until run end and CDP had already evicted the payment response body. Moli is an execution/completion failure: its 30th decision turn issued Place order, leaving no post-action observation, and CowAgent's max-step fallback still produced a success-shaped answer."}, + "owner": {"zh": "Chromium:browser-eval HAR 采集器;Moli:运行预算与 CowAgent 完成处理", "en": "Chromium: browser-eval HAR collector; Moli: run budget and CowAgent completion handling"}, + "proof": {"zh": "Chromium 已到成功页并记录订单号;evaluator 的响应、商品、付款和成功页检查均通过,只有购物车总额规则收到 null body,HAR 同时记录 Network.getResponseBody unavailable。Moli 没有付款信息或成功页事件,最后一回合 has_tool_calls=true 并点击 Place order。", "en": "Chromium reached the success page and recorded an order number; its evaluator passed response, product, payment, and success-page checks, while the cart-total rule alone received a null body and HAR logged Network.getResponseBody unavailable. Moli recorded no payment-information or success-page event; its final turn had has_tool_calls=true and clicked Place order."}, + "corrective_action": {"zh": "在 CDP loadingFinished 时立即保存 evaluator 所需 JSON 响应体;最后决策回合仍有工具调用时 fail closed;预注册 40 步上限,为最后动作保留观察与验证空间。", "en": "Capture evaluator-relevant JSON bodies immediately on CDP loadingFinished. Fail closed when the final decision turn still contains a tool call, and preregister a 40-turn ceiling so a last action can be observed and verified."}, + "verification": {"zh": "重跑 Chromium 与 Moli 的 #506。Chromium 必须把付款 JSON 保留到评测;Moli 必须在 40 步内确认成功页,否则返回 UNKNOWN_ERROR,不能报告未核验的成功。", "en": "Rerun Chromium and Moli task 506. Chromium must retain the payment JSON body through evaluation; Moli must either verify the success page within 40 turns or return UNKNOWN_ERROR rather than an unverified success."} + } + } +} diff --git a/research/browser-use-failure-cause-review.json b/research/browser-use-failure-cause-review.json new file mode 100644 index 000000000..5a7fac466 --- /dev/null +++ b/research/browser-use-failure-cause-review.json @@ -0,0 +1,117 @@ +{ + "schema": "lex_browser_use_layered_failure_cause_review/2", + "benchmark": "WebArena-Verified 1.2.3", + "cohorts": { + "gpt-5.6-sol": { + "reviewed_at": "2026-09-11", + "reviewer_class": "evidence analyst", + "bindings": { + "attempts_sha256": "73146500152db62d79f84b63c85a14027c005a6320c19bce7426651f167466bd", + "manifest_sha256": "5466f30619d19ffda87689a1b31c9c6782b8837f72e4acdf6020f48c2fe89422", + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b" + }, + "review_protocol": { + "unit": "one score-bearing logical result per browser and task", + "method": "manual root-cause verification against the official task contract, score-bearing final response, evaluator assertions, network evidence, and autonomous Agent event trajectory", + "rule": "automatic evaluator classification remains available; root-cause subcategory and reason remain empty until status is verified" + }, + "coverage": {"failed_results": 37, "verified_results": 37}, + "unresolved": [], + "results": { + "gpt-5.6-sol-chromium-124": {"status":"verified","subcategory":{"code":"agent_retrieval_coverage","zh":"检索覆盖不足","en":"Incomplete retrieval coverage"},"reason":{"zh":"Agent 只覆盖部分无线耳机,最高价返回 169.99,遗漏官方全集的 298.0 上界。","en":"The Agent covered only part of the wireless-earphone set, returning 169.99 and missing the official 298.0 upper bound."}}, + "gpt-5.6-sol-moli-124": {"status":"verified","subcategory":{"code":"agent_retrieval_coverage","zh":"检索覆盖不足","en":"Incomplete retrieval coverage"},"reason":{"zh":"Agent 只覆盖部分无线耳机,最高价返回 199.9,遗漏官方全集的 298.0 上界。","en":"The Agent covered only part of the wireless-earphone set, returning 199.9 and missing the official 298.0 upper bound."}}, + "gpt-5.6-sol-chromium-258": {"status":"verified","subcategory":{"code":"agent_navigation_semantics","zh":"导航语义错误","en":"Navigation-semantics error"},"reason":{"zh":"Agent 停在 GitLab Explore 总页 /explore,没有启用 Public 过滤参数 visibility_level=20。","en":"The Agent stopped at generic /explore without applying the Public filter visibility_level=20."}}, + "gpt-5.6-sol-moli-258": {"status":"verified","subcategory":{"code":"agent_navigation_semantics","zh":"导航语义错误","en":"Navigation-semantics error"},"reason":{"zh":"Agent 停在 GitLab Explore 总页 /explore,没有启用 Public 过滤参数 visibility_level=20。","en":"The Agent stopped at generic /explore without applying the Public filter visibility_level=20."}}, + "gpt-5.6-sol-chromium-279": {"status":"verified","subcategory":{"code":"agent_retrieval_coverage","zh":"检索覆盖不足","en":"Incomplete retrieval coverage"},"reason":{"zh":"Agent 仅收集 7/12 个 Sony 蓝牙耳机,遗漏条目同时包含 18.99 最低价和 406.0 最高价。","en":"The Agent collected only 7 of 12 Sony Bluetooth headphones, omitting both the 18.99 minimum and 406.0 maximum."}}, + "gpt-5.6-sol-moli-279": {"status":"verified","subcategory":{"code":"agent_retrieval_coverage","zh":"检索覆盖不足","en":"Incomplete retrieval coverage"},"reason":{"zh":"Agent 仅收集 6/12 个 Sony 蓝牙耳机,遗漏条目同时包含 18.99 最低价和 406.0 最高价。","en":"The Agent collected only 6 of 12 Sony Bluetooth headphones, omitting both the 18.99 minimum and 406.0 maximum."}}, + "gpt-5.6-sol-chromium-284": {"status":"verified","subcategory":{"code":"agent_comparison_strategy","zh":"候选比较错误","en":"Candidate-comparison error"},"reason":{"zh":"Agent 选择了可放 20 双鞋的 Kingpavonini,没有完成所有合格候选的价格比较;官方最低价是 OnlyEasy 24-pocket。","en":"The Agent chose a 20-pair Kingpavonini rack without completing the price comparison; the official cheapest qualifying item is OnlyEasy 24-pocket."}}, + "gpt-5.6-sol-moli-284": {"status":"verified","subcategory":{"code":"agent_comparison_strategy","zh":"候选比较错误","en":"Candidate-comparison error"},"reason":{"zh":"Agent 选择了 72 双容量的 Aeitc,没有完成所有合格候选的价格比较;官方最低价是 OnlyEasy 24-pocket。","en":"The Agent chose a 72-pair Aeitc rack without completing the price comparison; the official cheapest qualifying item is OnlyEasy 24-pocket."}}, + "gpt-5.6-sol-chromium-339": {"status":"verified","subcategory":{"code":"benchmark_route_contract","zh":"路由验收缺陷","en":"Route-acceptance defect"},"reason":{"zh":"Agent 已打开含 state=opened 与 label_name[]=bug 的正确页面;官方规则错误要求 /issues/.*,拒绝合法的无尾斜杠 /issues?... 路由。","en":"The Agent opened the correct filtered page; the official /issues/.* rule rejects the valid no-trailing-slash /issues?... route."}}, + "gpt-5.6-sol-moli-339": {"status":"verified","subcategory":{"code":"benchmark_route_contract","zh":"路由验收缺陷","en":"Route-acceptance defect"},"reason":{"zh":"Agent 已打开含 state=opened 与 label_name[]=bug 的正确页面;官方规则错误要求 /issues/.*,拒绝合法的无尾斜杠 /issues?... 路由。","en":"The Agent opened the correct filtered page; the official /issues/.* rule rejects the valid no-trailing-slash /issues?... route."}}, + "gpt-5.6-sol-chromium-464": {"status":"verified","subcategory":{"code":"agent_step_budget","zh":"步骤预算耗尽","en":"Step-budget exhaustion"},"reason":{"zh":"Agent 在 40 步内未完成统计评论、编辑描述和保存,证据中没有目标商品保存请求。","en":"The Agent did not finish counting reviews, editing, and saving within 40 steps; no target-product save request exists."}}, + "gpt-5.6-sol-moli-464": {"status":"verified","subcategory":{"code":"agent_step_budget","zh":"步骤预算耗尽","en":"Step-budget exhaustion"},"reason":{"zh":"Agent 在 40 步内未完成统计评论、编辑描述和保存,证据中没有目标商品保存请求。","en":"The Agent did not finish counting reviews, editing, and saving within 40 steps; no target-product save request exists."}}, + "gpt-5.6-sol-moli-448": {"status":"verified","subcategory":{"code":"evidence_capture","zh":"网络证据匹配缺陷","en":"Network-evidence matching defect"},"reason":{"zh":"HAR 有字段正确的 302 /profile POST,页面也已更新;同一提交又被记录为 200,evaluator 只选中 200 记录而误判。","en":"The HAR has a correct 302 /profile POST and the page is updated, but a duplicate 200 record is selected by the evaluator, causing a false negative."}}, + "gpt-5.6-sol-chromium-470": {"status":"verified","subcategory":{"code":"site_dependency_failure","zh":"站点依赖故障","en":"Site dependency failure"},"reason":{"zh":"确认取消后 Magento 返回 No alive nodes found in your cluster,订单仍为 Pending;本地搜索服务不可用。","en":"After cancellation confirmation Magento returned No alive nodes found in your cluster and the order remained Pending; the local search service was unavailable."}}, + "gpt-5.6-sol-moli-470": {"status":"verified","subcategory":{"code":"site_dependency_failure","zh":"站点依赖故障","en":"Site dependency failure"},"reason":{"zh":"确认取消后 Magento 返回 No alive nodes found in your cluster,订单仍为 Pending;本地搜索服务不可用。","en":"After cancellation confirmation Magento returned No alive nodes found in your cluster and the order remained Pending; the local search service was unavailable."}}, + "gpt-5.6-sol-chromium-475": {"status":"verified","subcategory":{"code":"benchmark_form_contract","zh":"表单验收缺陷","en":"Form-acceptance defect"},"reason":{"zh":"空项目已按正确名称、路径和命名空间创建;HTML 会省略未勾选的 initialize_with_readme,官方规则却要求显式值 0。","en":"The empty project was created with the correct name, path, and namespace; HTML omits unchecked initialize_with_readme, while the official rule requires an explicit 0."}}, + "gpt-5.6-sol-moli-475": {"status":"verified","subcategory":{"code":"benchmark_form_contract","zh":"表单验收缺陷","en":"Form-acceptance defect"},"reason":{"zh":"空项目已按正确名称、路径和命名空间创建;HTML 会省略未勾选的 initialize_with_readme,官方规则却要求显式值 0。","en":"The empty project was created with the correct name, path, and namespace; HTML omits unchecked initialize_with_readme, while the official rule requires an explicit 0."}}, + "gpt-5.6-sol-moli-506": {"status":"verified","subcategory":{"code":"agent_step_budget","zh":"步骤预算耗尽","en":"Step-budget exhaustion"},"reason":{"zh":"达到 40 步时仍在购物流程;证据只有购物车增删,没有付款或成功页事件。","en":"At the 40-step limit the Agent was still shopping; evidence has cart changes but no payment or success-page event."}}, + "gpt-5.6-sol-chromium-510": {"status":"verified","subcategory":{"code":"agent_product_selection","zh":"商品选择错误","en":"Incorrect product selection"},"reason":{"zh":"Agent 完成结账,但买的是 Hiquphon Tweeters,而非官方目标 Atlantic Technology FS-7.0-GLB;筛选排序执行错误。","en":"The Agent completed checkout with Hiquphon Tweeters instead of the official Atlantic Technology FS-7.0-GLB target; the ranking was applied incorrectly."}}, + "gpt-5.6-sol-moli-510": {"status":"verified","subcategory":{"code":"agent_step_budget","zh":"步骤预算耗尽","en":"Step-budget exhaustion"},"reason":{"zh":"Agent 在 40 步内反复比较和调整购物车,最终没有付款成功事件。","en":"The Agent repeatedly compared products and changed the cart within 40 steps, but never produced a successful payment event."}}, + "gpt-5.6-sol-chromium-528": {"status":"verified","subcategory":{"code":"benchmark_instruction_contract","zh":"任务与验收矛盾","en":"Instruction-evaluator contradiction"},"reason":{"zh":"Agent 正确填写退款表单并按“不要提交”停在提交前;官方 evaluator 却要求出现 dummy_bin POST,任务与验收相互矛盾。","en":"The Agent correctly filled the refund form and stopped before submission, but the evaluator requires a dummy_bin POST, contradicting the instruction."}}, + "gpt-5.6-sol-moli-528": {"status":"verified","subcategory":{"code":"benchmark_instruction_contract","zh":"任务与验收矛盾","en":"Instruction-evaluator contradiction"},"reason":{"zh":"Agent 正确填写退款表单并按“不要提交”停在提交前;官方 evaluator 却要求出现 dummy_bin POST,任务与验收相互矛盾。","en":"The Agent correctly filled the refund form and stopped before submission, but the evaluator requires a dummy_bin POST, contradicting the instruction."}}, + "gpt-5.6-sol-chromium-547": {"status":"verified","subcategory":{"code":"agent_step_budget","zh":"步骤预算耗尽","en":"Step-budget exhaustion"},"reason":{"zh":"Agent 在 40 步内未完成新增 Brown 色板和 S 码变体,两个必要保存请求都不存在。","en":"The Agent did not finish adding the Brown swatch and size-S variation within 40 steps; both required save requests are absent."}}, + "gpt-5.6-sol-moli-547": {"status":"verified","subcategory":{"code":"site_backend_failure","zh":"站点保存故障","en":"Site save failure"},"reason":{"zh":"Agent 到达变体保存阶段后 Magento 返回 The stock item was unable to be saved,S/Brown 变体未建立。","en":"At the variation-save stage Magento returned The stock item was unable to be saved, so the S/Brown variation was not created."}}, + "gpt-5.6-sol-moli-595": {"status":"verified","subcategory":{"code":"evidence_capture","zh":"网络证据缺失","en":"Missing network evidence"},"reason":{"zh":"页面显示已订阅且 HAR 有 subscribe.json 200 POST,但 Moli 记录缺少 Referer 和响应 JSON,evaluator 无法确认 subscribed=true。","en":"The page shows subscribed and the HAR has a 200 subscribe.json POST, but Moli lacks Referer and response JSON, so the evaluator cannot confirm subscribed=true."}}, + "gpt-5.6-sol-chromium-600": {"status":"verified","subcategory":{"code":"benchmark_route_contract","zh":"路由验收缺陷","en":"Route-acceptance defect"},"reason":{"zh":"合法的 /submit/gaming 302 POST 含正确标题和论坛 ID 10039;evaluator 只接受精确 /submit。","en":"The valid /submit/gaming 302 POST contains the correct title and forum ID 10039; the evaluator accepts only exact /submit."}}, + "gpt-5.6-sol-moli-600": {"status":"verified","subcategory":{"code":"benchmark_route_contract","zh":"路由验收缺陷","en":"Route-acceptance defect"},"reason":{"zh":"合法的 /submit/gaming 302 POST 含正确标题和论坛 ID 10039;evaluator 只接受精确 /submit。","en":"The valid /submit/gaming 302 POST contains the correct title and forum ID 10039; the evaluator accepts only exact /submit."}}, + "gpt-5.6-sol-moli-615": {"status":"verified","subcategory":{"code":"agent_field_omission","zh":"提交字段遗漏","en":"Submission-field omission"},"reason":{"zh":"发帖请求包含正确标题和 Connecticut 论坛 ID,但 submission[url] 为空,遗漏 Bald Eagle 图片 URL。","en":"The post has the correct title and Connecticut forum ID, but submission[url] is empty, omitting the Bald Eagle image URL."}}, + "gpt-5.6-sol-chromium-625": {"status":"verified","subcategory":{"code":"benchmark_route_contract","zh":"路由验收缺陷","en":"Route-acceptance defect"},"reason":{"zh":"合法的 /submit/technology 302 POST 含正确标题、正文和论坛 ID 10043;evaluator 只接受精确 /submit。","en":"The valid /submit/technology 302 POST contains the correct title, body, and forum ID 10043; the evaluator accepts only exact /submit."}}, + "gpt-5.6-sol-moli-625": {"status":"verified","subcategory":{"code":"benchmark_route_contract","zh":"路由验收缺陷","en":"Route-acceptance defect"},"reason":{"zh":"合法的 /submit/technology 302 POST 含正确标题、正文和论坛 ID 10043;evaluator 只接受精确 /submit。","en":"The valid /submit/technology 302 POST contains the correct title, body, and forum ID 10043; the evaluator accepts only exact /submit."}}, + "gpt-5.6-sol-chromium-676": {"status":"verified","subcategory":{"code":"benchmark_evaluator_selection","zh":"网络事件选择缺陷","en":"Network-event selection defect"},"reason":{"zh":"HAR 有完整 sales_order_grid 与 status=fraud 的 XHR。官方 evaluator 对 navigate + GET 只检查最后一次文档导航,随后又在同一路径事件中只取最后一条 notification_area,因而漏掉真正应验收的筛选请求;修复选择逻辑后,该证据可通过。","en":"The HAR contains the complete sales_order_grid XHR with status=fraud. For navigate-plus-GET tasks, the official evaluator checks only the last document navigation and then only the last same-route event, which skips the request that should be evaluated. The saved evidence passes after correcting event selection."}}, + "gpt-5.6-sol-moli-676": {"status":"verified","subcategory":{"code":"evidence_capture","zh":"网络证据缺失","en":"Missing network evidence"},"reason":{"zh":"HAR 有完整 sales_order_grid 与 status=fraud 请求,但 Moli 未记录 Referer,evaluator 无法匹配。","en":"The HAR has the complete sales_order_grid request with status=fraud, but Moli omits its Referer, preventing evaluator matching."}}, + "gpt-5.6-sol-chromium-694": {"status":"verified","subcategory":{"code":"site_backend_failure","zh":"站点保存故障","en":"Site save failure"},"reason":{"zh":"Agent 两次创建商品均被 Magento 以 The stock item was unable to be saved 拒绝,目录仍为空。","en":"Magento rejected both product-creation attempts with The stock item was unable to be saved, leaving the catalog empty."}}, + "gpt-5.6-sol-moli-694": {"status":"verified","subcategory":{"code":"site_backend_failure","zh":"站点保存故障","en":"Site save failure"},"reason":{"zh":"Agent 两次创建商品均被 Magento 以 The stock item was unable to be saved 拒绝,商品未创建。","en":"Magento rejected both product-creation attempts with The stock item was unable to be saved, so the product was not created."}}, + "gpt-5.6-sol-chromium-699": {"status":"verified","subcategory":{"code":"benchmark_evaluator_defect","zh":"evaluator 配置错误并伴随 Agent 选型错误","en":"Evaluator configuration defect with an Agent selection error"},"reason":{"zh":"官方 evaluator 把两个单值表单字段写成单元素 Alternatives,尚未比较请求就抛错。用正确字段重放后,证据显示 Agent 保存的是 Catalog Price Rule,而合同要求 Cart Price Rule,因此这条历史结果不能改判为通过。","en":"The official evaluator encodes two scalar form fields as one-item Alternatives and errors before comparing the request. Re-evaluation with corrected fields shows that the Agent saved a Catalog Price Rule while the contract requires a Cart Price Rule, so the historical result cannot be converted to a pass."}}, + "gpt-5.6-sol-moli-699": {"status":"verified","subcategory":{"code":"benchmark_evaluator_defect","zh":"evaluator 配置错误并伴随 Agent 选型错误","en":"Evaluator configuration defect with an Agent selection error"},"reason":{"zh":"官方 evaluator 把两个单值表单字段写成单元素 Alternatives,尚未比较请求就抛错。用正确字段重放后,证据显示 Agent 保存的是 Catalog Price Rule,而合同要求 Cart Price Rule,因此这条历史结果不能改判为通过。","en":"The official evaluator encodes two scalar form fields as one-item Alternatives and errors before comparing the request. Re-evaluation with corrected fields shows that the Agent saved a Catalog Price Rule while the contract requires a Cart Price Rule, so the historical result cannot be converted to a pass."}}, + "gpt-5.6-sol-chromium-771": {"status":"verified","subcategory":{"code":"agent_workflow_completeness","zh":"批量操作遗漏","en":"Incomplete batch workflow"},"reason":{"zh":"任务要求批准全部至少 4 星评论;Chromium 没有保存官方要求的评论 352 和 347,而 Moli 在相同条件下完成了两项。","en":"The task requires approving all reviews rated at least four stars; Chromium did not save required reviews 352 and 347, while Moli completed both under the same controls."}} + } + }, + "gpt-6-astra": { + "reviewed_at": "2026-09-11", + "reviewer_class": "evidence analyst", + "bindings": { + "attempts_sha256": "0659d227cf6d29fbfaf64d39f8e77a56bec73c87f7f6e1f2ac23ea13fd9b667c", + "manifest_sha256": "ea26ccbeb335126d91d8c2f9f229e445aca51137fbf31333d9b5db05f8508538", + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b" + }, + "review_protocol": { + "unit": "one score-bearing logical result per browser and task", + "method": "manual root-cause verification against the official task contract, score-bearing final response, evaluator assertions, network evidence, and autonomous Agent event trajectory", + "rule": "automatic evaluator classification remains available; root-cause subcategory and reason remain empty until status is verified" + }, + "coverage": {"failed_results": 38, "verified_results": 38}, + "unresolved": [], + "results": { + "gpt-6-astra-chromium-77": {"status":"verified","subcategory":{"code":"environment_concurrency_contamination","zh":"共享环境并发污染","en":"Shared-environment concurrency contamination"},"reason":{"zh":"准备完成后、Agent 读取前,共享 Shopping Admin 状态被并发执行改写:评论 352、347 从 Pending 变为 Approved,Pending 从合同基线 5 条降为 3 条。全新重建同一镜像可恢复 5 条,排除了镜像基线错误;Agent 对受污染页面的读取正确。","en":"After preparation and before the Agent read the page, a concurrent execution modified the shared Shopping Admin state: reviews 352 and 347 changed from Pending to Approved, reducing the contractual count from five to three. Recreating the same image restores five, ruling out a bad image baseline; the Agent correctly read the contaminated page."}}, + "gpt-6-astra-moli-77": {"status":"verified","subcategory":{"code":"environment_concurrency_contamination","zh":"共享环境并发污染","en":"Shared-environment concurrency contamination"},"reason":{"zh":"准备完成后、Agent 读取前,共享 Shopping Admin 状态被并发执行改写:评论 352、347 从 Pending 变为 Approved,Pending 从合同基线 5 条降为 3 条。全新重建同一镜像可恢复 5 条,排除了镜像基线错误;Agent 还通过显式筛选确认了受污染状态。","en":"After preparation and before the Agent read the page, a concurrent execution modified the shared Shopping Admin state: reviews 352 and 347 changed from Pending to Approved, reducing the contractual count from five to three. Recreating the same image restores five, ruling out a bad image baseline; the Agent also confirmed the contaminated state with an explicit filter."}}, + "gpt-6-astra-chromium-124": {"status":"verified","subcategory":{"code":"agent_retrieval_strategy","zh":"检索策略错误","en":"Retrieval-strategy error"},"reason":{"zh":"Agent 把 wireless earphones 放宽为 earphones,并把局部排序接口当作全集,返回 287.64,遗漏官方上界 298。","en":"The Agent broadened wireless earphones to earphones and treated a partial sorted endpoint as exhaustive, returning 287.64 instead of the official upper bound of 298."}}, + "gpt-6-astra-moli-124": {"status":"verified","subcategory":{"code":"agent_retrieval_strategy","zh":"检索策略错误","en":"Retrieval-strategy error"},"reason":{"zh":"Agent 把 wireless earphones 放宽为 earphones,并把局部排序接口当作全集,返回 287.64,遗漏官方上界 298。","en":"The Agent broadened wireless earphones to earphones and treated a partial sorted endpoint as exhaustive, returning 287.64 instead of the official upper bound of 298."}}, + "gpt-6-astra-chromium-163": {"status":"verified","subcategory":{"code":"agent_deduplication","zh":"结果去重遗漏","en":"Result-deduplication failure"},"reason":{"zh":"Magento 重叠分页重复出现同一条 Ernesto 20% 评论;Agent 没有按评论去重,最终重复输出。","en":"Overlapping Magento pages repeated the same Ernesto 20% review, and the Agent returned it twice instead of deduplicating reviews."}}, + "gpt-6-astra-moli-163": {"status":"verified","subcategory":{"code":"agent_deduplication","zh":"结果去重遗漏","en":"Result-deduplication failure"},"reason":{"zh":"Magento 重叠分页重复出现同一条 Ernesto 20% 评论;Agent 没有按评论去重,最终重复输出。","en":"Overlapping Magento pages repeated the same Ernesto 20% review, and the Agent returned it twice instead of deduplicating reviews."}}, + "gpt-6-astra-chromium-258": {"status":"verified","subcategory":{"code":"agent_navigation_semantics","zh":"导航语义错误","en":"Navigation-semantics error"},"reason":{"zh":"Agent 停在 GitLab Explore 总页 /explore,没有启用 Public 过滤参数 visibility_level=20。","en":"The Agent stopped at generic /explore without applying the Public filter visibility_level=20."}}, + "gpt-6-astra-moli-258": {"status":"verified","subcategory":{"code":"agent_navigation_semantics","zh":"导航语义错误","en":"Navigation-semantics error"},"reason":{"zh":"Agent 停在 GitLab Explore 总页 /explore,没有启用 Public 过滤参数 visibility_level=20。","en":"The Agent stopped at generic /explore without applying the Public filter visibility_level=20."}}, + "gpt-6-astra-chromium-279": {"status":"verified","subcategory":{"code":"mixed_agent_filtering_reference_state","zh":"检索遗漏与参考状态差异","en":"Retrieval omission and reference-state drift"},"reason":{"zh":"Agent 已抓取 H900N 却漏掉最终输出;同时当前 C300 名称比冻结参考多一个右括号,Agent 遗漏与参考状态漂移共同造成不匹配。","en":"The Agent fetched H900N but omitted it from the final output; the live C300 name also has an extra closing parenthesis relative to the frozen reference, so omission and reference-state drift jointly caused the mismatch."}}, + "gpt-6-astra-moli-279": {"status":"verified","subcategory":{"code":"mixed_agent_filtering_reference_state","zh":"检索遗漏与参考状态差异","en":"Retrieval omission and reference-state drift"},"reason":{"zh":"Agent 已抓取 H900N 却漏掉最终输出;同时当前 C300 名称比冻结参考多一个右括号,Agent 遗漏与参考状态漂移共同造成不匹配。","en":"The Agent fetched H900N but omitted it from the final output; the live C300 name also has an extra closing parenthesis relative to the frozen reference, so omission and reference-state drift jointly caused the mismatch."}}, + "gpt-6-astra-chromium-339": {"status":"verified","subcategory":{"code":"benchmark_route_contract","zh":"路由验收缺陷","en":"Route-acceptance defect"},"reason":{"zh":"Agent 已打开带 state=opened 与 bug 标签的合法 /issues?... 页面;官方规则只接受 /issues/.*,错误拒绝无尾斜杠路由。","en":"The Agent opened the valid /issues?... route with state=opened and the bug label, but the official /issues/.* rule incorrectly rejects the no-trailing-slash route."}}, + "gpt-6-astra-moli-339": {"status":"verified","subcategory":{"code":"benchmark_route_contract","zh":"路由验收缺陷","en":"Route-acceptance defect"},"reason":{"zh":"Agent 已打开正确筛选页;官方规则错误拒绝无尾斜杠 /issues?... 路由,且 Moli 记录还缺少 Referer。","en":"The Agent opened the correct filtered page; the official rule rejects the valid no-trailing-slash /issues?... route, and Moli's record also omits the Referer."}}, + "gpt-6-astra-chromium-448": {"status":"verified","subcategory":{"code":"evidence_capture","zh":"网络证据匹配缺陷","en":"Network-evidence matching defect"},"reason":{"zh":"个人资料更新已成功,HAR 含字段正确的 302 请求和重复的 200 记录;evaluator 选中后者而误判。","en":"The profile update succeeded. HAR contains the correct 302 request and a duplicate 200 record, but the evaluator selected the latter and produced a false negative."}}, + "gpt-6-astra-moli-448": {"status":"verified","subcategory":{"code":"evidence_capture","zh":"网络证据匹配缺陷","en":"Network-evidence matching defect"},"reason":{"zh":"个人资料更新已成功,HAR 含字段正确的 302 请求和重复的 200 记录;evaluator 选中后者而误判。","en":"The profile update succeeded. HAR contains the correct 302 request and a duplicate 200 record, but the evaluator selected the latter and produced a false negative."}}, + "gpt-6-astra-chromium-464": {"status":"verified","subcategory":{"code":"agent_field_omission","zh":"提交字段错误","en":"Submission-field error"},"reason":{"zh":"Agent 正确算出 1,却写入 product[description];官方要求的 product[short_description] 为空。后续库存保存错误不改变该独立失败原因。","en":"The Agent correctly derived 1 but wrote it to product[description], leaving the required product[short_description] empty. A later stock-save error does not change this independent failure."}}, + "gpt-6-astra-moli-464": {"status":"verified","subcategory":{"code":"agent_field_omission","zh":"提交字段错误","en":"Submission-field error"},"reason":{"zh":"Agent 正确算出 1,却写入 product[description];官方要求的 product[short_description] 为空。后续库存保存错误不改变该独立失败原因。","en":"The Agent correctly derived 1 but wrote it to product[description], leaving the required product[short_description] empty. A later stock-save error does not change this independent failure."}}, + "gpt-6-astra-chromium-470": {"status":"verified","subcategory":{"code":"site_dependency_failure","zh":"站点依赖故障","en":"Site dependency failure"},"reason":{"zh":"Agent 已提交取消请求;Magento 随后报告搜索集群无可用节点,订单仍为 Pending。","en":"The Agent submitted the cancellation, but Magento then reported no live search-cluster nodes and the order remained Pending."}}, + "gpt-6-astra-moli-470": {"status":"verified","subcategory":{"code":"site_dependency_failure","zh":"站点依赖故障","en":"Site dependency failure"},"reason":{"zh":"Agent 已提交取消请求;Magento 随后报告搜索集群无可用节点,订单仍为 Pending。","en":"The Agent submitted the cancellation, but Magento then reported no live search-cluster nodes and the order remained Pending."}}, + "gpt-6-astra-chromium-475": {"status":"verified","subcategory":{"code":"benchmark_form_contract","zh":"表单验收缺陷","en":"Form-acceptance defect"},"reason":{"zh":"空项目已按正确名称、路径和命名空间创建;未勾选的 HTML checkbox 按标准行为不提交字段,官方规则却要求 initialize_with_readme=0。","en":"The empty project was created correctly. Standard HTML omits an unchecked checkbox, while the official rule requires initialize_with_readme=0."}}, + "gpt-6-astra-moli-475": {"status":"verified","subcategory":{"code":"benchmark_form_contract","zh":"表单验收缺陷","en":"Form-acceptance defect"},"reason":{"zh":"空项目已按正确名称、路径和命名空间创建;未勾选的 HTML checkbox 按标准行为不提交字段,官方规则却要求 initialize_with_readme=0。","en":"The empty project was created correctly. Standard HTML omits an unchecked checkbox, while the official rule requires initialize_with_readme=0."}}, + "gpt-6-astra-moli-506": {"status":"verified","subcategory":{"code":"evidence_capture","zh":"响应证据缺失","en":"Missing response evidence"},"reason":{"zh":"Agent 清空旧购物车、加入正确商品 22490 并付款到成功页;Moli 把 /carts/mine/totals 响应正文记录为空,evaluator 无法确认商品。","en":"The Agent cleared the cart, added the correct product 22490, and completed payment. Moli recorded an empty /carts/mine/totals response body, preventing evaluator confirmation."}}, + "gpt-6-astra-chromium-510": {"status":"verified","subcategory":{"code":"agent_product_selection","zh":"商品选择错误","en":"Incorrect product selection"},"reason":{"zh":"Agent 完成结账,但购买商品 101685 Hiquphon Tweeters,而非合同要求的 75640 Atlantic Technology FS-7.0-GLB。","en":"The Agent completed checkout with product 101685, Hiquphon Tweeters, instead of required product 75640, Atlantic Technology FS-7.0-GLB."}}, + "gpt-6-astra-moli-510": {"status":"verified","subcategory":{"code":"agent_product_selection","zh":"商品选择错误","en":"Incorrect product selection"},"reason":{"zh":"Agent 完成结账,但购买商品 101685 Hiquphon Tweeters,而非合同要求的 75640 Atlantic Technology FS-7.0-GLB。","en":"The Agent completed checkout with product 101685, Hiquphon Tweeters, instead of required product 75640, Atlantic Technology FS-7.0-GLB."}}, + "gpt-6-astra-chromium-528": {"status":"verified","subcategory":{"code":"benchmark_instruction_contract","zh":"任务与验收矛盾","en":"Instruction-evaluator contradiction"},"reason":{"zh":"Agent 按要求填写退款表单并停在提交前;官方 evaluator 却要求出现 dummy_bin POST,与“不要提交”直接冲突。","en":"The Agent filled the refund form and stopped before submission, but the evaluator requires a dummy_bin POST, directly contradicting the instruction not to submit."}}, + "gpt-6-astra-moli-528": {"status":"verified","subcategory":{"code":"benchmark_instruction_contract","zh":"任务与验收矛盾","en":"Instruction-evaluator contradiction"},"reason":{"zh":"Agent 按要求填写退款表单并停在提交前;官方 evaluator 却要求出现 dummy_bin POST,与“不要提交”直接冲突。","en":"The Agent filled the refund form and stopped before submission, but the evaluator requires a dummy_bin POST, directly contradicting the instruction not to submit."}}, + "gpt-6-astra-chromium-547": {"status":"verified","subcategory":{"code":"agent_workflow_completeness","zh":"流程未完成","en":"Incomplete workflow"},"reason":{"zh":"Agent 仅执行 10 个决策回合,在打开配置向导后主动结束;没有新增 Brown 色板,也没有必要保存请求。","en":"The Agent voluntarily ended after 10 decision turns shortly after opening the configuration wizard; it created no Brown swatch and issued no required save request."}}, + "gpt-6-astra-moli-547": {"status":"verified","subcategory":{"code":"agent_workflow_completeness","zh":"流程未完成","en":"Incomplete workflow"},"reason":{"zh":"Agent 仅执行 10 个决策回合,在商品列表阶段主动结束;没有进入完整编辑与保存流程。","en":"The Agent voluntarily ended after 10 decision turns at the product-list stage without completing the edit and save workflow."}}, + "gpt-6-astra-chromium-585": {"status":"verified","subcategory":{"code":"agent_workflow_completeness","zh":"可恢复错误后提前终止","en":"Premature stop after recoverable error"},"reason":{"zh":"进入订单历史时出现可恢复的 snapshot 导航竞态;HAR 显示页面已返回 200,但 Agent 直接终止,未继续定位落地灯或提交评论。","en":"A recoverable snapshot/navigation race occurred while opening order history. HAR shows the page returned 200, but the Agent stopped without locating the floor lamp or submitting a review."}}, + "gpt-6-astra-moli-595": {"status":"verified","subcategory":{"code":"evidence_capture","zh":"网络证据缺失","en":"Missing network evidence"},"reason":{"zh":"订阅实际成功,页面变为 Unsubscribe,trace 响应为 subscribed=true;Moli HAR 缺 Referer 和响应 JSON,evaluator 因而无法匹配。","en":"The subscription succeeded: the UI changed to Unsubscribe and trace returned subscribed=true. Moli HAR omitted the Referer and response JSON, preventing evaluator matching."}}, + "gpt-6-astra-chromium-605": {"status":"verified","subcategory":{"code":"site_dependency_failure","zh":"站点会话故障","en":"Site session failure"},"reason":{"zh":"Agent 提交了正确论坛、标题和与新表单一致的 CSRF token;站点连续两次仍返回 CSRF token is invalid。","en":"The Agent submitted the correct forum, title, and CSRF token from the freshly loaded form, yet the site rejected both executions with CSRF token is invalid."}}, + "gpt-6-astra-chromium-676": {"status":"verified","subcategory":{"code":"benchmark_evaluator_selection","zh":"网络事件选择缺陷","en":"Network-event selection defect"},"reason":{"zh":"Agent 正确筛选 status=fraud,页面显示 Suspected Fraud,HAR 也有完整 XHR。官方 evaluator 的 navigate + GET 快捷逻辑只检查文档导航,且默认只取同路径最后一条事件,漏掉目标筛选请求;修复后保存证据可通过。","en":"The Agent correctly filtered status=fraud, the UI showed Suspected Fraud, and the HAR contains the complete XHR. The official navigate-plus-GET shortcut checks only document navigation and then defaults to the last same-route event, skipping the target filter request. The saved evidence passes after the fix."}}, + "gpt-6-astra-moli-676": {"status":"verified","subcategory":{"code":"evidence_capture","zh":"网络证据缺失","en":"Missing network evidence"},"reason":{"zh":"Agent 发出完整的欺诈订单筛选请求且页面显示 Suspected Fraud;Moli 未记录该 AJAX 请求的 Referer,evaluator 无法匹配。","en":"The Agent issued the complete fraudulent-order filter request and the UI showed Suspected Fraud, but Moli omitted the AJAX Referer, preventing evaluator matching."}}, + "gpt-6-astra-chromium-694": {"status":"verified","subcategory":{"code":"site_backend_failure","zh":"站点保存故障","en":"Site save failure"},"reason":{"zh":"合同字段均正确且网络验收通过;Magento 随后返回 The stock item was unable to be saved,商品未创建。","en":"All contract fields were correct and network acceptance passed, but Magento returned The stock item was unable to be saved and did not create the product."}}, + "gpt-6-astra-moli-694": {"status":"verified","subcategory":{"code":"site_backend_failure","zh":"站点保存故障","en":"Site save failure"},"reason":{"zh":"合同字段和 Top attribute set 均正确且网络验收通过;Magento 返回库存项保存失败,商品未创建。","en":"The contract fields and Top attribute set were correct and network acceptance passed, but Magento returned a stock-item save failure and did not create the product."}}, + "gpt-6-astra-chromium-699": {"status":"verified","subcategory":{"code":"benchmark_evaluator_defect","zh":"evaluator 配置错误并伴随 Agent 选型错误","en":"Evaluator configuration defect with an Agent selection error"},"reason":{"zh":"官方 evaluator 把两个单值表单字段写成单元素 Alternatives,尚未比较请求就抛错。用正确字段重放后,证据显示 Agent 保存的是 Catalog Price Rule,而合同要求 Cart Price Rule,因此这条历史结果不能改判为通过。","en":"The official evaluator encodes two scalar form fields as one-item Alternatives and errors before comparing the request. Re-evaluation with corrected fields shows that the Agent saved a Catalog Price Rule while the contract requires a Cart Price Rule, so the historical result cannot be converted to a pass."}}, + "gpt-6-astra-moli-699": {"status":"verified","subcategory":{"code":"benchmark_evaluator_defect","zh":"evaluator 配置错误并伴随 Agent 选型错误","en":"Evaluator configuration defect with an Agent selection error"},"reason":{"zh":"官方 evaluator 把两个单值表单字段写成单元素 Alternatives,尚未比较请求就抛错。用正确字段重放后,证据显示 Agent 保存的是 Catalog Price Rule,而合同要求 Cart Price Rule,因此这条历史结果不能改判为通过。","en":"The official evaluator encodes two scalar form fields as one-item Alternatives and errors before comparing the request. Re-evaluation with corrected fields shows that the Agent saved a Catalog Price Rule while the contract requires a Cart Price Rule, so the historical result cannot be converted to a pass."}}, + "gpt-6-astra-chromium-771": {"status":"verified","subcategory":{"code":"benchmark_instruction_contract","zh":"任务状态与验收矛盾","en":"State-evaluator contradiction"},"reason":{"zh":"最终执行开始时评论 352、347 已是 Approved,任务语义已经满足;evaluator 仍强制要求重复提交 status_id=1。","en":"Reviews 352 and 347 were already Approved at the start of the final execution, so the task was already satisfied, while the evaluator still required redundant status_id=1 submissions."}}, + "gpt-6-astra-moli-771": {"status":"verified","subcategory":{"code":"benchmark_instruction_contract","zh":"任务状态与验收矛盾","en":"State-evaluator contradiction"},"reason":{"zh":"最终执行开始时评论 352、347 已是 Approved,任务语义已经满足;evaluator 仍强制要求重复提交 status_id=1。","en":"Reviews 352 and 347 were already Approved at the start of the final execution, so the task was already satisfied, while the evaluator still required redundant status_id=1 submissions."}} + } + } + } +} diff --git a/research/browser-use-remediation.json b/research/browser-use-remediation.json new file mode 100644 index 000000000..7d31164a0 --- /dev/null +++ b/research/browser-use-remediation.json @@ -0,0 +1,44 @@ +{ + "schema": "lex_browser_use_remediation/1", + "updated_at": "2026-09-14", + "score_policy": "Raw evaluator outputs remain unchanged. A task is excluded from the formal comparison across every configuration when any verified result is affected by benchmark/evaluator, site, shared-environment, or mixed reference-state interference. Readmission requires a corrected contract or runtime, independent preflight, and a complete four-configuration rerun.", + "items": [ + { + "category": "shared_environment", + "task_ids": [77, 771], + "status": "implementation_fixed_rerun_required", + "action": "The full physical-attempt lifecycle is now protected by a cross-process lock. Rerun the affected task before admitting it to the clean comparison." + }, + { + "category": "benchmark_or_evaluator", + "task_ids": [339, 475, 600, 625, 676, 699], + "status": "correction_implemented_upstream_review_and_rerun_required", + "upstream_pr": "https://github.com/ServiceNow/webarena-verified/pull/45", + "action": "The corrected contracts and network-event selection logic pass regression tests and replay checks against saved HAR evidence. Merge the upstream correction, then rerun the complete four-configuration scope before readmission." + }, + { + "category": "benchmark_instruction_contract", + "task_ids": [528], + "status": "quarantined_pending_portable_evaluator", + "action": "Replace the synthetic dummy_bin POST requirement with a portable observable-state evaluator for the filled-but-unsubmitted form, then validate and rerun all four configurations." + }, + { + "category": "self_hosted_site", + "task_ids": [470, 547, 605, 694], + "status": "quarantined_pending_site_fix", + "action": "Repair the dependency or save path, add task-relevant read/write readiness checks, then rerun the complete four-configuration scope." + }, + { + "category": "reference_state_mixed", + "task_ids": [279], + "status": "quarantined_pending_reference_refreeze", + "action": "Refreeze the expected state before execution and rerun all configurations so the Agent contribution can be separated from reference drift." + }, + { + "category": "browser_evidence_capture", + "task_ids": [448, 506, 595, 676], + "status": "retained_as_browser_integration_signal", + "action": "Keep in the primary browser comparison because HAR completeness is part of the browser integration contract; improve response-body and Referer capture and rerun." + } + ] +} diff --git a/research/report-changes.json b/research/report-changes.json index dddb5c5ab..16e18dfce 100644 --- a/research/report-changes.json +++ b/research/report-changes.json @@ -1,6 +1,1158 @@ { - "schema_version": 1, + "schema_version": 2, "history_scope": "本页从有完整审计依据的修订开始记录。更早的站点版本仍保留在 Artifact Hub,但没有逐项变更说明,故不补编历史。", + "history_scope_i18n": { + "en": "This history begins with revisions supported by a complete audit trail. Earlier Artifact Hub versions remain available, but no itemized history is reconstructed where the supporting record is incomplete." + }, + "trajectory_replay_releases": [ + { + "id": "2026-09-15-webarena-verified-90-artifact-site", + "date": "2026-09-15", + "title": "发布 90 题浏览器轨迹重放评测", + "summary": "发布 Chromium 与 Moli 1.1.3 的 90 题固定轨迹重放结果:Chromium 90/90,Moli 54/90;36 个最终失败全部完成证据绑定复核。Agent 与模型调用为零。", + "publication": { + "service": "Artifact Site", + "site_slug": "7dF7uNK5SKIx", + "share_url": "https://artifact-hub.app.lexmount.com/v/hkZt38QFBpNIKtThKnqjpnBQk8At19PM", + "version": "ver_e92fe2d3c5a24001a36593fe6ca6fb4e" + }, + "effects": { + "score": "首次发布;Chromium 90/90,Moli 54/90。", + "denominator": "每个浏览器 90 题,共 180 个逻辑结果。", + "latency": "发布均值、P50 与 P95;共同通过的 54 题用于端到端、CPU 与内存比较。", + "evidence": "591 个公开文件、372 个 HTML 页面和 4,856 条本地链接通过检查;36/36 个失败已复核。", + "limitations": "每个浏览器每题只有一个正式逻辑结果;结论限于冻结任务、版本和本地环境。" + }, + "i18n": { + "en": { + "title": "Publish the 90-task browser trajectory-replay evaluation", + "summary": "Published fixed-trajectory replay results for Chromium and Moli 1.1.3 across 90 tasks: Chromium 90/90 and Moli 54/90. All 36 final failures have evidence-bound reviews. Agent and model calls are zero.", + "effects": { + "score": "Initial release: Chromium 90/90 and Moli 54/90.", + "denominator": "90 tasks per browser and 180 logical results in total.", + "latency": "Mean, P50, and P95 are reported; end-to-end, CPU, and memory comparisons use the 54 tasks both browsers passed.", + "evidence": "Checks passed for 591 public files, 372 HTML pages, and 4,856 local links; all 36 failures are reviewed.", + "limitations": "There is one formal logical result per browser and task. Conclusions are limited to the frozen tasks, versions, and local environment." + } + } + } + }, + { + "id": "2026-09-15-trajectory-replay-drilldown", + "date": "2026-09-15", + "title": "补齐轨迹重放报告的逐题诊断与站点导航", + "summary": "新增每题的目标、验收输入和标准动作页;结果页可筛选同题浏览器成败关系,并从失败结果下钻到确切报错、失败动作、复核原因、完整工具返回、页面截图和公开证据。核心指标增加资源与网络统计。", + "publication": { + "service": "Artifact Site", + "site_slug": "7dF7uNK5SKIx", + "share_url": "https://artifact-hub.app.lexmount.com/v/hkZt38QFBpNIKtThKnqjpnBQk8At19PM", + "version": "ver_0f4f162b973a4ba7afba2fb4e18d6173" + }, + "effects": { + "score": "Chromium 90/90、Moli 54/90,分数未变。", + "denominator": "仍为每浏览器 90 题、共 180 个逻辑结果。", + "latency": "原有耗时未变;补充共同通过 54 题的资源与网络统计。", + "evidence": "771 个公开文件、552 个 HTML 页面、10,154 条本地链接和 72 个视觉状态通过检查;Sites 嵌套详情导航已实测恢复。", + "limitations": "逐题证据用于定位本轮失败,不能从单次结果外推总体稳定性。" + }, + "i18n": { + "en": { + "title": "Add per-task replay diagnosis and repair hosted navigation", + "summary": "Added task pages for goals, evaluation input, and qualified actions. Results now filter same-task outcome relations and drill down to the exact error, failed action, reviewed cause, full tool response, last page, and public evidence. Core metrics add resource and network statistics.", + "effects": { + "score": "Scores remain Chromium 90/90 and Moli 54/90.", + "denominator": "Still 90 tasks per browser and 180 logical outcomes.", + "latency": "Measured timings are unchanged; resource and network statistics use the 54 shared passes.", + "evidence": "Checks covered 771 public files, 552 HTML pages, 10,154 local links, and 72 visual states. Hosted navigation from nested result pages was verified.", + "limitations": "Per-task evidence locates failures in this run; one outcome per task does not establish general reliability." + } + } + } + }, + { + "id": "2026-09-15-trajectory-replay-browser-comparison-and-submission-diagnosis", + "date": "2026-09-15", + "title": "扩展多浏览器对比并定位缺失的提交请求", + "summary": "同题结果改为可选择基准与多个浏览器的成败分组;Moli 第 464、465 题由笼统的验收失败细化为验收请求未出现,并链接到可能触发提交的动作及原始判定。", + "publication": { + "service": "Artifact Site", + "site_slug": "7dF7uNK5SKIx", + "share_url": "https://artifact-hub.app.lexmount.com/v/hkZt38QFBpNIKtThKnqjpnBQk8At19PM", + "version": "ver_ddc682f036c347a9a23c0831d611854c" + }, + "effects": { + "score": "Chromium 90/90、Moli 54/90;原始评分未变。", + "denominator": "每浏览器 90 题、共 180 个逻辑结果。", + "latency": "无计时或测量修改。", + "evidence": "771 个公开文件、552 个 HTML 页面、10,154 条本地链接和 72 个视觉状态通过检查;178 份逐题证据文件与前版相同,2 份补充诊断字段。", + "limitations": "缺失请求不能改判为通过;第 464、465 题的轨迹或浏览器行为修复仍需重新资格验证和正式比较。" + }, + "i18n": { + "en": { + "title": "Expand multi-browser comparison and locate absent submission requests", + "summary": "Same-task outcomes now group passes and failures with a selectable baseline and multiple comparison browsers. Moli tasks 464 and 465 identify an absent required request instead of a generic evaluator failure, with links to plausible trigger actions and the original finding.", + "effects": { + "score": "Scores remain Chromium 90/90 and Moli 54/90.", + "denominator": "90 tasks per browser and 180 logical outcomes.", + "latency": "No timing or measurement changes.", + "evidence": "Checks passed for 771 public files, 552 HTML pages, 10,154 local links, and 72 visual states. 178 result evidence files match the previous version; two add diagnosis fields.", + "limitations": "An absent required request cannot be rescored as a pass. Repairs to the trajectories or browser behavior for tasks 464 and 465 require renewed qualification and formal comparison." + } + } + } + }, + { + "id": "2026-09-15-trajectory-replay-dataset-filter-and-navigation", + "date": "2026-09-15", + "title": "补齐评测集筛选、逐题结果入口与核验轮次", + "summary": "评测集按任务目标、站点、类型和轨迹来源筛选;每题分别下钻到任务合同与全部浏览器实测结果。核验页按范围、发现、纠正和未完成记录两轮核验。结果页统一命名为“结果分析”,环境页统一命名为“环境与方法”;拓扑图中英文分别标注。长任务目标在列表中以两行预览,全文仍在详情页。", + "publication": { + "service": "Artifact Site", + "site_slug": "7dF7uNK5SKIx", + "share_url": "https://artifact-hub.app.lexmount.com/v/hkZt38QFBpNIKtThKnqjpnBQk8At19PM", + "version": "ver_fe02b959308441a1b88e01d3fa198ba4" + }, + "effects": { + "score": "Chromium 90/90、Moli 54/90;官方分数未变。", + "denominator": "每浏览器 90 题,共 180 个逻辑结果。", + "latency": "未修改计时或资源测量。", + "evidence": "771 个公开文件、552 个 HTML 页面、10,512 条站内链接和 72 个视觉状态通过检查;评测集到同题结果的点击路径已验证。", + "limitations": "列表预览不代替完整任务合同;每题的完整轨迹与验收输入须在任务详情查看。" + }, + "i18n": { + "en": { + "title": "Add dataset filters, exact-task drilldown, and review rounds", + "summary": "The dataset now filters by goal, site, task type, and trajectory source. Each row links separately to the task contract and all measured browser outcomes. The review page records two rounds by scope, finding, correction, and open items. Results analysis and Environment and method use consistent navigation labels; the topology is localized. Long goals are previewed in two lines while the task page retains the full text.", + "effects": { + "score": "Official scores remain Chromium 90/90 and Moli 54/90.", + "denominator": "90 tasks per browser and 180 logical outcomes in total.", + "latency": "No timing or resource measurement changes.", + "evidence": "Checks passed for 771 public files, 552 HTML pages, 10,512 local links, and 72 visual states. The dataset-to-outcome click path was exercised.", + "limitations": "The list preview does not replace the full task contract. Open a task page for its complete trajectory and acceptance input." + } + } + } + }, + { + "id": "2026-09-15-trajectory-replay-aligned-step-evidence", + "date": "2026-09-15", + "title": "逐步并排对照标准轨迹与浏览器执行", + "summary": "逐题结果页按动作序号并排展示已验证的固定轨迹与实际执行,保留未执行步骤;点击任一步可查看标准与实际参数、完整工具返回、日志及原始报错。最后页面截图明确不冒充该步即时截图。", + "publication": { + "service": "Artifact Site", + "site_slug": "7dF7uNK5SKIx", + "share_url": "https://artifact-hub.app.lexmount.com/v/hkZt38QFBpNIKtThKnqjpnBQk8At19PM", + "version": "ver_fcd443ac19fe403f8baa3442262246e1" + }, + "effects": { + "score": "Chromium 90/90、Moli 54/90;评分未变。", + "denominator": "每浏览器 90 题,共 180 个逻辑结果。", + "latency": "未修改计时或资源测量。", + "evidence": "标准轨迹和逐次执行证据按需读取;861 个公开文件、552 个 HTML 页面、10,872 条本地链接和 72 个视觉状态通过检查。", + "limitations": "轨迹是经验证的可行输入,不代表任务唯一正确操作路径;最后页面截图不等同于动作报错瞬间。" + }, + "i18n": { + "en": { + "title": "Align qualified actions with each browser's execution", + "summary": "Task results align the qualified fixed trajectory with actual execution by action index and retain steps that were not executed. Opening a step shows qualified and actual arguments, the complete tool response, logs, and exact error. The last observed page is not mislabeled as a screenshot of that step.", + "effects": { + "score": "Scores remain Chromium 90/90 and Moli 54/90.", + "denominator": "90 tasks per browser and 180 logical outcomes in total.", + "latency": "No timing or resource measurements changed.", + "evidence": "Qualified trajectories and execution evidence load on demand. Checks passed for 861 public files, 552 HTML pages, 10,872 local links, and 72 visual states.", + "limitations": "The trajectory is one qualified feasible input, not the only correct route. The last observed page is not necessarily the moment of an action error." + } + } + } + } + ], + "browser_use_reviews": [ + { + "id": "review-2026-09-14-complete-90-task-primary-result", + "date": "2026-09-14", + "title": "恢复完整 90 题官方结果主口径", + "status": "已处理", + "scope": "Browser Use 首页、核心指标表、同题矩阵、性能统计和结果分析页。", + "findings": "上一版误把排除 14 个已知外部干扰任务后的 76 题事后诊断子集写成正式主结果,造成报告看起来只有 76 题;首页还重复突出首次执行与重试恢复,掩盖了每题一个最终逻辑结果的正式口径。", + "corrections": "首页和结果分析恢复完整 90 题官方 evaluator 结果;四种配置的核心指标、同题矩阵和性能统计均使用 90 题。76 题仅保留为明确标注的事后诊断子集,不改分、不替代官方结果。首页删除首次/重试拆分模块;共同成功性能统计按最终逻辑结果及其全部物理执行计算,物理执行明细只在单题证据页下钻。", + "remaining": "14 个外部干扰任务的历史分数保持不变;只有修复后按冻结协议产生新的完整 90 题证据批次,才能更新官方结果。", + "i18n": {"en": { + "title": "Restore the complete 90-task official result as the primary view", + "status": "Resolved", + "scope": "The Browser Use homepage, core metrics, same-task matrix, performance statistics, and result analysis.", + "findings": "The prior version incorrectly presented the 76-task post-hoc diagnostic subset, which excludes 14 known interference tasks, as the formal primary result. It therefore appeared to contain only 76 tasks. The homepage also overemphasized first attempts and retry recovery instead of one final logical result per task.", + "corrections": "Restored the complete official 90-task evaluator result on the homepage and analysis page. Core metrics, the same-task matrix, and performance statistics now use all 90 tasks across all four configurations. The 76-task view remains only as an explicitly labeled post-hoc diagnostic subset and does not rescore or replace the official result. Removed the first-attempt/retry split from the homepage; common-success performance now uses each final logical result and all of its physical executions, while physical-execution details remain on task evidence pages.", + "remaining": "Historical scores for the 14 interference tasks remain unchanged. Only a new complete 90-task evidence cohort produced after corrections under the frozen protocol can update the official result." + }} + }, + { + "id": "review-2026-09-11-interpretation-reader-audit", + "date": "2026-09-11", + "title": "解读页、比较语义与双语版面复核", + "status": "已处理", + "scope": "报告首页、解读页、分析页、失败下钻、英文本地化和桌面/平板/手机断点。", + "findings": "原稿把不同模型配置放在一起标记优胜,混淆了程序判分位置与人工复核根因;重试累计也会扭曲性能对比。部分失败缺少直接下钻入口,英文页仍有中文残留,平板宽度过早堆成单列。", + "corrections": "优胜只在同一模型内比较浏览器;分别披露 43 个仅网络规则失败、20 个仅回答规则失败、12 个双规则失败,以及 29/24/10/8/2/2 的复核根因。性能资源改用四配置首次共同成功的 64 题且只取首次执行,并补齐逐结果链接、英文门禁和响应式布局。", + "remaining": "当前仍是每种配置一个正式样本。14 个已确认外部干扰任务需在修复后完整重跑;增加重复次数前不发布模型或浏览器总体排名。", + "i18n": {"en": { + "title": "Reader audit of interpretation, comparison semantics, and bilingual layout", + "status": "Resolved", + "scope": "The report home, guide, analysis, failure drill-down, English localization, and desktop, tablet, and phone breakpoints.", + "findings": "The draft marked winners across different model configurations, conflated programmatic failure location with human-reviewed root cause, and allowed retry accumulation to distort performance comparisons. Some failures lacked direct drill-down, English pages retained Chinese prose, and tablet layouts collapsed into one column too early.", + "corrections": "Winner marks now compare browsers only within one model. The report separately discloses 43 network-only, 20 response-only, and 12 dual-rule failures, plus the 29/24/10/8/2/2 reviewed root-cause split. Performance and resource statistics use the 64 tasks all four configurations passed on the first attempt and only the first execution; direct result links, English gates, and responsive layouts were added.", + "remaining": "Each configuration still has one formal sample. The 14 confirmed interference tasks require a complete rerun after fixes, and no overall model or browser ranking should be published before repetitions are added." + }} + }, + { + "id": "review-2026-09-11-astra-layer-and-environment-isolation", + "date": "2026-09-11", + "title": "Astra 模型层、逐执行证据与环境隔离核验", + "status": "已处理", + "scope": "GPT-5.6 Sol 与 GPT-6 Astra 两个模型层、360 个逻辑结果、438 次物理执行、75 个最终失败,以及共享 WebArena 站点栈的执行边界。", + "findings": "Sol 层为 Chromium 73/90、Moli 70/90;Astra 层两者均为 71/90。Astra Task 77 的两个失败来自共享 Shopping Admin 状态在准备后被并发执行改写,不是 Agent 误读或镜像基线错误。两个模型层使用的 browser-eval 提交不同,因此跨模型变化只能作观察性比较。", + "corrections": "所有模型、浏览器和任务仍只保留一个逻辑结果;公开包补齐每次物理执行的脱敏证据与哈希;75/75 个失败均绑定当前证据并核验根因。运行器现在用按 Docker daemon 区分的跨进程文件锁覆盖 reset、Agent、证据收尾和 evaluator;同一环境 lane 串行。", + "remaining": "历史官方分数不以人工核验覆盖;Task 77 只有在修复后的隔离环境中形成新的完整证据批次后才能更新成绩。真正并行执行仍要求每个 worker 拥有独立的完整站点栈、端口和浏览器容器。", + "i18n": {"en": { + "title": "Astra model layer, per-execution evidence, and environment-isolation review", + "status": "Resolved", + "scope": "The GPT-5.6 Sol and GPT-6 Astra model layers, 360 logical results, 438 physical executions, 75 final failures, and the execution boundary of the shared WebArena site stack.", + "findings": "In the Sol layer Chromium scored 73/90 and Moli 70/90; in the Astra layer both scored 71/90. The two Astra Task 77 failures came from concurrent modification of shared Shopping Admin state after preparation, not an Agent reading error or a bad image baseline. The model layers used different browser-eval commits, so cross-model changes are observational only.", + "corrections": "Kept one logical result per model, browser, and task; published sanitized evidence and hashes for every physical execution; and bound all 75 failures to verified root causes. The runner now holds a Docker-daemon-scoped cross-process lock across reset, Agent execution, evidence finalization, and evaluation, serializing every run within one environment lane.", + "remaining": "Human review does not override historical official scores. Task 77 can change only after a new complete evidence cohort is produced under the corrected isolation protocol. Safe parallel execution still requires a separate full site stack, ports, and browser containers for every worker." + }} + }, + { + "id": "review-2026-09-11-90-task-expansion", + "date": "2026-09-11", + "title": "90 题扩容与失败根因核验", + "status": "已处理", + "scope": "90 个预注册任务、180 个浏览器逻辑结果、220 次物理执行、环境准备回执、官方判分和全部最终失败结果。", + "findings": "Chromium 通过 73/90,Moli 通过 70/90;69 题共同通过、16 题共同失败、5 题结果不一致。37 个最终失败覆盖 Agent 检索与执行、站点依赖、官方验收合同及 Moli 网络证据采集四类责任层,不能用“模型能力不行”统一解释。", + "corrections": "逐结果绑定官方合同、最终回答、HAR 和 Agent 轨迹,完成 37/37 根因核验;报告改用 90 题动态口径并披露 220/220 环境准备回执。运行器对 Agent 启动前的瞬态 BrowserContextNotLoaded 增加一次仅限会话准备的恢复,不改变逻辑结果协议。", + "remaining": "Astra 未由当前 OpenAI 代理暴露,本轮保持 gpt-5.6-sol、reasoning=low。Wikipedia 与 Map 尚未进入实跑;官方 evaluator 的已核验缺陷需上游修复后以新证据复测。", + "i18n": {"en": { + "title": "90-task expansion and failure-cause review", + "status": "Resolved", + "scope": "The 90 preregistered tasks, 180 logical browser results, 220 physical executions, environment-preparation receipts, official scoring, and every final failure.", + "findings": "Chromium passed 73/90 and Moli passed 70/90. The paired results contain 69 joint passes, 16 joint failures, and five disagreements. The 37 final failures span agent retrieval or execution, site dependencies, official acceptance contracts, and Moli network-evidence collection; they do not support one blanket model-capability explanation.", + "corrections": "Bound each failed result to its official contract, final answer, HAR, and agent trajectory and completed 37/37 root-cause reviews. Updated the report to the 90-task cohort and disclosed 220/220 environment-preparation receipts. Session preparation now recovers once from a transient BrowserContextNotLoaded before the agent starts, without changing logical-result authority.", + "remaining": "This run remains on gpt-5.6-sol with reasoning=low because Astra was unavailable through the current model endpoint. Wikipedia and Map are not measured. Verified defects in the official evaluator require upstream correction and new execution evidence." + }} + }, + { + "id": "review-2026-09-10-shared-failure-and-expansion", + "date": "2026-09-10", + "title": "失败根因、修复与本机扩展核验", + "status": "已处理", + "scope": "十个失败逻辑结果的自动定位、逐结果根因核验、修复代码、结果详情信息层级、Colima 网络身份和 Mac 环境扩展范围。", + "findings": "相同官方分数掩盖了不同根因:Task 68 的官方 expected 与 schema 冲突;124 缺少候选语义核验;142 的类别验收边界未定义;168 的本项目响应协议与官方 null 合同冲突;506 中 Chromium 为 HAR 采集误判,Moli 为步数耗尽且未确认完成。", + "corrections": "保留 evaluator 的程序自动定位;新增按浏览器与任务记录、绑定 attempts 哈希的根因子类和原因核验,未核验字段保持为空,正式发布要求全部失败结果完成核验。另新增运行前合同一致性校验、候选语义与提交前检查、空结果状态协议、JSON 响应体即时 HAR 采集、末步工具调用 fail-closed 和可配置步数上限。历史分数不改写。", + "remaining": "修复后的 #124、#168、#506 尚需形成新执行证据;#68 与 #142 退出下一轮干净 A/B,直至合同被修复或验收口径可观察。Wikipedia 与 Map 尚未在本机实跑。", + "i18n": {"en": { + "title": "Failure causes, corrections, and local-environment expansion", + "status": "Resolved", + "scope": "Automatic failure location, evidence-bound root-cause review, runtime corrections, result-page hierarchy, Colima network identity, and the locally runnable environment set.", + "findings": "The ten failed logical results span task-contract defects, agent protocol or policy defects, and evidence-collection defects. Equal official scores can therefore have different causes, including different causes for the two browsers on the same task.", + "corrections": "Kept automatic evaluator location separate from reviewed causes; added per-browser, per-task cause records bound to the attempts hash and required complete failure coverage for release. Added contract admission checks, candidate and pre-submit validation, an explicit empty-result protocol, immediate JSON HAR-body capture, fail-closed final-step handling, and a configurable step ceiling. Historical scores remain unchanged.", + "remaining": "Tasks affected by runtime corrections require new execution evidence. Tasks with defective or unobservable contracts remain outside the next clean A/B until those contracts are corrected. Wikipedia and Map remain unmeasured locally." + }} + }, + { + "id": "review-2026-09-10-track-naming-and-release-gates", + "date": "2026-09-10", + "title": "轨道命名与发布门禁核验", + "status": "已处理", + "scope": "自主评测与固定轨迹评测的命名、优胜标记、英文界面属性、筛选状态和视觉发布门禁。", + "findings": "轨道名称按实现层命名,读者不易判断测量对象;并列最优和显示值相同的指标仍可能被标优;英文可访问属性与失效筛选链接缺少回归保护;SVG 和跨断点布局只受人工规范约束。", + "corrections": "统一命名为 Browser Use 与 Browser Trajectory Replay(轨迹重放),完成代码与命令迁移且不保留旧入口;优胜只允许显示值唯一领先;补充英文属性翻译、失效链接回退、SVG 拒绝和核心路由双语双主题宽窄屏渲染检查。", + "remaining": "视觉门禁验证结构性溢出、裁切和表头竖排,仍不能替代人工对信息层级与审美质量的判断。", + "i18n": {"en": { + "title": "Track naming and release-gate review", + "status": "Resolved", + "scope": "Names for autonomous and fixed-trajectory evaluation, winner marks, English accessibility text, filter state, and visual release gates.", + "findings": "Implementation-layer names obscured what each track measures; ties or display-equal metrics could be marked as winners; English accessibility attributes and invalid filter links lacked regression coverage; and responsive layout depended on manual checks.", + "corrections": "Standardized the names Browser Use and Browser Trajectory Replay, marked only a unique displayed winner, localized English accessibility attributes, restored invalid filters to valid defaults, rejected SVG assets, and added bilingual, two-theme desktop and narrow-screen checks for core routes.", + "remaining": "Automated visual gates detect structural overflow, clipping, and vertical headers, but do not replace human review of information hierarchy and visual quality." + }} + }, + { + "id": "review-2026-09-10-scalable-comparison-ui", + "date": "2026-09-10", + "title": "多浏览器比较与结果详情核验", + "status": "已处理", + "scope": "同题比较交互、BrowserTool 动作延迟和单结果详情首屏。", + "findings": "同题视图虽动态生成列,但交互仍以两浏览器关系为中心;动作表层级拥挤且对不同调用量的总耗时判优;完整 evaluator 错误被放入窄指标卡。", + "corrections": "同题比较改为基准浏览器加多选对比浏览器和明确关系条件;动作延迟改为按 action 分卡并只比较单次均值;详情首屏只保留简明规则定位,完整 expected、actual 和错误进入规则证据区。", + "remaining": "当前仅有两个参评浏览器;扩展行为已由三浏览器测试模型覆盖,新增真实浏览器仍需形成完整同题结果。", + "i18n": {"en": { + "title": "Multi-browser comparison and result-detail review", "status": "Resolved", + "scope": "Same-task comparison controls, BrowserTool action latency, and the opening section of result-detail pages.", + "findings": "The same-task view remained centered on a two-browser relationship; action tables ranked totals across unequal call counts; and full evaluator errors occupied narrow KPI cards.", + "corrections": "Introduced one baseline with multi-select comparison browsers and explicit outcome relations, compared action latency only by mean per call, and moved complete expected, actual, and error evidence out of the result summary.", + "remaining": "Only two browsers are measured. Three-browser fixtures cover interface scaling, but every new real browser still requires a complete same-task result set." + }} + }, + { + "id": "review-2026-09-10-report-model", + "date": "2026-09-10", + "title": "报告模型与比较口径核验", + "status": "已处理", + "scope": "统计摘要、优胜表达、结果下钻、核验记录和参评浏览器说明。", + "findings": "统计标签重复表达 P50;比较表缺少方向一致的优胜标记;结果页存在重复失败清单;核验页承担了逐结果清单职责。", + "corrections": "统一为均值、P50、P95;按指标方向突出唯一优胜者;删除重复失败清单;核验页改为按轮次记录范围、发现、处理和遗留事项。", + "remaining": "当前固定样本没有浏览器单边通过任务;优胜标记主要反映耗时与 Tokens,不改变官方任务成绩。", + "i18n": {"en": { + "title": "Report model and comparison review", "status": "Resolved", + "scope": "Summary statistics, winner presentation, result drill-down, review history, and browser-participant descriptions.", + "findings": "P50 was duplicated under another label, comparison tables lacked directionally consistent winner marks, the results page duplicated failure inventory, and the review page carried a per-result list.", + "corrections": "Standardized summaries on mean, P50, and P95; marked only unique directionally valid winners; removed the duplicate failure list; and made review history round-based.", + "remaining": "This fixed cohort has no one-browser-only pass. Winner marks mainly describe latency and token usage and do not alter official task scores." + }} + }, + { + "id": "review-2026-09-10-evidence-boundaries", + "date": "2026-09-10", + "title": "部署证据与计时边界核验", + "status": "已处理", + "scope": "本地站点部署、Colima 网络路径、环境准备证据、Agent 与 BrowserTool 计时。", + "findings": "环境拓扑与网关含义缺少集中说明;历史总耗时边界描述不准确;逐执行环境准备没有独立回执。", + "corrections": "补充部署流程图、端口和镜像身份;区分 Agent 链路与 BrowserTool 墙钟时间;新版 runner 保存环境准备和分阶段计时。", + "remaining": "本批冻结结果不具备逐执行准备回执,报告仅按现有证据声明可复现边界。", + "i18n": {"en": { + "title": "Deployment evidence and timing-boundary review", "status": "Resolved", + "scope": "Local-site deployment, the Colima network path, environment-preparation evidence, and Agent and BrowserTool timing.", + "findings": "The topology and gateway lacked a consolidated explanation, historical total-latency wording overstated its boundary, and per-execution preparation lacked an independent receipt.", + "corrections": "Added the deployment flow, ports, and image identities; separated agent-stack and BrowserTool wall time; and made the current runner retain environment-preparation and phase timings.", + "remaining": "This frozen run lacks per-execution preparation receipts, so reproducibility claims remain limited to the evidence actually retained." + }} + }, + { + "id": "review-2026-09-09-runner-integrity", + "date": "2026-09-09", + "title": "运行器与证据完整性核验", + "status": "已处理", + "scope": "模型响应校验、BrowserTool 清理、HAR 留存、证据目录和恢复执行。", + "findings": "异常清理可能遮蔽原始错误;HAR 停止操作缺少幂等保护;异常响应和原始证据目录需要更严格保护。", + "corrections": "隔离清理异常、使 HAR 停止幂等、加强模型响应结构校验,并阻止原始浏览器证据进入 Git。", + "remaining": "无阻塞项。", + "i18n": {"en": { + "title": "Runner and evidence-integrity review", "status": "Resolved", + "scope": "Model-response validation, BrowserTool cleanup, HAR retention, evidence directories, and resume behavior.", + "findings": "Cleanup exceptions could mask primary failures, HAR shutdown lacked idempotency, and malformed responses and raw-evidence directories required stronger protection.", + "corrections": "Isolated cleanup failures, made HAR shutdown idempotent, strengthened model-response validation, and prevented raw browser evidence from entering Git.", + "remaining": "No blocking items." + }} + } + ], + "browser_use_releases": [ + { + "id": "2026-09-14-restore-complete-90-task-results", + "date": "2026-09-14", + "title": "恢复完整 90 题正式结果并修正报告下钻", + "summary": "正式成绩恢复为完整 90 题、四种配置共 360 个逻辑结果;76 题仅保留为事后诊断子集。结果分析改为直接进入四配置同题矩阵和失败诊断,动作耗时在全部配置中同台比较,执行证据改为站内查看或下载。", + "attempts_sha256_by_model": { + "gpt-5.6-sol": "73146500152db62d79f84b63c85a14027c005a6320c19bce7426651f167466bd", + "gpt-6-astra": "0659d227cf6d29fbfaf64d39f8e77a56bec73c87f7f6e1f2ac23ea13fd9b667c" + }, + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "changes": [ + [ + "正式分母", + "上一版把 76 题事后诊断子集写成正式比较口径,与完整官方结果冲突。", + "恢复 90 题官方 evaluator 结果为唯一正式成绩;76 题只用于观察排除已知外部干扰后的差异,不改分。", + "四种配置分别为 73/90、70/90、71/90、71/90。" + ], + [ + "结果分析", + "首页与分析页重复核心总表,动作耗时只在同模型的两个浏览器之间标优。", + "首页保留核心总表;分析页直接展示四配置同题矩阵、全部配置动作耗时和逐结果诊断。", + "只调整信息结构和比较范围,不改变测量值。" + ], + [ + "执行证据", + "新标签链接在嵌入式公开阅读器中不能可靠打开,详情页还暴露未解释的内部字段。", + "证据查看改为站内页面,原始 JSON 改为下载;资源、校验和公开策略使用读者可理解的中文标签。", + "证据文件及其 SHA-256 不变。" + ] + ], + "i18n": { + "en": { + "title": "Restore the complete 90-task official results and repair drill-down", + "summary": "Official scores again use all 90 tasks and 360 logical results across four configurations; the 76-task set remains only a post-hoc diagnostic subset. Analysis now opens with the four-configuration same-task matrix and failure diagnosis, action latency compares every configuration together, and execution evidence opens in-site or downloads directly.", + "changes": [ + [ + "Official denominator", + "The previous release described the 76-task post-hoc subset as the formal comparison, conflicting with the complete official results.", + "Use the 90-task official evaluator results as the only formal scores; retain 76 tasks only for post-hoc diagnosis without rescoring.", + "The four configurations score 73/90, 70/90, 71/90, and 71/90." + ], + [ + "Results analysis", + "Home and analysis duplicated the core table, while action-latency winners were selected only between two browsers inside each model.", + "Keep the core table on home; make analysis focus on the four-configuration same-task matrix, all-configuration action latency, and per-result diagnosis.", + "Information structure and comparison scope changed; measurements did not." + ], + [ + "Execution evidence", + "New-tab links were unreliable in the embedded public reader and detail pages exposed unexplained internal fields.", + "Open the evidence viewer in-site, offer raw JSON as a download, and use reader-facing labels for resources, checks, and publication policy.", + "Evidence files and SHA-256 bindings are unchanged." + ] + ] + } + } + }, + { + "id": "2026-09-14-qualified-comparison-home", + "date": "2026-09-14", + "title": "首页改用审查合格比较集并合并报告解读", + "summary": "首页不再把含已确认合同、站点、共享状态或参考状态干扰的 90 题原始结果当作正式比较。四种配置统一改用 76 题审查合格分母;90 题原始 evaluator 输出完整保留在结果与核验页。报告解读并入首页,删除重复入口和独立阅读副本。", + "attempts_sha256_by_model": { + "gpt-5.6-sol": "73146500152db62d79f84b63c85a14027c005a6320c19bce7426651f167466bd", + "gpt-6-astra": "0659d227cf6d29fbfaf64d39f8e77a56bec73c87f7f6e1f2ac23ea13fd9b667c" + }, + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "changes": [ + [ + "正式比较分母", + "90 题汇总混入 14 题已确认的非目标变量干扰,首页同时展示诊断子集,读者无法判断哪组才是结论依据。", + "按整题、四配置一致排除 14 题,76 题成为唯一正式比较集;原始 90 题只作审计记录,不人工改分。", + "Sol 为 Chromium 71/76、Moli 67/76;Astra 为 Chromium 69/76、Moli 68/76。" + ], + [ + "Task 77", + "Astra 的两条 Task 77 证据在 reset 后被另一个进程改写共享 Shopping Admin 状态。", + "运行器已用按 Docker daemon 区分的跨进程锁覆盖 reset、Agent、证据收尾和 evaluator;同一环境 lane 串行,不同 lane 才能并行。旧污染结果退出分母,只有四配置新证据才能重新纳入。", + "Task 77 不再影响首页成绩;历史执行仍可下钻核对。" + ], + [ + "信息结构", + "首页与报告解读重复,读者需要在两个入口之间往返。", + "把解读、成绩、失败位置、审查边界和资源结论合并为首页,删除 guide 路由与独立副本。", + "数据集、结果、核验和复现仍保留独立下钻页面。" + ] + ], + "i18n": { + "en": { + "title": "Use a review-qualified comparison set on the unified homepage", + "summary": "The homepage no longer treats raw 90-task results containing confirmed contract, site, shared-state, or reference-state interference as the formal comparison. All four configurations now use one 76-task review-qualified denominator, while every raw evaluator output remains available in the result and review pages. The interpretation is merged into the homepage and duplicate routes are removed.", + "changes": [ + [ + "Formal denominator", + "The 90-task summary mixed 14 tasks with confirmed non-target interference and placed a diagnostic subset beside it, leaving the conclusion denominator ambiguous.", + "Exclude each affected task across all four configurations. The 76-task set becomes the sole formal comparison; raw 90-task results remain audit records and are not manually rescored.", + "Sol scores 71/76 on Chromium and 67/76 on Moli; Astra scores 69/76 on Chromium and 68/76 on Moli." + ], + [ + "Task 77", + "Both Astra Task 77 executions had shared Shopping Admin state modified by another process after reset.", + "A Docker-daemon-scoped cross-process lock now covers reset, agent execution, evidence finalization, and evaluation. Runs are serial within one environment lane and parallel only across independent lanes. The contaminated results leave the denominator and can return only through new four-configuration evidence.", + "Task 77 no longer affects the homepage scores; its historical executions remain available for inspection." + ], + [ + "Information architecture", + "The homepage and interpretation duplicated content and forced readers to switch between two entry points.", + "Merge interpretation, scores, failure locations, review boundaries, and resource findings into the homepage; remove the guide route and standalone copy.", + "Dataset, results, review, and reproduction remain separate drill-down pages." + ] + ] + } + } + }, + { + "id": "2026-09-11-browser-resource-comparison", + "date": "2026-09-11", + "title": "补齐四种配置的浏览器资源比较", + "summary": "首页新增四种模型×浏览器配置的 CPU、内存与网络统计,统一报告均值、P50 和 P95。数据来自现有浏览器容器测量,不修改任务成绩。", + "attempts_sha256_by_model": { + "gpt-5.6-sol": "73146500152db62d79f84b63c85a14027c005a6320c19bce7426651f167466bd", + "gpt-6-astra": "0659d227cf6d29fbfaf64d39f8e77a56bec73c87f7f6e1f2ac23ea13fd9b667c" + }, + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "changes": [ + [ + "资源统计", + "报告已保留浏览器容器的 CPU、内存和网络测量,但首页没有汇总,读者无法比较资源开销。", + "增加四种配置同台的 CPU 时间、平均单核 CPU、时间加权平均内存、峰值内存及网络收发量,全部展示均值、P50、P95。", + "资源只覆盖浏览器容器,不包含 CowAgent 或模型服务;模型会改变动作量,因此结果描述完整配置而非纯内核差异。" + ], + [ + "重试聚合", + "逻辑结果包含重试时,平均内存只取最后一次执行会丢失前次执行的资源贡献。", + "按每次执行的等间隔采样数加权合并平均内存;CPU 与网络继续累计全部物理执行。", + "只修正资源汇总口径;任务分数、分母和判分执行保持不变。" + ], + [ + "英文首页", + "失败干扰表和核心指标说明残留部分中文界面文案。", + "补齐动态失败摘要、责任范围、处理标签和指标说明的英文映射。", + "中英文页面结构和数据保持一致。" + ] + ], + "i18n": { + "en": { + "title": "Add browser-resource comparisons across all four configurations", + "summary": "The home page now compares CPU, memory, and network measurements across all four model-by-browser configurations using mean, P50, and P95. The values come from existing browser-container evidence and do not change task scores.", + "changes": [ + [ + "Resource statistics", + "Browser-container CPU, memory, and network measurements were retained but not summarized on the report home page.", + "Added CPU time, mean single-core CPU utilization, time-weighted mean memory, peak memory, and network receive/transmit totals for all four configurations, each with mean, P50, and P95.", + "Measurements cover only the browser container, excluding CowAgent and model services. Because models change workload, the table compares complete configurations rather than pure browser cores." + ], + [ + "Retry aggregation", + "For logical results with a retry, using only the final execution's mean memory omitted the earlier execution's resource contribution.", + "Mean memory is now combined across executions using equal-interval sample counts as weights; CPU and network totals continue to accumulate every physical execution.", + "Only the resource aggregation changes. Task scores, denominators, and score-authoritative executions remain unchanged." + ], + [ + "English home page", + "Some interface copy in the failure-interference table and core-metric notes remained in Chinese.", + "Completed the English mappings for the dynamic failure summary, owning scopes, treatment labels, and metric explanations.", + "The Chinese and English pages retain identical structure and data." + ] + ] + } + } + }, + { + "id": "2026-09-11-diagnostic-workbench-and-clean-subset", + "date": "2026-09-11", + "title": "将结果分析改为故障定位台", + "summary": "总体指标和动作延迟统一比较全部模型×浏览器配置;失败清单改为固定高度的横向诊断表并提供页内诊断弹窗;每个失败详情补齐失败位置、程序报错、复核根因、处理责任、轨迹、画面与可直接查看的执行证据。官方分数不变,另提供排除已证实外部干扰的诊断子集。", + "attempts_sha256_by_model": { + "gpt-5.6-sol": "73146500152db62d79f84b63c85a14027c005a6320c19bce7426651f167466bd", + "gpt-6-astra": "0659d227cf6d29fbfaf64d39f8e77a56bec73c87f7f6e1f2ac23ea13fd9b667c" + }, + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "changes": [ + [ + "总体与速度比较", + "旧版按模型分组标记浏览器优胜者,无法直接比较四种完整配置。", + "通过率、Agent 耗时、BrowserTool 耗时、Tokens 和逐动作单次延迟均改为四种配置同台比较,只标记唯一最优值。", + "总耗时与动作调用数仍只解释工作量,不参与浏览器快慢判定。" + ], + [ + "同题比较", + "旧页面要求先选基准浏览器再做两两比较,无法同时观察全部配置。", + "改为每题一行、全部模型×浏览器配置为列的动态矩阵。", + "新增模型或浏览器只增加配置列,不再复制比较模块。" + ], + [ + "失败定位", + "结果清单混入长原因和耗时字段,行高失控,且无法直接进入失败证据。", + "改为固定高度横向表;每行可打开页内诊断,直接查看失败位置、程序报错、复核根因、处理责任和证据入口。", + "完整 evaluator、轨迹、日志和原因留在结果详情;共享证据页按需读取 JSON,同时保留原始文件入口,避免在每个详情页重复完整载荷。" + ], + [ + "证据页面", + "失败详情缺少从判定到具体轨迹、日志和可见页面状态的连续入口。", + "增加失败诊断区、轨迹步骤锚点,并按 Playwright screencast 事件顺序从私有 trace 提取最后三张完整页面画面;原始 trace 仍不公开。", + "读者可从失败阶段直达相关轨迹,并核对日志与最后可见状态。" + ], + [ + "结果入口", + "浏览器详情页曾重复生成一套含长原因的结果表。", + "删除重复表,改为链接到预先筛选当前浏览器的唯一结果清单。", + "结果、筛选和诊断入口只有一套,新增浏览器不会复制页面结构。" + ], + [ + "干扰项处理", + "官方分数混合了模型/浏览器信号与已确认的题目、站点、共享环境和参考状态问题。", + "保留官方 90 题分数,同时按任务统一剔除已确认外部干扰,发布非官方诊断子集和修复登记。", + "诊断子集的四个配置使用相同分母;修复项必须完整重跑后才能重新纳入。" + ] + ], + "i18n": { + "en": { + "title": "Turn Results into a failure-diagnosis workbench", + "summary": "Overall metrics and action latency now compare every model-by-browser configuration together. The failure inventory is a fixed-height horizontal table with an in-page diagnosis dialog, and each failed result exposes its failure stage, exact program error, reviewed root cause, remediation owner, trajectory, screenshots, and directly readable execution evidence. Official scores are unchanged; a separate diagnostic subset removes confirmed external interference.", + "changes": [ + [ + "Overall and speed comparison", + "The previous version marked browser winners within each model, preventing direct comparison of all four complete configurations.", + "Pass rate, Agent latency, BrowserTool latency, tokens, and per-action per-call latency now compare all four configurations together and mark only a unique winner.", + "Total action latency and call counts remain workload context and are not used to rank browser speed." + ], + [ + "Same-task comparison", + "The previous page required a baseline browser and pairwise comparisons, preventing a simultaneous view of every configuration.", + "Replaced it with a dynamic matrix with one task per row and every model-by-browser configuration as columns.", + "New models or browsers add configuration columns without duplicating comparison modules." + ], + [ + "Failure location", + "The result inventory mixed long causes and latency fields, producing uncontrolled row heights and no direct route to failure evidence.", + "Replaced it with a fixed-height horizontal table whose rows open an in-page diagnosis with the failure stage, exact program error, reviewed cause, owner/treatment, and evidence links.", + "Complete evaluator output, trajectory, logs, and cause remain on the result detail page. A shared viewer loads JSON on demand while preserving a raw-file link, avoiding duplicate payloads in every detail page." + ], + [ + "Evidence pages", + "Failed-result pages lacked a continuous route from the verdict to the relevant trajectory, logs, and visible page state.", + "Added a diagnosis section, trajectory-step anchors, and the last three full-page frames selected in Playwright screencast-event order from the private trace while keeping the trace itself private.", + "Readers can jump from the failure stage to nearby trajectory evidence and inspect logs and final visible state." + ], + [ + "Result entry point", + "Browser detail pages duplicated a result table containing long cause text.", + "Removed the duplicate and linked to the single result inventory with the current browser preselected.", + "Results, filters, and diagnosis entry points now have one owner; adding a browser does not duplicate page structure." + ], + [ + "Interference treatment", + "Official scores combined model/browser signal with confirmed task, site, shared-environment, and reference-state problems.", + "Kept the official 90-task scores and added a non-official diagnostic subset that removes confirmed external interference at task level, plus a remediation registry.", + "All four configurations use the same diagnostic denominator; remediated tasks require a complete rerun before readmission." + ] + ] + } + } + }, + { + "id": "2026-09-11-reader-first-report-and-sites-migration", + "date": "2026-09-11", + "title": "重排结果首页并迁移到正式 Sites", + "summary": "首页先展示四组核心成绩与耗时,再给出同题差异、75 个失败的核验原因和三类任务成绩。此次只调整报告表达、检查流程与发布地址,不修改任何评测分数。", + "attempts_sha256_by_model": { + "gpt-5.6-sol": "73146500152db62d79f84b63c85a14027c005a6320c19bce7426651f167466bd", + "gpt-6-astra": "0659d227cf6d29fbfaf64d39f8e77a56bec73c87f7f6e1f2ac23ea13fd9b667c" + }, + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "changes": [ + [ + "首页结构", + "核心成绩被拆成多张卡片,关键对比不易浏览。", + "改为一张按模型和浏览器展开的核心指标表,并把它放在首页第一节。", + "通过数、通过率、Agent 总耗时、BrowserTool 耗时和 Tokens 可直接横向比较。" + ], + [ + "结果解释", + "部分短标签需要读者了解内部实现才能理解。", + "改用完整句子和明确列名说明同题关系、失败原因、统计范围与限制。", + "不改变计算口径;只降低理解成本。" + ], + [ + "失败原因", + "75 个失败原因以大量卡片堆叠,难以判断责任分布。", + "按 Agent、任务或 evaluator、测试网站、浏览器证据、共享环境和混合原因汇总为表格,并保留逐结果核验入口。", + "所有失败仍各计一次,逐结果核验结论和原始证据保持不变。" + ], + [ + "检查与发布", + "日常修改也会启动完整浏览器视觉检查,反馈较慢。", + "日常检查只验证数据、结构、链接和文案;正式打包发布时再执行全部语言、主题和屏幕宽度的视觉检查。", + "发布质量门槛不降低,常规迭代时间缩短。" + ] + ], + "i18n": {"en": { + "title": "Reorder the results report and migrate to production Sites", + "summary": "The home page now starts with the four core score and latency rows, followed by same-task differences, verified causes for 75 failures, and results for the three task types. This release changes presentation, checks, and the publication target only; no evaluation score changes.", + "changes": [ + ["Home-page structure", "Core results were split across cards, making comparison difficult.", "Replaced them with one model-by-browser core-metrics table at the start of the home page.", "Pass count, pass rate, agent total time, BrowserTool time, and tokens can now be compared directly."], + ["Result interpretation", "Several short labels required implementation context.", "Replaced them with explicit sentences and column names for paired outcomes, failure causes, scope, and limitations.", "Calculation rules are unchanged; the report is easier to understand."], + ["Failure causes", "The 75 failure causes were spread across a large card grid.", "Grouped them into a table covering agent, task or evaluator, test site, browser evidence, shared environment, and mixed causes, while retaining links to every reviewed result.", "Each failure is still counted once, and its reviewed cause and evidence remain unchanged."], + ["Checks and publication", "Routine edits also launched the complete browser-based visual suite.", "Routine checks now cover data, structure, links, and copy; packaging still runs every language, theme, and viewport visual check.", "Release quality gates remain intact while ordinary iteration is faster."] + ] + }} + }, + { + "id": "2026-09-11-astra-two-model-browser-comparison", + "date": "2026-09-11", + "title": "加入 GPT-6 Astra 模型层并完成双模型浏览器对照", + "summary": "同一 90 题样本按模型分层比较 Chromium 与 Moli:GPT-5.6 Sol 层为 73/90 对 70/90,GPT-6 Astra 层为 71/90 对 71/90。75 个最终失败全部完成证据绑定的根因核验;已确认的环境污染不人工改分。", + "attempts_sha256_by_model": { + "gpt-5.6-sol": "73146500152db62d79f84b63c85a14027c005a6320c19bce7426651f167466bd", + "gpt-6-astra": "0659d227cf6d29fbfaf64d39f8e77a56bec73c87f7f6e1f2ac23ea13fd9b667c" + }, + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "changes": [ + [ + "双模型分层", + "上一版只包含 GPT-5.6 Sol,无法观察更换模型后浏览器差异是否稳定。", + "加入 GPT-6 Astra low 的独立证据层;模型之间不混排,浏览器只在各自模型层内做同题配对。", + "Astra 层 Chromium 与 Moli 均为 71/90;两端各有 2 个单边通过,精确 McNemar p=1.0,未观察到显著通过率差异。" + ], + [ + "跨模型解释边界", + "两个模型层并非同一次运行,且 browser-eval 提交不同。", + "报告保留各模型层的运行 ID、提交、manifest 与 attempts 哈希,并把跨模型差异标为观察性结果。", + "不能把 Sol 与 Astra 的分数或耗时差异单独归因于模型版本。" + ], + [ + "失败根因", + "官方 evaluator 的失败信息只说明未满足哪条规则。", + "对 75 个最终失败逐项核对合同、最终回答、网络证据和自主轨迹;自动分类保留,核验前的详细根因为空。", + "75/75 已核验、0 项未决;发现 Agent、站点、环境隔离、evaluator 合同和浏览器证据采集等不同责任层。" + ], + [ + "环境隔离与并行", + "固定端口的共享站点栈允许另一个进程在 reset 后、Agent 读取前改写状态。", + "增加跨进程锁,覆盖 reset、浏览器会话、CowAgent、证据收尾与 evaluator;只有独立完整站点栈才允许并行。", + "该修复保护后续运行;历史 Task 77 官方分数保持原样,必须由新证据批次才能更正。" + ], + [ + "公开证据", + "逻辑结果累计了所有实际执行的耗时和成本,但旧页面只暴露判分执行的证据。", + "仍按模型×浏览器×任务展示一个逻辑结果,同时为 438 次物理执行分别发布脱敏回答、evaluator、事件、轨迹、指标与哈希。", + "读者可重算累计值;HAR、Playwright trace、Cookie 和凭据继续不公开。" + ] + ], + "i18n": {"en": { + "title": "Add GPT-6 Astra and complete a two-model browser comparison", + "summary": "Chromium and Moli are compared within two model layers on the same 90 tasks: 73/90 versus 70/90 with GPT-5.6 Sol, and 71/90 versus 71/90 with GPT-6 Astra. All 75 final failures have evidence-bound root-cause reviews; verified environment contamination does not manually change official scores.", + "changes": [ + [ + "Two model layers", + "The previous release contained only GPT-5.6 Sol, so it could not show whether browser differences persisted after changing the model.", + "Added an independent GPT-6 Astra low evidence layer. Models are not pooled; browsers are paired task by task only within each model layer.", + "In the Astra layer Chromium and Moli both scored 71/90, with two browser-only passes on each side and exact McNemar p=1.0; no significant pass-rate difference was observed." + ], + [ + "Cross-model boundary", + "The model layers were separate runs and used different browser-eval commits.", + "The report retains each layer's run ID, commit, manifest, and attempts hashes and labels cross-model differences as observational.", + "Sol–Astra score or latency changes cannot be attributed to the model version alone." + ], + [ + "Failure causes", + "Official evaluator messages identify unmet rules but not the responsible layer.", + "Reviewed the contract, final answer, network evidence, and autonomous trajectory for all 75 failed logical results. Automatic classification remains available, while detailed causes stay blank until verified.", + "All 75 failures are reviewed with none unresolved, spanning Agent, site, environment isolation, evaluator contracts, and browser evidence capture." + ], + [ + "Environment isolation and parallelism", + "The shared fixed-port site stack allowed another process to mutate state after reset and before the Agent observed it.", + "Added a cross-process lock across reset, browser session, CowAgent, evidence finalization, and evaluation. Parallel execution is allowed only with separate full site stacks.", + "The fix protects future runs. Historical Task 77 scores remain unchanged until replaced by a new evidence cohort." + ], + [ + "Public evidence", + "Logical results accumulated every execution's latency and cost, but the previous page exposed evidence only for the score-authoritative execution.", + "The report still shows one logical result per model, browser, and task, while publishing sanitized responses, evaluator output, events, trajectories, metrics, and hashes for all 438 physical executions.", + "Readers can recompute aggregates; HAR, Playwright trace, cookies, and credentials remain private." + ] + ] + }} + }, + { + "id": "2026-09-11-90-task-browser-use-comparison", + "date": "2026-09-11", + "title": "扩展到 90 个 Browser Use 任务", + "summary": "预注册样本扩展为 90 题,覆盖 easy 15、medium 30、hard 45;Chromium 通过 73 题,Moli 通过 70 题。全部 37 个最终失败结果已追到可复核根因,官方分数保持原样,不以人工核验改分。", + "attempts_sha256": "73146500152db62d79f84b63c85a14027c005a6320c19bce7426651f167466bd", + "selection_sha256": "2cb716552e36df7535e83ed6d2cf3f84283e3aca86072817cd6e039a38547c7b", + "changes": [ + [ + "评测范围", + "上一批只覆盖 30 题,无法同时形成足够的简单、中等和长链任务观察。", + "扩展为 90 个结果盲选任务,按最少可见 UI 动作预注册为 easy 15、medium 30、hard 45,覆盖 Shopping、Shopping Admin、Reddit 和 GitLab。", + "分母由 30 增至 90;难度标签是本项目操作长度分层,不是官方难度。" + ], + [ + "正式结果", + "上一批 Chromium 与 Moli 均为 25/30。", + "本批 Chromium 为 73/90,Moli 为 70/90;69 题共同通过、16 题共同失败、Chromium 单边通过 4 题、Moli 单边通过 1 题。", + "结果只适用于当前 CowAgent、gpt-5.6-sol low、四个本地站点与冻结运行协议,不外推完整 benchmark。" + ], + [ + "失败根因", + "程序化 evaluator 只能先定位失败规则,不能单独解释错误由哪一层产生。", + "对 37 个最终失败逻辑结果逐项核对合同、回答、网络证据和自主轨迹,记录具体根因子类与原因。", + "37/37 已核验、0 项未决;人工核验不覆盖官方分数。" + ], + [ + "环境证据", + "旧批次没有逐执行绑定的环境准备回执。", + "本批 220/220 次物理执行均保存 reset、ready、应用健康检查和登录准备状态,并继续披露 Colima 网关与本地镜像拓扑。", + "支持确认本批确在自托管站点执行;不代表 Wikipedia 与 Map 已部署。" + ] + ], + "i18n": {"en": { + "title": "Expand Browser Use to 90 tasks", + "summary": "The preregistered cohort now contains 90 tasks: 15 easy, 30 medium, and 45 hard. Chromium passed 73 and Moli passed 70. All 37 final failures have evidence-bound root-cause reviews; human review does not override official scores.", + "changes": [ + ["Evaluation scope", "The previous 30-task cohort did not provide enough observations across short, medium, and long workflows.", "Expanded to 90 outcome-blind tasks, preregistered by minimum visible UI actions as 15 easy, 30 medium, and 45 hard across Shopping, Shopping Admin, Reddit, and GitLab.", "The denominator increased from 30 to 90. Tiers describe project-defined operation length, not official difficulty."], + ["Official results", "The previous cohort gave Chromium and Moli 25/30 each.", "This cohort gives Chromium 73/90 and Moli 70/90: 69 joint passes, 16 joint failures, four Chromium-only passes, and one Moli-only pass.", "Results apply only to the current CowAgent, gpt-5.6-sol low, four local sites, and frozen run protocol; they do not generalize to the full benchmark."], + ["Failure causes", "The programmatic evaluator locates failed rules but cannot by itself identify the responsible layer.", "Reviewed the contract, answer, network evidence, and autonomous trajectory for all 37 failed logical results and recorded a specific cause subtype and rationale.", "All 37 failures are reviewed with none unresolved; human review does not change official scores."], + ["Environment evidence", "The prior cohort lacked preparation receipts bound to each execution.", "All 220 physical executions retain reset, readiness, application-health, and sign-in preparation status, alongside the Colima gateway and local image topology.", "This establishes that the cohort ran against self-hosted sites; it does not claim that Wikipedia or Map was deployed."] + ] + }} + }, + { + "id": "2026-09-10-shared-failure-diagnosis-and-mac-expansion", + "date": "2026-09-10", + "title": "追到失败根因并落实运行器修复", + "summary": "十个失败逻辑结果已完成逐项根因核验:两项任务合同问题、两项 Agent 协议或策略问题,以及 Task 506 中 Chromium 与 Moli 各不相同的采集和完成问题。自动 evaluator 定位与核验根因分层展示;核验记录绑定冻结证据哈希。运行器已修复合同准入、响应协议、候选核验、HAR 采集与超限处理;历史官方成绩仍为 Chromium 25/30、Moli 25/30,修复效果等待新执行验证。", + "attempts_sha256": "517b613adcc40fc0ebe41af2ab2115d76f49faaba5378b426cc196b5b633e9c7", + "selection_sha256": "ba3c344df003dd4eba3a5d1ef97162ee98df2ed728f7c29c4188bc4247982d38", + "changes": [ + [ + "报告解读", + "总分相同不足以解释结果,且相同零分不代表相同失败原因。", + "将根因、责任层、证据闭环、根治措施和复测门槛提升为首页与解读文章的核心结论。", + "改变结论层级,不改变官方分数、分母或冻结证据。" + ], + [ + "共同失败", + "旧解读停留在错误现象,没有定位产生错误的合同、策略或采集实现。", + "结合官方 expected/schema、最终回答、evaluator、HAR 与逐步事件,定位四类根因;Task 506 分浏览器分别归因。", + "不推断模型隐藏思维;根因由可复核证据闭环支持,官方历史分数不变。" + ], + [ + "失败原因核验", + "程序化 evaluator 能定位未通过的验收类型,但不能单独证明产生失败的根因。", + "按浏览器与任务保存根因子类和具体原因,并绑定当前 attempts 哈希;未核验时两个字段为空,完整发布要求覆盖全部失败逻辑结果。", + "本版 10/10 个失败逻辑结果已核验;Task 506 分浏览器记录不同根因,不改变官方分数。" + ], + [ + "运行器根治", + "错误合同会消耗正式调用;空结果协议、搜索候选聚合、HAR 延迟取体和超限总结会制造可避免的失败或误判。", + "新运行先验证 expected 是否满足公开 schema;统一要求候选语义和提交前核验;JSON 响应体在 loadingFinished 即时保存;末步仍调用工具时强制 UNKNOWN_ERROR;步数上限写入运行配置。", + "代码门禁已通过 61 个 Browser Use 单测、206 条报告路由和 104 个视觉状态;任务成败改善仍必须由新执行证明。" + ], + [ + "结果详情", + "合同、实际回答和规则结果在相邻区域重复,原始 JSON 抢占主要阅读层级。", + "合并为“任务要求与验收结果”,先展示可读规则结论,官方期望、本次观察和技术断言默认折叠。", + "原始公开证据和下载入口保持不变。" + ], + [ + "本地环境与扩展", + "Colima 网关角色和可继续运行的官方环境范围不够明确。", + "区分 VM 地址与宿主网关,说明四站实跑证据;纠正官方不存在 179 题子集,并给出 678 个当前可预检候选和 233 个 hard 候选。", + "Wikipedia、Map 仍标为尚未实跑,不把理论可部署写成已验证。" + ] + ], + "i18n": {"en": { + "title": "Trace failures to causes and apply runtime corrections", + "summary": "Ten failed logical results were reviewed across task contracts, agent protocol or policy, and evidence collection. Automatic failure location and reviewed causes are presented separately and bound to frozen evidence. Runtime corrections await validation by new executions; historical official scores remain 25/30 for each browser.", + "changes": [ + ["Report interpretation", "Equal totals did not explain outcomes, and equal zero scores did not imply equal causes.", "Made cause, owning layer, evidence chain, corrective action, and retest gate central to the report interpretation.", "The explanatory hierarchy changed; scores, denominators, and frozen evidence did not."], + ["Shared failures", "Earlier interpretation stopped at symptoms rather than locating the contract, policy, or collector responsible.", "Used official contracts, final answers, evaluator output, HAR, and events to identify four cause classes, including browser-specific causes on Task 506.", "No hidden model reasoning is inferred; historical official scores remain unchanged."], + ["Cause review", "Automatic evaluators locate an unmet acceptance type but do not prove its cause.", "Stored per-browser, per-task causes bound to the attempts hash; unreviewed fields remain blank and complete failure coverage is required for release.", "All ten failed logical results in this version are reviewed; this does not change official scores."], + ["Runtime corrections", "Invalid contracts and weaknesses in empty-result handling, candidate aggregation, HAR capture, and final-step handling could create avoidable failures or false negatives.", "Added contract admission, candidate and pre-submit checks, immediate JSON HAR capture, fail-closed final-step handling, and a recorded step ceiling.", "Code gates passed, but any task-outcome improvement still requires new execution evidence."], + ["Result details", "Contracts, answers, and rule results were duplicated while raw JSON dominated the reading order.", "Combined them into a task-and-acceptance section with readable conclusions first and technical values collapsed.", "Public evidence and download links remain unchanged."], + ["Local environment and expansion", "The Colima gateway and the scope of locally runnable official environments were unclear.", "Distinguished VM and host gateway roles, documented the four measured sites, and corrected the available candidate counts.", "Wikipedia and Map remain explicitly unmeasured; deployability is not presented as verification."] + ] + }} + }, + { + "id": "2026-09-10-track-naming-and-release-hardening", + "date": "2026-09-10", + "title": "统一轨道名称并加固发布检查", + "summary": "自主轨道统一为 Browser Use,固定动作轨道统一为 Browser Trajectory Replay(轨迹重放);旧代码路径和命令一次性迁移。优胜标记、英文可访问文本、筛选恢复、SVG 资产和跨断点布局新增自动门禁。官方成绩、分母和冻结证据不变。", + "attempts_sha256": "517b613adcc40fc0ebe41af2ab2115d76f49faaba5378b426cc196b5b633e9c7", + "selection_sha256": "ba3c344df003dd4eba3a5d1ef97162ee98df2ed728f7c29c4188bc4247982d38", + "changes": [ + [ + "轨道命名", + "旧名称混合了实现层与评测对象,难以直接区分自主任务与固定动作轨迹。", + "自主评测统一为 Browser Use;固定动作轨迹评测统一为 Browser Trajectory Replay(轨迹重放);代码包、示例、测试、命令、配置和发布路径同步迁移,不保留兼容入口。", + "只改变命名和入口,不改变任务、运行或判分。" + ], + [ + "比较与本地化", + "并列最优或舍入后显示相同的数值可能仍被标为优胜;英文页面部分可访问属性和动态标题保留中文。", + "只有按页面展示精度唯一领先的值才标优;英文标题、title、aria-label 和 placeholder 纳入本地化及完整性检查;失效的浏览器筛选链接回退到有效默认值。", + "修正读者界面,不改变测量值和官方分数。" + ], + [ + "防复发门禁", + "命名、图形格式和跨宽度布局主要依赖人工记忆。", + "CI 增加旧命名扫描;发布检查拒绝 SVG 图;视觉检查渲染中英文核心路由的桌面与窄屏、明暗主题组合,阻止页面级溢出、核心卡片裁切和表头竖排。", + "新增 104 个渲染状态的结构性视觉门禁;不改变报告数据。" + ] + ], + "i18n": {"en": { + "title": "Standardize track names and strengthen release checks", + "summary": "The autonomous track is Browser Use and the fixed-action track is Browser Trajectory Replay. Winner marks, English accessibility text, filter recovery, asset format, and responsive layout now have automated gates. Scores, denominators, and evidence are unchanged.", + "changes": [ + ["Track names", "Earlier names mixed implementation layers with evaluation subjects.", "Renamed the tracks Browser Use and Browser Trajectory Replay across code, commands, configuration, and publication paths without compatibility aliases.", "Only names and entry points changed; tasks, runs, and scoring did not."], + ["Comparison and localization", "Ties or display-equal values could be marked as winners, and some English attributes retained Chinese.", "Mark only uniquely better displayed values; localize titles and accessibility attributes; recover invalid browser filters to a valid default.", "Reader interface defects were corrected without changing measurements or scores."], + ["Regression gates", "Naming, graphics, and responsive layout relied mainly on manual memory.", "Added retired-name checks, SVG rejection, and bilingual light/dark desktop and narrow-screen rendering checks.", "The 104 structural render states do not change report data."] + ] + }} + }, + { + "id": "2026-09-10-comparison-and-review-presentation", + "date": "2026-09-10", + "title": "统一比较统计与可扩展浏览器对比", + "summary": "比较摘要统一展示均值、P50、P95;同题比较采用基准浏览器与多选浏览器组合;动作延迟按 action 分卡并只对单次均值判优;结果详情将长 evaluator 证据移出指标卡。Chromium 与 Moli 的官方成绩仍均为 25/30。", + "attempts_sha256": "517b613adcc40fc0ebe41af2ab2115d76f49faaba5378b426cc196b5b633e9c7", + "selection_sha256": "ba3c344df003dd4eba3a5d1ef97162ee98df2ed728f7c29c4188bc4247982d38", + "changes": [ + [ + "统计摘要", + "耗时和 Tokens 的集中趋势与尾部表现未采用统一展示结构。", + "每个指标统一展示均值、P50、P95;P50 不再以另一个名称重复出现。", + "只改变统计展示,官方任务分数、分母和证据不变。" + ], + [ + "比较表达", + "读者需要逐格判断数值方向。", + "通过数与通过率按更高值标优,耗时与 Tokens 按更低值标优;平局和调用次数保持中性。", + "优胜标记是描述性比较,不改变任务判定。" + ], + [ + "页面职责", + "失败清单与结果筛选重复,核验页混入逐结果覆盖表。", + "结果库保留成功/失败筛选和任务详情;核验页只呈现核验轮次、处理和遗留事项。", + "减少重复内容,不删除结果证据。" + ], + [ + "多浏览器对比", + "同题矩阵会随浏览器数量横向增宽,关系筛选也无法表达一个基准浏览器与多个对比浏览器的组合。", + "改为基准浏览器单选、对比浏览器多选和明确结果关系;任务行按通过与失败浏览器分组,浏览器标签链接完整结果。", + "页面结构可直接容纳新增浏览器;当前结果和分数不变。" + ], + [ + "动作延迟与详情页", + "动作表把三项指标压入浏览器单元格,并对调用量不同的总耗时判优;详情首屏的长错误文本破坏阅读层级。", + "每个 action 使用独立比较卡,只以单次均值判优并将总耗时、调用数保持中性;详情首屏只显示简明失败规则,完整诊断保留在逐规则证据区。", + "修正比较解释与页面层级,不改变测量值、判分或证据。" + ] + ], + "i18n": {"en": { + "title": "Standardize comparison statistics and scalable browser comparison", + "summary": "Comparison summaries now use mean, P50, and P95. Same-task comparison uses a baseline plus multiple comparison browsers. Action latency ranks mean per call only, and detailed evaluator evidence no longer appears in KPI cards. Both official scores remain 25/30.", + "changes": [ + ["Summary statistics", "Latency and token summaries lacked a consistent central and tail structure.", "Show mean, P50, and P95 for each metric without duplicating P50 under another name.", "Only presentation changed; scores, denominators, and evidence did not."], + ["Comparison presentation", "Readers had to infer metric direction cell by cell.", "Mark higher passes and pass rates and lower latency and token usage as better; keep ties and call counts neutral.", "Winner marks are descriptive and do not change task verdicts."], + ["Page responsibilities", "A failure list duplicated result filters, and review history contained a per-result inventory.", "Keep pass/fail filters and details in Results; keep only review rounds, corrections, and remaining items in Review.", "Duplicate presentation was removed without deleting evidence."], + ["Multi-browser comparison", "The matrix widened per browser and its filter could not compare one baseline with several browsers.", "Use one baseline, multi-select comparison browsers, explicit outcome relations, and linked pass/fail browser groups.", "The interface can add browsers without changing current results."], + ["Action latency and result details", "Action cells mixed three metrics and ranked totals despite unequal call counts; long errors disrupted result summaries.", "Use one card per action, rank only mean per call, keep totals and counts neutral, and retain full diagnostics in rule evidence.", "Interpretation and hierarchy changed; measurements, verdicts, and evidence did not."] + ] + }} + }, + { + "id": "2026-09-10-environment-timing-and-report-model-correction", + "date": "2026-09-10", + "title": "校正部署证据、计时边界与报告职责", + "summary": "报告明确本轮在 Colima/Docker 中部署四个 WebArena-Verified 官方站点环境,并解释浏览器容器通过私有宿主网关访问站点的网络拓扑;评测集页面恢复为纯任务定义,结果页新增 Agent 端到端与 BrowserTool 调用耗时,多浏览器比较改为可扩展矩阵和两两比较。Chromium 与 Moli 成绩仍均为 25/30。", + "attempts_sha256": "517b613adcc40fc0ebe41af2ab2115d76f49faaba5378b426cc196b5b633e9c7", + "selection_sha256": "ba3c344df003dd4eba3a5d1ef97162ee98df2ed728f7c29c4188bc4247982d38", + "changes": [ + [ + "冻结协议", + "上一版报告生成器默认使用最后一次执行作为结果,但本批冻结 manifest 声明使用原始正式结果,报告层无权改写证据协议。", + "生成器改为读取冻结 manifest 决定结果身份;当前批次恢复与原始证据一致的结果、回答和轨迹。后续新运行仍可使用新版协议,但必须在运行前冻结。", + "本批后续执行均未恢复成功,因此 Chromium 与 Moli 仍均为 25/30;修复的是结果身份和证据合同,不改变分数与分母。" + ], + [ + "部署环境", + "原图没有说明四个站点实际部署在哪里、私有网关是什么,也没有披露逐次环境准备缺少独立 receipt。", + "新增 macOS—Colima/Docker—站点容器—浏览器容器的拓扑说明、六类环境纳入状态、端口映射、镜像身份和证据边界;替换为精确标注的 ImageGen 图。", + "确认 30 题只覆盖 Shopping、Shopping Admin、Reddit、GitLab;Wikipedia 与 Map 未部署、未计分。现有结果不重跑。" + ], + [ + "耗时口径", + "旧说明错误地称端到端耗时包含站点重置,也没有分离浏览器工具调用时间。", + "明确 Agent 端到端计时不含此前的站点重置;从事件流聚合 BrowserTool 调用墙钟时间,并按 action 拆分。runner 为后续运行新增环境准备、会话准备、Agent、证据收尾和 evaluator 分阶段计时。", + "只新增诊断指标并纠正解释,不改变官方任务分数;BrowserTool 墙钟仍不等同于纯内核 CPU 时间。" + ], + [ + "报告模型", + "评测集页混入成功/失败,多浏览器结果结构写死为两列,方法介绍没有明确比较对象。", + "评测集页只保留任务与验收合同;结果页使用动态浏览器矩阵和自动两两比较;被测浏览器页列出 Chromium、Moli 及固定实验条件。", + "信息职责和未来扩展方式改变,当前两浏览器数据与成绩不变。" + ], + [ + "复核边界", + "证据完整性检查容易被误读为独立结果复核。", + "分别显示证据完整性 60/60 与独立语义复核 0/60,并移除固定样本上的总体推断式置信区间。", + "降低结论强度;不改变 evaluator 分数。" + ] + ], + "i18n": {"en": { + "title": "Correct deployment evidence, timing boundaries, and report responsibilities", + "summary": "The report documents the four WebArena-Verified sites deployed in Colima/Docker and the private host gateway used by browser containers. Dataset pages contain definitions only; result pages add agent end-to-end and BrowserTool latency. Both scores remain 25/30.", + "changes": [ + ["Frozen protocol", "The renderer chose the last execution even though the frozen manifest made the primary run authoritative.", "Read result authority from the frozen manifest and restore the matching result, answer, and trajectory.", "Neither browser recovered on later executions, so scores remain 25/30; only result identity and evidence semantics changed."], + ["Deployment environment", "The diagram did not locate the four sites, explain the private gateway, or disclose missing preparation receipts.", "Document the macOS, Colima/Docker, site-container, and browser-container topology, ports, image identities, and evidence boundary.", "The 30 tasks cover four sites only; Wikipedia and Map are neither deployed nor scored. No results were rerun."], + ["Timing boundary", "Earlier wording incorrectly included site reset in end-to-end latency and did not separate browser-tool time.", "Exclude the preceding site reset, aggregate BrowserTool wall time from events, and retain phase timings for future runs.", "Only diagnostics and wording changed; BrowserTool wall time still includes more than browser-core CPU time."], + ["Report model", "Dataset pages mixed outcomes with definitions, result tables assumed two browsers, and participants were unclear.", "Keep contracts in Dataset, render browsers dynamically in Results, and distinguish browser participants from shared controls.", "Information ownership and extensibility changed; current data and scores did not."], + ["Review boundary", "Evidence-integrity checks could be mistaken for independent semantic review.", "Show evidence integrity and semantic-review coverage separately and remove population-style confidence intervals from the fixed cohort.", "Claims were narrowed without changing evaluator scores."] + ] + }} + }, + { + "id": "2026-09-10-logical-results-and-publication-qa", + "date": "2026-09-10", + "title": "统一逻辑结果口径并完成全站可读性整改", + "summary": "公开报告统一为每题×浏览器一个最终逻辑结果,共 60 个结果;允许任务失败后再执行一次,最终一次决定结果,耗时与模型用量累计。Chromium 与 Moli 仍均为 25/30,分母和结论不变;高密度表格、代码对比度、流程图、窄屏布局和版本说明已按同一发布质量门禁整改。", + "attempts_sha256": "517b613adcc40fc0ebe41af2ab2115d76f49faaba5378b426cc196b5b633e9c7", + "selection_sha256": "ba3c344df003dd4eba3a5d1ef97162ee98df2ed728f7c29c4188bc4247982d38", + "changes": [ + [ + "结果口径", + "旧报告把底层执行次数直接暴露成主结果,并用不同标签区分,增加理解成本且与仓库统一逻辑结果原则不一致。", + "每题×浏览器只发布一个最终逻辑结果;任务失败后最多再执行一次,最终一次决定结果,所有执行的耗时与模型用量累计。", + "通过率仍为 Chromium 25/30、Moli 25/30,共 60 个逻辑结果;底层证据不删除,公开分数和分母不变,耗时统计改为完整成本口径。" + ], + [ + "全站视觉与信息密度", + "任务表和失败表的短标题被压成竖排,协议代码块对比度不足,文字型流程图在常规阅读尺寸下不清晰。", + "为核心表格设置语义化列宽和横向滚动;修复明暗主题代码色;用高清 ImageGen 信息图重绘部署—自主操作—双浏览器—验收流程;逐页检查桌面与窄屏。", + "只改变表达与可读性,不改变任务、合同、判定或原始证据。" + ], + [ + "版本说明", + "旧页面只展示当前变更表,缺少当前版本摘要、历史版本折叠、影响边界和结构化下载。", + "对齐 Web Fetch 的当前版本、按需展开依据、历史版本和范围下载结构。", + "版本历史成为正式报告的一部分;本次不改变分数、分母或证据。" + ], + [ + "可重复发布门禁", + "相同问题仅靠人工记忆修复,无法保证新环境生成报告时不复发。", + "把逻辑结果、禁用内部笔记、表格换行、代码对比度、跨断点视觉检查和变更记录要求写入生成器、自动检查、AGENTS.md 与仓库内 benchmark-report skill。", + "后续报告若重新出现这些已知问题,将在发布前检查中失败。" + ] + ], + "i18n": {"en": { + "title": "Unify logical-result semantics and complete report readability corrections", + "summary": "The public report now exposes one logical result per task and browser, with all physical-execution latency and model usage accumulated. Scores remain 25/30 for both browsers. Dense tables, code contrast, diagrams, narrow layouts, and release history now share one publication gate.", + "changes": [ + ["Logical results", "Physical executions were presented as separate headline results.", "Publish one logical result per task and browser; let the final allowed execution decide the outcome while accumulating every execution's cost.", "Scores and denominators remain unchanged and physical evidence is retained."], + ["Readability", "Short table headers wrapped vertically, code contrast was weak, and the text-heavy diagram was unreadable at normal size.", "Add semantic column widths and scrolling, correct theme contrast, replace the diagram with a readable raster asset, and inspect desktop and narrow layouts.", "Only presentation changed."], + ["Release history", "The page lacked a current summary, compact archive, effect boundaries, and a structured download.", "Use a report-style current release, expandable details, compact history, and structured data.", "Release history became part of the report without changing evidence or results."], + ["Repeatable publication gate", "Manual memory could not reliably prevent known regressions.", "Encode logical-result, privacy, table, contrast, responsive, and changelog requirements in automated release checks.", "Known regressions now fail before publication."] + ] + }} + }, + { + "id": "2026-09-10-deployment-and-autonomy-evidence", + "date": "2026-09-10", + "title": "补全本地部署与自主执行证据", + "summary": "首页标题改为 Chromium 与 Moli 的 Browser Use 能力实测;评测方法页新增 WebArena-Verified 本地站点、逐试次重置、CowAgent 闭环操作、官方验收和证据留存的完整流程图及实现证据。测量结果、分母和原始证据未改变。", + "attempts_sha256": "517b613adcc40fc0ebe41af2ab2115d76f49faaba5378b426cc196b5b633e9c7", + "selection_sha256": "ba3c344df003dd4eba3a5d1ef97162ee98df2ed728f7c29c4188bc4247982d38", + "changes": [ + [ + "报告标题", + "原标题把控制变量写成了评测对象,读者无法第一眼识别报告在比较哪两个浏览器。", + "主标题改为 Chromium 与 Moli 的 Browser Use 能力实测;CowAgent、模型、reasoning 和题集降为实验条件。", + "只修正叙事层级,不改变评测设计或结果。" + ], + [ + "部署与自主性证据", + "原报告只写“逐题重置”和“自主规划”,没有解释站点由谁部署、Agent 如何进入闭环,也不足以排除固定动作回放的误读。", + "新增双语流程图和逐环节证据表,明确运行器按试次启动官方站点镜像、健康检查、登录,再由 CowAgent 动态产生 BrowserTool 调用。", + "既有测量不重跑;新增说明直接绑定现有 manifest、事件、HAR 与 trace 哈希。" + ] + ], + "i18n": {"en": { + "title": "Complete deployment and autonomous-execution evidence", + "summary": "The title now identifies Chromium and Moli as the Browser Use participants. Methodology adds the full path from local WebArena-Verified deployment and per-execution reset through the CowAgent loop, official acceptance, and retained evidence. Measurements are unchanged.", + "changes": [ + ["Report title", "The earlier title foregrounded a control variable rather than the evaluated browsers.", "Make Chromium and Moli the title subjects and present CowAgent, model, reasoning, and tasks as conditions.", "Narrative hierarchy changed; design and results did not."], + ["Deployment and autonomy evidence", "Reset and autonomous planning were asserted without showing deployment ownership or the agent loop.", "Add a bilingual flow and evidence table linking site startup, health checks, sign-in, and dynamic BrowserTool calls.", "No measurements were rerun; the explanation binds to existing manifests, events, HAR, and trace hashes."] + ] + }} + }, + { + "id": "2026-09-10-report-evidence-restructure", + "date": "2026-09-10", + "title": "按证据链重构 Browser Use 报告", + "summary": "报告新增任务族覆盖、95% Wilson 区间、30 题同题横向对照和全路由英文镜像;将原“独立复核”纠正为确定性证据完整性审计,并明确五个共同失败尚无本轮成功路径,不能直接归因 Agent 或浏览器。测量结果、分母和原始证据未改变。", + "attempts_sha256": "517b613adcc40fc0ebe41af2ab2115d76f49faaba5378b426cc196b5b633e9c7", + "selection_sha256": "ba3c344df003dd4eba3a5d1ef97162ee98df2ed728f7c29c4188bc4247982d38", + "changes": [ + [ + "首页与结果分析", + "原首页只有总分和入口,无法同时判断样本覆盖、统计不确定性、调用完整性和同题关系。", + "增加三类任务族、95% Wilson 区间、可完成性覆盖、证据审计状态,以及可筛选的 30 题 Chromium/Moli 横向对照。", + "只改变解释和下钻结构,不改变 25/30 或 30 个配对。" + ], + [ + "核验命名与失败归因", + "原文把确定性记录一致性检查称为独立结果复核,并把共同失败过早归因给 Agent。", + "改名为证据完整性审计;五个没有成功轨迹的任务仅陈述官方期望与实际结果差异,要求另做非计分 oracle 或人工可完成性核验。", + "官方 evaluator 仍是唯一计分来源;共同失败不再被表述为已确认根因。" + ], + [ + "双语与移动端", + "英文只有摘要页,窄屏入口卡片会被固定跨列布局裁切。", + "为中文页面生成逐路由英文对应页并校验链接;移动端强制卡片单列,移除用 overflow 隐藏问题的做法。", + "英文与中文共享同一测量数据;206 个逻辑页面当前为零断链。" + ], + [ + "构建与发布门禁", + "正式报告可能在官方数据不可用时静默退化,日常检查也未默认要求完整证据。", + "正式构建绑定官方数据 checksum 并 fail closed;增加 current-report 发布 profile、完整证据默认检查和带 marker 的原子目录替换。", + "缺失或错配 selection、模型、reasoning、官方数据或证据审计时禁止发布。" + ] + ], + "i18n": {"en": { + "title": "Restructure the Browser Use report around the evidence chain", + "summary": "The report adds task-family coverage, Wilson intervals, 30 paired-task comparisons, and route-for-route English pages. It correctly labels deterministic evidence-integrity review and avoids attributing five shared failures without a successful path. Measurements are unchanged.", + "changes": [ + ["Home and results", "The home page did not jointly expose cohort coverage, uncertainty, call completeness, and paired outcomes.", "Add task families, Wilson intervals, solvability coverage, evidence-audit status, and a filterable 30-task Chromium/Moli comparison.", "Only explanation and drill-down changed; 25/30 scores and 30 pairs did not."], + ["Review terminology and attribution", "Deterministic consistency checks were called independent result review, and shared failures were attributed too early.", "Call the process evidence-integrity review and require separate non-scoring solvability review for tasks without a successful trajectory.", "The official evaluator remains the sole scoring authority; shared failures are no longer presented as confirmed causes."], + ["Bilingual and mobile", "English was summary-only and fixed card spans clipped narrow layouts.", "Generate an English counterpart for every Chinese route, verify links, and force cards to one column on narrow screens.", "Both languages use the same measured data; all 206 logical pages had valid links."], + ["Build and publication gates", "A formal build could silently degrade without official data, while routine checks did not require complete evidence.", "Bind builds to the official checksum and current publication profile, require complete evidence, and replace output atomically with a marker.", "Publication fails on missing or mismatched selection, model, reasoning, official data, or evidence review."] + ] + }} + }, + { + "id": "2026-09-09-webarena-verified-pilot", + "date": "2026-09-09", + "title": "完成 WebArena-Verified 自主 Agent 双浏览器对比", + "summary": "CowAgent 与 gpt-5.6-sol(reasoning=low)在预注册的 30 题上分别以 Chromium 和 Moli 形成 60 个逻辑结果;双方均通过 25 题,通过率 83.3%。全部结果证据通过确定性完整性审计,分歧与未决均为 0。", + "attempts_sha256": "517b613adcc40fc0ebe41af2ab2115d76f49faaba5378b426cc196b5b633e9c7", + "selection_sha256": "ba3c344df003dd4eba3a5d1ef97162ee98df2ed728f7c29c4188bc4247982d38", + "changes": [ + [ + "证据应用与独立解读", + "只有栏目分页仍不足以核对任务选择、逐次执行、复核覆盖、当前快照可完成性和结论依据。", + "首页作为决策入口;30 个任务页公开官方 ground truth、可完成性证据与两种浏览器结果对照;结果页公开实际回答、精确 assertion、规则级 expected/actual、CowAgent 自主轨迹、输出摘录和脱敏事件下载;增加英文摘要、证据索引和筛选状态保持。", + "出版结构与公开证据粒度改变;任务、分母、原始输出、分数与证据绑定均未修改。25 题有本轮成功轨迹,5 题只有官方合同;原始 HAR 与 Playwright trace 因含登录态只公开哈希和大小。" + ], + [ + "明确自主 Agent 轨道", + "仓库同时存在执行层 trajectory_replay harness 和自主 browser_use harness,若统称为“原 30 题”会误读运行方式。", + "本报告的 30 个 WebArena-Verified 任务明确由 CowAgent 自主观察、规划和操作;trajectory_replay 作为另一条执行层微基准单独说明。", + "本轮成绩只来自自主 Agent 轨道;两类 harness 不合并计分。" + ], + [ + "公开 benchmark 选型", + "模型厂商使用的 browser-use、computer-use、离线动作预测和 agentic search 口径混杂。", + "按一手机构采用情况梳理 Mind2Web、WebArena、WebVoyager,并比较 Online-Mind2Web、BU Bench、Odysseys 等评测对象和 judge。", + "第一阶段选择可自托管、确定性结果验收的 WebArena-Verified;实时网站和 LLM judge 留到后续泛化阶段。" + ], + [ + "完整正式运行", + "需要同时覆盖短、中、长操作链,且避免运行后换题或挑结果。", + "预注册 5/10/15 题操作长度分层;每个题×浏览器形成一个最终逻辑结果,任务失败后最多再执行一次。", + "Chromium 与 Moli 均为 25/30;共同失败 5 题、单边胜出 0 题,30 题 pilot 未观察到成功率差异。" + ], + [ + "证据与复核", + "最终分数必须能回到模型回答、网络副作用和官方 evaluator,不能人工修分。", + "保存回答、事件、HAR、Playwright trace 和 evaluator 明细;逐试次核对文件、哈希、分数、状态与模型请求尝试。", + "全部逻辑结果及其底层证据审计通过,记分分歧 0、未决 0;完整网络正文和登录凭据不公开。" + ], + [ + "Astra 可用性边界", + "用户代理公开的模型列表不等于当前团队有调用权限。", + "直接调用代理验证 gpt-6-astra,服务返回 team_model_access_denied;随后只用可用的 gpt-5.6-sol 完成流程。", + "本轮是 GPT-5.6 条件下的正式对比,不能写成 Astra 成绩;Astra 获权后必须新建独立运行。" + ] + ], + "i18n": {"en": { + "title": "Complete the WebArena-Verified autonomous-agent browser comparison", + "summary": "CowAgent with gpt-5.6-sol at reasoning=low produced 60 logical results across 30 preregistered tasks in Chromium and Moli. Each browser passed 25 tasks. Deterministic evidence-integrity checks covered every result with no unresolved inconsistency.", + "changes": [ + ["Evidence and interpretation", "Navigation alone did not make task selection, executions, review coverage, solvability, or conclusions auditable.", "Use the home page as a decision entry point and publish contracts, measured results, assertions, trajectories, sanitized events, an English edition, and an evidence index.", "Publication structure changed; tasks, denominator, outputs, scores, and evidence bindings did not."], + ["Autonomous-agent track", "The repository also contained a trajectory-replay harness, making the phrase 'the original 30 tasks' ambiguous.", "State that CowAgent autonomously observes, plans, and acts for this report, while trajectory replay is a separate microbenchmark.", "Only the autonomous-agent track contributes to these scores."], + ["Benchmark selection", "Browser-use, computer-use, offline action prediction, and agentic-search claims used incompatible evaluation units.", "Compare primary-source adoption and evaluation contracts across relevant benchmark families.", "Choose self-hosted WebArena-Verified for deterministic outcome evaluation; live-site and model-judge tracks remain future work."], + ["Complete formal run", "The cohort needed short, medium, and long workflows without post-run task selection.", "Preregister 5/10/15 operation-length tiers and one logical result per task-browser pair, with at most one declared execution after failure.", "Both browsers passed 25/30; five tasks failed jointly and none produced a one-browser-only pass."], + ["Evidence integrity", "Scores needed traceability to final answers, network side effects, and official evaluators without manual overrides.", "Retain responses, events, HAR, traces, and evaluator details and verify files, hashes, scores, statuses, and transport attempts.", "All logical results passed evidence-integrity checks with no scoring disagreement or unresolved item; sensitive network bodies remain private."], + ["Model-coverage boundary", "Availability in a model catalog did not establish access for this run.", "Record gpt-5.6-sol as the only measured model after Astra access was unavailable.", "This is a GPT-5.6 comparison, not an Astra result; Astra requires a separate run when available."] + ] + }} + } + ], "review_rounds": [ { "date": "2026-09-14", @@ -269,12 +1421,42 @@ "title": "说明按需发布架构,补全安装清单与公开安全门禁", "summary": "报告明确区分逻辑路由与线上物理文件,补齐双语入口、七种方法的安装清单和公开安全门禁;同时按固定最终轮纠正四题、三个 Moli 版本的 12 个异常耗时。任务输出、判定和分数不变,延迟汇总已重算。", "changes": [ - ["发布架构", "把 1,218 个可审计页面称为线上 HTML 文件,容易误解为重复部署并一次性加载。", "明确本地构建产出逻辑路由,Sites 包使用单个 HTML 外壳、共享运行时和内容寻址压缩块,页面与大证据按需加载。", "发布清单记录逻辑路由、源文件、压缩块和逐文件哈希;不减少证据,也不改变下载字节。"], - ["账号与安装", "旧页只显示唯一需要供应商账号的 Cloudflare,导致本地方法的取得方式和二进制配置看似消失。", "页面改为账号与安装清单:Cloudflare 列账号与凭据变量,其余六种本地方法列官方获取入口和二进制变量。", "只公开变量名称和官方入口,不保存值、本机路径、Cookie 或个人会话。"], - ["核验标识", "榜单只写已核验,读者无法直接看出核验主体。", "各方法审查列显示 100/100 · Codex 已核验,并继续链接到统一核验记录。", "只改展示文案;核验绑定、覆盖数和未决数不变。"], - ["中英版本", "原英文入口只是单页摘要,路由、章节和逐次证据解释均未与中文报告对齐。", "从同一报告模型生成逐页对应的中英文站点;导航按评测集、方法、分析、核验、解读、实现、复现、变更、安装排序,语言切换停留在当前路由。", "两种语言的路由、章节、表格、数值、筛选器和证据链接一一校验;界面与解释按英文语境改写,任务名、ID、URL、标准答案和原始输出保持原文且不重新评分。"], - ["公开安全门禁", "原发布扫描覆盖 Google、Artifact Hub、GitHub 与 OpenAI 等常见密钥形态,但未明确拦截 Authorization 值、本机用户目录、AWS 与 Slack 凭据。", "统一扫描每个重建后的公开文件,新增 Authorization、用户目录、AWS、Slack 和私钥形态检查;第三方 HTML 的既有 Google 形态脱敏仍保留双哈希。", "模式扫描不能证明不存在所有未知秘密;结构化凭据值仍禁止进入证据和报告,发布后还要逐文件字节复核。"], - ["Moli 时延纠正", "少量旧耗时异常会让 1.1.3 看起来存在未经证明的整体性能回退。", "对同一四题、三个版本做三轮轮换,统一采用第三轮的 12 个耗时并重算均值、P50、P95;不按结果择优。", "全部通过任务中,1.1.3 的均值由 4,000 降至 3,706 ms、P50 由 3,444 降至 3,340 ms、P95 由 9,833 降至 7,983 ms;1.0.6 与 1.1.1 也按同一规则更新。共保留 36 次物理调用;正文、评分和任务判定未改变。"] + [ + "发布架构", + "把 1,218 个可审计页面称为线上 HTML 文件,容易误解为重复部署并一次性加载。", + "明确本地构建产出逻辑路由,Sites 包使用单个 HTML 外壳、共享运行时和内容寻址压缩块,页面与大证据按需加载。", + "发布清单记录逻辑路由、源文件、压缩块和逐文件哈希;不减少证据,也不改变下载字节。" + ], + [ + "账号与安装", + "旧页只显示唯一需要供应商账号的 Cloudflare,导致本地方法的取得方式和二进制配置看似消失。", + "页面改为账号与安装清单:Cloudflare 列账号与凭据变量,其余六种本地方法列官方获取入口和二进制变量。", + "只公开变量名称和官方入口,不保存值、本机路径、Cookie 或个人会话。" + ], + [ + "核验标识", + "榜单只写已核验,读者无法直接看出核验主体。", + "各方法审查列显示 100/100 · Codex 已核验,并继续链接到统一核验记录。", + "只改展示文案;核验绑定、覆盖数和未决数不变。" + ], + [ + "中英版本", + "原英文入口只是单页摘要,路由、章节和逐次证据解释均未与中文报告对齐。", + "从同一报告模型生成逐页对应的中英文站点;导航按评测集、方法、分析、核验、解读、实现、复现、变更、安装排序,语言切换停留在当前路由。", + "两种语言的路由、章节、表格、数值、筛选器和证据链接一一校验;界面与解释按英文语境改写,任务名、ID、URL、标准答案和原始输出保持原文且不重新评分。" + ], + [ + "公开安全门禁", + "原发布扫描覆盖 Google、Artifact Hub、GitHub 与 OpenAI 等常见密钥形态,但未明确拦截 Authorization 值、本机用户目录、AWS 与 Slack 凭据。", + "统一扫描每个重建后的公开文件,新增 Authorization、用户目录、AWS、Slack 和私钥形态检查;第三方 HTML 的既有 Google 形态脱敏仍保留双哈希。", + "模式扫描不能证明不存在所有未知秘密;结构化凭据值仍禁止进入证据和报告,发布后还要逐文件字节复核。" + ], + [ + "Moli 时延纠正", + "少量旧耗时异常会让 1.1.3 看起来存在未经证明的整体性能回退。", + "对同一四题、三个版本做三轮轮换,统一采用第三轮的 12 个耗时并重算均值、P50、P95;不按结果择优。", + "全部通过任务中,1.1.3 的均值由 4,000 降至 3,706 ms、P50 由 3,444 降至 3,340 ms、P95 由 9,833 降至 7,983 ms;1.0.6 与 1.1.1 也按同一规则更新。共保留 36 次物理调用;正文、评分和任务判定未改变。" + ] ] }, { @@ -283,10 +1465,30 @@ "title": "统一方法组合对比,补充 Moli 时延诊断", "summary": "结果分析页改为一个可同时处理双方法和多方法的对比器,并重新排版任务与结果;issue #66 的回退线索经公网顺序轮换与本地固定页对照后,保留为未能单因归因的长尾现象。正式任务、分数和耗时不变。", "changes": [ - ["统一对比器", "双方法与多方法拆成两张大卡和两份结果表,控制项重复、页面拥挤。", "保留一个方法 A、可多选对比方法和一组统一关系条件;结果只渲染一张同题表。", "支持有分歧、A 成功或失败、全部成功或失败、仅所选方法成功;不生成新分数。"], - ["结果排版", "双栏压缩了任务类型和方法列,中文类型被逐字换行,横向阅读困难。", "组件改为全宽;任务类型并入任务单元格徽标,成功与失败方法用可换行状态标签展示。", "窄屏仍允许表格横向滚动,任务名称、ID 和证据链接完整保留。"], - ["Moli 时延线索", "正式单次结果显示 1.1.3 在少数任务上明显慢于旧版,但无法区分版本、网络和页面等待。", "选择四个差异大的不同路径任务做三轮轮换复测,再用本地固定页各五轮测固定开销。", "公网各题中位数有快有慢且波动区间重叠;本地固定开销未见 1.1.3 回退,因此不把 issue #66 升格为已确认回归。"], - ["纠正与失败重试", "把所有后续调用都禁止掉会让已证实的错误无法纠正;反过来,失败后挑选更好响应又会制造 best-of 偏差。", "区分两条路径:错误命令、版本、裁判或参考按明确范围 supersede 并重测;普通失败可在测量前声明最多重试一次,保留两次物理证据且最后一次无条件生效。", "当前 1,050 条结果没有启用重试,issue #66 的诊断复测也不替换正式结果;后续协议可按需启用。"] + [ + "统一对比器", + "双方法与多方法拆成两张大卡和两份结果表,控制项重复、页面拥挤。", + "保留一个方法 A、可多选对比方法和一组统一关系条件;结果只渲染一张同题表。", + "支持有分歧、A 成功或失败、全部成功或失败、仅所选方法成功;不生成新分数。" + ], + [ + "结果排版", + "双栏压缩了任务类型和方法列,中文类型被逐字换行,横向阅读困难。", + "组件改为全宽;任务类型并入任务单元格徽标,成功与失败方法用可换行状态标签展示。", + "窄屏仍允许表格横向滚动,任务名称、ID 和证据链接完整保留。" + ], + [ + "Moli 时延线索", + "正式单次结果显示 1.1.3 在少数任务上明显慢于旧版,但无法区分版本、网络和页面等待。", + "选择四个差异大的不同路径任务做三轮轮换复测,再用本地固定页各五轮测固定开销。", + "公网各题中位数有快有慢且波动区间重叠;本地固定开销未见 1.1.3 回退,因此不把 issue #66 升格为已确认回归。" + ], + [ + "纠正与失败重试", + "把所有后续调用都禁止掉会让已证实的错误无法纠正;反过来,失败后挑选更好响应又会制造 best-of 偏差。", + "区分两条路径:错误命令、版本、裁判或参考按明确范围 supersede 并重测;普通失败可在测量前声明最多重试一次,保留两次物理证据且最后一次无条件生效。", + "当前 1,050 条结果没有启用重试,issue #66 的诊断复测也不替换正式结果;后续协议可按需启用。" + ] ] }, { @@ -295,9 +1497,24 @@ "title": "增加同题方法组合对比,修复大输出卡顿", "summary": "结果分析页新增两方法差异和多方法交集;超过 256 KiB 的完整证据不再自动插入页面,改为按需加载或直接下载。正式任务、调用、判定、分数和耗时均未改变。", "changes": [ - ["同题方法对比", "旧页只能按方法或结果筛选 1,050 条调用,难以直接找到 A 成功而 B 失败、共同成功、共同失败或多方法分歧的任务。", "新增两方法差异与多方法交集;每行沿用同一任务的已核验主判定,并链接到各次标准答案、实际返回和失败原因。", "这是集合分析,不产生新分数,也不把不同成功样本的耗时混在一起比较。"], - ["大证据加载", "超过百万字符的输出自动下载并写入 pre 元素时,会长时间占用主线程,页面表现为卡死。", "超过 256 KiB 的文本先显示按需入口;用户点击后使用只读文本框承载完整内容,原始文件下载始终保留。", "只改变浏览器呈现时机和控件;发布包内证据字节及校验值不变。"], - ["版本与样本边界", "Moli 1.1.3 比 1.1.1 少两题可能被直接解读成版本能力回退;京东偶尔可打开也可能被用来否定正式失败。", "三轮配对诊断将瞬时请求、页面不稳定和当前主榜成绩分开;京东保留为网络与会话敏感的挑战样本。", "当前主榜未启用失败重试,也不用诊断调用挑选更好响应;后续可预先声明失败重试,已证实的错误可按完整范围纠正并替换权威逻辑结果。"] + [ + "同题方法对比", + "旧页只能按方法或结果筛选 1,050 条调用,难以直接找到 A 成功而 B 失败、共同成功、共同失败或多方法分歧的任务。", + "新增两方法差异与多方法交集;每行沿用同一任务的已核验主判定,并链接到各次标准答案、实际返回和失败原因。", + "这是集合分析,不产生新分数,也不把不同成功样本的耗时混在一起比较。" + ], + [ + "大证据加载", + "超过百万字符的输出自动下载并写入 pre 元素时,会长时间占用主线程,页面表现为卡死。", + "超过 256 KiB 的文本先显示按需入口;用户点击后使用只读文本框承载完整内容,原始文件下载始终保留。", + "只改变浏览器呈现时机和控件;发布包内证据字节及校验值不变。" + ], + [ + "版本与样本边界", + "Moli 1.1.3 比 1.1.1 少两题可能被直接解读成版本能力回退;京东偶尔可打开也可能被用来否定正式失败。", + "三轮配对诊断将瞬时请求、页面不稳定和当前主榜成绩分开;京东保留为网络与会话敏感的挑战样本。", + "当前主榜未启用失败重试,也不用诊断调用挑选更好响应;后续可预先声明失败重试,已证实的错误可按完整范围纠正并替换权威逻辑结果。" + ] ] }, { @@ -306,9 +1523,24 @@ "title": "迁移到 Sites,并固化评测报告工作流", "summary": "报告迁移到新的 Sites 域名并保留原站点;仓库新增跨评测类型的报告 Skill,覆盖评测设计、证据、独立核验、影响范围重跑和发布门禁。任务、输出、分数和耗时不变。", "changes": [ - ["发布目标", "旧发布脚本固定使用 artifact-hub.dev 域名,换会话可能继续更新旧站点。", "当前报告配置同时绑定 Sites 服务地址和站点 ID;发布、验证和正文链接从同一配置读取,目标不一致时拒绝更新。", "新站点为 UUAhhQMnrGwm;旧站点 bWkRP157EtF6 和 UeZE1vfcEOrM 均未覆盖。"], - ["发布身份", "仓库 .env 中的旧发布令牌可能优先于用户目录的新 Sites 令牌,导致已授权后仍返回 401。", "发布与验证脚本不再加载评测 .env,只读取显式进程环境或用户级令牌;设备授权时会备份不同的旧令牌。", "只改变发布凭据来源,不改变任何 provider 调用或报告数据。"], - ["跨评测工作流", "既有 benchmark-editorial 只负责正式解读文章,无法约束评测设计、评分核验、选择性重跑和报告发布。", "新增 benchmark-report Skill,并由根 AGENTS.md 强制路由;规则适用于 Web Fetch、Browser Use 及其他系统对比,不绑定本报告页面结构。", "这是流程和维护门禁;本轮主结果仍为 1,050 条,同窗测速仍为 140 条,未重新调用 provider。"] + [ + "发布目标", + "旧发布脚本固定使用 artifact-hub.dev 域名,换会话可能继续更新旧站点。", + "当前报告配置同时绑定 Sites 服务地址和站点 ID;发布、验证和正文链接从同一配置读取,目标不一致时拒绝更新。", + "新站点为 UUAhhQMnrGwm;旧站点 bWkRP157EtF6 和 UeZE1vfcEOrM 均未覆盖。" + ], + [ + "发布身份", + "仓库 .env 中的旧发布令牌可能优先于用户目录的新 Sites 令牌,导致已授权后仍返回 401。", + "发布与验证脚本不再加载评测 .env,只读取显式进程环境或用户级令牌;设备授权时会备份不同的旧令牌。", + "只改变发布凭据来源,不改变任何 provider 调用或报告数据。" + ], + [ + "跨评测工作流", + "既有 benchmark-editorial 只负责正式解读文章,无法约束评测设计、评分核验、选择性重跑和报告发布。", + "新增 benchmark-report Skill,并由根 AGENTS.md 强制路由;规则适用于 Web Fetch、Browser Use 及其他系统对比,不绑定本报告页面结构。", + "这是流程和维护门禁;本轮主结果仍为 1,050 条,同窗测速仍为 140 条,未重新调用 provider。" + ] ] }, { @@ -318,9 +1550,24 @@ "scoring_revision_input_sha256": "ea7b9d530566874af1b8e6b13796a29975d752564d6b143a9e540902d4ef8406", "summary": "默认重建现在只接受七方法、150 题和 1,050 条主调用的当前证据。机器可读路线改为本轮实际使用的原生 Markdown、Moli 自动原始响应和 node-html-markdown 2.0.0;分数、调用输出和耗时不变。", "changes": [ - ["默认重建", "默认证据目录仍是旧的十二方法评测,新会话可能在没有报错的情况下生成另一份报告。", "将已核验的七方法证据设为 current,并增加独立的当前报告配置文件;构建和证据检查会拒绝方法、数量、截止时间或路线不匹配的证据。", "主结果仍为 1,050/1,050 条核验,同窗测速仍为 140/140 条核验;旧证据移入 research 目录保留。"], - ["路线元数据", "下载数据仍把 Chromium 转换器写成已淘汰的 pyhtml2md 1.7.0,也没有写明三个 Moli 版本处理直接文档和媒体的路线。", "路线说明改为与运行器共用的当前定义:普通网页使用各自原生正文能力,Moli 直接文档和媒体使用自动原始响应,Chromium 使用 node-html-markdown 2.0.0。", "只更正非判定元数据;任务、参考、输出、分数和耗时均未变化。"], - ["复发门禁", "核验轮次依附在某一条版本记录中,新增版本时容易让核验页变空;通用 Moli 文件适配器仍可能给二进制响应附加 --dump html。", "核验轮次改为独立的结构化记录并在构建时校验;Moli 直接文件统一使用裸 fetch;文档命令统一使用锁定的 uv 运行入口。", "页面只展示每轮范围、发现、纠正和未完成项;不增加核验结果下载。"] + [ + "默认重建", + "默认证据目录仍是旧的十二方法评测,新会话可能在没有报错的情况下生成另一份报告。", + "将已核验的七方法证据设为 current,并增加独立的当前报告配置文件;构建和证据检查会拒绝方法、数量、截止时间或路线不匹配的证据。", + "主结果仍为 1,050/1,050 条核验,同窗测速仍为 140/140 条核验;旧证据移入 research 目录保留。" + ], + [ + "路线元数据", + "下载数据仍把 Chromium 转换器写成已淘汰的 pyhtml2md 1.7.0,也没有写明三个 Moli 版本处理直接文档和媒体的路线。", + "路线说明改为与运行器共用的当前定义:普通网页使用各自原生正文能力,Moli 直接文档和媒体使用自动原始响应,Chromium 使用 node-html-markdown 2.0.0。", + "只更正非判定元数据;任务、参考、输出、分数和耗时均未变化。" + ], + [ + "复发门禁", + "核验轮次依附在某一条版本记录中,新增版本时容易让核验页变空;通用 Moli 文件适配器仍可能给二进制响应附加 --dump html。", + "核验轮次改为独立的结构化记录并在构建时校验;Moli 直接文件统一使用裸 fetch;文档命令统一使用锁定的 uv 运行入口。", + "页面只展示每轮范围、发现、纠正和未完成项;不增加核验结果下载。" + ] ] }, { @@ -336,13 +1583,48 @@ "title": "完成七种方法实测、全量核验与同窗测速", "summary": "七种方法均完成同一组 150 个任务,共 1,050 次主调用;另有 20 题 × 7 种方法的 140 次同窗测速。Codex 任务代理在看不到机器结论和分数的条件下逐条核验 1,190/1,190 份输出,最终与当前评分一致,未决 0。", "changes": [ - ["七方法完整矩阵", "新增 Moli 1.0.6 后,旧的六方法分母和版本对比已不完整。", "七种方法各执行 150 个相同任务,每个方法—任务组合保留一个权威逻辑结果;Moli 1.0.6、1.1.1、1.1.3 分列。", "通过数依次为 Moli 1.1.1 135、Moli 1.0.6 134、Moli 1.1.3 133、Chromium 125、Lightpanda 119、Obscura 117、Kitesurf 112;总数只表示本评测集覆盖,不替代六类分榜。"], - ["独立核验账", "只检查失败项会漏掉假阳性,旧详情也曾把排版差异误判成内容缺失。", "Codex 任务代理逐条读取任务契约、标准答案和完整实际输出,主结果核验 1,050/1,050,同窗测速核验 140/140;机器结论和分数在独立判定前隐藏。", "当前自动假阴性 0、自动假阳性 0、未决 0;challenge-08、动态引文字段边界、飞书表格及直接文档路线等历史问题均以当前统一规则重算或重测。"], - ["同窗测速", "主矩阵来自多个执行批次,远程网络、排队和本地资源竞争使跨批次耗时不能公平排名。", "另取 20 个预声明、域名互异的网页,同一窗口并行执行七种方法;延迟只比较全部方法共同通过的 8 个相同任务。", "Moli 1.0.6 的样本 P50 最低;8 题属于探索性小样本,P95 只作样本诊断,不外推为稳定尾延迟或 SLA。"], - ["参考与路线纠正", "live-025 曾误用初始 HTTP 字节作为动态页面标准;Moli 直接 PDF/媒体若强加 --dump 会在取回产物前失败。", "live-025 改用独立浏览器渲染后的可见正文并重测七种方法;三个 Moli 版本对直接 PDF、DOCX 和媒体 URL 使用裸 fetch,正文题计入本地解析,文件题直接验收字节。", "标准、输入输出目标与评分入口对所有方法一致;各方法仍可使用自己最快的正式原生产物路径。"], - ["document-03 参考替换", "旧 W3C PDF 地址在本轮变成 Cloudflare 验证页,继续使用会把站点拦截误当作 PDF 解析能力。", "改用 Mozilla PDF.js 固定 Git 提交中的三页 PDF,并重新调用七种方法;未变化的 1,043 条主结果及核验保持原样。", "Moli 1.0.6、1.1.1 取得全部三页文字并通过;Moli 1.1.3 单次请求失败,其他方法未取得可读正文。该单次结果不能单独证明版本回归。"], - ["标题诊断", "标题指标先展示定义和空表,读者看不到输入、对齐和汇总过程。", "按读取条目、详情链接对齐、逐标题比较、可比结果汇总四步展示;没有可对照标题时明确显示不适用,不生成空表。", "标题诊断仍不参与任务通过判定,也不代表榜单召回、实时性或整页正文质量。"], - ["环境与复现", "旧环境页复述每个批次,账号页又容易让人误以为报告应保存正式密钥。", "环境和复现合并,只保留版本、二进制哈希、部署边界、配置文件和锁定命令;账号页仅列所需账号、变量名和官方获取入口。", "正式密钥只从仓库本地 .env 读取,不进入 HTML、Git 或发布包;Kitesurf 的托管后端版本和本地部署不可复现边界单独披露。"] + [ + "七方法完整矩阵", + "新增 Moli 1.0.6 后,旧的六方法分母和版本对比已不完整。", + "七种方法各执行 150 个相同任务,每个方法—任务组合保留一个权威逻辑结果;Moli 1.0.6、1.1.1、1.1.3 分列。", + "通过数依次为 Moli 1.1.1 135、Moli 1.0.6 134、Moli 1.1.3 133、Chromium 125、Lightpanda 119、Obscura 117、Kitesurf 112;总数只表示本评测集覆盖,不替代六类分榜。" + ], + [ + "独立核验账", + "只检查失败项会漏掉假阳性,旧详情也曾把排版差异误判成内容缺失。", + "Codex 任务代理逐条读取任务契约、标准答案和完整实际输出,主结果核验 1,050/1,050,同窗测速核验 140/140;机器结论和分数在独立判定前隐藏。", + "当前自动假阴性 0、自动假阳性 0、未决 0;challenge-08、动态引文字段边界、飞书表格及直接文档路线等历史问题均以当前统一规则重算或重测。" + ], + [ + "同窗测速", + "主矩阵来自多个执行批次,远程网络、排队和本地资源竞争使跨批次耗时不能公平排名。", + "另取 20 个预声明、域名互异的网页,同一窗口并行执行七种方法;延迟只比较全部方法共同通过的 8 个相同任务。", + "Moli 1.0.6 的样本 P50 最低;8 题属于探索性小样本,P95 只作样本诊断,不外推为稳定尾延迟或 SLA。" + ], + [ + "参考与路线纠正", + "live-025 曾误用初始 HTTP 字节作为动态页面标准;Moli 直接 PDF/媒体若强加 --dump 会在取回产物前失败。", + "live-025 改用独立浏览器渲染后的可见正文并重测七种方法;三个 Moli 版本对直接 PDF、DOCX 和媒体 URL 使用裸 fetch,正文题计入本地解析,文件题直接验收字节。", + "标准、输入输出目标与评分入口对所有方法一致;各方法仍可使用自己最快的正式原生产物路径。" + ], + [ + "document-03 参考替换", + "旧 W3C PDF 地址在本轮变成 Cloudflare 验证页,继续使用会把站点拦截误当作 PDF 解析能力。", + "改用 Mozilla PDF.js 固定 Git 提交中的三页 PDF,并重新调用七种方法;未变化的 1,043 条主结果及核验保持原样。", + "Moli 1.0.6、1.1.1 取得全部三页文字并通过;Moli 1.1.3 单次请求失败,其他方法未取得可读正文。该单次结果不能单独证明版本回归。" + ], + [ + "标题诊断", + "标题指标先展示定义和空表,读者看不到输入、对齐和汇总过程。", + "按读取条目、详情链接对齐、逐标题比较、可比结果汇总四步展示;没有可对照标题时明确显示不适用,不生成空表。", + "标题诊断仍不参与任务通过判定,也不代表榜单召回、实时性或整页正文质量。" + ], + [ + "环境与复现", + "旧环境页复述每个批次,账号页又容易让人误以为报告应保存正式密钥。", + "环境和复现合并,只保留版本、二进制哈希、部署边界、配置文件和锁定命令;账号页仅列所需账号、变量名和官方获取入口。", + "正式密钥只从仓库本地 .env 读取,不进入 HTML、Git 或发布包;Kitesurf 的托管后端版本和本地部署不可复现边界单独披露。" + ] ] }, { @@ -351,9 +1633,24 @@ "title": "区分文字存在与字段边界完整", "summary": "复核 Quotes 异步列表的两版 Moli 输出:10 条引文和作者均已取得,但相邻字段及多标签之间没有可读分隔。任务结果保持未通过,详情页改为按引文、作者、标签逐字段呈现,避免把边界损失笼统写成文字缺失。", "changes": [ - ["字段级证据", "旧详情把整条稳定记录标成未命中,即使其中引文和作者已经完整出现,读者无法知道真正失败的是标签列表边界。", "逐条展示引文、作者和标签的标准答案、实际解析值与判定;标签字符完整但分隔消失时明确标为字段边界不合格。", "只修正失败解释和分类,不改变任务契约、原始输出、耗时或分数。"], - ["两版 Moli 复核", "只看 Moli 1.1.3 会遗漏旧版是否存在同类问题。", "Moli 1.1.1 与 1.1.3 的该任务输出逐字节相同,均有 10 条完整且唯一记录;8 条多标签文字粘连,最后一条还混入 Loading 文本。", "两版均维持未通过;单标签的第 9 条正常命中,说明规则检查的是标签边界而非引文是否出现。"], - ["失败核验覆盖", "读者需要区分全量核验与只检查个别版本。", "主结果 900/900 全量盲审,包括 Moli 1.1.1 和 1.1.3 各 150 条;184 个机器失败项全部有非空独立审查证据。", "各方法失败核验覆盖均为 100%,无未裁定结果。"] + [ + "字段级证据", + "旧详情把整条稳定记录标成未命中,即使其中引文和作者已经完整出现,读者无法知道真正失败的是标签列表边界。", + "逐条展示引文、作者和标签的标准答案、实际解析值与判定;标签字符完整但分隔消失时明确标为字段边界不合格。", + "只修正失败解释和分类,不改变任务契约、原始输出、耗时或分数。" + ], + [ + "两版 Moli 复核", + "只看 Moli 1.1.3 会遗漏旧版是否存在同类问题。", + "Moli 1.1.1 与 1.1.3 的该任务输出逐字节相同,均有 10 条完整且唯一记录;8 条多标签文字粘连,最后一条还混入 Loading 文本。", + "两版均维持未通过;单标签的第 9 条正常命中,说明规则检查的是标签边界而非引文是否出现。" + ], + [ + "失败核验覆盖", + "读者需要区分全量核验与只检查个别版本。", + "主结果 900/900 全量盲审,包括 Moli 1.1.1 和 1.1.3 各 150 条;184 个机器失败项全部有非空独立审查证据。", + "各方法失败核验覆盖均为 100%,无未裁定结果。" + ] ] }, { @@ -363,11 +1660,36 @@ "title": "核对文档参考,纠正等价表格误判", "summary": "逐项核对 10 个文档任务的冻结源、完整正文参考、正文分段契约和 60 次实际输出;同时复查飞书帮助文档的表格判定。文档参考没有截断;飞书任务原先把 Markdown 表格语法当成内容要求,造成 3 次误判,现已按相同输入输出目标统一纠正。", "changes": [ - ["覆盖率展示", "位置详情把一小段人工定位样例放在长标准正文旁边,容易被误认为机器只用这段短文本打分。", "结果页就地说明分母、分子和完整输入:覆盖率使用上方完整实际返回计算;定位样例只帮助读者人工确认一次命中。", "评分实现、保存输出、耗时和分数均未改变。"], - ["最小 PDF 范围", "document-03 的源文件正文只有 Dummy PDF file,若沿用长文档任务说明,会被误解为参考正文缺失。", "报告明确标为最小文档解析样本:冻结源文件全部可提取正文就是 14 个字符,只检查能否取得短 PDF 文字。", "该任务不能证明多页、表格或附录能力;六种方法本轮都未取得这 14 个字符,结果仍全部失败。"], - ["文档参考审计", "嵌入文档任务只保存外围 HTML 时,读者无法追踪完整正文来自哪个 PDF。", "document-07 记录查看器实际加载的 PDF、既有冻结文件路径及双重哈希;document-08 记录冻结 HTML 中的嵌入地址、独立下载哈希和提取正文哈希。", "10 个文档参考与正文分段契约均核对通过,没有参考截断导致的分数变化。"], - ["Moli DOCX 口径", "报告把直接 DOCX 写成已经取得原始响应,但本轮 Moli 1.1.3 实际把该 DOCX 序列化为含包字节的 HTML。", "方法页改为如实描述自动选择结果:PDF 进入原始响应并解析,DOCX 没有形成可解析原文件并失败。", "不增加 curl 等外部兜底,不把其他下载器的成功归给 Moli。"], - ["等价表格输出", "飞书帮助文档原规则只接受 Markdown 管道表或 HTML table。Obscura 和两版 Moli 已完整返回三列表头及全部表格内容,但按阅读顺序展开为普通文本,因此被误判失败。", "删除与目标无关的表格语法限制;继续用 3 个稳定事实、五段正文覆盖和完整文本质量验收,同一规则作用于全部方法。", "沿用原始输出和耗时离线重评分,不重跑 provider;3 次结果由失败改为通过。"] + [ + "覆盖率展示", + "位置详情把一小段人工定位样例放在长标准正文旁边,容易被误认为机器只用这段短文本打分。", + "结果页就地说明分母、分子和完整输入:覆盖率使用上方完整实际返回计算;定位样例只帮助读者人工确认一次命中。", + "评分实现、保存输出、耗时和分数均未改变。" + ], + [ + "最小 PDF 范围", + "document-03 的源文件正文只有 Dummy PDF file,若沿用长文档任务说明,会被误解为参考正文缺失。", + "报告明确标为最小文档解析样本:冻结源文件全部可提取正文就是 14 个字符,只检查能否取得短 PDF 文字。", + "该任务不能证明多页、表格或附录能力;六种方法本轮都未取得这 14 个字符,结果仍全部失败。" + ], + [ + "文档参考审计", + "嵌入文档任务只保存外围 HTML 时,读者无法追踪完整正文来自哪个 PDF。", + "document-07 记录查看器实际加载的 PDF、既有冻结文件路径及双重哈希;document-08 记录冻结 HTML 中的嵌入地址、独立下载哈希和提取正文哈希。", + "10 个文档参考与正文分段契约均核对通过,没有参考截断导致的分数变化。" + ], + [ + "Moli DOCX 口径", + "报告把直接 DOCX 写成已经取得原始响应,但本轮 Moli 1.1.3 实际把该 DOCX 序列化为含包字节的 HTML。", + "方法页改为如实描述自动选择结果:PDF 进入原始响应并解析,DOCX 没有形成可解析原文件并失败。", + "不增加 curl 等外部兜底,不把其他下载器的成功归给 Moli。" + ], + [ + "等价表格输出", + "飞书帮助文档原规则只接受 Markdown 管道表或 HTML table。Obscura 和两版 Moli 已完整返回三列表头及全部表格内容,但按阅读顺序展开为普通文本,因此被误判失败。", + "删除与目标无关的表格语法限制;继续用 3 个稳定事实、五段正文覆盖和完整文本质量验收,同一规则作用于全部方法。", + "沿用原始输出和耗时离线重评分,不重跑 provider;3 次结果由失败改为通过。" + ] ] }, { @@ -382,14 +1704,54 @@ "title": "纠正文档与资源路线,并完成 900 条结果独立核验", "summary": "150 个任务 × 6 种方法共 900 次调用均有明确结果。Moli 1.1.3 的直接文档和媒体改用自动原始响应;Codex gpt-5.6-sol 盲审全部 900 条保存输出,分歧逐条裁定且无未决项。任务详情页直接展示标准答案、实际返回和判定证据。", "changes": [ - ["Moli 1.1.3 正确路线", "给直接 PDF 使用 --dump html/markdown 会在取得正文前报错,也无法真实衡量 1.1.3 的原始响应能力。", "普通网页仍使用原生 Markdown;直接文档与媒体使用裸 fetch,由 CLI 自动选择原始响应或序列化页面。PDF、PNG、WebM 和 MP4 在本轮取得原始字节,DOCX 被序列化为 HTML 并失败。", "只替换受实现影响的 7 个直接文档调用和 3 个直接文件调用;Moli 1.1.3 最终通过 133/150,旧版 Moli 1.1.1 为 123/150。"], - ["文档正文与文件分轨", "下载到 PDF 或 DOCX 不等于取得正文,反过来正文任务也不应强制所有方法先转成同一种中间格式。", "正文任务统一验收可读文字:方法直接给正文即可,只给原文件则把本地解析计入耗时;原始文件任务验收格式、长度和目标 SHA-256,不做无关 Markdown 转换。", "当前覆盖 PDF、DOCX、TXT 和嵌入文档;不声称支持旧 DOC、OCR、表格语义或任意办公格式。"], - ["全部结果独立核验", "只复核自动失败项会漏掉假阳性;文件任务若只交给代理空文本占位文件,也会制造假失败。", "Codex gpt-5.6-sol 在看不到机器判定和分数的条件下核对 900/900 条结果;文件任务同时提供真实二进制证据,协议缺陷影响的 3 个批次单独重审。", "所有分歧均有可观察证据和最终裁定,0 条未决;同窗测速 120/120 另行全量核验。"], - ["统一评分纠错", "Markdown 链接边界会让实际存在的 Reddit 社区说明被误判缺失;百分号编码的相对下载地址也会漏判。", "可见链接文字按排版边界还原;只把带文件后缀的尖括号相对目标按源页解析并 URL 解码,不把普通文件名 prose 当成地址。", "challenge-08 的两版 Moli 假阴性被纠正;Chromium 的 Blender 下载包地址改判通过。原输出、调用状态、耗时和内容哈希均不变。"], - ["乱码识别", "请求成功和较长输出不能证明文字可读;run-9b35581eb0fdbbdea4a6 含大量替换字符。", "统一检测 U+FFFD 的数量与占比,达到显著阈值时判正文解码损坏,并在详情页显示数量、比例和原始输出。", "该调用仍为失败,但失败原因由笼统内容缺失改为可核验的解码损坏,不把乱码当有效正文。"], - ["任务详情重做", "旧页把一份答案的五个位置写成 body-region,标准答案和实际输出又藏在大量表格与折叠项后。", "首屏显示任务 ID、方法、运行 ID 和结论;随后直接并排展示完整标准答案与实际返回。位置检查只保留汇总表,具体原文证据按需展开,运行字段统一折叠到页尾。", "只改变证据呈现,不修改任何任务、输出、分数或耗时。"], - ["版本追溯", "运行时版本字符串缺失时,仅写产品名无法证明测试的是哪个构建。", "本地实现按二进制 SHA-256 固定:Moli 1.1.1 与官方归档事后逐字节匹配,Moli 1.1.3 和 Chromium 保留运行时版本与哈希;Lightpanda 以精确 nightly 哈希标识。", "Kitesurf 是托管服务,服务端构建未公开,报告明确标为不可得,不伪造版本。"], - ["证据打包", "重新组装相同记录时改变 JSONL 行序会让独立核验哈希失效。", "组装器保留源证据顺序,并将参考修订、评分修订和结果核验记录随报告发布。", "900/900 指标可从保存输出复现;150 个任务和 6 种方法均无缺项。"] + [ + "Moli 1.1.3 正确路线", + "给直接 PDF 使用 --dump html/markdown 会在取得正文前报错,也无法真实衡量 1.1.3 的原始响应能力。", + "普通网页仍使用原生 Markdown;直接文档与媒体使用裸 fetch,由 CLI 自动选择原始响应或序列化页面。PDF、PNG、WebM 和 MP4 在本轮取得原始字节,DOCX 被序列化为 HTML 并失败。", + "只替换受实现影响的 7 个直接文档调用和 3 个直接文件调用;Moli 1.1.3 最终通过 133/150,旧版 Moli 1.1.1 为 123/150。" + ], + [ + "文档正文与文件分轨", + "下载到 PDF 或 DOCX 不等于取得正文,反过来正文任务也不应强制所有方法先转成同一种中间格式。", + "正文任务统一验收可读文字:方法直接给正文即可,只给原文件则把本地解析计入耗时;原始文件任务验收格式、长度和目标 SHA-256,不做无关 Markdown 转换。", + "当前覆盖 PDF、DOCX、TXT 和嵌入文档;不声称支持旧 DOC、OCR、表格语义或任意办公格式。" + ], + [ + "全部结果独立核验", + "只复核自动失败项会漏掉假阳性;文件任务若只交给代理空文本占位文件,也会制造假失败。", + "Codex gpt-5.6-sol 在看不到机器判定和分数的条件下核对 900/900 条结果;文件任务同时提供真实二进制证据,协议缺陷影响的 3 个批次单独重审。", + "所有分歧均有可观察证据和最终裁定,0 条未决;同窗测速 120/120 另行全量核验。" + ], + [ + "统一评分纠错", + "Markdown 链接边界会让实际存在的 Reddit 社区说明被误判缺失;百分号编码的相对下载地址也会漏判。", + "可见链接文字按排版边界还原;只把带文件后缀的尖括号相对目标按源页解析并 URL 解码,不把普通文件名 prose 当成地址。", + "challenge-08 的两版 Moli 假阴性被纠正;Chromium 的 Blender 下载包地址改判通过。原输出、调用状态、耗时和内容哈希均不变。" + ], + [ + "乱码识别", + "请求成功和较长输出不能证明文字可读;run-9b35581eb0fdbbdea4a6 含大量替换字符。", + "统一检测 U+FFFD 的数量与占比,达到显著阈值时判正文解码损坏,并在详情页显示数量、比例和原始输出。", + "该调用仍为失败,但失败原因由笼统内容缺失改为可核验的解码损坏,不把乱码当有效正文。" + ], + [ + "任务详情重做", + "旧页把一份答案的五个位置写成 body-region,标准答案和实际输出又藏在大量表格与折叠项后。", + "首屏显示任务 ID、方法、运行 ID 和结论;随后直接并排展示完整标准答案与实际返回。位置检查只保留汇总表,具体原文证据按需展开,运行字段统一折叠到页尾。", + "只改变证据呈现,不修改任何任务、输出、分数或耗时。" + ], + [ + "版本追溯", + "运行时版本字符串缺失时,仅写产品名无法证明测试的是哪个构建。", + "本地实现按二进制 SHA-256 固定:Moli 1.1.1 与官方归档事后逐字节匹配,Moli 1.1.3 和 Chromium 保留运行时版本与哈希;Lightpanda 以精确 nightly 哈希标识。", + "Kitesurf 是托管服务,服务端构建未公开,报告明确标为不可得,不伪造版本。" + ], + [ + "证据打包", + "重新组装相同记录时改变 JSONL 行序会让独立核验哈希失效。", + "组装器保留源证据顺序,并将参考修订、评分修订和结果核验记录随报告发布。", + "900/900 指标可从保存输出复现;150 个任务和 6 种方法均无缺项。" + ] ] }, { @@ -398,13 +1760,48 @@ "title": "完成五种浏览器最快 Markdown 路线的 750 次实测", "summary": "Kitesurf、Lightpanda、Obscura、Moli 使用原生 Markdown,Chromium 使用渲染 DOM 加 node-html-markdown 2.0.0;150 题 × 5 种方法全部执行、独立参考生命周期与离线复评分全部通过。", "changes": [ - ["方法口径", "旧草稿把五种方法强制写成相同 HTML 转换链路,掩盖了原生接口优势。", "对齐 URL、Markdown 输出、任务集、验收规则和 45 秒截止;每种方法使用最快的正式或自带 Markdown 路线。", "Kitesurf 走 Cloudflare 账号版 /markdown;三种本地 CLI 走原生 Markdown;只有无原生 Markdown 的 Chromium 做 DOM 转换。"], - ["Chromium 路线", "旧 pyhtml2md 路线在大页面上转换慢,且普通网页少通过一题。", "以同一批已保存 DOM 对候选转换器复评分,采用 node-html-markdown 2.0.0,并保留 HTML、版本、哈希和转换耗时。", "本轮 Chromium 普通网页 98/100;转换属于客户端端到端耗时,不冒充浏览器原生能力。"], - ["历史判定审查", "旧规则会漏判跨 Markdown 链接换行、原生接口链接排版、排版空白和部分动态表达,也曾放行过不完整正文。", "逐条对照 750 条保存输出、标准答案和任务契约,统一修正 43 个假阴性与 1 个假阳性;run-f4e6f2ee641e8baa9e3d 确认是误判。", "修正规则对所有方法生效,不增加 provider 特例;原输出、调用身份和耗时不改。"], - ["同步 main", "上游任务规则与报告缺陷修正会影响旧结论。", "合并最新 main 后复查影响范围;Django 教程参考变化,对五种方法全量同题重跑,其余仅在参考字节和评测契约等价时复用。", "Moli 该题新调用 browser exit,普通网页结果为 93/100;不沿用旧通过。"], - ["公平性", "远程 Kitesurf、本地进程、不同 readiness 条件和转换时间可能被误读为纯内核性能。", "速度只在双方共同通过的相同 URL 上配对,并明确客户端端到端边界。", "结果不能推出同机 CPU/RSS、网络剥离后的引擎速度、持续吞吐、成本或 SLA。"], - ["任务可识别性", "案例详情页只显示标题,读者无法从页面确认任务 ID。", "案例页首屏与逐次调用页均显示稳定任务 ID;运行 ID、规则、参考与输出继续可追溯。", "展示改动不改变任务、结果或分数。"], - ["发布隔离", "本报告与旧报告用途不同,覆盖旧站点会破坏历史链接。", "发布目标固定为独立站点 bWkRP157EtF6。", "旧站点 UeZE1vfcEOrM 保持不变。"] + [ + "方法口径", + "旧草稿把五种方法强制写成相同 HTML 转换链路,掩盖了原生接口优势。", + "对齐 URL、Markdown 输出、任务集、验收规则和 45 秒截止;每种方法使用最快的正式或自带 Markdown 路线。", + "Kitesurf 走 Cloudflare 账号版 /markdown;三种本地 CLI 走原生 Markdown;只有无原生 Markdown 的 Chromium 做 DOM 转换。" + ], + [ + "Chromium 路线", + "旧 pyhtml2md 路线在大页面上转换慢,且普通网页少通过一题。", + "以同一批已保存 DOM 对候选转换器复评分,采用 node-html-markdown 2.0.0,并保留 HTML、版本、哈希和转换耗时。", + "本轮 Chromium 普通网页 98/100;转换属于客户端端到端耗时,不冒充浏览器原生能力。" + ], + [ + "历史判定审查", + "旧规则会漏判跨 Markdown 链接换行、原生接口链接排版、排版空白和部分动态表达,也曾放行过不完整正文。", + "逐条对照 750 条保存输出、标准答案和任务契约,统一修正 43 个假阴性与 1 个假阳性;run-f4e6f2ee641e8baa9e3d 确认是误判。", + "修正规则对所有方法生效,不增加 provider 特例;原输出、调用身份和耗时不改。" + ], + [ + "同步 main", + "上游任务规则与报告缺陷修正会影响旧结论。", + "合并最新 main 后复查影响范围;Django 教程参考变化,对五种方法全量同题重跑,其余仅在参考字节和评测契约等价时复用。", + "Moli 该题新调用 browser exit,普通网页结果为 93/100;不沿用旧通过。" + ], + [ + "公平性", + "远程 Kitesurf、本地进程、不同 readiness 条件和转换时间可能被误读为纯内核性能。", + "速度只在双方共同通过的相同 URL 上配对,并明确客户端端到端边界。", + "结果不能推出同机 CPU/RSS、网络剥离后的引擎速度、持续吞吐、成本或 SLA。" + ], + [ + "任务可识别性", + "案例详情页只显示标题,读者无法从页面确认任务 ID。", + "案例页首屏与逐次调用页均显示稳定任务 ID;运行 ID、规则、参考与输出继续可追溯。", + "展示改动不改变任务、结果或分数。" + ], + [ + "发布隔离", + "本报告与旧报告用途不同,覆盖旧站点会破坏历史链接。", + "发布目标固定为独立站点 bWkRP157EtF6。", + "旧站点 UeZE1vfcEOrM 保持不变。" + ] ] }, { @@ -413,12 +1810,42 @@ "title": "完成文件任务评测,建立报告完整性发布门槛", "summary": "36 次文件任务全部实际调用并判定:Moli 与 Crawl4AI 各通过 3 次,共 6 次通过、30 次未通过、0 次未测。整份报告的 1,800 个计划调用均有唯一结果和明确判定。", "changes": [ - ["文件任务", "旧记录没有保存原文件字节,不能判定直接 PNG、WebM 和 MP4 是否取得。", "十二种方法各执行 3 个文件案例;最终配置对应的 36 次调用全部进行格式、长度和目标 SHA-256 验收。", "Moli 与 Crawl4AI 各通过 3 次,其余 30 次未通过;未通过保留为真实结果,不删除有效困难案例。"], - ["原生接口范围", "只测正文路径会漏掉产品公开提供的原生文件能力,并把局部配置误当成产品能力。", "Moli 文件任务使用原文件字节输出;Crawl4AI 文件任务预先选择其原生 HTTP crawler,正文仍使用浏览器路径。", "按任务开始前固定的公开接口评测,不做失败后兜底,也不把普通 HTTP 下载冒充 provider 结果。"], - ["计时边界", "“耗时”没有清楚区分服务端时间、下载、渲染、本地加工和报告处理。", "统一以客户端端到端计时:从发起 provider 调用到形成可评分最终答案;下载、渲染、提取、输入转换及文件哈希均计入。", "评分、证据落盘、限速等待和报告生成不计入;供应商服务端耗时不参与排名。"], - ["完整性门槛", "以实际调用数作分母会让缺少计划调用的 8/8 看起来完整;两个覆盖列也重复。", "改为“判分完整性 = 已明确判定数 / 计划调用数”,并检查每个 provider × case 恰有一个当前结果。", "任一缺失、重复或无法判定都会在报告全局高亮为不可靠;完整性检查、报告校验与发布打包均阻止不完整结果发布。"], - ["证据呈现", "成功文件保存在包内但不易从调用页直接核验,变更页又堆叠多次同日修订。", "成功调用页提供文件下载、格式、字节数和 SHA-256;同日修订合并为一个正式版本,较早版本折叠。", "只改善审计入口与阅读结构,不修改原始文件、分数或耗时。"], - ["首页结构", "类别概览藏在详细榜单之后,读者需要先穿过大表格。", "六类任务概览前置,直接列出样本数、完整判分的方法数、最高任务通过率及对应方法;详细榜单留在后方。", "不同类别任务要求不同,最高分不能跨类别直接比较。"] + [ + "文件任务", + "旧记录没有保存原文件字节,不能判定直接 PNG、WebM 和 MP4 是否取得。", + "十二种方法各执行 3 个文件案例;最终配置对应的 36 次调用全部进行格式、长度和目标 SHA-256 验收。", + "Moli 与 Crawl4AI 各通过 3 次,其余 30 次未通过;未通过保留为真实结果,不删除有效困难案例。" + ], + [ + "原生接口范围", + "只测正文路径会漏掉产品公开提供的原生文件能力,并把局部配置误当成产品能力。", + "Moli 文件任务使用原文件字节输出;Crawl4AI 文件任务预先选择其原生 HTTP crawler,正文仍使用浏览器路径。", + "按任务开始前固定的公开接口评测,不做失败后兜底,也不把普通 HTTP 下载冒充 provider 结果。" + ], + [ + "计时边界", + "“耗时”没有清楚区分服务端时间、下载、渲染、本地加工和报告处理。", + "统一以客户端端到端计时:从发起 provider 调用到形成可评分最终答案;下载、渲染、提取、输入转换及文件哈希均计入。", + "评分、证据落盘、限速等待和报告生成不计入;供应商服务端耗时不参与排名。" + ], + [ + "完整性门槛", + "以实际调用数作分母会让缺少计划调用的 8/8 看起来完整;两个覆盖列也重复。", + "改为“判分完整性 = 已明确判定数 / 计划调用数”,并检查每个 provider × case 恰有一个当前结果。", + "任一缺失、重复或无法判定都会在报告全局高亮为不可靠;完整性检查、报告校验与发布打包均阻止不完整结果发布。" + ], + [ + "证据呈现", + "成功文件保存在包内但不易从调用页直接核验,变更页又堆叠多次同日修订。", + "成功调用页提供文件下载、格式、字节数和 SHA-256;同日修订合并为一个正式版本,较早版本折叠。", + "只改善审计入口与阅读结构,不修改原始文件、分数或耗时。" + ], + [ + "首页结构", + "类别概览藏在详细榜单之后,读者需要先穿过大表格。", + "六类任务概览前置,直接列出样本数、完整判分的方法数、最高任务通过率及对应方法;详细榜单留在后方。", + "不同类别任务要求不同,最高分不能跨类别直接比较。" + ] ] }, { @@ -428,13 +1855,48 @@ "title": "统一验收口径,纠正参考、成绩与报告结论", "summary": "保留 150 个案例、1,800 条原始调用;92 条任务判定改变,36 条缺少文件字节证据的历史记录从误记失败改为未测。原始输出和耗时不变。", "changes": [ - ["正文参考", "CSS 标准正文目录被误删,Django 文档混入站点外围内容。", "按独立保存的源页面重新确定正文边界,并记录来源与参考哈希。", "全文 F1 随参考修正;旧低分不能直接归因于供应商正文噪声。"], - ["统一评分", "明确的名称—链接字段和合法 Markdown 软换行没有统一进入验收。", "所有方法共用同一答案转换和评分入口,排除请求回显与诊断日志。", "修正格式导致的假失败,不按 provider 名称设置评分分支。"], - ["正文覆盖", "只检查开头关键词和最小长度可能让极短摘录通过长文任务。", "有完整独立参考的任务增加最多五个连续区域的词语覆盖检查。", "它是内容覆盖下限,不替代全文 F1,也不保证语义或事实正确。"], - ["动态字段", "商品任务要求的分期和配送字段没有被完整验收。", "按独立参考检查字段归属、数量、类型和唯一性;易变价格与排序不冻结为永久答案。", "字段齐全不等于变化值真实或最新。"], - ["截止时间", "超出登记预算的结果仍可能计为成功。", "新调用采用统一端到端截止;历史记录仅按已登记耗时和预算重判。", "不声称历史调用当时已经受新硬截止控制。"], - ["文件缺测", "36 条没有字节证据的文件记录被当成失败。", "先改为未测并退出通过率分母,随后在 2026-09-04 通过真实 provider 调用补齐。", "不以 URL、文本或其他方法的下载结果代替原方法文件证据。"], - ["结果解读", "不同成功子集的条件耗时被直接当成同题速度,部分文章结论超出证据。", "增加共同通过子集诊断,并明确单次、跨批次与参考范围限制。", "不能从本轮样本外推全网能力、稳定性或生产 SLA。"] + [ + "正文参考", + "CSS 标准正文目录被误删,Django 文档混入站点外围内容。", + "按独立保存的源页面重新确定正文边界,并记录来源与参考哈希。", + "全文 F1 随参考修正;旧低分不能直接归因于供应商正文噪声。" + ], + [ + "统一评分", + "明确的名称—链接字段和合法 Markdown 软换行没有统一进入验收。", + "所有方法共用同一答案转换和评分入口,排除请求回显与诊断日志。", + "修正格式导致的假失败,不按 provider 名称设置评分分支。" + ], + [ + "正文覆盖", + "只检查开头关键词和最小长度可能让极短摘录通过长文任务。", + "有完整独立参考的任务增加最多五个连续区域的词语覆盖检查。", + "它是内容覆盖下限,不替代全文 F1,也不保证语义或事实正确。" + ], + [ + "动态字段", + "商品任务要求的分期和配送字段没有被完整验收。", + "按独立参考检查字段归属、数量、类型和唯一性;易变价格与排序不冻结为永久答案。", + "字段齐全不等于变化值真实或最新。" + ], + [ + "截止时间", + "超出登记预算的结果仍可能计为成功。", + "新调用采用统一端到端截止;历史记录仅按已登记耗时和预算重判。", + "不声称历史调用当时已经受新硬截止控制。" + ], + [ + "文件缺测", + "36 条没有字节证据的文件记录被当成失败。", + "先改为未测并退出通过率分母,随后在 2026-09-04 通过真实 provider 调用补齐。", + "不以 URL、文本或其他方法的下载结果代替原方法文件证据。" + ], + [ + "结果解读", + "不同成功子集的条件耗时被直接当成同题速度,部分文章结论超出证据。", + "增加共同通过子集诊断,并明确单次、跨批次与参考范围限制。", + "不能从本轮样本外推全网能力、稳定性或生产 SLA。" + ] ] } ] diff --git a/research/trajectory-replay-failure-review.json b/research/trajectory-replay-failure-review.json new file mode 100644 index 000000000..bab37cc50 --- /dev/null +++ b/research/trajectory-replay-failure-review.json @@ -0,0 +1,135 @@ +{ + "schema": "lex_webarena_trajectory_replay_failure_review/1", + "benchmark": "WebArena-Verified 1.2.3 · Browser Trajectory Replay", + "bindings": { + "effective_results_sha256": "abaaa0e90aced74a831093f2d6d7c3371639acb984b3a2c057f3cc39632ea935" + }, + "review_protocol": { + "unit": "one final logical result per browser and task", + "method": "compare the fixed trajectory, BrowserTool result, ordered trace, network evidence, official evaluator output, and the passing reference-browser execution", + "score_effect": "none; the official replay verdict remains unchanged", + "rule": "the programmatic failure location and reviewed root cause are separate fields" + }, + "coverage": { + "failed_results": 36, + "verified_results": 36 + }, + "cause_groups": [ + { + "code": "moli_snapshot_actionability_mismatch", + "label_zh": "快照与可操作节点不一致", + "label_en": "Snapshot/actionability mismatch", + "responsibility": "browser", + "remediation_owner": "Moli BrowserTool adapter", + "remediation_status": "open", + "results": [ + {"engine":"moli","task_id":29,"reason_zh":"Moli 快照已经列出 DIY 新帖链接,但紧接着对同一 href 的点击等待 60 秒仍找不到可操作节点;同一固定轨迹在 Chromium 通过。","reason_en":"Moli's snapshot listed the DIY new-post link, but the immediately following click on the same href could not resolve an actionable node within 60 seconds; the fixed trajectory passes in Chromium."}, + {"engine":"moli","task_id":66,"reason_zh":"Moli 快照已经列出 Books 版块链接,但随后点击同一 href 超时;同一固定轨迹在 Chromium 通过。","reason_en":"Moli's snapshot listed the Books forum link, but the following click on the same href timed out; the fixed trajectory passes in Chromium."}, + {"engine":"moli","task_id":404,"reason_zh":"Moli 快照已经列出 Books 版块链接,但随后点击同一 href 超时;同一固定轨迹在 Chromium 通过。","reason_en":"Moli's snapshot listed the Books forum link, but the following click on the same href timed out; the fixed trajectory passes in Chromium."}, + {"engine":"moli","task_id":595,"reason_zh":"Moli 快照已经列出 Space 版块链接,但随后点击同一 href 超时;同一固定轨迹在 Chromium 通过。","reason_en":"Moli's snapshot listed the Space forum link, but the following click on the same href timed out; the fixed trajectory passes in Chromium."}, + {"engine":"moli","task_id":620,"reason_zh":"Moli 快照已经列出 relationship_advice 版块链接,但随后点击同一 href 超时;同一固定轨迹在 Chromium 通过。","reason_en":"Moli's snapshot listed the relationship_advice forum link, but the following click on the same href timed out; the fixed trajectory passes in Chromium."}, + {"engine":"moli","task_id":640,"reason_zh":"Moli 快照已经列出 Books 版块链接,但随后点击同一 href 超时;同一固定轨迹在 Chromium 通过。","reason_en":"Moli's snapshot listed the Books forum link, but the following click on the same href timed out; the fixed trajectory passes in Chromium."}, + {"engine":"moli","task_id":645,"reason_zh":"Moli 快照已经列出 Books 版块链接,但随后点击同一 href 超时;同一固定轨迹在 Chromium 通过。","reason_en":"Moli's snapshot listed the Books forum link, but the following click on the same href timed out; the fixed trajectory passes in Chromium."}, + {"engine":"moli","task_id":714,"reason_zh":"Moli 快照已经列出 Gadgets 当日热门链接,但随后点击同一 href 超时;同一固定轨迹在 Chromium 通过。","reason_en":"Moli's snapshot listed the Gadgets daily-top link, but the following click on the same href timed out; the fixed trajectory passes in Chromium."} + ] + }, + { + "code": "moli_visible_target_state_divergence", + "label_zh": "目标控件未进入可见可操作状态", + "label_en": "Target control did not become actionable", + "responsibility": "browser", + "remediation_owner": "Moli rendering and BrowserTool integration", + "remediation_status": "open", + "results": [ + {"engine":"moli","task_id":108,"reason_zh":"固定轨迹到达筛选步骤后,Moli 未暴露可见的 Filters 按钮,点击超时;Chromium 同轨迹通过。","reason_en":"At the filter step, Moli did not expose an actionable visible Filters button and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":156,"reason_zh":"Moli 未暴露固定轨迹所需的 Assigned merge requests 链接实例,点击超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the Assigned merge-requests link instance required by the fixed trajectory; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":163,"reason_zh":"Moli 未暴露评论列表第二页链接,点击超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the review-list page-two link and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":243,"reason_zh":"Moli 在菜单导航后未暴露 Customers 链接,点击超时;Chromium 同轨迹通过。","reason_en":"After menu navigation, Moli did not expose the Customers link and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":374,"reason_zh":"Moli 未暴露 Magento Blank 主题的编辑链接,点击超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the Magento Blank theme edit link and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":399,"reason_zh":"Moli 未暴露目标论坛用户的个人页链接,点击超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the target forum user's profile link and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":415,"reason_zh":"Moli 未暴露可见的评论文本框,填入动作超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose an actionable visible comment textarea and the fill timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":418,"reason_zh":"Moli 未暴露 Set status 按钮,点击超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the Set status button and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":441,"reason_zh":"Moli 未暴露目标仓库的 index.html 文件链接,点击超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the target repository's index.html link and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":448,"reason_zh":"Moli 未暴露 GitLab 个人资料入口,点击超时;重试仍在同一步失败,而 Chromium 同轨迹通过。","reason_en":"Moli did not expose the GitLab profile entry; the retry failed at the same step while Chromium passes the same trajectory."}, + {"engine":"moli","task_id":506,"reason_zh":"Moli 在结账流程中未暴露下一步按钮,点击超时;Chromium 同轨迹完成并通过验收。","reason_en":"Moli did not expose the Next button during checkout and the click timed out; Chromium completes and passes the same trajectory."}, + {"engine":"moli","task_id":571,"reason_zh":"Moli 未暴露可见的州/省下拉框,选择动作超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose an actionable visible region selector and the selection timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":585,"reason_zh":"Moli 未暴露 Submit Review 按钮,点击超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the Submit Review button and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":590,"reason_zh":"Moli 未暴露里程碑标题输入框,填入动作超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the milestone-title input and the fill timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":658,"reason_zh":"Moli 未暴露议题标题输入框,填入动作超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the issue-title input and the fill timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":731,"reason_zh":"Moli 未暴露目标论坛帖的编辑链接,点击超时;Chromium 同轨迹通过。","reason_en":"Moli did not expose the target forum post's edit link and the click timed out; Chromium passes the same trajectory."} + ] + }, + { + "code": "moli_navigation_readiness_race", + "label_zh": "导航切换与后续动作竞态", + "label_en": "Navigation/readiness race", + "responsibility": "browser", + "remediation_owner": "Moli navigation lifecycle", + "remediation_status": "open", + "results": [ + {"engine":"moli","task_id":173,"reason_zh":"前一步快照已报告导航导致执行上下文销毁,Moli 随后仍未恢复到可点击 Search 的稳定文档;Chromium 同轨迹通过。","reason_en":"The preceding snapshot reported execution-context destruction during navigation, and Moli did not recover to a stable document with an actionable Search link; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":196,"reason_zh":"前一步发生导航切换后,Moli 未在下一动作前恢复 Clear all 控件,点击超时;Chromium 同轨迹通过。","reason_en":"After the preceding navigation transition, Moli did not restore the Clear all control before the next action and the click timed out; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":204,"reason_zh":"前一步导航尚未形成稳定文档,Moli 未暴露目标订单链接,点击超时;Chromium 同轨迹通过。","reason_en":"The preceding navigation did not settle into a stable document in Moli, so the target order link never became actionable; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":313,"reason_zh":"Moli 在按键动作执行时仍处于文档导航切换,CDP 返回 Navigation is changing the document;Chromium 同轨迹通过。","reason_en":"Moli was still changing documents when the keypress was dispatched, and CDP returned 'Navigation is changing the document'; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":389,"reason_zh":"前一步快照遭遇导航上下文销毁,Moli 未恢复目标 merge request 链接,点击超时;Chromium 同轨迹通过。","reason_en":"The preceding snapshot lost its execution context to navigation, and Moli did not recover the target merge-request link; Chromium passes the same trajectory."}, + {"engine":"moli","task_id":666,"reason_zh":"前一步快照遭遇导航上下文销毁,Moli 未恢复新建 merge request 链接,点击超时;Chromium 同轨迹通过。","reason_en":"The preceding snapshot lost its execution context to navigation, and Moli did not recover the new merge-request link; Chromium passes the same trajectory."} + ] + }, + { + "code": "moli_dom_cardinality_difference", + "label_zh": "可见节点数量与参考浏览器不同", + "label_en": "Visible-node cardinality differs", + "responsibility": "browser", + "remediation_owner": "Moli DOM projection", + "remediation_status": "open", + "results": [ + {"engine":"moli","task_id":314,"reason_zh":"固定轨迹要求第二个可见的 /primer/design 链接;Moli 只暴露不足两个匹配节点,Chromium 同轨迹通过。","reason_en":"The trajectory selects the second visible /primer/design link; Moli exposes fewer than two matching nodes while Chromium passes the same trajectory."}, + {"engine":"moli","task_id":349,"reason_zh":"固定轨迹要求第二个可见的目标项目链接;Moli 只暴露不足两个匹配节点,Chromium 同轨迹通过。","reason_en":"The trajectory selects the second visible target-project link; Moli exposes fewer than two matching nodes while Chromium passes the same trajectory."} + ] + }, + { + "code": "moli_dynamic_suggestion_missing", + "label_zh": "动态候选未生成", + "label_en": "Dynamic suggestion missing", + "responsibility": "browser", + "remediation_owner": "Moli page event integration", + "remediation_status": "open", + "results": [ + {"engine":"moli","task_id":576,"reason_zh":"输入用户名前缀后,Moli 页面没有生成 abisubramanya27 候选,固定选择脚本因候选不存在而失败;Chromium 同轨迹通过。","reason_en":"After entering the username prefix, Moli did not generate the abisubramanya27 suggestion, so the fixed selection script failed; Chromium passes the same trajectory."} + ] + }, + { + "code": "moli_async_side_effect_lost", + "label_zh": "异步保存副作用未发生", + "label_en": "Asynchronous save side effect was lost", + "responsibility": "browser", + "remediation_owner": "Moli script and lifecycle integration", + "remediation_status": "open", + "results": [ + {"engine":"moli","task_id":464,"reason_zh":"七个 BrowserTool 动作均返回成功,但由 setTimeout 调度的 Magento 保存没有出现在 Moli 网络记录中,官方验收因此找不到目标 POST;Chromium 同轨迹产生该请求并通过。","reason_en":"All seven BrowserTool actions returned success, but the Magento save scheduled through setTimeout never appeared in Moli's network record, so the evaluator found no target POST; Chromium emits it and passes."} + ] + }, + { + "code": "moli_ambiguous_link_activation", + "label_zh": "同一链接选择器激活了不同目标", + "label_en": "Same link selector activated a different target", + "responsibility": "browser", + "remediation_owner": "Moli DOM ordering and click dispatch", + "remediation_status": "open", + "results": [ + {"engine":"moli","task_id":465,"reason_zh":"固定轨迹点击第三个可见 href=# 链接;Chromium 进入心愿单并产生验收所需请求,Moli 留在搜索页且没有心愿单 POST,说明可见节点顺序或点击分派不同。","reason_en":"The trajectory clicks the third visible href=# link. Chromium opens the wishlist and emits the required request, while Moli remains on the search page with no wishlist POST, showing different visible-node ordering or click dispatch."} + ] + }, + { + "code": "moli_json_document_dom_shape", + "label_zh": "JSON 文档 DOM 结构不同", + "label_en": "JSON-document DOM shape differs", + "responsibility": "browser", + "remediation_owner": "Moli document rendering", + "remediation_status": "open", + "results": [ + {"engine":"moli","task_id":784,"reason_zh":"Moli 已以 200 打开 API 且快照含完整 JSON 文本,但页面没有 Chromium 提供的 pre 包装节点,后续 get_text(pre) 超时。","reason_en":"Moli opened the API with status 200 and its snapshot contains the JSON text, but the document lacks Chromium's pre wrapper, so get_text(pre) timed out."} + ] + } + ] +} diff --git a/research/webarena-mac-expansion.json b/research/webarena-mac-expansion.json new file mode 100644 index 000000000..08ed7c3b7 --- /dev/null +++ b/research/webarena-mac-expansion.json @@ -0,0 +1,23 @@ +{ + "schema": "lex_webarena_mac_expansion/1", + "verified_on": "2026-09-10", + "official_counts": { + "full": 812, + "hard": 258, + "current_four_sites_single_site": 655, + "current_four_sites_including_cross_site": 678, + "current_four_sites_hard_including_cross_site": 233 + }, + "environments": [ + ["Shopping", "已在本机实跑", "187", "56", "amd64 镜像;已通过 Colima 转译运行"], + ["Shopping Admin", "已在本机实跑", "182", "55", "amd64 镜像;已通过 Colima 转译运行"], + ["Reddit", "已在本机实跑", "106", "42", "amd64 镜像;已通过 Colima 转译运行"], + ["GitLab", "已在本机实跑", "180", "57", "amd64 镜像;已通过 Colima 转译运行"], + ["Wikipedia", "可部署,尚未实跑", "0", "0", "arm64 镜像;需约 100 GB 数据;只出现在跨站任务"], + ["Map", "可尝试,尚未验证", "109", "0", "amd64 镜像;需约 60 GB 数据与 Apple Silicon 转译"] + ], + "sources": [ + "https://github.com/ServiceNow/webarena-verified", + "https://servicenow.github.io/webarena-verified/dev/environments/" + ] +} diff --git a/scripts/authorize_artifact_hub.py b/scripts/authorize_artifact_hub.py index 0067bea6f..87941fb99 100644 --- a/scripts/authorize_artifact_hub.py +++ b/scripts/authorize_artifact_hub.py @@ -1,4 +1,4 @@ -"""Authorize Artifact Hub and save its token in the user-scoped credential file.""" +"""Authorize Artifact Site and save its token in the user-scoped credential file.""" import argparse import json import os @@ -18,10 +18,15 @@ def main(): action.add_argument('--persist-existing', action='store_true') parser.add_argument('--timeout', type=int, default=540) parser.add_argument('--replace-existing', action='store_true', help='Explicitly back up and replace a different user token') - parser.add_argument('--base-url',default=os.environ.get('ARTIFACT_HUB_BASE_URL','https://artifact-hub.app.lexmount.com')) + parser.add_argument('--base-url', default=os.environ.get( + 'ARTIFACT_SITE_BASE_URL', + os.environ.get('ARTIFACT_HUB_BASE_URL', 'https://artifact-hub.app.lexmount.com'), + )) args = parser.parse_args() if args.persist_existing: - backup=save_token(os.environ.get('ARTIFACT_HUB_TOKEN', '').strip(),base_url=args.base_url,replace_existing=args.replace_existing) + token = (os.environ.get('ARTIFACT_SITE_TOKEN', '').strip() + or os.environ.get('ARTIFACT_HUB_TOKEN', '').strip()) + backup=save_token(token,base_url=args.base_url,replace_existing=args.replace_existing) if backup:print('Previous credential backed up:',backup) print('Existing authorization saved to the user credential file; token not displayed.') return diff --git a/scripts/build_browser_use_replay_seeds.py b/scripts/build_browser_use_replay_seeds.py new file mode 100644 index 000000000..8a71b39bf --- /dev/null +++ b/scripts/build_browser_use_replay_seeds.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +"""Build private fixed-trajectory candidates from measured Browser Use evidence.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from harness.trajectory_replay.browser_use_seeds import build_replay_seed_manifest + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--profile", default="evaluation/browser-use/current-report.json") + parser.add_argument("--evidence", action="append", required=True) + parser.add_argument("--output", required=True) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + profile = json.loads(Path(args.profile).read_text(encoding="utf-8")) + manifest = build_replay_seed_manifest(args.evidence, profile) + output = Path(args.output).expanduser().resolve() + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8") + print(json.dumps({"output": str(output), **manifest["counts"]}, ensure_ascii=False)) + + +if __name__ == "__main__": + main() diff --git a/scripts/build_browser_use_report.py b/scripts/build_browser_use_report.py new file mode 100644 index 000000000..24f8da1c0 --- /dev/null +++ b/scripts/build_browser_use_report.py @@ -0,0 +1,6 @@ +"""Build the autonomous browser-use report.""" + +from harness.browser_use.report import main + +if __name__ == "__main__": + main() diff --git a/scripts/build_webarena_expert_seeds.py b/scripts/build_webarena_expert_seeds.py new file mode 100644 index 000000000..ce30128e6 --- /dev/null +++ b/scripts/build_webarena_expert_seeds.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Merge reviewed expert repair candidates into a private replay seed manifest.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from harness.trajectory_replay.expert_seeds import build_expert_seed_manifest + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument( + "--base", + "--base-seeds", + dest="base", + required=True, + help="Existing Browser Use replay seed manifest", + ) + parser.add_argument( + "--repairs", + "--repair-spec", + dest="repairs", + required=True, + help="Explicit expert repair specification", + ) + parser.add_argument("--output", required=True, help="Destination merged seed manifest") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + manifest = build_expert_seed_manifest(args.base, args.repairs) + output = Path(args.output).expanduser().resolve() + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8") + print( + json.dumps( + { + "output": str(output), + "expert_repair_candidates": manifest["counts"]["expert_repair_candidates"], + "replay_ready_tasks": manifest["counts"]["replay_ready_tasks"], + }, + ensure_ascii=False, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/build_webarena_replay_report.py b/scripts/build_webarena_replay_report.py new file mode 100644 index 000000000..acb70516e --- /dev/null +++ b/scripts/build_webarena_replay_report.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +"""Build the current WebArena trajectory replay report.""" + +from __future__ import annotations + +import argparse +import json + +from harness.trajectory_replay.webarena_report import build_webarena_replay_report + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--results", required=True) + parser.add_argument("--output", default="reports/trajectory-replay") + args = parser.parse_args() + manifest = build_webarena_replay_report( + results_path=args.results, + dataset_path="evaluation/trajectory-replay/webarena-verified/trajectories.json", + review_path="research/trajectory-replay-failure-review.json", + profile_path="evaluation/trajectory-replay/webarena-verified/current-run.json", + output_dir=args.output, + ) + print(json.dumps(manifest, ensure_ascii=False)) + + +if __name__ == "__main__": + main() diff --git a/scripts/check_browser_use_report.py b/scripts/check_browser_use_report.py new file mode 100644 index 000000000..6b0fa79c6 --- /dev/null +++ b/scripts/check_browser_use_report.py @@ -0,0 +1,900 @@ +#!/usr/bin/env python3 +"""Validate the autonomous browser-use report and its measured evidence.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +from html.parser import HTMLParser +from pathlib import Path + +from PIL import Image + + +class _VisibleTextParser(HTMLParser): + def __init__(self) -> None: + super().__init__() + self.hidden_depth = 0 + self.parts: list[str] = [] + + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + if tag.lower() in {"script", "style", "pre", "code"}: + self.hidden_depth += 1 + + def handle_endtag(self, tag: str) -> None: + if tag.lower() in {"script", "style", "pre", "code"} and self.hidden_depth: + self.hidden_depth -= 1 + + def handle_data(self, data: str) -> None: + if not self.hidden_depth: + self.parts.append(data) + + +def _visible_text(html: str) -> str: + parser = _VisibleTextParser() + parser.feed(html) + return " ".join(parser.parts) + +from harness.browser_use.failure_cause_review import ( + DEFAULT_REVIEW_PATH as FAILURE_CAUSE_REVIEW_PATH, +) +from harness.browser_use.failure_cause_review import ( + load_failure_cause_review, + validate_failure_cause_review, +) +from harness.browser_use.failure_cause_review import ( + sha256 as file_sha256, +) +from harness.browser_use.publication import evidence_metadata +from harness.browser_use.report import ( + PUBLIC_EXECUTION_ARTIFACTS, + _attempt_id, + _configure_engine_labels, + _configure_models, + _load_evidence_bundles, + _logical_results, + _public_attempt, + _public_execution, +) +from harness.browser_use.selection import load_pilot + + +FORBIDDEN_READER_TERMS = ( + "模型内完整配对", + "不等于共同根因", + "仅基准通过", + "仅对比通过", +) + + +def _check_reader_facing_ux( + report_dir: Path, rendered_pages: dict[Path, str] +) -> None: + """Enforce stable, data-independent report presentation contracts.""" + + for page, text in rendered_pages.items(): + term = next((term for term in FORBIDDEN_READER_TERMS if term in text), None) + if term: + raise ValueError(f"formal report contains unexplained internal term {term}: {page}") + for homepage in (report_dir / "index.html", report_dir / "en/index.html"): + text = rendered_pages[homepage] + if not re.search(r']*\bid=["\']home-core-metrics["\']', text, flags=re.I): + raise ValueError(f"report home page is missing the core metrics table: {homepage}") + + +def _sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _check_execution_artifacts( + report_dir: Path, + logical_rows: list[dict], + public_results: list[dict], + evidence_index: dict, +) -> None: + """Verify that every physical execution has one safe, hash-bound public projection.""" + + logical_by_id = {_attempt_id(row): row for row in logical_rows} + public_by_id = {str(row.get("id")): row for row in public_results} + indexed_rows = evidence_index.get("results") or [] + indexed_by_id = {str(row.get("result_id")): row for row in indexed_rows} + expected_ids = set(logical_by_id) + if len(public_by_id) != len(public_results) or set(public_by_id) != expected_ids: + raise ValueError("public result projection must identify every logical result exactly once") + if len(indexed_by_id) != len(indexed_rows) or set(indexed_by_id) != expected_ids: + raise ValueError("public evidence index must cover every logical result exactly once") + if evidence_index.get("schema") != "lex_browser_use_public_evidence_index/4": + raise ValueError("public evidence index schema does not describe physical executions") + + expected_files: set[Path] = set() + for result_id, logical in logical_by_id.items(): + sources = logical.get("_source_executions") or [logical] + authority = int(logical["attempt"]) + public = public_by_id[result_id] + indexed = indexed_by_id[result_id] + detail_text = (report_dir / "results" / f"{result_id}.html").read_text(encoding="utf-8") + if f"包含 {len(sources)} 次物理执行" not in detail_text: + raise ValueError(f"result page omits physical execution count: {result_id}") + expected_public = _public_attempt(logical) + for key in ( + "execution_count", + "authoritative_execution", + "score", + "duration_ms", + "action_count", + "browser_tool_duration_ms", + "browser_tool_duration_by_action_ms", + "browser_tool_count_by_action", + "model_usage", + ): + if public.get(key) != expected_public.get(key): + raise ValueError( + f"public logical aggregate cannot be recomputed: {result_id}/{key}" + ) + for key in ( + "model", + "engine", + "task_id", + "score", + "execution_count", + "authoritative_execution", + ): + if indexed.get(key) != public.get(key): + raise ValueError(f"evidence index logical binding differs: {result_id}/{key}") + if public.get("execution_count") != len(sources): + raise ValueError(f"public result has wrong physical execution count: {result_id}") + if public.get("authoritative_execution") != authority: + raise ValueError(f"public result changed score authority: {result_id}") + if indexed.get("execution_count") != len(sources): + raise ValueError(f"evidence index has wrong physical execution count: {result_id}") + if indexed.get("authoritative_execution") != authority: + raise ValueError(f"evidence index changed score authority: {result_id}") + executions = indexed.get("executions") or [] + if len(executions) != len(sources): + raise ValueError(f"evidence index omitted a physical execution: {result_id}") + indexed_by_number = {int(item.get("execution") or 0): item for item in executions} + if len(indexed_by_number) != len(executions): + raise ValueError(f"evidence index duplicates a physical execution: {result_id}") + for source in sources: + number = int(source["attempt"]) + indexed_execution = indexed_by_number.get(number) + if indexed_execution is None: + raise ValueError(f"evidence index omitted execution {number}: {result_id}") + expected_measurements = _public_execution(source, authoritative=number == authority) + measured = {key: indexed_execution.get(key) for key in expected_measurements} + if measured != expected_measurements: + raise ValueError(f"published execution measurements differ: {result_id}/{number}") + artifacts = indexed_execution.get("artifacts") or {} + if set(artifacts) != set(PUBLIC_EXECUTION_ARTIFACTS): + raise ValueError(f"execution artifact inventory differs: {result_id}/{number}") + for label, filename in PUBLIC_EXECUTION_ARTIFACTS.items(): + relative = Path("artifacts") / result_id / "executions" / str(number) / filename + artifact = artifacts[label] + if ( + set(artifact) != {"path", "sha256"} + or artifact.get("path") != relative.as_posix() + ): + raise ValueError( + f"unsafe or incorrect artifact link: {result_id}/{number}/{label}" + ) + path = report_dir / relative + expected_files.add(path) + if not path.is_file(): + raise ValueError(f"missing physical execution artifact: {path}") + if artifact.get("sha256") != _sha256(path): + raise ValueError(f"physical execution artifact hash differs: {path}") + if ( + f'href="{relative.as_posix()}"' not in detail_text + or str(artifact["sha256"]) not in detail_text + ): + raise ValueError( + f"result page omits execution artifact link or hash: " + f"{result_id}/{number}/{label}" + ) + screenshots = indexed_execution.get("screenshots") or [] + if number != authority and screenshots: + raise ValueError(f"only the authoritative execution may publish frames: {result_id}") + for screenshot in screenshots: + if set(screenshot) != { + "path", + "sha256", + "label", + "source", + "width", + "height", + } or screenshot.get("source") != "playwright-screencast-frame": + raise ValueError(f"trace frame lacks screencast provenance: {result_id}") + relative = ( + Path("artifacts") + / result_id + / "executions" + / str(number) + / str(screenshot.get("path") or "") + ) + path = report_dir / relative + if path.suffix.lower() not in {".jpeg", ".jpg", ".png"}: + raise ValueError(f"unsafe screenshot type: {relative}") + expected_files.add(path) + if not path.is_file() or screenshot.get("sha256") != _sha256(path): + raise ValueError(f"trace frame binding differs: {relative}") + with Image.open(path) as image: + dimensions = image.size + if dimensions != (screenshot.get("width"), screenshot.get("height")): + raise ValueError(f"trace frame dimensions differ: {relative}") + if dimensions[0] < 800 or dimensions[1] < 450: + raise ValueError(f"trace frame is not a full page image: {relative}") + if relative.as_posix() not in detail_text: + raise ValueError(f"result page omits trace frame: {relative}") + bundle_path = ( + report_dir + / "artifacts" + / result_id + / "executions" + / str(number) + / PUBLIC_EXECUTION_ARTIFACTS["execution_evidence"] + ) + bundle = json.loads(bundle_path.read_text(encoding="utf-8")) + if ( + bundle.get("schema") != "lex_browser_use_public_execution_evidence/3" + or bundle.get("result_id") != result_id + or bundle.get("execution") != number + or bundle.get("authoritative") != (number == authority) + or bundle.get("measurements") != expected_measurements + ): + raise ValueError(f"execution evidence binding differs: {result_id}/{number}") + for key in ("agent_response", "evaluation", "agent_events", "trajectory"): + if key not in bundle: + raise ValueError( + f"execution evidence payload is incomplete: {result_id}/{number}/{key}" + ) + source_files = (bundle.get("source_evidence") or {}).get("files") or {} + if source_files != (evidence_metadata(source).get("files") or {}): + raise ValueError(f"source evidence hashes differ: {result_id}/{number}") + missing_public_sources = { + label + for label in ("response", "evaluation", "events") + if not (source_files.get(label) or {}).get("present") + } + if missing_public_sources: + raise ValueError( + f"physical execution lacks publishable source evidence: " + f"{result_id}/{number}/{sorted(missing_public_sources)}" + ) + for private_label in ("network_har", "agent_trace"): + if ( + private_label in source_files + and source_files[private_label].get("publication") + != "private_only_session_sensitive" + ): + raise ValueError(f"private {private_label} was marked publishable") + + artifact_root = report_dir / "artifacts" + actual_files = {path for path in artifact_root.rglob("*") if path.is_file()} + if actual_files != expected_files: + extra = sorted(str(path) for path in actual_files - expected_files) + missing = sorted(str(path) for path in expected_files - actual_files) + raise ValueError(f"execution artifact tree differs; extra={extra}, missing={missing}") + forbidden_text = re.compile( + r"(?:Password\.123|admin1234|test1234|hello1234|" + r"-----BEGIN [A-Z ]*PRIVATE KEY-----|\bBearer\s+[A-Za-z0-9._~+/=-]{12,}|" + r"\bsk-[A-Za-z0-9_-]{12,}|(?-i:/(?:Users|home|private|tmp|var|opt)/[^\s\"']+))", + flags=re.I, + ) + for path in actual_files: + if path.suffix.lower() in {".har", ".zip"} or path.name in { + "network.har", + "agent_trace.zip", + }: + raise ValueError(f"private HAR or trace was copied into the public report: {path}") + if path.suffix.lower() in {".json", ".jsonl", ".html", ".css", ".js"} and forbidden_text.search(path.read_text(encoding="utf-8")): + raise ValueError(f"published execution artifact contains sensitive text: {path}") + + +def _check_public_privacy(report_dir: Path) -> None: + forbidden_text = re.compile( + r"(?:Password\.123|admin1234|test1234|hello1234|" + r"-----BEGIN [A-Z ]*PRIVATE KEY-----|\bBearer\s+[A-Za-z0-9._~+/=-]{12,}|" + r"\bsk-[A-Za-z0-9_-]{12,}|(?-i:/(?:Users|home|private|tmp|var|opt)/[^\s\"']+))", + flags=re.I, + ) + text_suffixes = {".html", ".json", ".jsonl", ".js", ".css"} + for path in report_dir.rglob("*"): + if not path.is_file(): + continue + if path.suffix.lower() == ".har" or ( + "trace" in path.name.lower() and path.suffix.lower() == ".zip" + ): + raise ValueError(f"private HAR or trace was copied into the public report: {path}") + if path.suffix.lower() in text_suffixes and forbidden_text.search( + path.read_text(encoding="utf-8") + ): + raise ValueError(f"published report contains sensitive text: {path}") + + +def check( + report_dir: Path, + evidence_dir: Path | list[Path], + selection_path: Path, + *, + require_complete: bool, +) -> None: + profile_path = Path("evaluation/browser-use/current-report.json") + profile = json.loads(profile_path.read_text(encoding="utf-8")) + _configure_engine_labels(profile) + _configure_models(profile) + engines = tuple(str(engine) for engine in profile.get("engines") or []) + models = tuple(str(model) for model in profile.get("models") or []) + if not engines or len(engines) != len(set(engines)): + raise ValueError("current Browser Use profile must register unique browser engines") + page_names = ( + "index.html", + "changes/index.html", + "datasets/index.html", + "methodology/index.html", + "analysis/index.html", + "review/index.html", + "adoption/index.html", + "reproduce/index.html", + "accounts/index.html", + "evidence-viewer.html", + *(f"engines/{engine}.html" for engine in engines), + ) + pages = tuple(report_dir / name for name in page_names) + index = pages[0] + audit_path = report_dir / "report-data.json" + bundles = _load_evidence_bundles(evidence_dir) + task_pages = tuple( + report_dir / "datasets" / f"{task_id}.html" + for task_id in sorted(task.task_id for task in load_pilot(selection_path).tasks) + ) + measured_rows = [row for bundle in bundles for row in bundle["source_attempts"]] + logical_rows = [ + row + for bundle in bundles + for row in _logical_results( + bundle["source_attempts"], bundle["manifest"].get("protocol") or {} + ) + ] + result_pages = tuple( + report_dir / "results" / f"{_attempt_id(row)}.html" for row in logical_rows + ) + chinese_pages = (*pages, *task_pages, *result_pages) + english_pages = tuple( + report_dir / "en" / page.relative_to(report_dir) for page in chinese_pages + ) + data_files = tuple( + report_dir / "data" / name + for name in ( + "selection.json", + "results.jsonl", + "summary.json", + "environment.json", + "review.json", + "publication-privacy.json", + "evidence-index.json", + "failure-cause-review.json", + ) + ) + artifact_files = tuple( + report_dir + / "artifacts" + / _attempt_id(row) + / "executions" + / str(source["attempt"]) + / filename + for row in logical_rows + for source in (row.get("_source_executions") or [row]) + for filename in PUBLIC_EXECUTION_ARTIFACTS.values() + ) + if not all( + path.is_file() + for path in ( + *chinese_pages, + *english_pages, + *data_files, + *artifact_files, + audit_path, + ) + ): + raise ValueError("report is missing one or more required report pages") + selection = load_pilot(selection_path) + audit = json.loads(audit_path.read_text(encoding="utf-8")) + if audit.get("selection_sha256") != selection.sha256: + raise ValueError("report was not built from the current pre-registered selection") + rows = measured_rows + selected_ids = {task.task_id for task in selection.tasks} + expected_primary = { + (model, engine, task_id, 1) + for model in models + for engine in engines + for task_id in selected_ids + } + by_key: dict[tuple[str, str, int, int], dict] = {} + for row in rows: + key = ( + str(row["model"]), + str(row["engine"]), + int(row["task_id"]), + int(row["attempt"]), + ) + if key in by_key: + raise ValueError(f"duplicate measured attempt: {key}") + if key[0] not in set(models) or key[1] not in set(engines) or key[2] not in selected_ids: + raise ValueError(f"attempt outside the registered cohort: {key}") + if key[3] not in {1, 2}: + raise ValueError(f"more than one retry is not allowed: {key}") + by_key[key] = row + for model, engine, task_id, attempt in by_key: + if attempt == 2: + primary = by_key.get((model, engine, task_id, 1)) + if primary is None or float(primary["score"]) == 1.0: + raise ValueError( + f"retry is allowed only after a failed primary: {(model, engine, task_id)}" + ) + rendered_pages = { + page: page.read_text(encoding="utf-8") for page in (*chinese_pages, *english_pages) + } + _check_reader_facing_ux(report_dir, rendered_pages) + if any('' not in rendered_pages[page] for page in english_pages): + raise ValueError("every Chinese report route must have an English mirror") + cjk = re.compile(r"[\u3400-\u9fff]") + localized_attribute = re.compile(r'(?:aria-label|placeholder|title)="([^"]*)"', flags=re.I) + untranslated_interface = ( + "该逻辑结果包含", + "物理执行", + "判分权威", + "单次调用墙钟时间", + "查看该浏览器的全部结果", + "完整结果只保留一份", + "该执行没有可公开的 trace 页面画面", + "个物理执行均记录", + "个物理执行均保存", + "本轮剔除", + ) + for page in english_pages: + text = rendered_pages[page] + title = re.search(r"(.*?)", text, flags=re.I | re.S) + if title and cjk.search(title.group(1)): + raise ValueError(f"English report title is not localized: {page}") + if any(cjk.search(value) for value in localized_attribute.findall(text)): + raise ValueError(f"English report accessibility text is not localized: {page}") + if any(marker in text for marker in untranslated_interface): + raise ValueError(f"English report interface prose is not localized: {page}") + excluded_lists = re.findall(r"tasks excluded in this run: ([^<]+)", text) + if any("、" in value for value in excluded_lists): + raise ValueError(f"English excluded-task list uses Chinese punctuation: {page}") + for relative, allowed in { + "index.html": {"中文"}, + }.items(): + page = report_dir / "en" / relative + visible = re.sub(r"\s+", " ", _visible_text(rendered_pages[page])) + for token in allowed: + visible = visible.replace(token, "") + match = cjk.search(visible) + if match: + excerpt = visible[max(0, match.start() - 40) : match.start() + 80] + raise ValueError( + f"English reader-facing page contains Chinese prose: {page}: {excerpt}" + ) + for engine in engines: + engine_page = rendered_pages[report_dir / "engines" / f"{engine}.html"] + if "engine-result-table" in engine_page or f"f_engine={engine}" not in engine_page: + raise ValueError("browser pages must link to the single filtered result inventory") + if task_pages and any( + "Ground truth · 官方验收合同" not in rendered_pages[page] + or "查看本题运行结果" not in rendered_pages[page] + or "最终结果对照" in rendered_pages[page] + for page in task_pages + ): + raise ValueError( + "task pages must publish frozen contracts without embedding measured comparisons" + ) + dataset_index = rendered_pages[report_dir / "datasets/index.html"] + if "data-result=" in dataset_index or "可完成性证据" in dataset_index: + raise ValueError("dataset inventory must not mix measured results into task definitions") + review_index = rendered_pages[report_dir / "review/index.html"] + if ( + "核验与修订记录" not in review_index + or "核验轮次" not in review_index + or "review-result-table" in review_index + or "逐结果覆盖" in review_index + ): + raise ValueError("review page must present review rounds, not a per-result audit table") + methodology_index = rendered_pages[report_dir / "methodology/index.html"] + for marker in ("192.168.5.2", "Wikipedia", "Map", "环境准备证据"): + if marker not in methodology_index: + raise ValueError(f"methodology is missing deployment evidence boundary: {marker}") + analysis_index = rendered_pages[report_dir / "analysis/index.html"] + for marker in ( + "comparison-winner", + "完整 90 题结果入口", + "BrowserTool 动作延迟", + "action-card-grid", + "全部配置 · 单次调用墙钟时间", + "同题全配置对比", + 'id="configuration-matrix"', + "查看诊断", + 'class="diagnosis-dialog"', + "程序报错", + ): + if marker not in analysis_index: + raise ValueError(f"analysis is missing timing or scalable comparison: {marker}") + + guide_index = rendered_pages[report_dir / "index.html"] + for marker in ( + "90 题核心指标", + "Agent 总耗时", + "BrowserTool 耗时", + "均值", + "P50", + "P95", + "comparison-winner", + "四种完整配置中统一比较", + "已通过结果没有再做独立语义盲审", + "程序判分落点与复核根因", + 'class="verdict-strip"', + 'class="cause-bars"', + 'id="home-qualified-results"', + 'class="audit-ladder"', + 'id="home-resource-table"', + 'results/gpt-6-astra-chromium-699.html#verification', + ): + if marker not in guide_index: + raise ValueError(f"interpretation is missing an evidence-backed reading aid: {marker}") + if "首次成功与重试恢复" in guide_index: + raise ValueError("homepage must present one final logical result, not retry narration") + for forbidden in ( + "中位数", + "失败结果定位", + "是否属于目标变量", + 'id="failure-table"', + 'id="pair-table"', + 'class="finding-list"', + ): + if forbidden in analysis_index: + raise ValueError(f"analysis contains a redundant or retired presentation: {forbidden}") + participant_index = rendered_pages[report_dir / "adoption/index.html"] + if ( + "被测浏览器与固定实验条件" not in participant_index + or "参评选手" in participant_index + ): + raise ValueError("participant page must use neutral reader-facing terminology") + if result_pages and any( + marker not in rendered_pages[page] + for page in result_pages + for marker in ( + "任务要求与验收结果", + "查看官方期望与本次观察", + "CowAgent 自主执行轨迹", + "运行、日志与证据", + "runtime-strip", + "任务与验收", + "在页面中查看完整执行证据", + "下载原始 JSON", + ) + ): + raise ValueError("every result page must expose result, evaluator, trajectory and logs") + evidence_viewer = report_dir / "evidence-viewer.html" + english_evidence_viewer = report_dir / "en/evidence-viewer.html" + if any(path not in rendered_pages for path in (evidence_viewer, english_evidence_viewer)): + raise ValueError("report is missing the shared HTML execution-evidence viewer") + for marker in ( + 'id="evidence-viewer-content"', + "execution-evidence[.]json", + "fetch(new URL(source,document.baseURI)", + ): + if marker not in rendered_pages[evidence_viewer]: + raise ValueError(f"execution-evidence viewer is incomplete: {marker}") + failed_pages = [ + report_dir / "results" / f"{_attempt_id(row)}.html" + for row in logical_rows + if float(row["score"]) < 1 + ] + if failed_pages and any( + marker not in rendered_pages[page] + for page in failed_pages + for marker in ( + "失败诊断", + 'class="failure-lead"', + "程序报错", + "程序判定", + "复核根因", + "负责范围", + "失败执行的最后页面画面", + ) + ): + raise ValueError("every failed result must expose a complete diagnosis path") + if any( + marker in rendered_pages[page] + for page in result_pages + for marker in ("Ground truth 与实际最终回答", "官方 evaluator 逐规则判定") + ): + raise ValueError("result pages must not duplicate answer and evaluator presentations") + html = rendered_pages[index] + article_heading = "程序判分落点与复核根因" + if article_heading not in rendered_pages[report_dir / "index.html"]: + raise ValueError("homepage must contain the authored interpretation") + public_results = [ + json.loads(line) + for line in (report_dir / "data/results.jsonl").read_text(encoding="utf-8").splitlines() + if line + ] + if len(public_results) != len(logical_rows): + raise ValueError("public result projection must cover every logical result") + evidence_index = json.loads( + (report_dir / "data/evidence-index.json").read_text(encoding="utf-8") + ) + _check_execution_artifacts(report_dir, logical_rows, public_results, evidence_index) + _check_public_privacy(report_dir) + forbidden = { + "agent_response", + "paths", + "model_transport_errors", + } + if any(forbidden & set(item) for item in public_results): + raise ValueError("public result projection contains private evidence fields") + forbidden_report_terms = ( + "诊断重试", + "失败重试", + "首轮试次", + "assistant reasoning", + "用户提示词", + "我的思考", + "内部工作日志", + "另一个覆盖更多站点的 179 题", + "后续 179 题", + "需要注意:", + "代理实测返回 team_model_access_denied", + "证据支持的诊断", + "共享链路诊断", + "失败归因:", + ) + for page, text in rendered_pages.items(): + if any(term in text for term in forbidden_report_terms): + raise ValueError( + f"formal report contains forbidden notebook or execution-label text: {page}" + ) + if re.search( + r"]+(?:src|data)=[\"'][^\"']*\.svg(?:[?#][^\"']*)?[\"']", + text, + flags=re.I, + ): + raise ValueError( + f"Browser Use reports must use raster or Mermaid diagrams, not SVG: {page}" + ) + svg_assets = [ + path for path in report_dir.rglob("*") if path.is_file() and path.suffix.lower() == ".svg" + ] + if svg_assets: + raise ValueError(f"Browser Use report contains SVG assets: {svg_assets[0]}") + style = (report_dir / "assets/style.css").read_text(encoding="utf-8") + for required_style in ( + "#task-table{min-width:1500px}", + "background:var(--code-bg)!important", + ".method-figure img{min-width:960px", + ".detail-grid .kv{overflow-wrap:anywhere", + ".comparison-winner{", + ".browser-comparison{", + ".action-card-grid{", + ".attempt-summary{", + "#result-table{min-width:1580px;table-layout:fixed}", + ".diagnosis-grid{", + ".screenshot-grid{", + ".review-round{", + ".page-main{overflow-x:clip}", + ): + if required_style not in style: + raise ValueError(f"report is missing publication readability guard: {required_style}") + if require_complete: + if not str(profile.get("logical_result") or "").startswith( + "score authority follows the frozen run manifest" + ): + raise ValueError("current report profile must defer score authority to the manifest") + authorities = audit.get("score_authority") or {} + review_bindings = (audit.get("review") or {}).get("bindings") or {} + for bundle in bundles: + evidence_path = bundle["path"] + manifest_path = evidence_path / "manifest.json" + run_config_path = evidence_path / "run-config.json" + attempts_path = evidence_path / "attempts.jsonl" + review_path = evidence_path / "result-review.json" + if not manifest_path.is_file() or not run_config_path.is_file(): + raise ValueError("complete evidence requires manifest.json and run-config.json") + manifest, run_config = bundle["manifest"], bundle["config"] + model = bundle["model"] + official = (manifest.get("environment") or {}).get("official_evaluator") or {} + if model not in models: + raise ValueError( + f"current report profile does not register measured model: {model}" + ) + for key, expected in { + "selection_sha256": selection.sha256, + "reasoning_effort": (manifest.get("protocol") or {}).get("reasoning_effort"), + "official_data_checksum": official.get("data_checksum"), + }.items(): + if profile.get(key) != expected: + raise ValueError(f"current report profile mismatch for {key}: {model}") + browser_versions = set( + (manifest.get("environment") or {}).get("browser_versions") or {} + ) + if set(engines) != browser_versions: + raise ValueError(f"current report engines differ from measured identities: {model}") + authority = (manifest.get("protocol") or {}).get( + "official_score_uses" + ) or "final execution for each browser and task" + if authorities.get(model) != authority: + raise ValueError(f"report renderer changed frozen score authority: {model}") + if manifest.get("selection_sha256") != selection.sha256: + raise ValueError("manifest selection hash differs from the registered cohort") + if run_config.get("selection_sha256") != selection.sha256: + raise ValueError("run configuration differs from the registered cohort") + if run_config.get("reasoning_effort") != profile.get("reasoning_effort"): + raise ValueError("run configuration reasoning effort differs from the profile") + if not run_config.get("browser_eval_commit"): + raise ValueError("run configuration must identify the browser-eval commit") + if manifest.get("protocol", {}).get("browser_eval_commit") != run_config.get( + "browser_eval_commit" + ): + raise ValueError("manifest and run configuration disagree on browser-eval commit") + if manifest.get("protocol", {}).get("model_transport_retries") != 1: + raise ValueError("pilot allows exactly one model transport retry") + if not review_path.is_file(): + raise ValueError("complete publication requires result-review.json per model") + bundle_review = bundle["review"] + bindings = bundle_review.get("bindings") or {} + if bindings.get("attempts_sha256") != _sha256(attempts_path): + raise ValueError(f"result review is not bound to current attempts: {model}") + if bindings.get("manifest_sha256") != _sha256(manifest_path): + raise ValueError(f"result review is not bound to current manifest: {model}") + if bindings.get("selection_sha256") != selection.sha256: + raise ValueError("result review is not bound to the registered selection") + model_rows = [row for row in rows if row["model"] == model] + coverage = bundle_review.get("coverage") or {} + if coverage != { + "recorded_attempts": len(model_rows), + "reviewed_attempts": len(model_rows), + }: + raise ValueError(f"result review does not cover every attempt: {model}") + if bundle_review.get("disagreements") or bundle_review.get("unresolved"): + raise ValueError(f"result review has disagreements or unresolved checks: {model}") + reviewed_keys = { + (model, str(item["engine"]), int(item["task_id"]), int(item["attempt"])) + for item in bundle_review.get("attempts") or [] + } + measured_keys = {key for key in by_key if key[0] == model} + if reviewed_keys != measured_keys: + raise ValueError(f"result review inventory differs from evidence: {model}") + if review_bindings.get(model) != bindings: + raise ValueError(f"rendered report review binding differs: {model}") + missing = expected_primary - set(by_key) + if missing: + raise ValueError(f"missing {len(missing)} primary attempts") + missing_retries = { + (model, engine, task_id, 2) + for model, engine, task_id, _ in expected_primary + if float(by_key[(model, engine, task_id, 1)]["score"]) != 1.0 + and (model, engine, task_id, 2) not in by_key + } + if missing_retries: + raise ValueError(f"missing {len(missing_retries)} permitted failure retries") + if "测量尚未完整" in html: + raise ValueError("complete publication cannot contain an incomplete-results notice") + cause_profile = profile.get("failure_cause_review") or {} + cause_required = bool(cause_profile.get("required_for_publication")) + default_cause_review = load_failure_cause_review() + verified_causes: dict[str, dict] = {} + for bundle in bundles: + model = bundle["model"] + failed_ids = { + _attempt_id(row) + for row in logical_rows + if row["model"] == model and float(row.get("score") or 0) < 1 + } + verified_causes.update( + validate_failure_cause_review( + default_cause_review, + model_id=model, + attempts_sha256=bundle["attempts_sha256"], + failed_result_ids=failed_ids, + require_complete=cause_required, + ) + ) + if cause_required: + failed_result_ids = { + _attempt_id(row) for row in logical_rows if float(row.get("score") or 0) < 1 + } + if set(verified_causes) != failed_result_ids: + raise ValueError("every failed logical result must have a verified root cause") + configured_hash = cause_profile.get("source_sha256") + if not isinstance(configured_hash, str) or configured_hash != file_sha256( + FAILURE_CAUSE_REVIEW_PATH + ): + raise ValueError("current profile is not bound to the failure-cause review") + published_cause_review = json.loads( + (report_dir / "data/failure-cause-review.json").read_text(encoding="utf-8") + ) + if published_cause_review.get("results") != verified_causes: + raise ValueError("published failure causes differ from verified reviews") + interpretation = json.loads( + (report_dir / "data/interpretation-evidence.json").read_text(encoding="utf-8") + ) + scoring_claim = next( + ( + item + for item in interpretation.get("claims") or [] + if item.get("claim") == "官方 evaluator 的失败规则组合" + ), + None, + ) + if not scoring_claim: + raise ValueError("interpretation evidence is missing programmatic failure locations") + scoring_basis = scoring_claim.get("basis") or {} + scoring_classes = scoring_basis.get("classes") or {} + failed_count = sum(1 for row in logical_rows if float(row.get("score") or 0) < 1) + if scoring_basis.get("denominator") != failed_count or sum( + int(scoring_classes.get(key) or 0) + for key in ("network_only", "response_only", "both") + ) != failed_count: + raise ValueError("programmatic failure locations do not cover every final failure") + remediation = json.loads( + (Path(__file__).parents[1] / "research/browser-use-remediation.json").read_text( + encoding="utf-8" + ) + ) + if remediation.get("schema") != "lex_browser_use_remediation/1": + raise ValueError("unsupported Browser Use remediation schema") + external_prefixes = ("benchmark_", "site_", "environment_", "mixed_") + expected_quarantine = { + int(result_id.rsplit("-", 1)[-1]) + for result_id, diagnosis in verified_causes.items() + if str((diagnosis.get("subcategory") or {}).get("code") or "").startswith( + external_prefixes + ) + } + registered_quarantine = { + int(task_id) + for item in remediation.get("items") or [] + if item.get("category") + in { + "shared_environment", + "benchmark_or_evaluator", + "benchmark_instruction_contract", + "self_hosted_site", + "reference_state_mixed", + } + for task_id in item.get("task_ids") or [] + } + if registered_quarantine != expected_quarantine: + raise ValueError("remediation quarantine differs from verified external interference") + for bundle in bundles: + for _page, text in rendered_pages.items(): + if str(bundle["path"]) in text: + raise ValueError( + "published pages must not contain machine-local evidence paths" + ) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("report", type=Path) + parser.add_argument("--evidence", required=True, action="append", type=Path) + parser.add_argument( + "--selection", + type=Path, + default=Path("evaluation/browser-use/webarena-verified-pilot.json"), + ) + parser.add_argument("--require-complete", action="store_true") + args = parser.parse_args() + check( + args.report, + args.evidence, + args.selection, + require_complete=args.require_complete, + ) + print("Browser Use report integrity check passed") + + +if __name__ == "__main__": + main() diff --git a/scripts/check_browser_use_visual.py b/scripts/check_browser_use_visual.py new file mode 100644 index 000000000..de103ec77 --- /dev/null +++ b/scripts/check_browser_use_visual.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Render every Browser Use route class and reject structural layout failures.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import shutil +from pathlib import Path + +from playwright.sync_api import sync_playwright + + +def report_tree_sha256(report: Path) -> str: + digest = hashlib.sha256() + for path in sorted(item for item in report.rglob("*") if item.is_file()): + relative = path.relative_to(report).as_posix().encode() + digest.update(len(relative).to_bytes(8, "big")) + digest.update(relative) + digest.update(hashlib.sha256(path.read_bytes()).digest()) + return digest.hexdigest() + + +def _browser_executable() -> str: + configured = os.environ.get("BROWSER_USE_VISUAL_BROWSER") or os.environ.get( + "CHROMIUM_BINARY" + ) + candidates = ( + configured, + "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", + shutil.which("google-chrome"), + shutil.which("chromium"), + shutil.which("chromium-browser"), + ) + for candidate in candidates: + if candidate and Path(candidate).is_file(): + return candidate + raise ValueError( + "visual QA requires Chrome/Chromium; set BROWSER_USE_VISUAL_BROWSER to its executable" + ) + + +def _route_classes(report: Path) -> list[Path]: + fixed = [ + "index.html", + "datasets/index.html", + "methodology/index.html", + "analysis/index.html", + "review/index.html", + "changes/index.html", + "adoption/index.html", + "reproduce/index.html", + "accounts/index.html", + ] + dynamic = [] + for folder in ("datasets", "results", "engines"): + pages = sorted((report / folder).glob("*.html")) + if pages: + dynamic.append(pages[0].relative_to(report).as_posix()) + if folder == "results": + failed_page = next( + (page for page in pages if 'id="diagnosis"' in page.read_text(encoding="utf-8")), + None, + ) + if failed_page: + failed_route = failed_page.relative_to(report).as_posix() + if failed_route not in dynamic: + dynamic.append(failed_route) + chinese = [report / name for name in (*fixed, *dynamic)] + routes = chinese + [report / "en" / page.relative_to(report) for page in chinese] + missing = [path for path in routes if not path.is_file()] + if missing: + raise ValueError(f"visual QA route is missing: {missing[0]}") + return routes + + +def check(report: Path, receipt: Path) -> None: + states = [] + failures = [] + executable = _browser_executable() + routes = _route_classes(report) + with sync_playwright() as playwright: + browser = playwright.chromium.launch(executable_path=executable, headless=True) + try: + for route in routes: + for width, height in ((1440, 1000), (820, 1180), (390, 844)): + for theme in ("light", "dark"): + page = browser.new_page(viewport={"width": width, "height": height}) + page.goto(route.resolve().as_uri(), wait_until="load") + page.evaluate( + "value => { document.documentElement.dataset.theme = value; }", theme + ) + result = page.evaluate( + """() => { +const root=document.documentElement; +const overflow=root.scrollWidth-root.clientWidth; +const clipped=[]; +for(const element of document.querySelectorAll('.filters input,.filters select,.attempt-fact,.evidence-card,.review-round,.action-card')){ + if(!element.getClientRects().length)continue; + const box=element.getBoundingClientRect(); + if(box.left < -2 || box.right > innerWidth + 2)clipped.push(element.className || element.tagName); +} +const vertical=[]; +for(const button of document.querySelectorAll('th .sort-button')){ + const box=button.getBoundingClientRect(),style=getComputedStyle(button),line=parseFloat(style.lineHeight)||16; + if(button.textContent.trim().length>2 && box.height>line*2.6 && box.width<52)vertical.push(button.textContent.trim()); +} +const textOverflow=[]; +for(const element of document.querySelectorAll('.hero h1,.hero .lead')){ + if(element.scrollWidth > element.clientWidth + 2)textOverflow.push(element.tagName+'.'+element.className); +} +return {overflow,clipped:[...new Set(clipped)],vertical,textOverflow}; +} +""" + ) + state = { + "route": route.relative_to(report).as_posix(), + "width": width, + "height": height, + "theme": theme, + **result, + } + states.append(state) + if ( + result["overflow"] > 2 + or result["clipped"] + or result["vertical"] + or result["textOverflow"] + ): + failures.append(state) + page.close() + finally: + browser.close() + receipt.parent.mkdir(parents=True, exist_ok=True) + receipt.write_text( + json.dumps( + { + "schema": "lex_browser_use_visual_qa/1", + "report_tree_sha256": report_tree_sha256(report), + "browser_executable": executable, + "route_classes": len(routes), + "states_checked": len(states), + "failures": failures, + }, + ensure_ascii=False, + indent=2, + ), + encoding="utf-8", + ) + if failures: + first = failures[0] + raise ValueError( + f"visual QA failed at {first['route']} {first['width']}px {first['theme']}: {first}" + ) + print(f"Browser Use visual QA passed: {len(states)} rendered states") + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("report", type=Path) + parser.add_argument( + "--receipt", + type=Path, + default=Path("publish/browser-use-visual-qa.json"), + ) + args = parser.parse_args() + check(args.report, args.receipt) + + +if __name__ == "__main__": + main() diff --git a/scripts/check_report_links.py b/scripts/check_report_links.py index 5246be5d9..8aaca7145 100755 --- a/scripts/check_report_links.py +++ b/scripts/check_report_links.py @@ -12,8 +12,12 @@ def __init__(self) -> None: super().__init__() self.values: list[str] = [] self.ids: set[str] = set() + self.base_href: str | None = None def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + if tag == "base": + self.base_href = next((value for key, value in attrs if key == "href" and value), None) + return for key, value in attrs: if key in {"href", "src"} and value: self.values.append(value) @@ -21,8 +25,8 @@ def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None self.ids.add(value) -def main() -> int: - root = Path(sys.argv[1]).resolve() +def check(root: Path) -> None: + root = root.resolve() missing: list[str] = [] pages = list(root.rglob("*.html")) parsed_pages: dict[Path, Links] = {} @@ -35,11 +39,16 @@ def parse_page(path: Path) -> Links: for page in pages: parser = parse_page(page) + base_dir = page.parent + if parser.base_href: + parsed_base = urlsplit(parser.base_href) + if not parsed_base.scheme: + base_dir = (page.parent / unquote(parsed_base.path)).resolve() for value in parser.values: parsed = urlsplit(value) if parsed.scheme or value.startswith("mailto:"): continue - target = page if not parsed.path else (page.parent / unquote(parsed.path)).resolve() + target = page if not parsed.path else (base_dir / unquote(parsed.path)).resolve() if not target.exists(): missing.append(f"{page.relative_to(root)} -> {value}") continue @@ -48,10 +57,16 @@ def parse_page(path: Path) -> Links: if fragment not in parse_page(target).ids: missing.append(f"{page.relative_to(root)} -> {value} (missing fragment)") if missing: - print("Broken local links:") - print("\n".join(missing)) - return 1 + raise ValueError("Broken local links:\n" + "\n".join(missing)) print(f"Report link check: {len(pages)} logical report routes, 0 broken local links") + + +def main() -> int: + try: + check(Path(sys.argv[1])) + except ValueError as error: + print(error) + return 1 return 0 diff --git a/scripts/check_track_naming.py b/scripts/check_track_naming.py new file mode 100644 index 000000000..73f11578d --- /dev/null +++ b/scripts/check_track_naming.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +"""Reject retired track names from maintained repository files.""" + +from __future__ import annotations + +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +EXCLUDED_ROOTS = { + ".git", + "node_modules", + "publish", + "reports", + "evaluation/dataset.json", + "evaluation/sources", +} +EXCLUDED_PARTS = {".pytest_cache", ".ruff_cache", ".venv", "__pycache__"} +TEXT_SUFFIXES = {"", ".css", ".html", ".js", ".json", ".md", ".py", ".sh", ".toml", ".txt", ".yaml", ".yml"} +FORBIDDEN = ( + "browser_" + "agent", + "browser-" + "agent", + "Browser " + "Agent", + "browser_" + "kernel", + "browser-" + "kernel", + "Browser " + "Kernel", + "Browser-" + "agent", + "Browser-" + "kernel", +) + + +def main() -> None: + findings: list[str] = [] + for path in ROOT.rglob("*"): + if not path.is_file() or path.suffix.lower() not in TEXT_SUFFIXES: + continue + relative = path.relative_to(ROOT) + name = relative.as_posix() + if EXCLUDED_PARTS.intersection(relative.parts): + continue + if any(name == root or name.startswith(root + "/") for root in EXCLUDED_ROOTS): + continue + text = path.read_text(encoding="utf-8", errors="ignore") + for term in FORBIDDEN: + if term in text: + findings.append(f"{name}: retired track name {term!r}") + if findings: + raise SystemExit("\n".join(findings)) + print("Track naming check passed") + + +if __name__ == "__main__": + main() diff --git a/scripts/check_webarena_replay_report.py b/scripts/check_webarena_replay_report.py new file mode 100644 index 000000000..5d3704833 --- /dev/null +++ b/scripts/check_webarena_replay_report.py @@ -0,0 +1,268 @@ +#!/usr/bin/env python3 +"""Run deterministic release checks for the WebArena replay report.""" + +from __future__ import annotations + +import argparse +import json +import re +from html.parser import HTMLParser +from pathlib import Path +from urllib.parse import unquote, urlsplit + +from harness.trajectory_replay.webarena_report import LOCAL_PATH_RE + + +class _VisibleTextParser(HTMLParser): + def __init__(self) -> None: + super().__init__() + self.hidden_depth = 0 + self.parts: list[str] = [] + + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + if tag.lower() in {"script", "style", "pre", "code"}: + self.hidden_depth += 1 + + def handle_endtag(self, tag: str) -> None: + if tag.lower() in {"script", "style", "pre", "code"} and self.hidden_depth: + self.hidden_depth -= 1 + + def handle_data(self, data: str) -> None: + if not self.hidden_depth: + self.parts.append(data) + + +def _visible_text(html: str) -> str: + parser = _VisibleTextParser() + parser.feed(html) + return " ".join(parser.parts) + + +def _check_english_localization(pages: list[Path]) -> None: + cjk = re.compile(r"[\u3400-\u9fff]") + localized_attribute = re.compile( + r'(?:aria-label|placeholder|title)=["\']([^"\']*)', + flags=re.I, + ) + for page in pages: + text = page.read_text(encoding="utf-8") + title = re.search(r"(.*?)", text, flags=re.I | re.S) + if title and cjk.search(title.group(1)): + raise ValueError(f"English report title is not localized: {page}") + if any(cjk.search(value) for value in localized_attribute.findall(text)): + raise ValueError(f"English report accessibility text is not localized: {page}") + visible = _visible_text(text).replace("中文", "") + match = cjk.search(visible) + if match: + excerpt = visible[max(0, match.start() - 40) : match.start() + 80] + raise ValueError( + f"English reader-facing page contains Chinese prose: {page}: {excerpt}" + ) + + +def _local_target(page: Path, raw: str) -> Path | None: + parsed = urlsplit(raw) + if parsed.scheme or raw.startswith(("#", "mailto:", "javascript:")): + return None + return (page.parent / unquote(parsed.path)).resolve() + + +def check(report_dir: Path) -> dict[str, int]: + root = report_dir.resolve() + manifest = json.loads((root / "report-manifest.json").read_text(encoding="utf-8")) + results = json.loads((root / "data/results.json").read_text(encoding="utf-8")) + required_manifest_fields = ( + "task_count", + "participant_count", + "participants", + "logical_results", + "failed_results", + "reviewed_failures", + ) + missing_manifest_fields = [ + field for field in required_manifest_fields if field not in manifest + ] + if missing_manifest_fields: + raise ValueError( + "report manifest is missing required fields: " + + ", ".join(missing_manifest_fields) + ) + participants = manifest.get("participants") or [] + task_count = int(manifest.get("task_count") or 0) + participant_count = int(manifest.get("participant_count") or 0) + expected_results = task_count * participant_count + if ( + task_count < 1 + or participant_count < 1 + or len(participants) != participant_count + or len(set(participants)) != participant_count + or manifest["logical_results"] != expected_results + or len(results) != expected_results + ): + raise ValueError("report must contain the complete manifest-declared participant-task matrix") + task_ids = {int(item["task_id"]) for item in results} + result_keys = {(str(item["engine"]), int(item["task_id"])) for item in results} + expected_keys = {(participant, task_id) for participant in participants for task_id in task_ids} + if len(task_ids) != task_count or result_keys != expected_keys: + raise ValueError("report results do not match the manifest-declared participant-task matrix") + failures = [item for item in results if item["status"] != "pass"] + if ( + manifest["failed_results"] != len(failures) + or manifest["reviewed_failures"] != len(failures) + ): + raise ValueError("report must contain a review for every final failure") + if any(not item["reason_zh"] or not item["reason_en"] for item in failures): + raise ValueError("every failed result needs a bilingual reviewed cause") + + zh_details = {path.name for path in (root / "result").glob("*.html")} + en_details = {path.name for path in (root / "en/result").glob("*.html")} + if zh_details != en_details or len(zh_details) != expected_results: + raise ValueError("Chinese and English result routes must match one-to-one") + zh_tasks = {path.name for path in (root / "task").glob("*.html")} + en_tasks = {path.name for path in (root / "en/task").glob("*.html")} + if zh_tasks != en_tasks or zh_tasks != {f"{task_id}.html" for task_id in task_ids}: + raise ValueError("every task needs matching Chinese and English contract routes") + for task_id in task_ids: + frozen_path = root / "data/trajectory" / f"{task_id}.json" + if not frozen_path.is_file(): + raise ValueError(f"qualified trajectory is missing for task {task_id}") + frozen = json.loads(frozen_path.read_text(encoding="utf-8")) + action_count = len(frozen.get("tool_calls") or []) + if frozen.get("task_id") != task_id or action_count < 1: + raise ValueError(f"qualified trajectory is invalid for task {task_id}") + for participant in participants: + key = f"{participant}-{task_id}" + for language_root in (root, root / "en"): + page = language_root / "result" / f"{key}.html" + text = page.read_text(encoding="utf-8") + relative_root = "../" if language_root == root else "../../" + if ( + text.count('class="aligned-step ') != action_count + or f'data-standard-url="{relative_root}data/trajectory/{task_id}.json"' not in text + or f'data-evidence-url="{relative_root}data/evidence/{key}.json"' not in text + or 'id="step-dialog"' not in text + ): + raise ValueError(f"result detail lacks aligned step evidence: {page}") + core_names = ( + "index.html", + "dataset.html", + "results.html", + "environment.html", + "review.html", + "evidence-viewer.html", + ) + core = [ + *(root / name for name in core_names), + *(root / "en" / name for name in core_names), + ] + forbidden = ( + "用户提示词", + "我的思考", + "assistant reasoning", + "内部工作日志", + "中位数", + "median", + "后续 179 题", + ) + for page in core: + text = page.read_text(encoding="utf-8") + if any(term in text for term in forbidden): + raise ValueError(f"formal report contains internal or retired prose: {page}") + if "Browser Trajectory Replay", + ">Dataset", + ">Results", + ">Environment", + ">Review history", + ">Evidence viewer", + ">Action", + ">Target", + ">Time", + ">Status", + ">Result", + " calls", + ) + for page in [*core[: len(core_names)], *(root / "result").glob("*.html")]: + text = page.read_text(encoding="utf-8") + if any(marker in text for marker in untranslated_zh_markers): + raise ValueError(f"Chinese report contains untranslated interface text: {page}") + _check_english_localization( + [*core[len(core_names) :], *(root / "en/result").glob("*.html"), *(root / "en/task").glob("*.html")] + ) + + checked_links = 0 + for page in root.rglob("*.html"): + text = page.read_text(encoding="utf-8") + for raw in re.findall(r'(?:href|src)=["\']([^"\']+)', text): + target = _local_target(page, raw) + if target is None: + continue + checked_links += 1 + if target.is_dir(): + target = target / "index.html" + if not target.is_file() or (target != root and root not in target.parents): + raise ValueError(f"broken or escaping local link in {page}: {raw}") + + distributable = "\n".join( + path.read_text(encoding="utf-8", errors="ignore") + for path in root.rglob("*") + if path.is_file() and path.suffix in {".html", ".json", ".js", ".css"} + ) + if LOCAL_PATH_RE.search(distributable) or re.search( + r"sk-[A-Za-z0-9_-]{12,}", distributable + ): + raise ValueError("report contains a local path or credential-like token") + css = (root / "assets/report.css").read_text(encoding="utf-8") + if ".inventory{height:" not in css or "overflow:auto" not in css: + raise ValueError("result inventory must be fixed-height and horizontally scrollable") + results_html = (root / "results.html").read_text(encoding="utf-8") + if '