Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Benchmark ownership

Browser Eval owns execution adapters, evidence handling and report tooling. `lexmount/lexbench-webfetch` owns Web Fetch task populations, task models, deterministic acceptance scoring, compilation and frozen definition files. Import its pinned package directly; do not duplicate or re-export retired evaluator modules here. Resolve frozen historical catalogs/manifests through `lexbench_webfetch.inputs.definition_path`; keep experiment profiles and compact hash bindings in `evaluation/`. WebMainBench, WCXB and WebArena retain their own upstream task authorities.

# Report maintenance

Repository documentation and contribution guidance use English. Keep the root README and `docs/README.zh-CN.md` aligned; additional README translations are welcome, but other guides need no parallel translation. Preserve original-language source quotations and hash-bound acceptance/evidence artifacts. Report-generation languages follow the track contracts below.
Expand Down Expand Up @@ -78,3 +82,5 @@ These requirements apply to **all** report tracks; browser-specific rules below
- English routes may preserve measured task names, IDs and evidence verbatim, but all report-authored interface and explanatory prose must be localized. Release checks must reject known untranslated interface markers and untranslated accessibility attributes.
- Before release, inspect every core route class in each explicitly generated language at desktop and narrow widths in both themes, plus the English not-generated notice when applicable. Prevent vertical table labels, clipped controls and unreadable evidence; use ImageGen raster assets or Mermaid rather than SVG for Browser Use diagrams.
- Keep routine report checks fast and deterministic; they cover structure, evidence, links, terminology, and static readability guards without launching a browser. Run the separate full release check before packaging or publishing; it adds the complete cross-language, cross-theme, desktop and narrow-width visual gate.

All code changes go through a feature branch and pull request. Never push directly to main or merge a pull request.
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing

PRs are welcome for adapters, tasks, tests, scoring fixes, documentation and translations. Run the [offline example](README.md#quickstart), then open the guide for the track you want to change.
PRs are welcome for adapters, execution fixes, tests, documentation and translations. Self-authored Web Fetch task requirements are maintained in [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch); see [benchmark boundaries](docs/benchmarks.md). Run the [offline example](README.md#quickstart), then open the guide for the track you want to change.

## Report a bug

Expand All @@ -11,7 +11,7 @@ Include the task ID, method and versions, OS, command and sanitized error. Remov
| Change | Where it belongs |
| --- | --- |
| Browser or provider adapter | Existing execution modules; see the [provider contract](docs/provider-contract.md) |
| Task requirements | Dataset configuration, applied to every method |
| LexBench Web Fetch task requirements | [Benchmark repository](https://github.com/lexmount/lexbench-webfetch), applied to every method |
| Scoring fix | Shared scorer, scenario regressions and a recorded correction reason |
| New browser, version or sample size | Configuration or registry; reuse the runner and renderer |
| Reusable logic | `harness/`; `scripts/` stays a thin command entry point |
Expand Down
42 changes: 36 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,45 @@
<div align="center">

<img src="docs/assets/overview.svg" alt="Browser Eval evaluation workflow" width="100%">

# Browser Eval

**English** | [简体中文](docs/README.zh-CN.md)
### Run the same tasks. Keep the evidence.

An evaluation harness for web fetching, recorded browser actions and autonomous agents.

**English** · [简体中文](docs/README.zh-CN.md)

[![CI](https://github.com/lexmount/browser-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/lexmount/browser-eval/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/code-MIT-51735b)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-356a91)](pyproject.toml)

[Quickstart](#quickstart) · [Benchmarks](docs/benchmarks.md) · [Reports](#read-and-share-a-report) · [Contributing](CONTRIBUTING.md)

Reproducible browser benchmarks for web fetching, trajectory replay and AI agent automation. Compare task completion, latency and resource use across browsers, using shared task rules and saved evidence. Browser Eval is maintained as a companion to [Moli](https://github.com/lexmount/moli) and supports other browsers.
</div>

---

Browser Eval connects benchmark tasks to browser and service adapters, executes declared comparisons, calls the benchmark scorer on saved outputs and builds inspectable reports. It supports Web Fetch, Browser Trajectory Replay and Browser Use. Task populations and acceptance requirements come from the selected benchmark.

[LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) owns the self-authored Web Fetch task suites. WebMainBench, WCXB and WebArena supply other benchmark inputs. Browser Eval provides the execution and reporting machinery; each benchmark retains its own scope, versions and attribution. See [repository boundaries and integration](docs/benchmarks.md).

You get an HTML report with comparison tables, task details, responses and logs. Read it locally or publish it through [Artifact Site](https://github.com/lexmount/artifact-site) so others can explore the results online.

[What it tests](#what-it-tests) · [Quickstart](#quickstart) · [Reports](#read-and-share-a-report) · [Related projects](#related-repositories) · [Contribute](#contribute)

```mermaid
flowchart LR
A[Benchmark tasks and contracts] --> B[Browser Eval]
B --> C[Browser or provider adapters]
C --> D[Saved outputs and timings]
D --> E[Scoring and evidence checks]
E --> F[Inspectable HTML report]
```

## What it tests

Current evaluation tracks:
Supported execution tracks:

| Track | Question | Start here |
| --- | --- | --- |
Expand Down Expand Up @@ -41,7 +70,7 @@ Support varies by task type. Check [adapter requirements](docs/provider-contract
| Suite | What it measures | Availability and entry |
| --- | --- | --- |
| Offline fixture / local article | Harness operation, selection, scoring and evidence; no performance ranking | Included; commands below |
| Live Web Fetch suites | Ordinary pages, dynamic content, documents, images/video, popular sites and access restrictions | [Track guide](harness/web_fetch/README.md); complete historical references require an approved [external input set](docs/reference-inputs.md) |
| LexBench Web Fetch | Ordinary pages, dynamic content, documents, images/video, popular sites and access restrictions | [Benchmark repository](https://github.com/lexmount/lexbench-webfetch) · [Integration](docs/benchmarks.md); complete historical references require approved external inputs |
| WebMainBench / WCXB | Extraction fidelity on independently bound frozen HTML and reference content | [Frozen-HTML guide](harness/web_fetch/WEBMAINBENCH.md); obtain source data under its terms and use the suite's certification pipeline |
| Curated trajectory suite | Identical fixed actions on selected public pages | [Trajectory run guide](examples/trajectory_replay/README.md); uses Docker and live sites |
| WebArena human-recording replay | Qualified fixed trajectories and official task evaluation on self-hosted sites | [Reproduction guide](docs/webarena-fixed-trajectory-reproduction.md); external datasets and substantial deployment resources |
Expand Down Expand Up @@ -122,6 +151,7 @@ Example runs are diagnostic. For a formal benchmark release, follow the selected

| Repository | Role in this work |
| --- | --- |
| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | Independent task suites and acceptance contracts for self-authored Web Fetch evaluations |
| [Moli](https://github.com/lexmount/moli) | Browser evaluated alongside the other implementations |
| [Artifact Site](https://github.com/lexmount/artifact-site) | Report hosting, versions and sharing; self-hosting supported |
| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | Separate benchmark for protocols, automation drivers and web-platform semantics |
Expand All @@ -135,7 +165,7 @@ See [dependency sources and pins](docs/dependencies.md) for exact versions. Inst

## Contribute

PRs are welcome for adapters, tasks, tests, scoring fixes, documentation and translations. Start with [CONTRIBUTING](CONTRIBUTING.md), or [open an issue](https://github.com/lexmount/browser-eval/issues) with a reproducible example.
PRs are welcome for adapters, execution fixes, tests, documentation and translations. Changes to LexBench task requirements belong in [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch). Start with [CONTRIBUTING](CONTRIBUTING.md), or [open an issue](https://github.com/lexmount/browser-eval/issues) with a reproducible example.

```bash
make setup-test
Expand All @@ -148,7 +178,7 @@ Project code is [MIT licensed](LICENSE). [THIRD_PARTY](THIRD_PARTY.md) covers da

## Cite Browser Eval

If you use the harness, task definitions or evaluation methodology in published work, cite the repository and record the tag or commit used. GitHub's citation menu reads [CITATION.cff](CITATION.cff). For the initial release:
If you use the harness in published work, cite this repository and record the tag or commit used. Cite LexBench Web Fetch or other benchmark owners separately for their task definitions and methodology. GitHub's citation menu reads [CITATION.cff](CITATION.cff). For the initial release:

```bibtex
@software{browser_eval_2026,
Expand Down
42 changes: 36 additions & 6 deletions docs/README.zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,45 @@
<div align="center">

<img src="assets/overview.svg" alt="Browser Eval evaluation workflow" width="100%">

# Browser Eval

[English](../README.md) | **简体中文**
### 执行同一组任务,留下可核查的证据。

用于网页抓取、固定浏览器操作和自主 Agent 的评测脚手架。

[English](../README.md) · **简体中文**

[![CI](https://github.com/lexmount/browser-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/lexmount/browser-eval/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/code-MIT-51735b)](../LICENSE)
[![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-356a91)](../pyproject.toml)

[快速开始](#快速开始) · [Benchmark 接入](benchmarks.md) · [报告](#查看与分享报告) · [参与贡献](../CONTRIBUTING.md)

可复现的浏览器评测,涵盖网页抓取、轨迹重放和 AI Agent 自动化:检查网页内容能否完整获取、记录好的操作能否完成、Agent 能否自主完成任务,并比较耗时和资源消耗。Browser Eval 是 [Moli](https://github.com/lexmount/moli) 的配套评测项目,也支持其他浏览器;各方法使用统一的任务规则,结果保留原始证据。
</div>

---

Browser Eval 把 benchmark 任务接到浏览器或服务适配器上,执行预先声明的比较,调用 benchmark 评分器检查保存的输出,并生成可查看证据的报告。它支持 Web Fetch、轨迹重放和 Browser Use;任务范围和验收要求由所选 benchmark 决定。

自建的 Web Fetch 任务集由 [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) 维护。WebMainBench、WCXB 和 WebArena 提供其他评测输入。Browser Eval 负责执行与报告,各 benchmark 分别维护范围、版本和来源说明。详见[仓库分工与接入](benchmarks.md)。

运行后得到 HTML 报告,里面有比较表、任务详情、响应和日志。可以在本地查看,也可以通过 [Artifact Site](https://github.com/lexmount/artifact-site) 发布,让其他人在线查看结果。

[评测内容](#评测内容) · [快速开始](#快速开始) · [查看与分享报告](#查看与分享报告) · [相关项目](#相关仓库导航) · [参与贡献](#参与贡献)

```mermaid
flowchart LR
A[Benchmark 任务与验收规则] --> B[Browser Eval]
B --> C[浏览器或服务适配器]
C --> D[保存的响应与耗时]
D --> E[评分与证据检查]
E --> F[可查看证据的 HTML 报告]
```

## 评测内容

目前支持以下评测方式:
支持以下执行方式:

| 轨道 | 回答什么问题 | 从这里开始 |
| --- | --- | --- |
Expand Down Expand Up @@ -41,7 +70,7 @@
| 任务集 | 评测内容 | 可用条件与入口 |
| --- | --- | --- |
| 离线 fixture / 本地文章 | 检查运行、任务选择、评分和证据,不提供性能排名 | 随仓库提供,命令见下文 |
| 实时 Web Fetch 任务集 | 普通网页、动态内容、文档、图片/视频、热门网站、访问限制 | [轨道说明](../harness/web_fetch/README.md);完整历史参考需要获准使用的[外部输入](../docs/reference-inputs.md) |
| LexBench Web Fetch | 普通网页、动态内容、文档、图片/视频、热门网站、访问限制 | [Benchmark 仓库](https://github.com/lexmount/lexbench-webfetch) · [接入说明](benchmarks.md);完整历史参考需要获准使用的外部输入 |
| WebMainBench / WCXB | 独立绑定的冻结 HTML 与参考内容上的提取质量 | [冻结 HTML 说明](../harness/web_fetch/WEBMAINBENCH.md);按原数据许可获取输入,并使用对应认证流程 |
| 项目维护的轨迹任务集 | 在指定公开网页上执行相同固定操作 | [轨迹运行说明](../examples/trajectory_replay/README.md);需要 Docker 和网站访问 |
| WebArena 人工轨迹重放 | 自建网站上的已通过资格检查的固定轨迹与官方任务评估 | [复现说明](../docs/webarena-fixed-trajectory-reproduction.md);需要外部数据和较多部署资源 |
Expand Down Expand Up @@ -122,6 +151,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run --language en

| 仓库 | 在这项工作中的用途 |
| --- | --- |
| [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) | 自建 Web Fetch 评测使用的独立任务集和验收规则 |
| [Moli](https://github.com/lexmount/moli) | 与其他实现一起接受评测的浏览器 |
| [Artifact Site](https://github.com/lexmount/artifact-site) | 报告托管、版本管理和分享,支持自行部署 |
| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 独立评测浏览器协议、自动化驱动和 Web 平台语义 |
Expand All @@ -135,7 +165,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run --language en

## 参与贡献

欢迎为适配器、任务、测试、评分修正、文档和翻译提交 PR。先看[贡献指南](../CONTRIBUTING.md),或提交带有复现示例的 [issue](https://github.com/lexmount/browser-eval/issues)。
欢迎为适配器、执行修复、测试、文档和翻译提交 PR。LexBench 任务要求的修改请提交到 [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch)。先看[贡献指南](../CONTRIBUTING.md),或提交带有复现示例的 [issue](https://github.com/lexmount/browser-eval/issues)。

```bash
make setup-test
Expand All @@ -148,7 +178,7 @@ make ci

## 引用 Browser Eval

在公开作品中使用本仓库的工具、任务定义或评测方法时,请引用仓库,并记录实际使用的 tag 或提交。GitHub 的引用入口读取 [CITATION.cff](../CITATION.cff)。首个版本可这样引用:
在公开作品中使用本脚手架时,请引用仓库,并记录实际使用的 tag 或提交。任务定义与评测方法请另行引用 LexBench Web Fetch 或对应 benchmark。GitHub 的引用入口读取 [CITATION.cff](../CITATION.cff)。首个版本可这样引用:

```bibtex
@software{browser_eval_2026,
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Use this page to find the code responsible for a change. For running evaluations

[Data flow](#data-flow) · [Code owners](#code-boundaries) · [Stored inputs](#repository-and-external-inputs) · [Report releases](#reports-and-publication) · [Review tools](#verification)

Browser Eval benchmarks web fetching, trajectory replay and AI agent automation. These tracks compare task completion, latency and resource use, and share execution and reporting utilities. Each has its own tasks, evaluator and publication target. The current tracks are listed below.
Browser Eval supplies execution, evidence handling and reporting for web fetching, trajectory replay and AI agent automation. Benchmarks supply task populations and acceptance requirements. [LexBench Web Fetch](https://github.com/lexmount/lexbench-webfetch) owns the self-authored Web Fetch suites; see [benchmark boundaries](benchmarks.md). Each experiment has its own evaluator binding and publication target. The supported tracks are listed below.

| Track | Implementation | Running and reproduction |
| --- | --- | --- |
Expand Down Expand Up @@ -56,7 +56,7 @@ through configuration, registries or explicit parameters.
| JSONL framing and finalized/live readers | `harness/record_io.py` |
| Web Fetch task execution and certification | `benchmark_runner.py`, `certification.py`, `reference_lifecycle.py` |
| Browser adapters and results | [Provider contract](provider-contract.md), `harness/web_fetch/providers/` |
| Web Fetch scoring and task acceptance | `scoring.py`, `metrics.py`, task configuration |
| Web Fetch scoring and task acceptance | Pinned `lexbench_webfetch.scoring`, `lexbench_webfetch.metrics`, task configuration |
| Bound artifact copying and aggregation | `artifacts.py`, `evidence.py`, `assembly.py` |
| Reference and scoring corrections | `reference_revision.py`, `scoring_revision.py`, `rescoring.py` |
| WebMainBench certification, scoring and aggregation | `webmainbench_certification.py`, `webmainbench_scoring.py`, `webmainbench_assembly.py` |
Expand Down
22 changes: 22 additions & 0 deletions docs/assets/overview.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading