LexBench Web Fetch

What does a fetch actually return?

Versioned tasks and acceptance rules for retrieving web content, structured records and files.

English · 简体中文

Tasks · Quickstart · Evaluation protocol · Contributing

---

LexBench Web Fetch defines what a retrieval system must deliver: page text, named records with usable links, expected resources, or a requested file. The versioned task rules distinguish a successful request from a usable answer.

The benchmark is independent of any browser, provider or execution harness. It has no Browser Eval dependency. Use its JSON task definitions with your own runner, or export verified inputs for Browser Eval.

flowchart LR
    A[Versioned task and acceptance rules] --> B[Your retrieval system]
    B --> C[Saved response or delivered file]
    A --> D[Evaluate the declared requirements]
    C --> D
    E[Hash-bound reference inputs] --> D

Loading

Task suites

Suite	Task entries	Scope
webfetch	150	Web pages, documents, images, video and challenge pages
dynamic-content	100	Content and records loaded after the initial page response
popular-sites	100	Task-relevant content on widely used websites
access-restrictions	100	URLs where access restrictions were previously observed

These are four separately versioned populations, with 450 task entries in total. They are not a deduplicated 450-site leaderboard. The access-restriction label describes historical observations; a URL may behave differently today. WebMainBench, WCXB and WebArena remain separate upstream benchmarks.

Each suite ships a readable tasks.json, a catalog with original task-definition hashes, and a manifest binding the exact executable dataset and reference files. See data layout and rights. Task wording and frozen acceptance rules retain their original language.

Quickstart

Install uv, then:

git clone https://github.com/lexmount/lexbench-webfetch.git
cd lexbench-webfetch
uv sync --frozen
uv run --frozen lexbench-webfetch list
uv run --frozen lexbench-webfetch validate

This checks the public definitions without a browser, API key or Browser Eval checkout. It does not fetch websites or compute benchmark scores.

To reproduce a frozen suite, obtain an approved copy of its inputs. The required files and hashes are in src/lexbench_webfetch/definitions/registry.json. Third-party snapshots are not redistributed here; no public download bundle is currently provided.

uv run --frozen lexbench-webfetch verify-inputs \
  --suite webfetch --source /absolute/path/approved-inputs
uv run --frozen lexbench-webfetch export \
  --suite webfetch --source /absolute/path/approved-inputs \
  --output /absolute/path/new-input-directory

The export includes only manifest-listed files and prints the executable dataset path. Missing files, changed bytes and symlinks fail validation. Existing output directories are never reused.

Score a saved answer

The benchmark includes its own scorer. It needs no Browser Eval installation:

uv run --frozen lexbench-webfetch score --task examples/task.json --response examples/answer.txt
uv run --frozen lexbench-webfetch score --task examples/task.json --response examples/missing.txt

The first answer passes; the second fails. Use --metadata /absolute/path/metadata.json to include structured answer fields such as links or images; the file must contain a JSON object. Metadata cannot supply a file-acceptance verdict. Supply the complete task JSON from a verified dataset for a real result. Public task projections bind reference bodies by hash and cannot replace those complete rules. File tasks use repeated --file /absolute/path/file arguments; acceptance is derived from the actual bytes, signature, length and declared hash, not a supplied pass flag. Optional ordered-tree tasks require uv sync --frozen --extra tree.

Compile the audited access-restriction rules with independently captured references:

uv run --frozen lexbench-webfetch compile-access-restrictions \
  --evidence-home /absolute/path/reference-data-home --output /absolute/path/new-dataset.json

Run an evaluation

A runner receives each task's URL and prompt, saves its response or files, and evaluates the declared requirements against the verified reference inputs. Record the benchmark commit, suite ID, input hashes, method versions, execution policy and scorer revision. Follow the evaluation protocol when reporting scores or timing.

With Browser Eval installed in a separate checkout:

uv run --frozen browser-harness-eval validate \
  --dataset /absolute/path/new-input-directory/dataset.json
uv run --frozen browser-harness-eval run \
  --dataset /absolute/path/new-input-directory/dataset.json \
  --provider-config configs/providers.quickstart.json --providers http

Run these two commands from Browser Eval. Ordinary runs are diagnostic; a formal live-site comparison additionally requires its pre-call/post-call reference certification. The benchmark input checker verifies frozen bytes, not present-day website validity. See the Browser Eval integration guide.

Repository contents

Path	Purpose
src/lexbench_webfetch/definitions/	Task definitions, catalogs and frozen input manifests
src/lexbench_webfetch/benchmark.py	Independent definition checks and verified input export
src/lexbench_webfetch/scoring.py	Authoritative task acceptance and content diagnostics
src/lexbench_webfetch/risk_control_dataset.py	Compile audited access-restriction task requirements
docs/protocol.md	Acceptance, evidence, timing and result-reporting rules
tests/	Accepted/rejected input scenarios and export behavior

Browser adapters, credentials, measured runs, report renderers and hosting configuration belong to the execution harness or its evidence storage. This repository owns task models, deterministic acceptance scoring, file-byte checks and task compilation. Browser Eval imports these implementations as a pinned dependency. This repository publishes no measured leaderboard.

Contribute and cite

Read CONTRIBUTING before changing a task or proposing a result. Run make check to validate the definitions and test the input tools. Report security issues through SECURITY.

Cite LexBench Web Fetch and record the exact commit and suite. Cite any execution harness separately. Project-owned code and definitions use the MIT license; third-party source material retains its own terms, described in THIRD_PARTY.