Own Web Fetch definitions and standalone evaluation

.github/workflows/ci.yml

@@ -10,16 +10,24 @@ jobs:
         python: ["3.11", "3.12", "3.13", "3.14"]
     steps:
       - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
       - uses: astral-sh/setup-uv@v6
         with:
           version: "0.7.8"
-      - run: uv sync --frozen --python ${{ matrix.python }}
-      - run: uv run --frozen python -m unittest discover -s tests -v
+      - run: uv sync --frozen --extra test --python "$(command -v python)"
+      - run: uv run --frozen --extra test python -m pytest tests -q
       - run: uv run --frozen lexbench-webfetch validate
-      - run: uv build --sdist --wheel
+      - run: uv build --sdist
+      - run: uv build dist/*.tar.gz --wheel
       - name: Validate an isolated wheel installation
         run: |
-          uv venv /tmp/lexbench-wheel --python ${{ matrix.python }}
-          uv pip install --python /tmp/lexbench-wheel/bin/python dist/*.whl
+          uv venv /tmp/lexbench-wheel --python "$(command -v python)"
+          for lexbench_wheel in dist/*.whl; do
+            uv pip install --python /tmp/lexbench-wheel/bin/python "${lexbench_wheel}[test]"
+          done
+          cp -R tests /tmp/lexbench-wheel-tests
           cd /tmp
           /tmp/lexbench-wheel/bin/lexbench-webfetch validate
+          uv run --no-project --no-sync --python /tmp/lexbench-wheel/bin/python python -m pytest /tmp/lexbench-wheel-tests -q

AGENTS.md

@@ -1,7 +1,9 @@
 # Repository guidance
 
-This repository owns Web Fetch task definitions, acceptance contracts, frozen input bindings and independent input tools. Keep browser/provider adapters, generic scoring implementations, execution manifests, measured outputs and report renderers in the execution harness or external evidence storage.
+This repository owns Web Fetch task definitions, acceptance contracts, frozen input bindings and independent scoring/input tools. Keep browser/provider adapters, execution manifests, measured outputs and report renderers in the execution harness or external evidence storage.
 
 Use English for code documentation and contribution guides. Keep README.md and docs/README.zh-CN.md aligned. Preserve original-language tasks, quotes and hash-bound artifacts. Keep prose concise and reader-facing.
 
 Run Python through `uv run --frozen python` or Make. Run `make check` after changes. Tests must cover accepted and rejected input scenarios. Preserve frozen input versions and add provenance for revisions. Never add credentials, third-party captures or run-local evidence to Git.
+
+All code changes go through a feature branch and pull request. Never push directly to main or merge a pull request.

CITATION.cff

@@ -6,4 +6,4 @@ authors:
   - name: LexBench Web Fetch contributors
 repository-code: "https://github.com/lexmount/lexbench-webfetch"
 license: MIT
-version: 0.1.0
+version: 0.2.0

CONTRIBUTING.md

@@ -1,8 +1,8 @@
 # Contributing
 
-Task definitions and acceptance requirements belong here. Browser adapters, execution scheduling, generic scoring code and report rendering belong in the selected execution harness. Include a task ID and a reproducible accepted/rejected example when reporting a rule problem.
+Task definitions and acceptance requirements belong here. Task models, acceptance scoring, file checks and compilation also belong here. Browser adapters, scheduling, reference-capture orchestration and report rendering belong in the selected execution harness. Include a task ID and a reproducible accepted/rejected example when reporting a rule problem.
 
-Use Python 3.11 or newer and [uv](https://docs.astral.sh/uv/). Run `uv sync --frozen`, then `make check`. Tests use synthetic inputs and need no browser, network access or historical source bundle. Keep the English and Chinese READMEs aligned; other guides use English.
+Use Python 3.11 or newer and [uv](https://docs.astral.sh/uv/). Run `uv sync --frozen`, then `make check`. Run `uv sync --frozen --extra test` to install the test tools. Tests use synthetic inputs and need no browser, network access or historical source bundle. Keep the English and Chinese READMEs aligned; other guides use English.
 
 A task revision must explain the intended requirement, preserve the prior input version, and include scenarios that should pass and fail. Update the original dataset, catalog, public task definitions, input manifest and registry together after independent source verification. Do not regenerate old hashes to disguise an input correction. Frozen quotes retain their original language.
 

MANIFEST.in

@@ -1,3 +1,5 @@
 include AGENTS.md CITATION.cff CONTRIBUTING.md SECURITY.md THIRD_PARTY.md provenance.json uv.lock Makefile
 recursive-include docs *.md *.svg
 recursive-include tests *.py
+recursive-include src/lexbench_webfetch/definitions *.json *.md
+recursive-include examples *.json *.txt

Makefile

@@ -1,4 +1,4 @@
 .PHONY: check
 check:
 	uv run --frozen lexbench-webfetch validate
-	uv run --frozen python -m unittest discover -s tests -v
+	uv run --frozen --extra test python -m pytest tests -q

README.md

@@ -60,7 +60,7 @@ uv run --frozen lexbench-webfetch validate
 
 This checks the public definitions without a browser, API key or Browser Eval checkout. It does not fetch websites or compute benchmark scores.
 
-To reproduce a frozen suite, obtain an approved copy of its inputs. The required files and hashes are in `src/lexbench_webfetch/suites/<suite>/source-inputs.json`. Third-party snapshots are not redistributed here; no public download bundle is currently provided.
+To reproduce a frozen suite, obtain an approved copy of its inputs. The required files and hashes are in `src/lexbench_webfetch/definitions/registry.json`. Third-party snapshots are not redistributed here; no public download bundle is currently provided.
 
 ```sh
 uv run --frozen lexbench-webfetch verify-inputs \
@@ -72,6 +72,24 @@ uv run --frozen lexbench-webfetch export \
 
 The export includes only manifest-listed files and prints the executable dataset path. Missing files, changed bytes and symlinks fail validation. Existing output directories are never reused.
 
+## Score a saved answer
+
+The benchmark includes its own scorer. It needs no Browser Eval installation:
+
+```sh
+uv run --frozen lexbench-webfetch score --task examples/task.json --response examples/answer.txt
+uv run --frozen lexbench-webfetch score --task examples/task.json --response examples/missing.txt
+```
+
+The first answer passes; the second fails. Use `--metadata /absolute/path/metadata.json` to include structured answer fields such as `links` or `images`; the file must contain a JSON object. Metadata cannot supply a file-acceptance verdict. Supply the complete task JSON from a verified dataset for a real result. Public task projections bind reference bodies by hash and cannot replace those complete rules. File tasks use repeated `--file /absolute/path/file` arguments; acceptance is derived from the actual bytes, signature, length and declared hash, not a supplied pass flag. Optional ordered-tree tasks require `uv sync --frozen --extra tree`.
+
+Compile the audited access-restriction rules with independently captured references:
+
+```sh
+uv run --frozen lexbench-webfetch compile-access-restrictions \
+  --evidence-home /absolute/path/reference-data-home --output /absolute/path/new-dataset.json
+```
+
 ## Run an evaluation
 
 A runner receives each task's URL and prompt, saves its response or files, and evaluates the declared requirements against the verified reference inputs. Record the benchmark commit, suite ID, input hashes, method versions, execution policy and scorer revision. Follow the [evaluation protocol](docs/protocol.md) when reporting scores or timing.
@@ -92,12 +110,14 @@ Run these two commands from Browser Eval. Ordinary runs are diagnostic; a formal
 
 | Path | Purpose |
 | --- | --- |
-| `src/lexbench_webfetch/suites/` | Task definitions, catalogs and frozen input manifests |
+| `src/lexbench_webfetch/definitions/` | Task definitions, catalogs and frozen input manifests |
 | `src/lexbench_webfetch/benchmark.py` | Independent definition checks and verified input export |
+| `src/lexbench_webfetch/scoring.py` | Authoritative task acceptance and content diagnostics |
+| `src/lexbench_webfetch/risk_control_dataset.py` | Compile audited access-restriction task requirements |
 | `docs/protocol.md` | Acceptance, evidence, timing and result-reporting rules |
 | `tests/` | Accepted/rejected input scenarios and export behavior |
 
-Browser adapters, credentials, measured runs, report renderers and hosting configuration belong to the execution harness or its evidence storage. This repository publishes no measured leaderboard. Changes to task requirements belong here; generic execution or scorer bugs belong in the harness that implements them.
+Browser adapters, credentials, measured runs, report renderers and hosting configuration belong to the execution harness or its evidence storage. This repository owns task models, deterministic acceptance scoring, file-byte checks and task compilation. Browser Eval imports these implementations as a pinned dependency. This repository publishes no measured leaderboard.
 
 ## Contribute and cite
 

docs/README.zh-CN.md

@@ -60,7 +60,7 @@ uv run --frozen lexbench-webfetch validate
 
 这一步检查公开任务定义，无需浏览器、API Key 或 Browser Eval。它不访问网站，也不计算评测分数。
 
-复现冻结任务集需要取得获准使用的原始输入。所需文件及哈希列在 `src/lexbench_webfetch/suites/<suite>/source-inputs.json` 中。本仓库不分发第三方页面快照，目前也没有公开的完整输入下载包。
+复现冻结任务集需要取得获准使用的原始输入。所需文件及哈希列在 `src/lexbench_webfetch/definitions/registry.json` 中。本仓库不分发第三方页面快照，目前也没有公开的完整输入下载包。
 
 ```sh
 uv run --frozen lexbench-webfetch verify-inputs \
@@ -72,6 +72,24 @@ uv run --frozen lexbench-webfetch export \
 
 导出工具只复制清单内的文件，并打印可执行数据集路径。缺文件、内容变化或符号链接都会使校验失败，已有输出目录不会被复用。
 
+## 对保存的响应评分
+
+本仓库自带评分器，无需安装 Browser Eval：
+
+```sh
+uv run --frozen lexbench-webfetch score --task examples/task.json --response examples/answer.txt
+uv run --frozen lexbench-webfetch score --task examples/task.json --response examples/missing.txt
+```
+
+第一份响应通过，第二份失败。若回答含有结构化链接或图片等字段，可通过 `--metadata /absolute/path/metadata.json` 提供，文件内容须为 JSON 对象。元数据不能指定文件验收结果。实际评分时，请使用已校验数据集中的完整任务 JSON。公开任务文件中的参考正文只有哈希绑定，不能替代完整验收输入。文件任务通过可重复的 `--file /absolute/path/file` 提供真实文件；评分器检查字节签名、长度和声明的哈希，不接受调用者自报的通过标记。可选的树结构任务需要先运行 `uv sync --frozen --extra tree`。
+
+使用独立采集的参考输入编译已审定的访问受限任务规则：
+
+```sh
+uv run --frozen lexbench-webfetch compile-access-restrictions \
+  --evidence-home /absolute/path/reference-data-home --output /absolute/path/new-dataset.json
+```
+
 ## 执行评测
 
 执行工具读取任务网址与提示词，保存响应或文件，再按任务规则和已校验的参考输入验收。记录 benchmark 提交版本、任务集 ID、输入哈希、待测方法版本、执行策略和评分器版本。分数与耗时的报告方式见[评测协议](protocol.md)。
@@ -92,12 +110,14 @@ uv run --frozen browser-harness-eval run \
 
 | 路径 | 用途 |
 | --- | --- |
-| `src/lexbench_webfetch/suites/` | 任务定义、目录和冻结输入清单 |
+| `src/lexbench_webfetch/definitions/` | 任务定义、目录和冻结输入清单 |
 | `src/lexbench_webfetch/benchmark.py` | 独立的定义检查与输入导出 |
+| `src/lexbench_webfetch/scoring.py` | 统一的任务验收与内容诊断 |
+| `src/lexbench_webfetch/risk_control_dataset.py` | 将已审定的访问受限任务规则编译为数据集 |
 | `docs/protocol.md` | 验收、证据、耗时和结果报告规则 |
 | `tests/` | 输入接受、拒绝及导出行为测试 |
 
-浏览器适配器、凭据、实测结果、报告生成器和托管配置由执行脚手架或外部证据存储管理。本仓库没有实测排行榜。任务要求的修改在这里维护；通用执行或评分实现的问题在对应脚手架修复。
+浏览器适配器、凭据、实测结果、报告生成器和托管配置由执行脚手架或外部证据存储管理。任务模型、确定性验收评分、文件字节校验和任务编译由本仓库维护，Browser Eval 通过固定版本的依赖直接调用。本仓库没有实测排行榜。
 
 ## 参与贡献与引用
 

docs/data.md

@@ -1,16 +1,16 @@
 # Data layout and rights
 
-The suite registry is `src/lexbench_webfetch/suites/registry.json`. It binds every shipped suite file by SHA-256 and records each population's ID and size.
+The suite registry is `src/lexbench_webfetch/definitions/registry.json`. Its suite directories contain the current inputs. It binds every shipped suite file by SHA-256 and records each population's ID and size.
 
 | File within a suite | Role |
 | --- | --- |
 | `tasks.json` | Public task instructions and acceptance requirements, with body-reference text replaced by SHA-256/character-count bindings, and no full reference bodies or capture logs |
 | `dataset-catalog.json` | Original ordered task inventory, source URLs and complete task-definition hashes |
 | `source-inputs.json` | Exact executable dataset and external reference-file hashes |
 
-The four initial versions were imported from Browser Eval commit `894b38f8`. The full source commit and per-file hashes are in `provenance.json`. Original catalog and manifest bytes are retained, including original dataset IDs and task languages. Their names may contain historical method labels; those labels identify frozen input versions and confer no method-specific acceptance exception.
+The initial versions were imported from Browser Eval commit `894b38f8`. The current access-restriction suite selects the later frozen version declared in the registry. Source commits and per-file hashes are in `provenance.json`. Original catalog and manifest bytes are retained, including original dataset IDs and task languages. Their names may contain historical method labels; those labels identify frozen input versions and confer no method-specific acceptance exception.
 
-Public task requirements are maintained in this repository. Browser Eval retains hash-bound historical records needed by existing experiments; those records preserve old experiments and are not a second source for new task revisions.
+Public task requirements are maintained in this repository. Frozen historical catalogs and manifests also live under `definitions/evaluation/` with their original relative paths and bytes. Browser Eval resolves its historical logical paths through this installed package. It retains experiment profiles and compact hash bindings, rather than duplicate definition files.
 
 ## External inputs
 

docs/protocol.md

@@ -8,7 +8,7 @@ Use the original dataset verified by the suite's `source-inputs.json`. The publi
 
 A method receives the task URL and prompt. Follow the declared task kind: return body text, structured records with the required identities and links, discover the specified resources, or deliver the requested file. A successful HTTP request or nonempty response alone is insufficient. Method-specific alternate task requirements are not allowed.
 
-Task requirements are data: expected markers, fact groups, forbidden output, minimum length, body coverage, structure and resource/file contracts are selected by the task. Missing reference coverage is unavailable, not zero or a perfect score. Binary task acceptance and text-quality diagnostics are separate. Browser Eval implements generic scoring through `score_attempt` and the shared `task_passed` predicate; other harnesses must disclose their evaluator implementation and demonstrate equivalent task-rule behavior before comparing scores.
+Task requirements are data: expected markers, fact groups, forbidden output, minimum length, body coverage, structure and resource/file contracts are selected by the task. Missing reference coverage is unavailable, not zero or a perfect score. Binary task acceptance and text-quality diagnostics are separate. LexBench owns `score_attempt`, the shared `task_passed` predicate and file-byte acceptance. Browser Eval calls the same installed implementations. Other harnesses can use `lexbench_webfetch.evaluation.evaluate_response` or the `score` command without Browser Eval; disclose the benchmark/scorer revision when comparing scores.
 
 ## Live sources and frozen evidence
 

examples/answer.txt

@@ -0,0 +1 @@
+Alpha article

examples/missing.txt

@@ -0,0 +1 @@
+Please sign in to continue

examples/task.json

@@ -0,0 +1 @@
+{"id":"article","name":"Local article","url":"https://example.test/article","category":"web","expected_any":["Alpha article"],"min_content_chars":1}