feat(reporting): bind reports and calls to benchmark editions

CONTRIBUTING.md

@@ -27,6 +27,11 @@ Test both accepted and rejected scenarios. Fix scoring rules rather than editing
 
 Write documentation in English. Keep the [Chinese README](docs/README.zh-CN.md) aligned when setup or usage changes; other docs do not need duplicate translations. Preserve original task text, quotations and frozen acceptance inputs. Reports follow their track's language settings.
 
+Benchmark changes follow [benchmark editions](docs/benchmarks.md#benchmark-editions).
+Keep human-readable editions, immutable dataset/scorer hashes and the package
+commit together. A minor scoring correction requires common-edition rescoring;
+changed task populations or requests require a major edition.
+
 ## PR and commit titles
 
 Use `<type>(<scope>): <summary>` for PR titles and new commit subjects. The scope is optional; use the module or feature affected, such as `scoring` or `webfetch`.

README.md

@@ -106,6 +106,11 @@ Install an extra with `uv sync --frozen --extra NAME`. Browser binaries, Docker,
 
 ## Read and share a report
 
+Reports retain the selected benchmark edition and exact dataset/scorer hashes.
+Package upgrades leave historical results unchanged. See
+[benchmark editions](docs/benchmarks.md#benchmark-editions) for patch, minor and
+major comparison rules.
+
 | Output | What you will find |
 | --- | --- |
 | `report/en/index.html` | English comparisons, task results and evidence links |

docs/README.zh-CN.md

@@ -106,6 +106,10 @@ uv run --frozen browser-harness-eval run --delay 0
 
 ## 查看与分享报告
 
+报告固定绑定所选评测版本，以及题目和评分源码的精确哈希。升级软件包不会改变历史结果。
+补丁版可以直接比较；评分修正的小版本要统一重评；题目或任务要求改变的大版本分别展示。
+具体规则见[评测版本](benchmarks.md#benchmark-editions)。
+
 | 输出 | 内容 |
 | --- | --- |
 | `report/en/index.html` | 英文比较、任务结果和证据入口 |

docs/benchmarks.md

@@ -51,6 +51,58 @@ Change task requirements in LexBench, freeze a new dataset/catalog/manifest vers
 
 The retained `harness/lexbench/` module concerns historical headless-browser comparisons. It is unrelated to the `lexbench-webfetch` input package and remains an execution/report adapter.
 
+## Benchmark editions
+
+Use a benchmark version together with exact task/reference and scoring-source
+hashes. Keep the package's full Git commit pinned separately. A package update
+does not select a new dataset for an existing report.
+
+Use the same `benchmark-vMAJOR.MINOR.PATCH` tag prefix in both repositories.
+Reports reference their benchmark edition; report rebuilds use the hosting
+platform's version ID rather than a separate semantic-version sequence.
+
+| Change | Edition | Result comparison |
+| --- | --- | --- |
+| Documentation or presentation only | Patch | Direct when execution controls also match |
+| Reviewed scoring/reference correction with the same task inputs | Minor | Re-score both saved cohorts under one edition |
+| Added/removed cases, changed requests, output scope or routes | Major | Separate full-population results |
+
+The task package owns `lexbench-webfetch release` and `compare-releases`.
+See its [version contract](https://github.com/lexmount/lexbench-webfetch/blob/benchmark-v2.0.1/docs/versions.md).
+Freeze `benchmark-release.json` beside the compiled inputs before calls.
+The certified Web Fetch runner accepts `--benchmark-release`; it also discovers
+the sidecar beside the selected dataset. Release creation records the full scorer
+commit alongside the independently computed source hash. The runner checks tasks and the active scorer
+before external work and retains the binding in execution metadata. The original
+call edition remains in benchmark-input-release.json; a report may select a newer
+scoring edition after rescoring, retaining both files.
+
+An instance profile can select an edition explicitly:
+
+```json
+{
+  "benchmark_release": {
+    "path": "benchmark-release.json",
+    "sha256": "<SHA-256 of the selected sidecar>"
+  }
+}
+```
+
+Paths are relative to the external profile. The existing build copies this
+edition into the report manifest and package and displays a link beside the
+home-page scope. Checks reject mismatched populations, acceptance rules,
+scoring sources, missing sidecars and incorrect version labels. Historical
+hash-bound reports remain usable without inventing a version label. An
+explicit historical edition preserves the saved scorer identity during a
+presentation rebuild; it does not run the current scorer over old results.
+
+Reuse unchanged saved outputs for a minor correction, including prior
+successes, then recompute scores under one edition. Use the existing revision
+receipts; do not overwrite the older report. Major editions may retain
+diagnostic task-overlap analyses, but their complete throughputs/pass rates
+are separate measurements. Latency still requires matching runtime controls
+and the existing same-window comparison contract.
+
 ## Report instances
 
 Each report track uses its maintained build, check and package entry points.

docs/report-hosting.md

@@ -11,6 +11,13 @@ Make targets. A new instance reuses the same commands with different inputs.
 Historical full-cohort formats use an explicit `--authority-root` bundle that
 retains their original relative paths and hash-bound profile bytes.
 
+An optional `benchmark_release` path/hash binding in the external profile fixes
+the selected benchmark edition. Builds and packages retain
+`benchmark-release.json`; presentation rebuilds keep its dataset and scoring
+identity. See [benchmark editions](benchmarks.md#benchmark-editions) for the
+version and comparison rules. A newer task package must not silently update
+an older report.
+
 ## Report languages
 
 The Web Fetch CLI defaults to English. `run --language zh` generates Chinese only; `report <run> --language en` rebuilds both editions and opens English. A rebuild without `--language` keeps the saved choice, including Chinese-only reports created by older versions.

evaluation/lexbench-webfetch.json

@@ -2,7 +2,7 @@
   "schema_version": 1,
   "kind": "external_benchmark_binding",
   "repository": "https://github.com/lexmount/lexbench-webfetch",
-  "revision": "6bd263ed98030a883e66ac75a47ac4e7fbae4ad1",
+  "revision": "2ea8c52df67c39eee2ea5113a63f5da34f102160",
   "ownership": {
     "benchmark": "Task populations, task model, acceptance scoring, file checks, compiler and frozen definitions",
     "harness": "Execution, reference lifecycle, evidence and reports"
@@ -80,6 +80,7 @@
     "evaluation/risk-control/source-inputs.json": "77c5a4b62c07d59ed57154a833e85f9e7a97bc739730f51cd4ca3a698edb9258",
     "evaluation/scoring-regressions.json": "432ddabe23d045d0ce4c23aef69ca3337ba4d240fe3c2061932cee26b7dc00dc",
     "evaluation/source-inputs.json": "c202abc4624811b938a837402bbaf0ecbe716e375fbca70567346d2edb3b1708",
-    "annotation-revisions/serialization-20261021.json": "e0fe42179e584abb73782cbc459f521cab9cb09d3442de1ec4a6ff54d1b1235a"
+    "annotation-revisions/serialization-20261021.json": "e0fe42179e584abb73782cbc459f521cab9cb09d3442de1ec4a6ff54d1b1235a",
+    "benchmark-release.json": "96ac7f1044601f4a3f768be814a49a48ff46e3ad9802f7f9e17a524460b57c4b"
   }
 }

harness/web_fetch/certification.py

@@ -140,6 +140,8 @@ def main(argv: list[str] | None = None) -> None:
         return
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--dataset", type=Path, default=ROOT / "evaluation/dataset.json")
+    parser.add_argument("--benchmark-release", type=Path,
+                        help="Frozen benchmark edition; checked before any external calls")
     parser.add_argument("--output", type=Path, required=True)
     parser.add_argument("--work-root", type=Path, required=True)
     parser.add_argument("--providers", required=True)
@@ -215,6 +217,16 @@ def main(argv: list[str] | None = None) -> None:
             publication_state="diagnostic",
             unqualified_reference_cases=sorted(unqualified),
         )
+    release_raw = None
+    release_path = args.benchmark_release or args.dataset.parent / 'benchmark-release.json'
+    if args.benchmark_release or release_path.exists():
+        from lexbench_webfetch.scoring import SCORING_SOURCE_SHA256
+        from lexbench_webfetch.versions import verify_release
+        if release_path.is_symlink():
+            parser.error('Benchmark release rejects symlinks')
+        release_raw = release_path.read_bytes()
+        verify_release(json.loads(release_raw), dataset, exact_dataset=False,
+                       scoring_source_sha256=SCORING_SOURCE_SHA256)
     # Reject malformed execution contracts before doing any external captures.
     from harness.web_fetch.reference_lifecycle import (
         requires_browser_observation,
@@ -320,6 +332,13 @@ def main(argv: list[str] | None = None) -> None:
     if args.provider_config:
         benchmark.extend(["--provider-config", str(args.provider_config)])
     run(benchmark_runner.main, benchmark)
+    if release_raw is not None:
+        (args.output / 'benchmark-input-release.json').write_bytes(release_raw)
+        execution_path = args.output / 'execution.json'
+        execution = json.loads(execution_path.read_text())
+        execution['aggregate']['benchmark_release'] = {
+            'path': 'benchmark-input-release.json', 'sha256': hashlib.sha256(release_raw).hexdigest()}
+        execution_path.write_text(json.dumps(execution, ensure_ascii=False, indent=2) + '\n')
 
     # Re-capture HTTP evidence after every provider call has completed, then
     # choose browser fallbacks from that post-call evidence. Reusing the

harness/web_fetch/published_snapshot.py

@@ -64,6 +64,11 @@ def validate_snapshot_audit(source, profile_path):
     records = load_jsonl(inputs['results'])
     receipt = json.loads(inputs['receipt'].read_text())
     audit = json.loads(inputs['review'].read_text())
+    scoring_source = SCORING_SOURCE_SHA256
+    if config.get('benchmark_release'):
+        from .report_version import load_release
+        _, release = load_release(source, dataset, records, profile=profile_path)
+        scoring_source = release['scoring_source_sha256']
     old_cases = {row['id']: row for row in baseline['cases']}
     cases = {row['id']: row for row in dataset['cases']}
     old_rows = {row['attempt_id']: row for row in previous}
@@ -75,7 +80,7 @@ def validate_snapshot_audit(source, profile_path):
     if (receipt.get('kind') != 'published_snapshot_rescore' or receipt.get('provider_calls_added') != 0
             or receipt.get('input_dataset_sha256') != file_sha256(source / 'dataset.json')
             or receipt.get('input_results_sha256') != file_sha256(source / 'results.jsonl')
-            or receipt.get('scoring_source_sha256') != SCORING_SOURCE_SHA256
+            or receipt.get('scoring_source_sha256') != scoring_source
             or receipt.get('corrected_dataset_sha256') != canonical_json_sha256(dataset)):
         raise ValueError('Snapshot audit does not bind the saved calls and current scorer')
     correction_provenance(**config['correction_provenance'])
@@ -96,7 +101,7 @@ def validate_snapshot_audit(source, profile_path):
         if (view['path'] != old['artifact_path'] or view['published_sha256'] != file_sha256(artifact)
                 or view['original_sha256'] != old['metrics']['content_sha256']
                 or row['metrics']['content_sha256'] != view['scored_sha256']
-                or row['metrics']['scoring_source_sha256'] != SCORING_SOURCE_SHA256
+                or row['metrics']['scoring_source_sha256'] != scoring_source
                 or row.get('scoring', {}).get('artifact_view') != view['view']):
             raise ValueError('Snapshot audit artifact identity changed: ' + identifier)
         expected = {'original': view['original_sha256'], 'recovered_original': view['original_sha256'],
@@ -212,6 +217,8 @@ def build_snapshot_audit(source, output, *, profile):
         (staging / 'dataset.json').write_text(json.dumps(dataset, ensure_ascii=False, indent=2)+'\n')
         shutil.copy2(inputs['results'], staging / 'results.jsonl')
         batches = json.loads((staging / 'execution.json').read_text())['batches']
+        from .report_version import bind_release
+        release_binding = bind_release(source, staging, dataset, records, profile=profile)
         generate(staging, dataset, records, batches, authored_interpretation=True, frozen_scores=True)
         write_frozen_result_summary(staging, dataset, records)
         (staging / '.generated-report').write_text('webfetch\n')
@@ -221,6 +228,8 @@ def build_snapshot_audit(source, output, *, profile):
             'results_sha256':file_sha256(staging/'results.jsonl'),
             'generated_languages':list(generated_languages(staging)),
             'score_audit':comparison['score_audit'], 'profile_sha256':file_sha256(profile)}
+        if release_binding:
+            manifest['benchmark_release'] = release_binding
         (staging/'report-manifest.json').write_text(json.dumps(manifest,ensure_ascii=False,indent=2)+'\n')
         check_snapshot_audit(staging)
         _install_report_tree(staging, output)
@@ -241,6 +250,8 @@ def check_snapshot_audit(root):
             raise ValueError('Snapshot audit input changed: '+name)
     dataset = json.loads((root/'dataset.json').read_text())
     rows = load_jsonl(root/'results.jsonl')
+    from .report_version import check_release
+    check_release(root, dataset, rows)
     if dataset.get('definition_revision'):
         revision_path=_bound(root,dataset['definition_revision'])
         historical=root/'audit/history/dataset.json'

harness/web_fetch/report.py

@@ -2215,6 +2215,10 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english
   description=f'{len(cases)} 个案例 · {len(names)} 种方法 · {track_label} 类任务。'
  if local_pass_rate_ranking(dataset):
   description=model_text(f'{len(cases)} 个案例 · {len(names)} 种方法 · 按本轮任务通过率排序。',f'{len(cases)} cases · {len(names)} methods · Ordered by task pass rate for this sample.')
+ release_path=out/'benchmark-release.json'
+ if release_path.is_file():
+  release=json.loads(release_path.read_text())
+  description='<span>'+description+'</span> · <a data-benchmark-release="'+esc(release['version'])+'" href="benchmark-release.json">'+esc(release['benchmark']+' v'+release['version'])+'</a>'
  home='<div class="home-page">'+hero('Web Fetch 评测报告',description)
  from .report_explainer import render_explainer
  TITLE='本轮调用链检查与日志范围' if dataset.get('comparison',{}).get('call_chain_audit') else '本报告的任务范围与验收规则'

harness/web_fetch/report_build.py

@@ -250,7 +250,7 @@ def _install_report_tree(staging: Path, output: Path):
 
 def _build_frozen_presentation(source, output, presentation_binding, *,
                                include_english, language, authored_interpretation,
-                               show_local_status_banners):
+                               show_local_status_banners, profile_path=None):
     """Render a bound historical report using saved scores and public evidence."""
     from harness.report_languages import select_report_language
     from .safety import evidence_tree_identity, copy_evidence_file
@@ -291,6 +291,10 @@ def _build_frozen_presentation(source, output, presentation_binding, *,
                 if relative in before:
                     metadata["public_" + kind + "_path"] = relative
                     metadata["public_" + kind + "_sha256"] = before[relative]
+        from .report_version import bind_release
+        release_binding = bind_release(source, staging, dataset, records, profile=profile_path)
+        if release_binding:
+            manifest['benchmark_release'] = release_binding
         generate(staging, dataset, records, batches, include_english=include_english,
                  authored_interpretation=authored_interpretation,
                  show_local_status_banners=show_local_status_banners, frozen_scores=True)
@@ -380,7 +384,7 @@ def build_report(source: Path, output: Path, *, include_english: bool = False,
         return _build_frozen_presentation(
             source, output, presentation_binding, include_english=include_english,
             language=language, authored_interpretation=authored_interpretation,
-            show_local_status_banners=show_local_status_banners)
+            show_local_status_banners=show_local_status_banners, profile_path=profile_path)
     dataset_probe = json.loads(contained_path(source, 'dataset.json').read_text())
     diagnostic_preview = (diagnostic_preview and
         dataset_probe.get('comparison', {}).get('kind') == 'web_fetch_diagnostic')
@@ -771,6 +775,8 @@ def build_report(source: Path, output: Path, *, include_english: bool = False,
             }, ensure_ascii=False, indent=2))
         from .report_reading import copy_archived_reading
         copy_archived_reading(source,staging)
+        from .report_version import bind_release
+        release_binding = bind_release(source, staging, dataset, records, profile=profile_path)
         generate(staging,dataset,records,manifests,include_english=include_english,
                  authored_interpretation=authored_interpretation,
                  show_local_status_banners=show_local_status_banners)
@@ -782,6 +788,8 @@ def build_report(source: Path, output: Path, *, include_english: bool = False,
             'dataset_sha256': file_sha256(staging/'dataset.json'),
             'results_sha256': file_sha256(staging/'results.jsonl'),
             'generated_languages': generated_languages(staging)}
+        if release_binding:
+            template_manifest['benchmark_release'] = release_binding
         if dataset.get('comparison', {}).get('kind') == 'uniform_native_local_measurement':
             template_manifest.update(evidence_state='local_measurement', certified=False,
                 languages=template_manifest['generated_languages'], provider_calls_added=0)
@@ -1501,6 +1509,8 @@ def validate_local_presentation(output: Path, binding_path: Path):
         require_host_metadata_projection(output, binding['publication_projection'])
     dataset = json.loads((output / 'dataset.json').read_text())
     records = load_jsonl(contained_path(output, 'results.jsonl'))
+    from .report_version import check_release
+    check_release(output, dataset, records)
     if binding.get('publication_evidence_views'):
         from .publication_privacy import require_public_evidence_view_inventory
         require_public_evidence_view_inventory(output, binding['publication_evidence_views'], records)

harness/web_fetch/report_check.py

@@ -300,6 +300,10 @@ def check(root: Path, *, authority_root: Path, profile: Path | None = None, pres
         profile_path = (
             (repository / profile).resolve() if not profile.is_absolute() else profile.resolve()
         )
+    if (root / 'dataset.json').is_file() and (root / 'results.jsonl').is_file():
+        from .report_version import check_release
+        check_release(root, json.loads((root / 'dataset.json').read_text()),
+                      load_jsonl(root / 'results.jsonl'), profile=profile_path)
     if (root / 'report-manifest.json').is_file() and json.loads((root / 'report-manifest.json').read_text()).get('evidence_state') == 'published_snapshot_score_audit':
         from .published_snapshot import check_snapshot_audit
         if profile_path and file_sha256(profile_path) != json.loads((root / 'report-manifest.json').read_text()).get('profile_sha256'):