report(webmainbench): publish complete seven-method content audit

.agents/skills/benchmark-report/references/report-and-release.md

@@ -88,6 +88,7 @@ These rules apply to every public benchmark report, regardless of provider, task
 - **Component denominators:** distinguish “a reference exists” from “that reference contains this component.” If a denominator is the union of reference-side and output-side content, show the counted categories, zero-score additions and pre-scoring exclusions with measured counts. Do not describe an output-only component as a missing reference.
 - **Metric labels:** use parallel names and units (Mean latency, P50 latency, P95 latency). Explain their common population once beside the table. Define P50 as the median and P95 as the 95th percentile at first use; do not attach “successful” only to the mean while silently applying it to all three. Label unavailable values separately from zero.
 - **Highlights:** derive unique directional extrema from raw numeric values and the complete displayed participant registry. Ties remain neutral. Green denotes a column extreme in the declared scope, not statistical significance or an overall ranking. Keep the same markers in all languages, themes, and copies of the comparison. Conditional success coverage and post-hoc shared-success subsets remain explicit.
+- **Table interaction:** choose controls by the table's reader task. Large result and dataset inventories must provide search or relevant filters plus stable sorting for decision-relevant columns. Compact comparison tables keep the complete participant set visible and provide accessible sorting for useful numeric columns; do not add filters that can hide participants and change the apparent comparison population. Small definition, provenance, or lookup tables may remain static. Sortable headers use buttons and `aria-sort`, cells expose raw machine-readable sort values where rendered text is formatted, unavailable values stay after real values in either direction, and ties retain source order. Sorting or filtering must never recompute directional highlights from the visible subset.
 - **One fact, one owner:** use one result summary containing outcome and scalar facts. Distinguish request status, valid output, and task acceptance, but do not repeat them as several summary sections. Keep inventories compact and filterable. Put full output comparisons, scoring payloads, provenance, and logs behind labeled dialogs or disclosures; never flatten them into cards or inventory cells.
 - **Repairable failures:** connect the failure location and program error category to exact error text, retained logs, relevant output or step, reviewed cause, and remediation evidence when recorded. Keep unknown fields explicitly unavailable; a warning or nearby successful action is not automatically the root cause. Preserve complete raw evidence, but do not duplicate a log stream in multiple visible sections.
 - **Evidence interaction:** dialogs must have accessible names, close controls, Escape support, focus restoration, readable bounded scrolling, and usable narrow layouts. Verify a real failure from inventory to exact error/log and back, plus a successful output comparison. Test hosted navigation as well as local paths.

AGENTS.md

@@ -21,8 +21,8 @@ These requirements apply to **all** report tracks; browser-specific rules below
 - Run repository Python only through `make` targets or `uv run --frozen python`. `.python-version`, `uv.lock`, the Makefile and CI are the runtime contract; never fall back to system Python.
 - Run every publishable provider cohort through `scripts/run_certified_benchmark.py`, which freezes references before calls and verifies them again afterward. Direct `run_benchmark.py` output and incomplete cohorts are diagnostic evidence only; never assemble or publish them.
 - For the default Web Fetch report, treat `evaluation/current-report.json` as the matrix and publication-target authority. The independent WCXB report uses only `evaluation/wcxb/current-report.json` for its matrix and target. Browser Use and Browser Trajectory Replay have separate contracts under their own `evaluation/` namespaces; never reuse a publication target across reports or tracks. Publishing must use the relevant bound site and expected remote version; do not copy target IDs from prose, old receipts or earlier reports.
-- For the complete WebMainBench report, `evaluation/webmainbench-full/current-report.json` is the immutable five-method pre-call profile bound by its cohort certificate. The repaired Moli sixth version has its own pre-call profile and certificate at `evaluation/webmainbench-full/moli-fixed-report.json`; the report joins both complete 545-page cohorts by a hash-bound six-method receipt. Their `publication_state` records frozen pre-call state. The live site, share URL, and expected remote version are bound in `evaluation/webmainbench-full/publication-target.json` together with both profile hashes; update this target after publication without changing or re-certifying historical calls.
-- For the current WCXB report, the source-qualified 504-case manifest is the entire evaluation population for quality, delivery, and latency. Keep source-invalid cases outside the current manifest, scored evidence, public package, and future provider calls; preserve earlier physical calls only in private historical evidence. The dataset page owns frozen task inputs, source URLs and human reference scope; it must not show measured browser scores. The home page owns aggregate comparisons, the result analysis owns the filterable method-case inventory, and case result pages own full responses. Keep a direct link from each dataset row to a safe frozen-page preview and its human reference. Default result filters must show a nonempty, clearly labeled diagnostic population when one exists, and filtered tables must shrink rather than leave a large blank viewport.
+- For the complete WebMainBench report, `evaluation/webmainbench-full/current-report.json` is the immutable baseline pre-call profile bound by its cohort certificate. Independently certified variant cohorts are registered in `evaluation/webmainbench-full/`; discover the current variant registry and each variant's profile and certificate from that directory rather than hard-coding names. The method-binding file joins the baseline and all certified variant cohorts through locked profile hashes. Their `publication_state` records frozen pre-call state. The live site, share URL, and expected remote version are bound in `evaluation/webmainbench-full/publication-target.json` together with the baseline and all variant profile hashes; update this target after publication without changing or re-certifying historical calls.
+- For the current WCXB report, the source-qualified manifest is the entire evaluation population for quality, delivery, and latency. Keep source-invalid cases outside the current manifest, scored evidence, public package, and future provider calls; preserve earlier physical calls only in private historical evidence. The dataset page owns frozen task inputs, source URLs and human reference scope; it must not show measured browser scores. The home page owns aggregate comparisons, the result analysis owns the filterable method-case inventory, and case result pages own full responses. Keep a direct link from each dataset row to a safe frozen-page preview and its human reference. Default result filters must show a nonempty, clearly labeled diagnostic population when one exists, and filtered tables must shrink rather than leave a large blank viewport.
 - Persist provider calls outside Git under `~/.browser-eval/` (or `BROWSER_EVAL_DATA_HOME`). Git may contain compact benchmark definitions, annotations, independent source evidence and public release metadata, but never measured outputs, execution manifests, provider artifacts, rerun directories, credentials or internal work logs.
 - Web Fetch keeps one current authoritative logical result per method and task. A declared failure retry retains every physical call, retains total retry execution cost separately from the profile-defined primary latency and uses the final response, never the best response. A proven wrong command, version, scorer, reference or timing is a correction: preserve the prior evidence, supersede the complete declared scope and disclose score, denominator and latency effects.
 - Aggregate only declared, hash-bound evidence files; reject symlinks and never publish worker directories, local paths or internal run trees. Every auxiliary and retry artifact named by a result must be materialized and hash-verified at the aggregate boundary.
@@ -39,7 +39,7 @@ These requirements apply to **all** report tracks; browser-specific rules below
 - Maintain `research/report-changes.json` for substantive releases and `research/provider-pricing.json` for public pricing. Release notes explain reader-visible or result-visible effects without narrating assistant workflow; prices retain verification dates, commitments, billing units and unknowns.
 - Render Web Fetch release history, including WebMainBench and the public benchmark research edition, from `research/report-changes.json`; keep Chinese and English release text together in each entry. Do not duplicate release prose or past measurements in report renderers or exact-string translation maps. Release checks compare structured records with rendered routes rather than matching fixed phrases. Browser Use and trajectory replay keep their existing structured release models.
 - Use `Browser Use` for the autonomous track and `Browser Trajectory Replay` (Chinese: `轨迹重放`) for the fixed BrowserTool-action track. Their implementation roots are `harness/browser_use/` and `harness/trajectory_replay/`; do not add aliases or compatibility paths using the retired names.
-- The public WebArena human-recording replay is isolated under `harness/trajectory_replay/public_webarena/` with its own examples, tests, `evaluation/browser-use-public/current-report.json` and release script. It must not change autonomous `harness/browser_use/` or share a report target or denominator with the repository-curated 30-call replay suite.
+- The public WebArena human-recording replay is isolated under `harness/trajectory_replay/public_webarena/` with its own examples, tests, `evaluation/browser-use-public/current-report.json` and release script. It must not change autonomous `harness/browser_use/` or share a report target or denominator with the repository-curated replay suite.
 - Browser Trajectory Replay canonical inputs must not contain run-local snapshot refs. Compile each ref to a hash-bound stable selector while preserving the original ref action semantics; do not silently replace a DOM ref click/fill/select with BrowserTool's Playwright selector fallback because that adds different viewport, overlay and navigation-wait behavior.
 - Browser Trajectory Replay reports keep the dataset separate from measured verdicts. Every task needs a contract-and-qualified-trajectory page; every result needs a same-task all-browser comparison, a reader-facing failure locator, the exact failed-action or evaluator error, reviewed cause, complete expandable tool responses, and on-demand public evidence. The result inventory stays compact, fixed-height, filterable and linked to a diagnosis dialog. The same-task explorer uses a registry-driven baseline, multi-select comparison browsers and explicit outcome relations; group pass/fail links instead of adding one result column per future browser. Keep Chinese and English task/result routes aligned.
 - A BrowserTool call that returns success does not prove its intended site mutation occurred. When the official evaluator rejects a completed trajectory for an absent required network event, locate the last plausible trigger action and publish the exact missing-event assertion, but do not mark that action as failed or change the official verdict. Repair asynchronous or position-dependent trajectory inputs only through new qualification evidence and a frozen full-scope comparison rerun.

evaluation/webmainbench-full/case-correction-binding.json

@@ -1,5 +1,5 @@
 {
   "schema_version": 1,
-  "relative_path": "evidence/webmainbench-545-origin-correction-20260924",
-  "sha256": "24ec49dba0948b85c0d980b3caf04b016f8fd486ceaa36eef43385b18cb4c9ab"
+  "relative_path": "evidence/webmainbench-545-origin-correction-seven-pr919-e944d066-rescored-20260929",
+  "sha256": "4bd72bf64c56f72566f7085cf6bdbb8107b1e34efe2b0f4845546e6fa0d9fc0e"
 }

evaluation/webmainbench-full/content-binding.json

@@ -1,5 +1,5 @@
 {
   "schema_version": 1,
-  "relative_path": "report-content.json",
-  "sha256": "c5094f07e653f323a2d4914b69a4533f510fe9e0cb9d7fb75041007ddae6e1cb"
+  "relative_path": "seven-method-moli-repairs/report-content.json",
+  "sha256": "92307a6be0ea8ec3bdae29045d19750fc4c66945dd901c9746eb8349647c054b"
 }

evaluation/webmainbench-full/illustrations.json

@@ -2,16 +2,100 @@
   "schema_version": 1,
   "assets": [
     {
-      "name": "overview.png",
-      "sha256": "1166017830d43cf56876e7cfc771d34dcd2b77998cac38d07bb84fa4e869d177",
+      "name": "seven-method-overview.png",
+      "role": "overview",
+      "sha256": "dde509a23ff5e0694df68c7b91974c388ddae3ff9441df6e161caddc6912f06e",
+      "generator": "deterministic HTML/CSS render",
+      "scope": "seven methods after certified origin, reference, TEDS and shared-population corrections",
+      "claims": {
+        "scores_percent": {
+          "moli": "36.41",
+          "lightpanda": "35.00",
+          "kitesurf": "33.43",
+          "chromium": "27.02",
+          "obscura": "31.03",
+          "moli-pr785-fixed": "36.53",
+          "moli-content-fixed": "39.41"
+        },
+        "machine_issues": 25,
+        "request_failures": 15,
+        "empty_outputs": 10,
+        "error_pages": 0
+      }
+    },
+    {
+      "name": "method-error-landscape.png",
+      "role": "quality-risk-landscape",
+      "sha256": "4b32244509a42764602dfbf16b13cb8d8f3351e5f7547b5489292698e124bbb5",
+      "generator": "deterministic HTML/CSS render",
+      "scope": "fixed-population quality thresholds and reviewed delivery failures for seven methods",
+      "claims": {
+        "denominators": {
+          "delivery": 545,
+          "text": 545,
+          "code": 110,
+          "formula": 257,
+          "table": 179,
+          "teds": 179
+        },
+        "reviewed_runtime_issues": 25
+      }
+    },
+    {
+      "name": "seven-method-error-taxonomy.png",
+      "role": "error-taxonomy",
+      "sha256": "a6d63aa9e38545f4b46d507ed7d8b7190cbb3e67879631b08e3f49d34e887c16",
+      "generator": "ImageGen",
+      "scope": "conceptual separation of delivery failures and content-quality gaps across seven methods"
+    },
+    {
+      "name": "moli-repair-mechanisms.png",
+      "role": "repair-mechanisms",
+      "sha256": "9db091bb41d693aa22862f2fe7e8f316ebfaaba1d2d1938fc515df915a8a329a",
+      "generator": "deterministic HTML/CSS render",
+      "scope": "observed defect to shared cause, implementation layer and verification"
+    },
+    {
+      "name": "moli-fix-01-visibility-disclosure.png",
+      "role": "moli-fix-type",
+      "sha256": "d7cb94763bdc1be0d8c479117e7d1e7eccb54d43392cf5d743a14e0e4615b8c1",
+      "generator": "ImageGen",
+      "scope": "visibility, disclosure content and hidden-subtree boundaries"
+    },
+    {
+      "name": "moli-fix-02-fallback-fragments.png",
+      "role": "moli-fix-type",
+      "sha256": "7d1dfec266bd0f6a070fc47e918806dce8fc6bf27d6cd32fbaffa3a3b12f1568",
+      "generator": "ImageGen",
+      "scope": "fallback conversion, escaping and fragment aliases"
+    },
+    {
+      "name": "moli-fix-03-live-form-state.png",
+      "role": "moli-fix-type",
+      "sha256": "852e50109f160588f6df5fe34ac66a39cfa81f341fb6a53b13b40360bbf3a571",
+      "generator": "ImageGen",
+      "scope": "live form state instead of serialized initial attributes"
+    },
+    {
+      "name": "moli-fix-04-media-candidates.png",
+      "role": "moli-fix-type",
+      "sha256": "5d89b58c2ba3618ef0ef04ecd02cba7f57e837c36ee11ea2594c433cec0dbbc4",
+      "generator": "ImageGen",
+      "scope": "standard and lazy media candidate precedence with URL safety"
+    },
+    {
+      "name": "moli-fix-05-semantic-boundaries.png",
+      "role": "moli-fix-type",
+      "sha256": "eadaf97a6703c5649674f75986a200701159ff4c882a48898c2c3a539ebfb6ce",
       "generator": "ImageGen",
-      "scope": "six methods after certified origin correction"
+      "scope": "math, code and table semantic boundaries"
     },
     {
-      "name": "error-mechanisms.png",
-      "sha256": "c53b86571b5c20976f9a54ccfe33d065111732399fceeb4a148c74c520d3d843",
+      "name": "moli-fix-06-parser-resume-ownership.png",
+      "role": "moli-fix-type",
+      "sha256": "720259b73c1cbc7d0eccecc4cc8618cefa09c20902bb5d2fd34500c47716ad70",
       "generator": "ImageGen",
-      "scope": "six methods after certified origin correction"
+      "scope": "parser suspension resume ownership"
     }
   ]
 }

evaluation/webmainbench-full/moli-content-fixed-report.json

@@ -0,0 +1,28 @@
+{
+  "schema_version": 1,
+  "dataset_id": "webmainbench-545-moli-fixed",
+  "comparison_kind": "webmainbench_frozen_html_to_markdown",
+  "case_count": 545,
+  "case_ids_sha256": "5c0cf5810602b70e8f7c1e16723e24a49c3ea207d54865df5c03c38f3d88ffd5",
+  "source_dataset_sha256": "03bc6370a2b37427eba94f95268cf3c935ac89b15b184dfc47f84b78b97fe18e",
+  "providers": ["moli"],
+  "versions": {"moli": "1.1.11"},
+  "binary_sha256": "e146b853bb06ea02bb8847407a76e28eb2bc31ba4ce741af3a7fdaabb194e209",
+  "source_commit": "e944d066bc12493bc05e737f07c5721d5be32cf6",
+  "timeout_seconds": 45,
+  "runs_per_case": 1,
+  "concurrency": 1,
+  "max_failure_retries": 1,
+  "reference_binding_mode": "complete_webmainbench_545_official_frozen_html_and_groundtruth_content",
+  "official_scorer_commit": "f7cb97df2dfb5ac10ebc97cd0b8f9d73ab40f50b",
+  "metric_config": {"use_llm": false},
+  "official_metrics": ["overall", "text_edit", "code_edit", "formula_edit", "table_edit", "table_TEDS"],
+  "input_contract": {
+    "body": "official html field encoded as UTF-8 without mutation",
+    "execution": "inline scripts may run; non-loopback subresources and data access are blocked; local engine receives the same body from an OS-isolated loopback HTTPS fixture with a per-run CA",
+    "original_url": "retained as evidence metadata only"
+  },
+  "publication_state": "pending",
+  "require_independent_result_review": false,
+  "conversion": "Native Markdown from Moli PR #919 after shared fallback semantics, live form state, proven interactive disclosures, bounded one-shot fade-in final-state recovery, media candidate and fragment-target fixes, based on main cb56b3b33"
+}

evaluation/webmainbench-full/moli-variants.json

@@ -0,0 +1,23 @@
+{
+  "schema_version": 1,
+  "variants": [
+    {
+      "provider": "moli-pr785-fixed",
+      "profile": "moli-pr785-fixed-report.json",
+      "evidence_alias": "moli-fixed",
+      "label": "Moli 1.1.9 · PR785 修复版",
+      "short_label": "PR785 修复版",
+      "role": "prior_repair",
+      "pull_request": "https://github.com/lexmount/moli/pull/785"
+    },
+    {
+      "provider": "moli-content-fixed",
+      "profile": "moli-content-fixed-report.json",
+      "evidence_alias": "moli-fixed",
+      "label": "Moli 1.1.11 · PR919 修复版",
+      "short_label": "PR919 修复版",
+      "role": "content_repair",
+      "pull_request": "https://github.com/lexmount/moli/pull/919"
+    }
+  ]
+}

evaluation/webmainbench-full/publication-target.json

@@ -1,9 +1,12 @@
 {
   "schema_version": 1,
   "profile_sha256": "8cbceaf9694e88e7c1ddbba3263573dcccf69b1ac89d7e9ea5aa098f2a3bab02",
-  "moli_fixed_profile_sha256": "db8e7fa408edf3dc2e19d2b0f35645448d0e1efd3ac86926d0bcc9e1a3fb3848",
+  "variant_profile_sha256": {
+    "moli-pr785-fixed": "db8e7fa408edf3dc2e19d2b0f35645448d0e1efd3ac86926d0bcc9e1a3fb3848",
+    "moli-content-fixed": "e3a4b3abd03e29ef3aab13e27e11b614d06f0d822fad86b08c61dbde1501b6bf"
+  },
   "host": "https://artifact-site.lexmount.cn",
   "site_slug": "zrqPjO672dgg",
   "share_url": "https://artifact-site.lexmount.cn/v/Avz3qzo6EEujI0iln1LAzCUF93I47OEC",
-  "expected_remote_version": "ver_8cbf0496af25497283fdcb750756b422"
+  "expected_remote_version": "ver_9440fb2764424af387e7a3c41cf6720f"
 }