Skip to content

feat(webfetch): support reusable seven-family native comparisons - #42

Merged
lanyue-llk merged 172 commits into
mainfrom
codex/webfetch-430-refresh
Oct 16, 2026
Merged

lanyue-llk merged 172 commits into
mainfrom
codex/webfetch-430-refresh

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

The Web Fetch harness could not fairly run and explain a browser-native comparison across seven task families. Native Markdown, rendered HTML, embedded resources, direct files, retries, scoring, and failure review were handled by separate paths, and the first implementation also stored one report's profile, rerun scope, release prose, and participant identities in Git.

This change makes those boundaries reusable:

  • Route webpage and document-body tasks through each browser's declared native content interface; Chromium alone uses the fixed HTML-to-Markdown converter. Embedded-media tasks inspect addresses already present in that output, while direct-media tasks verify original bytes. The harness does not switch interfaces after a failure or add an external document parser.
  • Keep dynamic completion at task-family or registered-contract scope. Per-case selectors, answer-dependent waits, and provider-specific scoring branches remain rejected.
  • Preserve visible record ownership through HTML and Markdown normalization. Image alternative text cannot replace a required visible name, and fields from adjacent cards, lists, wrappers, or asides cannot complete another record.
  • Normalize bounded main-navigation transport failures before the shared retry decision, preserve native DNS context for attachment calls, retain explicit loopback fixtures, and keep experiment-specific DNS settings out of portable defaults.
  • Build the seven-family, 430-case dataset from hash-bound reusable sources and corrections. A report profile is now an explicit external input; the repository no longer stores this report's profile or rerun receipt.
  • Read report release history and pricing overrides from hash-bound report-assets when supplied. The root release registry is a legacy fallback, and the shared pricing catalog contains product facts rather than one run's wording.
  • Use generic Moli release/candidate configuration slots. Actual versions, commits, executable hashes, review payloads, prose, and publication state are bound by external evidence.

The published report remains on Artifact Site, with only a compact version/profile-hash archive binding in Git: https://artifact-site.lexmount.cn/v/IqGpMUYHWat-dw4n8usQttTptoPzboAo

Reproduce

uv run --frozen python scripts/assemble_webfetch_dataset.py \
  --assembly evaluation/webfetch-seven-family-430/dataset-assembly.json \
  --profile <external-profile.json> \
  --output <new-dataset.json>

The external profile declares participants, case inventory, run policy, executable identities, and report output. The assembly rejects a missing profile, stale source hash, mismatched case ownership, unregistered correction control, or report-local input embedded in the reusable authority.

Validation

  • Focused architecture, scoring, report-content, pricing, and participant tests: 83 passed, 14 subtests passed.
  • All 240 evaluation, research, and configuration JSON files parse successfully.
  • Ruff F checks and git diff --check pass.
  • The existing public report remains unchanged; this review fix moves its private instance data out of the repository and does not rerun browser calls or alter stored results.

@lanyue-llk lanyue-llk changed the title fix(webfetch): preserve cohort contracts and bound Markdown parsing Fix frozen Web Fetch reruns and native dynamic execution Oct 9, 2026
@lanyue-llk lanyue-llk changed the title Fix frozen Web Fetch reruns and native dynamic execution Fix frozen browser benchmark reruns and fresh report cohorts Oct 9, 2026
@lanyue-llk
lanyue-llk force-pushed the codex/webfetch-430-refresh branch from 1a7f2e3 to da3f01b Compare October 9, 2026 16:44
@lanyue-llk lanyue-llk changed the title Fix frozen browser benchmark reruns and fresh report cohorts Correct native Web Fetch routes and publish five-browser 430-task report Oct 9, 2026
@lanyue-llk
lanyue-llk marked this pull request as ready for review October 9, 2026 18:12
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@lanyue-llk lanyue-llk changed the title Correct native Web Fetch routes and publish five-browser 430-task report 扩充五浏览器430题评测并硬化原生接口与发布规则 Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@lanyue-llk lanyue-llk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查结论:建议修正下列问题后再合入。

本 PR 将 Web Fetch 扩展为五浏览器、七类任务、每浏览器 430 题,并统一原生接口、正文评分和发布校验。共享任务投影与统一路由选择器是合适的抽象;保留动态任务分类也合理,但调用校验没有同步迁移。资源投影还把单个 URL 当成 srcset 列表解析,导致合法资源地址损坏。

确认两处功能回退,均做了基线对照:目标分支通过,PR 失败;在隔离副本中验证最小修复后,相关测试 63 项通过。现有相关测试分两组共 478 项通过、13 项跳过;启用四种真实本地浏览器后,资源、取消和文件测试 24 项通过、1 项失败,失败原因见第三条。进程清理失败后的隔离行为符合现有安全合同,不列为缺陷。

这次审查覆盖代码、调用合同及相关测试,不改变报告中独立逐结果复核 0/2,150 的范围。

Comment thread harness/web_fetch/retrieval_hints.py
Comment thread harness/web_fetch/task_outputs.py Outdated
Comment thread tests/test_headless_files.py Outdated
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Claude encountered an error after 3s —— View job


I'll analyze this and get back to you.

@lanyue-llk
lanyue-llk force-pushed the codex/webfetch-430-refresh branch from 7607b18 to 2166c94 Compare October 10, 2026 08:29
@lanyue-llk lanyue-llk changed the title 扩充五浏览器430题评测并硬化原生接口与发布规则 Use native Web Fetch interfaces and frozen benchmark inputs Oct 10, 2026
@lanyue-llk
lanyue-llk force-pushed the codex/webfetch-430-refresh branch from cd0b669 to 0598c6a Compare October 10, 2026 11:08
@lanyue-llk lanyue-llk changed the title Use native Web Fetch interfaces and frozen benchmark inputs Use native Web Fetch interfaces and publish the complete 430-task comparison Oct 10, 2026
@lanyue-llk
lanyue-llk force-pushed the codex/webfetch-430-refresh branch from 85f8524 to caa269e Compare October 12, 2026 03:19
@lanyue-llk lanyue-llk changed the title Use native Web Fetch interfaces and publish the complete 430-task comparison Use native Web Fetch interfaces for the complete 430-task comparison Oct 13, 2026
@lanyue-llk
lanyue-llk force-pushed the codex/webfetch-430-refresh branch from 06b86f4 to b61c880 Compare October 16, 2026 09:03
Comment thread evaluation/risk-control/history/v2/dataset-catalog.json Outdated
Comment thread research/browser-native-430-fairness-pre-call-revision.json Outdated
Comment thread research/provider-pricing.json
@lanyue-llk lanyue-llk changed the title feat(webfetch): publish reviewed six-method 430-case report feat(webfetch): support reusable seven-family native comparisons Oct 16, 2026
@lanyue-llk
lanyue-llk merged commit 90c4a2f into main Oct 16, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant