Skip to content

ci: retest audited Web Fetch regressions fairly - #1429

Closed
lanyue-llk wants to merge 18 commits into
mainfrom
codex/webfetch-comparison-ci
Closed

lanyue-llk wants to merge 18 commits into
mainfrom
codex/webfetch-comparison-ci

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

The earlier Moli 1.1.15-versus-candidate comparison ran on different dates and used an evaluation adapter that changed Moli's native timeout and response-size behavior. It reported 45 pass-to-fail cases, but a complete offline audit found that five were scoring or reference defects: three hidden-menu labels had been treated as visible body text, and two live-editor pages had passed from values present only in source examples while the rendered result still showed Loading.

This adds a manual mode to the existing CI workflow for the remaining 40 cases. It builds the exact requested candidate, verifies the official Moli 1.1.15 archive and binary hashes, and runs both versions serially in case-alternating order. Both use the same native task routes, native internal timeout defaults, a common 45-second outer deadline, one formal attempt per task, and no selector or answer assistance.

If the formal same-window result still contains a stable-pass-to-candidate-fail case, the workflow creates a separate diagnostic cohort and calls both versions once more on that case. It retains every physical call and never replaces the formal verdict with the better attempt. This distinguishes repeatable regressions from transient page or network behavior without selecting scores after the fact.

The source dataset is 04e40811fdfcb2bc83c1ea8f5091c4ea916b8389667411a143062bc2f390d59a. It is derived mechanically from the corrected 430-case evidence and contains exactly the 40 audited regressions: 2 web, 18 dynamic, 10 popular-site/challenge, and 10 risk-control tasks. The parent dataset and results are bound as 6d1274a0c24fe53ffd2511976eacecb8d33df538e612af8ca54b9a0fe2a79539 and 76a40788748eecc7fddc2a7fb0acbfca6ac9b52a6499b7f3ff0886661d55a40b.

The unredacted frozen inputs remain outside Git. The preparation job downloads webfetch-regression-40-scoped-04e40811.tar.gz from the draft release webfetch-regression-ci-input-04e40811, verifies archive SHA-256 ba83c0cb6509ca85e802cda5dc7c1bf09e1a3d485254ca77095e17f28aabf345, rejects unsafe paths and symlinks, verifies the inner dataset hash, and passes the bundle to the comparison job as a one-day Actions artifact. The archive contains only the 40 selected task definitions and their 15 existing task-scoped source directories; unrelated frozen cases and prior provider outputs are excluded.

The runner is pinned to browser-eval commit c6c8bcbd79101ecaf5fee4e22a3e3781d6edab2f from browser-eval #42. It keeps the 20 MB limit on retained final output without turning it into a page or subresource download limit, preserves Moli's native internal timeouts, requires output-section patterns in the final task verdict, and prevents a scorer implementation change from being mixed into a task-rule revision receipt.

The intended candidate remains the top of the product stack at #1432, commit 1d827c940866566a6e63aae189b2103f30eb4dfa. The product stack is #1334 → #1335 → #1336 → #1337 → #1380 → #1428 → #1431 → #1432. This CI PR targets main independently and changes no browser product code.

The workflow uploads the formal evidence, the complete difference inventory, and any diagnostic retry evidence even if a later verification step fails. It fails closed on an incomplete matrix, identity mismatch, overlapping calls, route drift, missing or truncated logs, unsafe evidence paths, or a retry appearing in the formal cohort.

@lanyue-llk
lanyue-llk force-pushed the codex/webfetch-comparison-ci branch from df51935 to 9331d99 Compare October 12, 2026 07:45
@lanyue-llk lanyue-llk changed the title ci: add certified 430-case Web Fetch comparison ci: retest audited Web Fetch regressions fairly Oct 12, 2026
@lanyue-llk lanyue-llk closed this Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant