Repository navigation
ci: retest audited Web Fetch regressions fairly - #1429
Closed
lanyue-llk wants to merge 18 commits into
Closed
lanyue-llk wants to merge 18 commits into
lanyue-llk wants to merge 18 commits into
Conversation
This was referenced Oct 12, 2026
lanyue-llk
force-pushed
the
codex/webfetch-comparison-ci
branch
from
October 12, 2026 07:45
df51935 to
9331d99
Compare
added 6 commits
October 12, 2026 16:01
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The earlier Moli 1.1.15-versus-candidate comparison ran on different dates and used an evaluation adapter that changed Moli's native timeout and response-size behavior. It reported 45 pass-to-fail cases, but a complete offline audit found that five were scoring or reference defects: three hidden-menu labels had been treated as visible body text, and two live-editor pages had passed from values present only in source examples while the rendered result still showed
Loading.This adds a manual mode to the existing CI workflow for the remaining 40 cases. It builds the exact requested candidate, verifies the official Moli 1.1.15 archive and binary hashes, and runs both versions serially in case-alternating order. Both use the same native task routes, native internal timeout defaults, a common 45-second outer deadline, one formal attempt per task, and no selector or answer assistance.
If the formal same-window result still contains a stable-pass-to-candidate-fail case, the workflow creates a separate diagnostic cohort and calls both versions once more on that case. It retains every physical call and never replaces the formal verdict with the better attempt. This distinguishes repeatable regressions from transient page or network behavior without selecting scores after the fact.
The source dataset is
04e40811fdfcb2bc83c1ea8f5091c4ea916b8389667411a143062bc2f390d59a. It is derived mechanically from the corrected 430-case evidence and contains exactly the 40 audited regressions: 2 web, 18 dynamic, 10 popular-site/challenge, and 10 risk-control tasks. The parent dataset and results are bound as6d1274a0c24fe53ffd2511976eacecb8d33df538e612af8ca54b9a0fe2a79539and76a40788748eecc7fddc2a7fb0acbfca6ac9b52a6499b7f3ff0886661d55a40b.The unredacted frozen inputs remain outside Git. The preparation job downloads
webfetch-regression-40-scoped-04e40811.tar.gzfrom the draft releasewebfetch-regression-ci-input-04e40811, verifies archive SHA-256ba83c0cb6509ca85e802cda5dc7c1bf09e1a3d485254ca77095e17f28aabf345, rejects unsafe paths and symlinks, verifies the inner dataset hash, and passes the bundle to the comparison job as a one-day Actions artifact. The archive contains only the 40 selected task definitions and their 15 existing task-scoped source directories; unrelated frozen cases and prior provider outputs are excluded.The runner is pinned to browser-eval commit
c6c8bcbd79101ecaf5fee4e22a3e3781d6edab2ffrom browser-eval #42. It keeps the 20 MB limit on retained final output without turning it into a page or subresource download limit, preserves Moli's native internal timeouts, requires output-section patterns in the final task verdict, and prevents a scorer implementation change from being mixed into a task-rule revision receipt.The intended candidate remains the top of the product stack at #1432, commit
1d827c940866566a6e63aae189b2103f30eb4dfa. The product stack is #1334 → #1335 → #1336 → #1337 → #1380 → #1428 → #1431 → #1432. This CI PR targetsmainindependently and changes no browser product code.The workflow uploads the formal evidence, the complete difference inventory, and any diagnostic retry evidence even if a later verification step fails. It fails closed on an incomplete matrix, identity mismatch, overlapping calls, route drift, missing or truncated logs, unsafe evidence paths, or a retry appearing in the formal cohort.