Skip to content

docs: lead README benchmarks with LexBench WebFetch - #1609

Open
lanyue-llk wants to merge 1 commit into
mainfrom
codex/readme-lexbench-webfetch
Open

lanyue-llk wants to merge 1 commit into
mainfrom
codex/readme-lexbench-webfetch

Conversation

@lanyue-llk

@lanyue-llk lanyue-llk commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The README leads with the historical 192-URL crawl sample. The reviewed LexBench WebFetch comparison is missing from the project overview.

Change

Replace that sample with the public five-browser, 430-task table. Show one Moli result, task counts, pass rates and the full report link. The native Markdown disclosure opens by default. English, Chinese, German, Spanish, French, Japanese and Korean use the same measurements.

This PR targets main directly and changes only seven README files. It does not include #1380's WebMainBench table, benchmark reorganization or runtime changes. Other existing benchmark sections remain unchanged.

Reproduce and validate

Open this branch's README at the benchmark section. The LexBench WebFetch disclosure should already be open, with five rows and exactly one Moli row: 385/430 (89.53%). The report link opens the complete task-level results. Click the disclosure or focus its summary and press Enter to close or reopen it.

All seven editions pass GitHub Markdown rendering checks for the same values, five rows, one Moli result and the default open state. No WebMainBench table or internal build history remains in this change. The result counts are copied from the published report; no evaluation calls or measured evidence were changed. git diff --check passes.

Report: LexBench WebFetch.

Comment thread README.md Outdated
Comment thread docs/README.de.md
@lanyue-llk
lanyue-llk changed the base branch from codex/webfetch-observers to main October 16, 2026 07:14
@lanyue-llk
lanyue-llk force-pushed the codex/readme-lexbench-webfetch branch from 286d531 to 8ca9e66 Compare October 16, 2026 07:14
@github-actions

Copy link
Copy Markdown

Spider Bench A/B

✅ All benchmark browser service runs completed; results are informational.

Public HEAD: 184 / 240 rows; 38 / 48 sites produced rows; 10 unexpected empty sites.

✅ Deterministic fixture contract is clean.

Common ancestor 3c0cdbc5c0b5 → HEAD 8ca9e66a37c0; benchmark order: head-first; workflow: success.

Public 48-site run · informational

Metric Base HEAD Δ
Run status success success —
Extracted rows / contract 179 / 240 184 / 240 +5 (+2.79%)
Contract fill 74.58% 76.67% +2.09 pp (+2.80%)
Sites with rows 37 / 48 38 / 48 +1 (+2.70%)
Sites meeting row contract 35 / 48 36 / 48 +1 (+2.86%)
Unexpected empty sites 11 10 -1 (-9.09%)
Partial-row sites 2 2 0 (0.00%)
Site outcomes extracted: 36; HTTP error + rows: 1; empty: 7; HTTP error + empty: 1; navigation error + empty: 1; snapshot error + empty: 2 extracted: 36; HTTP error + rows: 2; empty: 7; HTTP error + empty: 1; snapshot error + empty: 2 —
Observed duration 57.11 s 50.24 s -6.88 s (-12.04%)
Estimated CPU 50.65 s 48.14 s -2.51 s (-4.96%)
Peak PSS 286.43 MiB 283.83 MiB -2.59 MiB (-0.91%)
Peak RSS 290.05 MiB 287.30 MiB -2.75 MiB (-0.95%)
Peak threads 26 27 +1 (+3.85%)
Mismatched sites 0 0 0
Timeout with items 0 0 0
Category rows Base HEAD Δ
news 35 / 40 35 / 40 0
stocks 20 / 40 20 / 40 0
tech 35 / 40 35 / 40 0
sports 27 / 40 27 / 40 0
games 37 / 40 37 / 40 0
life 25 / 40 30 / 40 +5

Public-site content, timing, and memory are noisy. This single A/B run reports evidence only; site outcome counts explain missing rows without treating them as a deterministic regression.

CPU and memory timelines

These are bounded 40-point views of the same complete process-tree samples used by the HTML report. CPU uses 100% per occupied logical core; memory lines are RSS first and PSS second when complete PSS samples are available.

Base

xychart-beta
    title "Base CPU"
    x-axis "Elapsed seconds" [1.537, 3.539, 5.041, 5.541, 6.542, 8.045, 10.048, 11.55, 13.052, 15.054, 16.055, 16.556, 18.058, 19.56, 21.563, 23.065, 24.567, 25.569, 26.57, 28.073, 29.576, 31.078, 33.081, 34.583, 36.083, 38.085, 39.587, 41.089, 42.589, 44.593, 46.093, 47.593, 49.593, 51.096, 52.596, 54.099, 56.102, 57.103]
    y-axis "CPU percent" 0 --> 500
    line [113.71, 135.94, 159.91, 199.98, 151.75, 115.81, 135.83, 141.79, 149.83, 2, 0, 123.9, 153.83, 2, 133.96, 115.85, 139.87, 107.75, 141.96, 107.75, 145.81, 129.81, 139.8, 109.89, 28.01, 3.99, 121.86, 115.87, 8, 5.99, 0, 6, 7.99, 0, 126.02, 167.73, 109.74, 147.95]
Loading
xychart-beta
    title "Base memory: RSS"
    x-axis "Elapsed seconds" [0.036, 1.537, 3.539, 5.041, 5.541, 6.542, 8.045, 10.048, 11.55, 13.052, 15.054, 16.055, 16.556, 18.058, 19.56, 21.563, 23.065, 24.567, 25.569, 26.57, 28.073, 29.576, 31.078, 33.081, 34.583, 36.083, 38.085, 39.587, 41.089, 42.589, 44.593, 46.093, 47.593, 49.593, 51.096, 52.596, 54.099, 56.102, 57.103, 57.115]
    y-axis "Memory MiB" 0 --> 500
    line [12.66, 103.22, 126.41, 158.79, 174.37, 190.03, 198.51, 190, 176.86, 177.07, 155.6, 155.35, 162.09, 177.79, 148.23, 183.14, 198.02, 165.76, 290.05, 204.61, 226.2, 159.3, 185.58, 172.97, 166.62, 152.78, 154.7, 242.25, 226.24, 181.96, 179.95, 179.96, 179.95, 179.95, 179.95, 170.43, 195.16, 191.57, 0, 0]
Loading

HEAD

xychart-beta
    title "HEAD CPU"
    x-axis "Elapsed seconds" [1.535, 2.538, 3.039, 4.04, 5.543, 7.044, 8.547, 10.048, 11.049, 12.55, 14.053, 15.556, 17.057, 17.558, 18.058, 19.56, 21.063, 22.564, 24.067, 25.568, 26.57, 28.071, 29.573, 31.075, 32.578, 33.579, 35.08, 36.584, 38.084, 39.587, 40.587, 42.089, 43.592, 45.093, 46.596, 47.097, 47.598, 49.099]
    y-axis "CPU percent" 0 --> 500
    line [113.96, 111.7, 163.73, 113.89, 183.57, 153.94, 121.99, 173.8, 139.76, 109.85, 103.8, 71.87, 5.99, 0, 4, 101.89, 17.96, 2, 5.99, 61.89, 69.92, 115.88, 103.81, 99.97, 97.97, 129.81, 71.85, 79.85, 75.93, 7.99, 144.03, 91.85, 7.99, 79.89, 111.75, 209.58, 27.95, 111.96]
Loading
xychart-beta
    title "HEAD memory: RSS then PSS"
    x-axis "Elapsed seconds" [0.035, 1.535, 2.538, 3.039, 4.04, 5.543, 7.044, 8.547, 10.048, 11.049, 12.55, 14.053, 15.556, 17.057, 17.558, 18.058, 19.56, 21.063, 22.564, 24.067, 25.568, 26.57, 28.071, 29.573, 31.075, 32.578, 33.579, 35.08, 36.584, 38.084, 39.587, 40.587, 42.089, 43.592, 45.093, 46.596, 47.097, 47.598, 49.099, 50.236]
    y-axis "Memory MiB" 0 --> 500
    line [12.65, 114.92, 123.57, 137.32, 142.02, 152.49, 190.02, 159.36, 187.13, 170.34, 200.52, 173.02, 166.87, 153.19, 153.47, 152.91, 166.61, 150.9, 150.64, 149.59, 165.59, 174.74, 174.11, 287.3, 211.3, 159.82, 163.07, 193.56, 154.12, 168.17, 151.09, 195.28, 173.58, 151.01, 165.36, 184.88, 168.92, 165.88, 167.46, 0]
    line [9.24, 111.21, 119.87, 133.44, 138.34, 149.13, 186.47, 155.79, 183.58, 166.77, 196.95, 169.46, 163.26, 149.58, 149.87, 149.3, 163.14, 147.3, 147.04, 145.99, 162.22, 171.16, 170.35, 283.83, 207.7, 156.28, 159.36, 189.89, 150.69, 164.68, 147.5, 186.54, 170, 147.42, 161.74, 181.34, 165.82, 162.21, 163.78, 0]
Loading

Deterministic fixture · required contract

The fixture exercises 15 routes: 8 are expected to emit rows and 7 intentionally exercise empty, loading, or timeout behavior.

Metric Base HEAD Δ
Run status success success —
Extracted rows / contract 40 / 40 40 / 40 0 (0.00%)
Contract fill 100.00% 100.00% 0.00 pp (0.00%)
Sites with rows 8 / 15 8 / 15 0 (0.00%)
Sites meeting row contract 15 / 15 15 / 15 0 (0.00%)
Unexpected empty sites 0 0 0
Partial-row sites 0 0 0
Site outcomes extracted: 7; HTTP error + rows: 1; empty: 3; timeout + empty: 4 extracted: 7; HTTP error + rows: 1; empty: 3; timeout + empty: 4 —
Observed duration 19.91 s 20.01 s +0.10 s (+0.52%)
Estimated CPU 0.48 s 0.44 s -0.04 s (-7.90%)
Peak PSS 66.10 MiB 66.18 MiB +0.08 MiB (+0.12%)
Peak RSS 70.16 MiB 70.18 MiB +0.01 MiB (+0.02%)
Peak threads 18 18 0 (0.00%)
Mismatched sites 0 0 0
Timeout with items 0 0 0

Full HTML, JSON, CSV, logs, and page snapshots: workflow run and full spider-bench-results artifact.

The public-web A/B is informational. The exact HEAD fixture contract runs as a separate required CI check; timing and public-site content deltas remain non-blocking.

@github-actions

Copy link
Copy Markdown

CI Regression Report

Source CI run · source state at render: in_progress · artifacts: 6/6

Check Status Signal
Release regression ✅ HEAD/base failures 0/0; raw binary 0.00%
Frontend differential ✅ 1,020/1,020 cases matched; 0 issues
Agent episodes ✅ 8/8 Moli episodes passed; 0 failures
Runtime/CDP contracts ✅ 26 contract cases; 0 failures
CDP smoke ✅ 52/52 groups passed; 621 scenarios
WebMainBench ✅ 545/545 completed; 0 unexpected failures
Release regression — ✅ HEAD/base failures 0/0; raw binary 0.00%

Package and image size

Metric Base HEAD Delta Delta %
Raw binary 153.04 MiB 153.04 MiB 0 B 0.00%
Stripped binary 106.36 MiB 106.36 MiB 0 B 0.00%
gzip binary 52.71 MiB 52.71 MiB −1 B −0.000002%
Rootfs 159.94 MiB 159.94 MiB 0 B 0.00%
gzip rootfs 61.75 MiB 61.75 MiB −2 B −0.000003%

Startup latency and PSS

Case Base p50 HEAD p50 Delta Base PSS HEAD PSS Delta
Serve ready 3.10 ms 27.27 ms +24.17 ms 9.66 MiB 4.32 MiB −5.34 MiB
CDP first page 65.38 ms 50.59 ms −14.78 ms 9.12 MiB 4.34 MiB −4.78 MiB
CDP warm page 17.14 ms 12.35 ms −4.79 ms 50.27 MiB 50.65 MiB +381.00 KiB
CLI about:blank 27.52 ms 15.66 ms −11.86 ms 6.53 MiB 4.36 MiB −2.17 MiB
CLI local JS 18.23 ms 19.47 ms +1.24 ms 4.33 MiB 3.71 MiB −636.50 KiB

Concurrency matrix

Concurrency Base p50 HEAD p50 Delta Base PSS HEAD PSS Delta HEAD failures Unstable B/H
1 17.51 ms 18.06 ms +0.55 ms 747.00 KiB 825.00 KiB +78.00 KiB 0 0/0
5 20.98 ms 21.68 ms +0.69 ms 28.43 MiB 18.67 MiB −9.77 MiB 0 0/0
10 19.77 ms 20.78 ms +1.00 ms 23.61 MiB 27.92 MiB +4.31 MiB 0 0/0
25 19.68 ms 21.47 ms +1.79 ms 13.47 MiB 28.18 MiB +14.71 MiB 0 0/0
100 20.64 ms 21.10 ms +0.46 ms 13.64 MiB 27.80 MiB +14.16 MiB 0 0/0
Frontend differential — ✅ 1,020/1,020 cases matched; 0 issues
Match DOM mismatch Diagnostic mismatch Moli error Reference error Infrastructure error Mismatched frames Duration
1,020 0 0 0 0 0 0 6m 42.88s
Agent episodes · Moli vs Chromium — ✅ 8/8 Moli episodes passed; 0 failures
Fast contract Episodes Passed Assertions Ready p50 Episode p50
Moli 8 8 47/47 4.18 ms 26.60 ms
Chromium 8 8 47/47 275.19 ms 93.57 ms
Operation p50 Chromium Moli Delta Delta %
navigate 44.96 ms 5.76 ms −39.21 ms −87.20%
observe 0.89 ms 0.95 ms +0.06 ms +6.36%
fill 3.98 ms 3.61 ms −0.38 ms −9.46%
click 6.14 ms 4.36 ms −1.78 ms −28.97%
Idle-resource episode Peak RSS Peak PSS Average CPU Peak CPU Peak processes Sampler
Moli 66.65 MiB 63.96 MiB 0.74% 10.00% 1 ✅
Chromium 1287.96 MiB 335.96 MiB 2.00% 93.72% 13 ⚠️ PSS partial
Runtime and CDP session contracts — ✅ 26 contract cases; 0 failures
Suite Cases Failures Median case p50 Median case p95 Median peak PSS p50
Synthetic fetch 13 0 18.68 ms 20.55 ms 647.00 KiB
Long-lived CDP session 13 0 19.58 ms 20.43 ms —
CDP smoke — ✅ 52/52 groups passed; 621 scenarios

Workers: 4 · cumulative group time: 196.63 s · failed groups: 0

Slowest group Scenarios Duration Status
scrollbar-visibility 22 28.59 s ✅
target-lifecycle 24 17.04 s ✅
dom-input 27 13.11 s ✅
multi-page 45 11.46 s ✅
chromium-cdp 56 11.45 s ✅
WebMainBench · 545 pages — ✅ 545/545 completed; 0 unexpected failures
Completed Successful Expected DNS failure Unexpected failures Panics Crashes Timeouts Empty output
545/545 544 1 0 0 0 0 0

Frozen HTML · Linux · --wait done · 45 seconds per page · no retries. Content quality is evaluated separately.

Diagnostics: webmainbench-results — every page's Markdown, stderr, and exit status. Source run and artifacts

All artifact fields are parsed by the trusted default-branch renderer; missing or invalid inputs remain visible as unavailable.

@github-actions

Copy link
Copy Markdown

Sequential Navigation Soak A/B

✅ HEAD completed the 200-navigation resilience and memory observation.

One browser process, one target, and one CDP session navigate CSDN → SegmentFault → Huaban → example.com repeatedly. This run issued 200 Page.navigate commands per binary.

Common ancestor 3c0cdbc5c0 → HEAD 8ca9e66a37; benchmark order: head-first; workflow: success.

Session resilience

Metric Base HEAD
Attempted / planned 200 / 200 200 / 200
Direct observable pass 199 / 200 200 / 200
Failures / recovered 1 / 1 0 / 0
Recovery failures 0 0
Lifecycle order violations 0 0
Network order violations 0 0
Wall time 439.8 s 461.0 s

A public-page failure is reported separately from an unrecoverable session. The soak requires all navigation attempts, zero failed recovery, zero lifecycle/network ordering violations, and complete resource evidence.

Process-tree memory

Metric Base HEAD Δ
Boundary samples 200 200 0
Boundary RSS samples 200 200 0
Boundary PSS samples 200 200 0
Periodic peak RSS 263.3 MiB 242.2 MiB -21.0 MiB
Periodic peak PSS 260.5 MiB 240.2 MiB -20.3 MiB
RSS · first 10 avg 177.6 MiB 173.9 MiB -3.8 MiB
RSS · last 10 avg 196.2 MiB 185.9 MiB -10.3 MiB
RSS · last − first window +18.6 MiB +12.0 MiB -6.6 MiB
RSS · warm slope / 100 nav +3.2 MiB +3.2 MiB -0.0 MiB
PSS · first 10 avg 174.4 MiB 170.9 MiB -3.5 MiB
PSS · last 10 avg 193.5 MiB 182.8 MiB -10.7 MiB
PSS · last − first window +19.1 MiB +11.9 MiB -7.2 MiB
PSS · warm slope / 100 nav +3.5 MiB +3.8 MiB +0.3 MiB
Peak file descriptors 58 62 4
Peak threads 29 31 2

HEAD memory by 50-navigation quarter

HEAD range Avg RSS Final RSS Peak RSS Avg PSS Final PSS Peak FD
1–50 182.6 MiB 165.2 MiB 233.6 MiB 179.3 MiB 161.2 MiB 54
51–100 182.9 MiB 138.5 MiB 237.3 MiB 179.2 MiB 134.9 MiB 55
101–150 189.0 MiB 171.7 MiB 237.8 MiB 186.0 MiB 168.8 MiB 55
151–200 188.1 MiB 175.5 MiB 241.9 MiB 185.1 MiB 172.3 MiB 54

Raw reports and per-navigation boundary samples: workflow run and full sequential-navigation-soak-results artifact.

Public-site timing and memory are observational. Use the A/B deltas and trend shape as evidence, not as a deterministic performance threshold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant