Repository navigation
feat(fetch): allow explicit interaction before page dump - #1305
Conversation
Spider Bench A/B✅ All benchmark browser service runs completed; results are informational. Public HEAD: 184 / 240 rows; 38 / 48 sites produced rows; 10 unexpected empty sites. ✅ Deterministic fixture contract is clean. Common ancestor Public 48-site run · informational
Public-site content, timing, and memory are noisy. This single A/B run reports evidence only; site outcome counts explain missing rows without treating them as a deterministic regression. CPU and memory timelinesThese are bounded 40-point views of the same complete process-tree samples used by the HTML report. CPU uses 100% per occupied logical core; memory lines are RSS first and PSS second when complete PSS samples are available. Basexychart-beta
title "Base CPU"
x-axis "Elapsed seconds" [2.039, 4.043, 5.544, 7.546, 9.547, 11.55, 12.051, 13.051, 15.054, 17.056, 19.058, 21.061, 22.562, 24.564, 26.565, 28.566, 30.068, 32.072, 34.076, 36.078, 38.08, 39.582, 41.583, 43.584, 45.587, 47.091, 47.591, 49.094, 51.097, 53.099, 55.102, 56.604, 58.605, 60.608, 62.609, 64.11, 64.611, 66.111]
y-axis "CPU percent" 0 --> 500
line [129.76, 119.77, 151.92, 149.88, 93.89, 143.76, 0, 0, 0, 0, 0, 0, 127.86, 125.96, 13.99, 21.96, 73.89, 9.99, 139.6, 95.9, 4, 0, 0, 0, 0, 105.82, 105.82, 101.79, 49.94, 114, 103.7, 51.93, 147.85, 23.96, 49.99, 99.81, 197.74, 138.01]
xychart-beta
title "Base memory: RSS then PSS"
x-axis "Elapsed seconds" [0.036, 2.039, 4.043, 5.544, 7.546, 9.547, 11.55, 12.051, 13.051, 15.054, 17.056, 19.058, 21.061, 22.562, 24.564, 26.565, 28.566, 30.068, 32.072, 34.076, 36.078, 38.08, 39.582, 41.583, 43.584, 45.587, 47.091, 47.591, 49.094, 51.097, 53.099, 55.102, 56.604, 58.605, 60.608, 62.609, 64.11, 64.611, 66.111, 67.709]
y-axis "Memory MiB" 0 --> 500
line [12.07, 113.9, 138.68, 155.94, 164.48, 167.07, 155.63, 155.63, 141.7, 140.69, 140.61, 140.61, 140.58, 159.79, 144.34, 149.83, 166.02, 150.57, 142.55, 173.54, 157.57, 139.01, 139.01, 139.01, 139.01, 139.01, 232.84, 297.35, 210.36, 159.2, 173.37, 156.15, 152.5, 214.1, 174.65, 178.13, 189.9, 182.23, 184.96, 0]
line [8.73, 110.22, 135.08, 152.24, 160.81, 163.1, 151.95, 151.95, 138.02, 137, 136.93, 136.93, 136.9, 155.83, 140.65, 146.12, 162.31, 146.87, 138.84, 169.95, 153.87, 135.3, 135.3, 135.3, 135.3, 135.3, 229.18, 294.85, 206.66, 155.44, 169.59, 152.52, 148.77, 207.89, 170.97, 174.41, 187.65, 175.54, 181.23, 0]
HEADxychart-beta
title "HEAD CPU"
x-axis "Elapsed seconds" [2.039, 4.542, 5.042, 6.544, 8.547, 10.55, 13.052, 15.055, 17.056, 19.058, 21.562, 23.566, 25.568, 27.57, 30.074, 32.076, 34.078, 36.078, 38.582, 40.582, 42.586, 44.589, 47.093, 49.095, 51.098, 53.099, 53.6, 55.601, 57.604, 59.607, 61.61, 64.113, 66.115, 68.117, 70.121, 72.623, 74.626, 75.628]
y-axis "CPU percent" 0 --> 200
line [113.77, 3.99, 0, 2, 115.77, 145.95, 89.89, 151.76, 4, 2, 0, 0, 57.99, 151.72, 7.98, 165.78, 4, 39.99, 0, 58.05, 0, 0, 2, 2, 2, 103.91, 105.82, 131.96, 139.82, 141.8, 147.79, 101.94, 49.93, 69.95, 17.97, 165.64, 143.75, 189.64]
xychart-beta
title "HEAD memory: RSS then PSS"
x-axis "Elapsed seconds" [0.036, 2.039, 4.542, 5.042, 6.544, 8.547, 10.55, 13.052, 15.055, 17.056, 19.058, 21.562, 23.566, 25.568, 27.57, 30.074, 32.076, 34.078, 36.078, 38.582, 40.582, 42.586, 44.589, 47.093, 49.095, 51.098, 53.099, 53.6, 55.601, 57.604, 59.607, 61.61, 64.113, 66.115, 68.117, 70.121, 72.623, 74.626, 75.628, 76.207]
y-axis "Memory MiB" 0 --> 500
line [12.13, 137.28, 111.86, 111.86, 111.86, 127.24, 162.18, 167.32, 165.79, 144.66, 144.56, 144.48, 144.39, 159.57, 181.92, 138.26, 154.69, 138.41, 140.74, 137.72, 173.77, 145.25, 144.26, 144.26, 144.26, 144.26, 218.88, 280.44, 204.88, 184.61, 184.53, 168.12, 152.25, 158.2, 196.2, 166.09, 188.55, 172.41, 191.6, 0]
line [8.79, 133.71, 108.16, 108.16, 108.16, 123.59, 158.49, 163.77, 161.94, 140.98, 140.88, 140.8, 140.71, 155.93, 178.86, 134.76, 152.19, 134.71, 137.03, 134.01, 170.06, 141.52, 140.54, 140.54, 140.54, 140.54, 215.2, 277.35, 201.24, 180.22, 180.94, 164.4, 148.52, 154.49, 190.81, 162.36, 185.93, 168.7, 187.98, 0]
Deterministic fixture · required contractThe fixture exercises 15 routes: 8 are expected to emit rows and 7 intentionally exercise empty, loading, or timeout behavior.
Full HTML, JSON, CSV, logs, and page snapshots: workflow run and full The public-web A/B is informational. The exact HEAD fixture contract runs as a separate required CI check; timing and public-site content deltas remain non-blocking. |
Sequential Navigation Soak A/B✅ HEAD completed the 200-navigation resilience and memory observation. One browser process, one target, and one CDP session navigate CSDN → SegmentFault → Huaban → example.com repeatedly. This run issued 200 Common ancestor Session resilience
A public-page failure is reported separately from an unrecoverable session. The soak requires all navigation attempts, zero failed recovery, zero lifecycle/network ordering violations, and complete resource evidence. Process-tree memory
HEAD memory by 50-navigation quarter
Raw reports and per-navigation boundary samples: workflow run and full Public-site timing and memory are observational. Use the A/B deltas and trend shape as evidence, not as a deterministic performance threshold. |
CI Regression ReportSource CI run · source state at render:
Release regression — ✅ HEAD/base failures 0/0; raw binary +0.000283%Package and image size
Startup latency and PSS
Concurrency matrix
Frontend differential — ✅ 1,020/1,020 cases matched; 0 issues
Agent episodes · Moli vs Chromium — ✅ 8/8 Moli episodes passed; 0 failures
Runtime and CDP session contracts — ✅ 26 contract cases; 0 failures
CDP smoke — ✅ 52/52 groups passed; 619 scenariosWorkers: 4 · cumulative group time: 188.33 s · failed groups: 0
WebMainBench · 545 pages — ✅ 545/545 completed; 0 unexpected failures
Frozen HTML · Linux · Diagnostics: All artifact fields are parsed by the trusted default-branch renderer; missing or invalid inputs remain visible as unavailable. |
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
审查版本：cc89392f56f7549041c9efff71055f190fb4a5db。结论：建议修改后再合入。确认两项功能缺陷，已分别留下行内评论。
需求是否真实、是否应改 moli
--eval与--dump互斥是真实的 CLI 限制；允许明确交互后复用标准导出器，符合 moli 的浏览器自动化定位。无需增加网站特例、猜测 handler 或自动点击所有按钮，这个组合方案总体够小、通用。- 但 WebMainBench
8798f93a-240c-46d7-a8d1-a5fb011279fc（mcubed 的 Mets 连胜/连败页）不证明 Markdown 缺少这项能力。冻结 HTML 确有 91 个listN，页面脚本用hideallids()隐藏它们；详情已经在 DOM 中。准确 PR 二进制对同一冻结 HTML 默认导出、显式点击 91 个展开链接后导出，均为 53,412 字节、91 个WINNER，逐字节相同，SHA256 均为f807ca1ab1dbfa0ce99dbbcb93528eaa22a377014484f2f90901f1911eb6f76d。这与moli-html2md/src/visibility.rs保留折叠详情的设计及现有测试一致。 - 因而不需要为“导出这页已有的比赛明细”修改 moli；这个 PR 应定位为通用 CLI 组合改进。现有
--wait-script也可以显式操作再导出，不过把一次准备动作放在 eval 中，比把副作用塞进 readiness predicate 更清楚。请更正描述中该案例与能力缺口的因果关系；如要以真实网站证明新增价值，应补充交互后才创建/加载正文的案例及完整复现条件。不能将这项功能当作已有只读成绩的修复。
仓库要求、实现与回归
- 标题、使用说明和 7 个文件的范围清楚；仓库未提供额外 PR 模板。Promise、异常、返回值和评测边界写明了，但正文对“返回值被丢弃”的承诺与实现不符，导航行为也缺少说明。
- 参数来源互斥、纯 eval 的无效正文选项拒绝、raw 文档拒绝、标准导出器复用均合理，未发现网站耦合或为该数据集定制的代码。原有值输出路径保留。
- 新增 DOM 返回值测试不足以证明任意返回值被忽略；同页 DOM 修改测试不足以覆盖点击链接、表单或 location 导航。需要按这些用户场景补回归测试。
验证证据
当前提交的 CI Format、Clippy、Nextest 全部成功；Nextest 日志为 19,701 passed、16 skipped。本机准确 PR 版本格式和 diff 检查通过，20 个 eval 相关测试、82 个 CLI 测试通过；额外实际 CLI 测试覆盖两种脚本来源 × HTML/Markdown/JSON、同步抛错/Promise 拒绝、循环返回对象、getter、同源导航和冻结比赛页。
最小修正是在 execute 中复用已有 evaluate_runtime_expression_with_await_async(expression, true)，保留纯值输出的 ByValue 路径。我在隔离副本验证了这个单行替换：循环对象不再失败、返回对象 getter 不执行、导航导出目标页面，原有 20 个 eval 测试和异常零 stdout 契约继续通过。无需新建底层 API。
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
复审版本：cfd1087dbe47b90f8d692c17ac0acf23f750eafe。本轮扩大检查后，确认剩余 1 项 P2：交互进入目标页后，load handler 发起的下一次导航会被遗漏，详见行内评论。建议修正此项后再合入。
上轮两项问题已修复：准备执行不再按值序列化普通返回对象；首次交互导航可导出目标页。循环对象、普通 getter、Promise 返回对象及链接/location 导航已有对应回归测试。PR 描述也已正确区分 CLI 组合能力与比赛页已有内容，不再将那 91 组明细当作默认采集缺陷的依据。
本轮没有只重跑新增测试，另外检查和实测了：
--eval、--eval-file、stdin；HTML、Markdown、JSON、两种 semantic tree、截图、全页截图、PDF。- 表单 requestSubmit、导航前 await 定时器、history.pushState、HTTP 302 跳转、目标页 load 再跳转及连续两次 load 跳转。
- JSON 最终 URL、状态、标题和正文；交互/导航后截图的实际颜色像素，确认导出的是更新后的页面。
--with-base、--with-frames、strip、network trace、disable-js 组合。- 同步异常、Promise 拒绝、异常同时请求导航；204/205、下载响应、404 和 JSON 目标。204/205/下载保留原文档是既有正确语义；失败路径均不输出 dump。
准确提交的仓库 CI 全部成功，本机 Format/diff 检查通过。使用仓库要求的 Nextest 对全部 fetch CLI/参数测试运行，249/249 通过。普通 cargo test 的共享进程方式会因全局日志订阅及并发时序使其他用例互相干扰，独立进程 Nextest 排除了这项干扰；它不是此 PR 的回退。
新发现已做根因与修正验证：直接 fetch 中间页会继续到最终页，但通过 eval 进入同一页时停在中间页，且 HTML 中已出现 load handler 写入的标记，证明 handler 执行过、只是导航观察器提前返回。在隔离副本给 Follow 续跑路径复用现有生命周期/待导航 reconciliation 后，两种表达式来源 × HTML/Markdown/JSON，以及连续两次 load 跳转均正确输出最终正文，249 项回归测试继续全部通过。无需定时等待、按钮特例或新导航 API。验证改动已撤回，没有提交或推送代码。
b682173 to
1fa7cb1
Compare
2 participants
问题
fetch --eval能执行页面交互，却只能输出表达式的值；fetch --dump能输出页面，但两者互斥。调用者无法直接组合一次脚本执行与标准 Markdown、HTML 或 JSON 导出，例如通过按钮创建正文、等待异步交互完成，或者导航至目标页面后导出。这是通用 CLI 组合能力，与 #919 的内容转换、CSS 和 parser 修复独立。
--wait-script可以执行副作用，但它是可能重复执行的就绪判定；明确的一次性操作应使用--eval。本 PR 基于 main 的默认 Markdown 导出已经保留 WebMainBench8798f93a…的 91 组明细，展开前后输出相同，因此该案例不能证明本分支默认采集缺失正文，也不是此 PR 修复成绩的依据。方案
允许组合已有的
--eval/--eval-file与--dump，复用现有等待 Promise、跟随导航的执行接口和标准导出路径。初始加载与交互后的导航续跑共享同一规则：即使达到 load，若该事件已排队后续导航，仍先完成下一跳，再导出最终文档：查看完整机制图
moli fetch --eval 'document.querySelector("button.show-details").click()' \ --dump markdown https://example.com--dump时，表达式值的原有输出契约保留；此时拒绝正文专用参数，避免静默忽略。b21b4f05615c1b9aa71ea2074084646895d82be7，没有搬入 fix(fetch): preserve rendered document content #919 的 renderer / html2md 修改，也没有新增底层执行接口。验收
公开 CLI 测试覆盖：
组合能力的旧二进制复现为参数冲突。原提交
cc89392的仓库 CI 全部通过，包括 Format、Clippy 和 Nextest。两条 review 问题在原实现上分别复现了返回值序列化失败/副作用和导航后仍导出原页；复用现有接口后，独立构建二进制的 20 个 eval 测试以及 11 个新增回归场景通过。修复提交cfd1087的仓库 CI 已全部通过，包括 Format、Clippy、Nextest、545 页 WebMainBench、协议冒烟及导航检查。导航续跑修复
b682173复用既有生命周期判断，没有引入延时或新导航 API。同一修复的独立构建已通过 249 项 CLI 回归及六种连续导航组合；提交b682173的 19 项 CI 检查已全部通过，包括 Nextest 和 545 页 WebMainBench。评测边界
此 PR 不改变已发布的 545 页成绩。若比较交互采集，应预先固定同一操作和任务范围，对所有支持交互的方法执行一致流程，单独认证并展示；不能仅给 Moli 添加交互后替换正式报告结果。已有七方法报告继续绑定其原始调用、二进制和参考。