ci: expose test stall stacks and bound fixture progress IDs

[loopx-agent]

Long pytest runs need an observable executing stack, while large implicit parameter IDs can expand one progress line to hundreds of thousands of characters. This keeps diagnostic output in the existing pytest timing provider and gives the affected text-boundary cases short labels.

The backpressure defect reported in the exact-head review is addressed: the original observer blocked in os.write and pytest waited indefinitely in timer.join() when stderr stopped being read. The same real unread-pipe fixture now finishes without draining it, preserving native passing/failing results.

• --ci-stall-after 120 captures up to 24 frames from the executing test thread through its original stderr destination, without reading locals. A minimal isolated writer uses the selected interpreter, a 100ms self-exit deadline and a 500ms parent communication timeout with kill/reap. The writer also self-exits if only pytest is killed. Shared stderr flags are unchanged; timers drain before the parent descriptor closes. Native process startup can add latency, and failed/backpressured snapshots may be dropped.
• The option defaults to 0, which creates no timer/writer. Full Linux shards explicitly opt in. Six complete partitions, two workers per shard, the 60-minute ceiling, native JUnit/coverage, fork permissions and failure gates retain their existing owners. Cooperative snapshots cannot diagnose native code that indefinitely holds the GIL, and a stack does not establish a stall's cause.
• Twelve Lark parameter cases retain their exact values and boundaries; the largest collected ID drops from 375,081 to 134 characters. This is an output improvement, not a measured CPU speedup or old timing-cache reuse.
• English/Chinese guidance documents activation, disable, output loss and process-lifecycle limits. Placement/future-facing pass: keep this OS transport in the existing Python pytest provider, release frame references before transport, and avoid a new module, persistent queue, vocabulary or control-plane decision owner.

Validation

Tested revision: e93ea84ce7bfc5c16f36c256b87ffc8db22ebdd1, with main merged through 3c681f1132fe719e01f75e5a41bfd854dfa9e346. Written by: OpenAI Codex model agent. Inputs: synthetic/public fixtures; run state: finished.

Check	Result and boundary
Full affected workflow/timing/failure-context/Lark modules	648 passed, 1 existing skip: Python 3.14 template-string syntax on Python 3.11.17
Failing-before / recovery	Old enabled observer times out at 4 seconds on a full unread stderr pipe; default-off exits. New helper deadline and an independently unresponsive-helper parent-timeout case both finish without any reader resuming; helper processes are settled and original FD flags remain blocking
Interrupted parent	Killing only pytest leaves the writer to its own deadline; process inspection verifies it is no longer active, and inspection errors fail the regression
Real production entry	Actual pytest/xdist complete, failed and killed runs; pre-kill stacks; native duration parity; all six synthetic split/coverage executions and missing-coverage refusal; cancellation, invalid delays/FDs, default-off and IO failure
Fixture parity	Actual original base/head collection: 21 collected items and 12 identical parameter values/hashes/character and UTF-8 lengths; both fixture blobs remain byte-identical to reviewed 290a117
Static / boundary	Ruff, repository-configured mypy (19 source files), advisory before full semantic drift check, seven-path public scan, DCO and clean diff
Supplemental typing	python -m mypy --strict --follow-imports=silent --disable-error-code=no-untyped-call scripts/ci/pytest_timing.py passes its one-file scope; it suppresses imported-helper diagnostics and untyped-call errors. The unrestricted python -m mypy --strict scripts/ci/pytest_timing.py fails: 10 errors across two files, including two calls from this provider into the untyped failure-context helper. This is not full strict typing coverage.
Original source-scope quality and risk	Passed against the stated base/head with no blockers. Managed premerge: 4 direct checks, 10 catalog canaries, 8 risk-profile smokes and the seven-path public-boundary check; no failures, timeouts, side effects or manual holds.

Source-only CI observation adapter: no new frontend/Lark operation or installed product behavior is claimed. The independent model review approves this unchanged head via the author-account COMMENTED fallback. Native approval closeout is clear; raw review decision remains REVIEW_REQUIRED. Required hosted CI remains open; the review and local validation do not grant merge authority.

Hosted CI and continuation

The earlier natural run 38625209094 remains a failed/queued observation of merge c35e259 at the old head: six full Python shards had no assigned runner then. Its TS consumer-pin failure has existing #6431; browser Origin fixes entered main through #6456; the independently reproduced legacy extension-status defect has source proposal #6463. Those source changes and local results do not qualify this new head's hosted run. The historical long-stall cause and release-runner priority remain open, and no required check is waived.

中文：既有 pytest observer 提供显式开启的协作式栈快照，默认 0 关闭；不读取局部变量。已修复 stderr 管道满时诊断阻塞测试的问题：隔离 writer 自带退出期限，父进程另设超时并回收，pytest 被单独终止后 writer 也能退出，不改变共享 stderr 标志。648 项通过、1 项既有版本语法跳过；六分片、双 worker、60 分钟及完整报告/覆盖率门禁保持。当前 head 已获独立模型评审的 APPROVE 结论（作者账号 COMMENTED 回退），原生 approval closeout 为 clear；这不是 GitHub 正式批准或合并授权。仓库配置 mypy 的19文件通过，限定单文件检查排除了导入检查和 untyped-call；完整 strict 仍有10个错误，不得称全量 strict 通过。完整 hosted CI、历史停滞根因与发布优先级仍待验收。

Current-head hosted follow-up: Windows job 116102476139 at the merge of e93ea84 into main 3c681f1 passes the runtime lifecycle, sidecar and peer-input steps, then fails the legacy extension migration case with extension revision history is invalid (26 other cases pass). That status-readback correction is already proposed in #6463; no duplicate repair is added here. The later Windows SQLite/frontend steps are skipped. The six Python shards in the observed run are pending; full hosted CI remains open. Independent review is now recorded at the unchanged head as linked above.

A fresh author readback at 2026-10-14 22:15 UTC confirms the same head and natural run 38634947433. All six full Python shards were pending with no assigned runner in that snapshot, approximately 73 minutes after their pending timestamp; this is queue time, not test execution or runner cost. No complete duration evidence or source-proposal adoption is inferred. The existing raw Windows/TS/browser failures remain recorded; related forwarding-fixture #6460 has since merged into main, outside this run's original base. No manual workflow, cancellation, new source push or merge was performed for this readback.

Additional completed-run stall evidence

Natural run 38550223334 is now complete. Its checkout logs identify merge source 51feaa5426770cdfea3d4ab39b6cb0b1f47701ba (run metadata head 5841f9e). Shard 3 and shard 6 both carry the hosted 60-minute deadline annotation and neither produced JUnit. The existing strict timing reader accepts both streams as incomplete: 2,531/3,776 and 1,358/3,776 cases finished, respectively. Setup before pytest was 33 and 40 seconds; the last available log lines precede the configured deadline by about 37 and 44 minutes.

This narrows the evidence gap to execution/reporting silence well before the deadline. It does not establish a blocked stack, CPU exhaustion, worker death, log-transport failure or timing imbalance, and does not justify increasing the timeout or reducing the collection. The three reported setup phases without matching completion identify investigation targets only. This older source did not contain the proposed stack observer; its results do not qualify this PR's current head or imply that the observer fixes the underlying stall. Full current-head hosted qualification and release-priority acceptance remain open.

The unmodified archive of that exact merge source passes all three full modules containing the unfinished phase identities on macOS: 244 passed, 0 skipped, about 165 seconds. An additional isolated Linux amd64 container under emulation on an arm64 host, using Python 3.11.17, Node 24.21.0, two xdist workers, the existing timing provider and coverage, also passes 244 tests with no skips in 499.1 seconds. JUnit verifies all three exact unfinished case identities passed; the strict timing reader confirms 244 selected, observed and finished cases with no invalid durations. The complete input archive was verified byte-for-byte against a fresh archive of 51feaa5. These targeted runs did not reproduce the long silence. They do not qualify the full split collection, native hosted Ubuntu, runner or logging conditions, and the emulated duration is not a hosted performance comparison. The separate digest-owner assertion in the old shard is reproducible at that old source and passes on current main after existing owner reuse (d11dca9f8); it is not conflated with the long silence.

中文：新增完成轮次的两个分片在约67%与35%后无更多日志，距60分钟上限尚有约37与44分钟；已有严格计时读取器判定两者均不完整，未产出JUnit。同一未改源码的三个完整模块在macOS及arm64宿主上的Linux amd64仿真容器中，均以双worker、既有timing与coverage通过244项、零跳过；Linux回放499.1秒，JUnit及严格timing回读完整，三个精确尾部用例均通过。整个输入归档与该Git提交重新导出的字节一致。这仍未复现原长时间静默，不能证明原生hosted Ubuntu完整分片已恢复，也不能把仿真时延用于CPU或性能比较。继续复用本PR的有界栈诊断，等待真实执行位置证据；不据此调超时、减测试、宣称根因已修复或关闭发布优先级验收。

A second completed natural run 38553506775 checked out cba1e3c3d8669dfd6e903a888f1c501b247875bf (metadata head a33a523). The strict timing reader confirms complete execution of the selected collections in shards 1/2/4/5, with pytest exit codes 1/1/1/0. Their three assertions are already corrected on main by the existing heartbeat JSON assertion (ccd8df2ba), claimed executor fixture with an unclaimed-route negative (#6419), and direction-source UTF-8 decoding (6361d3cc4); they are separate from the long silence and need no duplicate patch here.

Shard 3 remains incomplete at 2,760/3,776 finished cases, with one observed setup lacking a call/teardown for the steward notice-freshness test. Its last timing record precedes termination by about 21 minutes. Its annotation identifies supersession by a same-PR waiting candidate, not this job's deadline. Shard 6 has the 60-minute deadline annotation, but its individual log endpoint returns HTTP 404, the completed-run log ZIP also omits that shard, and it has no JUnit; missing observations stay missing. This older source also lacks the proposed stack observer. A last phase or silent log cannot establish the executing stack, root cause or current-head recovery.

A targeted replay of that second-run source now resolves the unfinished full-node hash to the notice case [continuation-blocker-file]. In an isolated Linux amd64 container under arm64-host emulation, the unmodified cba1e3c module passes 31 tests, zero skips, including that exact case (9.034 seconds); native JUnit records 151.187 seconds and the existing strict reader confirms 31 selected/observed/finished cases with no invalid durations. Python 3.11.17, Node 24.21.0, two xdist workers and coverage are retained. The whole disposable run, including provisioning, takes 245 seconds. Separately, the five available original timing streams contain 17,864 observed full-node hashes with no overlap between jobs. This does not qualify the missing/unfinished collection, reproduce the full shard or identify the hosted stall cause; current-head hosted evidence remains required. No unchanged local replay is planned without a new causal input.

中文：第二个自然轮次在实际 checkout cba1e3c 上，四片完成既定 collection、pytest 退出码分别为1/1/1/0；三项断言已有主干修复，复用原 owner，不重复开补丁。片3仅完成2760/3776，最后计时距终止约21分钟，且平台明确因同PR新候选替换取消；片6才有60分钟超时记录，但单job日志404、整轮日志ZIP也未包含该片、无JUnit。旧源尚无本PR栈诊断，不能把尾部setup、日志静默或缺失记录当成根因或完整CI已恢复。

中文补充：原 cba1e3c 的完整31例模块在Linux amd64仿真中零跳过通过，精确尾部参数 continuation-blocker-file 用例9.034秒；原生JUnit151.187秒、严格计时31例完整，含准备的容器总时长245秒。五份原日志中17864个已观测完整node hash跨job无重叠；片3未完、片6缺失，仍不证明全片并集/互斥完整或托管停滞已恢复。没有新条件不重放相同局部模块，继续等待真实执行栈。

[loopx-agent]

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=xhigh

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

Reviewed head: 290a117cdb6804f88106ef8e58db76ff50bf2a60. 当前作者账号的独立评审以 COMMENTED 记录。

动机

排查长时间 pytest 运行的开发者，需要及时看到执行位置，而不是只看到最后一个测试名。 大文本用例原先会打印极长参数名；测试卡顿时缺少执行栈。本 PR 缩短标签并显式输出栈，但日志 stderr 停止读取时，一个已执行完的测试会被新诊断线程拖住。 12 个改标签载荷逐项不变，最长 ID 从 375081 降到 134；正常诊断通过，但满管道反例停在 join，恢复读取后才退出。 只评审诊断源码与测试输出边界，不认定 hosted 长停滞根因、CPU 加速或发布容量问题已解决。

改动思路

将协作式栈输出放在既有 Python pytest provider 是合适的：pytest 仍拥有测试选择、阶段结果与报告；观察器只检查当前执行线程，不读取 locals，也不检查其他进程。保留既有 provider/typed owner，不需要另建框架或持久化词汇。当前 PR 只验收有界诊断输出；必须先修复背压反例，hosted 根因和完整持续运行资格另验。

开启/关闭由本次 pytest 命令的 --ci-stall-after 控制，本地默认0；全套 Linux 分片显式加120。正值在 configure 时保留原 stderr descriptor，每个测试启动一次 timer；结束时取消并 join。这个最后步骤目前能把可选观察器变成执行阻塞。只是把 join 改成 timeout 仍需处理留下的写线程和 FD 生命周期；应在原 owner 内保证非阻塞写与有界清理，不修改共享 stderr 的全局 flags。

具体改动

完整7路径 +209/-31。workflow 只增加显式120参数并把过期注释改为六条 lane；实际六分片、双 worker、60分钟、JUnit/coverage与失败汇总没有减弱。timing provider 新增 option、StashKey FD、线程栈输出及资源关闭，同时给原 hooks 补类型。两个 Lark 测试文件只给12个大文本参数加短ID；我用真实 base/head collection 核对21个条目顺序和值，四组修改的12个载荷哈希、字符和UTF-8长度全部一致。最大ID375081→134是输出收益，不是CPU提速。原 timing/workflow测试加入0/启用、缺失FD、非法延迟、OSError、取消、被杀controller与六分片实际coverage场景。英文和中文指南保留诊断、关闭及GIL限制。

关键代码讲解

pytest_configure 的 finite/nonnegative检查和正值FD获取拒绝非法配置；0不要求FD。pytest_runtest_protocol 的 Timer 取强frame引用，最多24帧，然后调用阻塞 os.write。测试 finally 的 timer.join() 无上限，已进入写系统调用的线程不能由 cancel() 或 Event 唤回。旧 TimingObserver 输出/统计仍由 controller 执行。

依据变更前 docs/development/testing-and-quality.md，固定版本 de9dc656d7a0c040e8594085921442dcd4967e4b：Local Validation Environment 的原生report/coverage边界保留；Risk-Based Review 的真实失败恢复在本head尚不成立，不能用当前指南自证。

对主干的风险

[P2] 可选栈输出在日志背压时阻塞已完成测试。 在真实 pytest 子进程中，stdout独立，原stderr pipe填满65536字节且不读取。一个100ms测试主体已完成；启用10ms诊断后，4秒仍不退出。独立重复并采集线程栈：timer停在 pytest_timing.py:54 的 os.write，主线程停在第68行join；恢复管道读取后才exit0。相同fixture的旧base默认、当前无参数及0都在一秒内exit0。现有“抛出 OSError”的测试不覆盖系统调用阻塞。最低修复是让诊断写入与清理有界，增加不恢复读取也能正常结束的真实pipe回归；不把诊断失败变成测试失败或留下descriptor复用竞态。

完整独立测试643passed/1既有版本syntax skip；Ruff通过。semantic advisory在全量语义检查之前运行，native premerge4direct+19selected/executed全过，零失败、timeout、sideeffect或manual hold，7个公开路径扫描干净。这些通过不会覆盖刚发现的背压缺陷。已验证默认关闭保持原结果，invalid/nonfinite与无原FD启用时在测试前拒绝；没有改Quota、调度或授权。未查询、轮询、等待或启动GitHub CI，不认证当前hosted运行。

我的整体评价

REQUEST_CHANGES。源码诊断和标签压缩有实际价值、范围也合适，但long_horizon/user_experience在日志异常下出现回归，不能把报告无法完成当成“仅诊断”。future-facing pass已保持既有provider并补native类型；修复同owner的有界IO/cleanup即可，不必增加通用层或改其它控制面。修复后重跑相同满管道反例与原 timing/workflow suite。当前测试通过、统计减少和本评审均不授合并权限。

English verdict: REQUEST_CHANGES - 290a117; a full unread original stderr pipe blocks the timer at os.write and pytest at unbounded join after the test completes. Default-off/base finish; draining recovers exit0.643tests passed,1existing skip and4direct+19native checks passed, but bounded diagnostic IO/cleanup needs repair.

[loopx-agent]

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=xhigh

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

Reviewed head: e93ea84ce7bfc5c16f36c256b87ffc8db22ebdd1。共用作者账号以 COMMENTED 发布独立评审结论。

动机

维护 Python 测试的贡献者需要在测试迟迟不返回时看到正在执行的位置，同时让原测试继续正常结束。 旧补丁启用栈观察后，未被读取的 stderr 管道填满会让输出线程堵住，已经完成的测试也等不到退出；当前补丁把诊断写入放到有期限的独立子进程，满管道时允许丢弃快照并释放测试。 相同真实 pytest 满管道反例中，旧 head 在四秒后仍存活，当前 head 约 0.48 秒正常退出且无需恢复读取；正常执行、失败报告和参数值保持原有意义。 这里交付可选诊断的背压修复和整个诊断补丁的验证，不认定未采样的 hosted CI 根因，不承诺 native 调用期间的绝对时间上限，也不改变测试或覆盖验收。

改动思路

诊断属于既有 pytest provider 的可选 IO，复用原 hook 和结果 owner；独立短命 writer 是处理共享阻塞 FD 且不修改其 flags 的有界修复，不另建控制面规则。 当前 PR 验收完整诊断路径、背压退出、默认关闭及原报告意义；native/GIL 和 hosted 运行资格保持明确边界。

最强反对理由是观察器本身让测试永远无法结束；仅把线程标成 daemon 不能解决主线程 join，也不能改变 duplicate FD 的 flags，否则会影响共享 stderr。当前选择在触发快照时使用原解释器的短命子进程，并同时约束它自身退出和父进程等待。不开启时没有 timer、dup 或 subprocess。已有 timing/failure owner 继续决定报告，无需新建 TypeScript 控制面能力或统一诊断框架。

具体改动

整包7路径 +346/-31，重新评估 base 到 head，而非只看上一条 finding。pytest provider 新增 finite/nonnegative 的 --ci-stall-after，默认0；full Linux shards显式120秒，原六个 partition、每组2worker、60分钟和 coverage/JUnit门槛保留。两个 Lark测试只对12组参数提供短展示ID，载荷和业务断言不变。英文/中文测试指南说明开启、关闭、Python线程/GIL范围和背压丢弃；原4→6文档数字修正为 base 已存在的六分片，没有扩大执行集。

关键代码讲解

pytest_runtest_protocol:44 只观察当前执行线程，最多24层且不读取 locals；组装行后释放 frame引用。subprocess.run:66 使用选中的 sys.executable -I -S，子进程自身100ms退出、父等待500ms超时并 kill/reap，原FD未设 nonblocking。原 finally取消并join后才能 unconfigure关闭FD，故障写入可丢弃且不替换pytest结果。pytest_configure:99只对有效正值 dup原stderr；缺失FD、负数、NaN/Inf在执行测试前明确拒绝。原 TimingObserver:165、xdist集合与subtest数量、失败身份和结果仍由原owner负责。

依据变更前 docs/development/testing-and-quality.md，固定版本 3c681f1132fe719e01f75e5a41bfd854dfa9e346：Local Validation Environment 的源码/解释器来源成立；Risk-Based Review 的原生结果、真实边界、风险正反例及资格区分成立。新文档按条款比较了先执行后观察、默认关闭、失败/IO恢复、覆盖和权限条件。栈快照是可选诊断，仓库 native test/coverage仍是可执行义务；未把机器要求改称“指导”。

对主干的风险

旧 review 5523970168 的唯一背压 finding 被逐项映射到当前实现；完整旧正文及 inline 都读回，没有删除讨论或继承旧结论。再次运行同一真实pytest fixture：65536字节满 stderr，无reader、100ms测试、10ms观察。旧 290a117cdb6804f88106ef8e58db76ff50bf2a60 在4秒仍存活，恢复读取后才退出；当前 .478秒退出0且没有读取管道，默认/.0也正常退出。当前原生回归同时覆盖自身deadline、parent timeout、pytest被杀后writer不再运行、FD flags不变、failure/xdist/JUnit/coverage及正常栈可见。只证明终止，不冒称父进程死亡后由它完成reap。

实际当前源码 collect21items，与 immutable base参数hash、顺序、字符/UTF8长度完全相同，最长nodeID由375081降至134。定向5模块648passed、1个既有Python3.14模板语法用例在3.13跳过；仓库配置 python -m mypy 的19源文件通过，Ruff通过。补充 mypy --strict scripts/ci/pytest_timing.py 未通过：head10errors/2files，base29errors/2files；包含未变的failure_context helper未标注和新typing暴露的调用诊断，不能说所有失败签名相同。此额外strict命令不在仓库配置19文件gate内，provider严格类型覆盖仍有限，未把它记为通过。advisory先于全量semantic；premerge4direct+19selected/executed通过，0fail/timeout/manualhold，7公开路径扫描干净。未查询、轮询、等待或启动CI。

我的整体评价

APPROVE。当前完整head让可选诊断不会因已演示的阻塞管道拖住测试，默认关闭和原结果/coverage意义有独立证据。future-facing pass已复用单一provider/IO边界并释放frame引用；相邻 failure_context严格typing可以沿原owner改进，但不以新增抽象扩大本次修复。诊断仍可能在背压下丢失，native/GIL和process startup不具绝对时限；这不是hosted故障根因或持续运行资格。COMMENTED结论不授予合并权限，approval closeout和merge readiness分别回读。

English verdict: APPROVE - e93ea84; the old full unread-stderr counterexample is independently reproduced and the current real pytest process completes without draining. Native outcomes, default-off behavior, parameter values and coverage gates remain intact. Configured typing and risk-based checks pass; supplemental strict provider typing and hosted/native latency remain explicitly limited.