Skip to content

fix(test): watchdog silent stand-down on missing heartbeat + teardown heartbeats - #317

Merged
LeXwDeX merged 1 commit into
devfrom
fix/httpapi-teardown-watchdog
Aug 17, 2026
Merged

fix(test): watchdog silent stand-down on missing heartbeat + teardown heartbeats#317
LeXwDeX merged 1 commit into
devfrom
fix/httpapi-teardown-watchdog

Conversation

@LeXwDeX

@LeXwDeX LeXwDeX commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Closes 首个 #316 交付物(挂起可诊断化);#316 根因继续开放。

问题

2026-08-17 CI:httpapi exerciser 在 summary pass=229 后静默挂起 15 分钟被 step timeout 杀掉(run 31994077328,同 commit 孪生 run 绿)。 已武装进程外看门狗,但全程未开火

根因之一(已修)

看门狗仔脚本把心跳文件缺失当健康

if (!mtime || Date.now() - mtime <= timeoutMs) return  // 文件没了 = 永久静默

CI runner 回收 tmpdir 文件即解除武装。本机独立验证复现:正常场景 3 秒开火 ✅;删除心跳文件后静默不作为(漏洞)→ 修复后 3 秒开火 ✅。

修复

  1. 缺失即停滞:记录 missing-since,超过 timeout 照杀——静默路径不存在了
  2. teardown 心跳:disposeApps 逐 app + cleanup + complete 各写心跳标签——下次挂起直接点名卡在 teardown 哪一段

验证

  • packages/opencode bun typecheck
  • --mode effect --include dag --progress:exit 0、22 pass、零误触发 ✅
  • 缺文件场景:3 秒开火并报告 last activity: <none>

#316 的意义

下一个真实挂起将被看门狗在 120s 内终结并带出现场归因(哪个 app 的 dispose、哪段 cleanup),根因从现场钉死,不再依赖低命中率复现。

… heartbeats

The out-of-process watchdog treated a missing heartbeat file as healthy
(!mtime → return forever), so a CI runner reclaiming the tmpdir file left
the guard silently disarmed — the 2026-08-17 teardown hang burned the full
15m step timeout with no kill despite --progress arming the watchdog. Track
missing-since and kill after the timeout instead.

Also heartbeat the teardown phases (disposeApps per-app, cleanup, complete):
the 2026-08-17 hang sat after 'summary pass=229' with zero attribution; the
next hang names the teardown stage it froze in.
@LeXwDeX
LeXwDeX merged commit a9af24a into dev Aug 17, 2026
5 checks passed
@LeXwDeX
LeXwDeX deleted the fix/httpapi-teardown-watchdog branch August 17, 2026 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant