Repository navigation
test(watchdog): park the alarm children's daemon thread - #16688
Conversation
The real-alarm child processes polled every 0.05 s, so the alarm often landed while the loop-stall-watchdog daemon thread was running Python. faulthandler walks every thread's frames from the signal handler with no GIL, a frame changed under the walk, and the child died by SIGSEGV part way through its dump instead of by SIGALRM after it. Under CI load this failed test_alarm_dump_survives_a_temporary_sigalrm_owner_that_ restores_sig_dfl with (-11, ...) != -14. stall_after (30 s) outlives every child, so no poll in these children can act. A 60 s poll keeps the daemon thread blocked in Event.wait for the child's whole life, and the dump only walks stable frames. Repro, the test's child script run directly, 48 in parallel with 16 busy loops: poll 0.05 -> 8 of 6000 SIGSEGV (dump cut off in the daemon thread's frames); poll 60.0 -> 0 of 6000, every one SIGALRM with a full dump.
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Before merge, confirm someone has filed a follow-up issue for the same faulthandler race on the gateway's 5 s poll, which this test-only PR leaves open by design. [DESIGN-REVIEWED] 4d79309 Description read[DESCRIPTION-READ] 9c5c985a07a5e87691cc8e0f710b54544cd32c125fff118b8ab8e1bd8cf8304d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Merge override (release driver, coordinator-approved): the only red checks on 4d79309 are main-wide failures unrelated to this test-only diff, which touches only |
…, #16688) (#16694) * test: give the shell-probe cancel test its own probe caches (#16139) (cherry picked from commit 5583d36) * test(watchdog): park the alarm children's daemon thread The real-alarm child processes polled every 0.05 s, so the alarm often landed while the loop-stall-watchdog daemon thread was running Python. faulthandler walks every thread's frames from the signal handler with no GIL, a frame changed under the walk, and the child died by SIGSEGV part way through its dump instead of by SIGALRM after it. Under CI load this failed test_alarm_dump_survives_a_temporary_sigalrm_owner_that_ restores_sig_dfl with (-11, ...) != -14. stall_after (30 s) outlives every child, so no poll in these children can act. A 60 s poll keeps the daemon thread blocked in Event.wait for the child's whole life, and the dump only walks stable frames. Repro, the test's child script run directly, 48 in parallel with 16 busy loops: poll 0.05 -> 8 of 6000 SIGSEGV (dump cut off in the daemon thread's frames); poll 60.0 -> 0 of 6000, every one SIGALRM with a full dump. (cherry picked from commit 4d79309)
Problem / Motivation
Goal: The real-alarm watchdog tests pass on a loaded CI runner instead of failing now and then with SIGSEGV.
test_alarm_dump_survives_a_temporary_sigalrm_owner_that_restores_sig_dflfails intermittently with(-11, ...)/assert -11 == -14: the child dies by SIGSEGV instead of SIGALRM. The real-alarm children intest/test_loop_watchdog.pypoll every 0.05 s, so the alarm often lands while theloop-stall-watchdogdaemon thread is running Python. faulthandler walks every thread's frames from inside the signal handler, with no GIL, and a frame that changes under the walk crashes it part way through the dump.Why it matters
This test ran red in a release-candidate test job, and that job gates every publish job of a release run. It also failed the tagged insider candidate on its rerun.
Not a goal
loop_watchdogor faulthandler behaviour. The gateway's default poll is 5 s, so the window is far smaller there, and a SIGSEGV during the dump still ends the stalled process. This PR is test-only.What changed (motivation → approach → change)
test/test_loop_watchdog.py_child_scriptpoll_intervalgoes from 0.05 to 60.0, with a docstring that explains whystall_after(30 s) outlives every child, so no poll in these children can ever act. A 60 s poll keeps the daemon thread blocked inEvent.waitfor the child's whole life, so the alarm dump only walks stable frames. All four callers of_child_scriptassert only on the alarm exit, the dump and stderr; none depends on the poll.Backwards compatibility
Compatible: test-only change; no product code is touched.
Removes nothing: only a test helper's poll interval changes.
Tests
line 0 in _note_resume). At poll 60.0, 0 of 6000 do: every run ends by SIGALRM with a full dump.pytest -n 8 --timeout=120 test/test_loop_watchdog.py: 36 passed.Manual verification
N/A: the stress repro above is the check.
Related Issues
Fixes #16687
Pattern harvest
Rule candidate: review-prompt
Pattern: a test child that ends by a faulthandler all-threads dump must not have another thread running Python when the signal lands.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)