Skip to content

test(watchdog): park the alarm children's daemon thread - #16688

Merged
bolichen97 merged 1 commit into
mainfrom
fix/watchdog-child-parks-poll
Oct 4, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/watchdog-child-parks-poll

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The real-alarm watchdog tests pass on a loaded CI runner instead of failing now and then with SIGSEGV.

test_alarm_dump_survives_a_temporary_sigalrm_owner_that_restores_sig_dfl fails intermittently with (-11, ...) / assert -11 == -14: the child dies by SIGSEGV instead of SIGALRM. The real-alarm children in test/test_loop_watchdog.py poll every 0.05 s, so the alarm often lands while the loop-stall-watchdog daemon thread is running Python. faulthandler walks every thread's frames from inside the signal handler, with no GIL, and a frame that changes under the walk crashes it part way through the dump.

Why it matters

This test ran red in a release-candidate test job, and that job gates every publish job of a release run. It also failed the tagged insider candidate on its rerun.

Not a goal

  • Changing loop_watchdog or faulthandler behaviour. The gateway's default poll is 5 s, so the window is far smaller there, and a SIGSEGV during the dump still ends the stalled process. This PR is test-only.

What changed (motivation → approach → change)

Where What
test/test_loop_watchdog.py _child_script poll_interval goes from 0.05 to 60.0, with a docstring that explains why

stall_after (30 s) outlives every child, so no poll in these children can ever act. A 60 s poll keeps the daemon thread blocked in Event.wait for the child's whole life, so the alarm dump only walks stable frames. All four callers of _child_script assert only on the alarm exit, the dump and stderr; none depends on the poll.

Backwards compatibility

Compatible: test-only change; no product code is touched.

Removes nothing: only a test helper's poll interval changes.

Tests

  • Repro, with the test's child script run directly, 48 in parallel beside 16 busy loops: at poll 0.05, 8 of 6000 runs die by SIGSEGV, each with a dump cut off in the daemon thread's frames (line 0 in _note_resume). At poll 60.0, 0 of 6000 do: every run ends by SIGALRM with a full dump.
  • pytest -n 8 --timeout=120 test/test_loop_watchdog.py: 36 passed.
  • The black gate, isort, flake8 and the comment-history gate (scoped to this change) pass.

Manual verification

N/A: the stress repro above is the check.

Related Issues

Fixes #16687

Pattern harvest

Rule candidate: review-prompt
Pattern: a test child that ends by a faulthandler all-threads dump must not have another thread running Python when the signal lands.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

The real-alarm child processes polled every 0.05 s, so the alarm often
landed while the loop-stall-watchdog daemon thread was running Python.
faulthandler walks every thread's frames from the signal handler with
no GIL, a frame changed under the walk, and the child died by SIGSEGV
part way through its dump instead of by SIGALRM after it. Under CI load
this failed test_alarm_dump_survives_a_temporary_sigalrm_owner_that_
restores_sig_dfl with (-11, ...) != -14.

stall_after (30 s) outlives every child, so no poll in these children
can act. A 60 s poll keeps the daemon thread blocked in Event.wait for
the child's whole life, and the dump only walks stable frames.

Repro, the test's child script run directly, 48 in parallel with 16
busy loops: poll 0.05 -> 8 of 6000 SIGSEGV (dump cut off in the
daemon thread's frames); poll 60.0 -> 0 of 6000, every one SIGALRM
with a full dump.
@bolichen97
bolichen97 requested a review from a team as a code owner October 4, 2026 06:29
@bolichen97
bolichen97 requested a review from CrysisDeu October 4, 2026 06:29
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 4, 2026
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4d79309744b3d0fd96b684db31f255230dbdbf1b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merge, confirm someone has filed a follow-up issue for the same faulthandler race on the gateway's 5 s poll, which this test-only PR leaves open by design.

[DESIGN-REVIEWED] 4d79309

Description read

[DESCRIPTION-READ] 9c5c985a07a5e87691cc8e0f710b54544cd32c125fff118b8ab8e1bd8cf8304d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4d79309744b3d0fd96b684db31f255230dbdbf1b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4d79309

Verdict parsed from the review's SHA-scoped output markers for commit 4d79309744b3d0fd96b684db31f255230dbdbf1b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4d79309744b3d0fd96b684db31f255230dbdbf1b: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4d79309744b3d0fd96b684db31f255230dbdbf1b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4d79309

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4d79309744b3d0fd96b684db31f255230dbdbf1b: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 4, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Merge override (release driver, coordinator-approved): the only red checks on 4d79309 are main-wide failures unrelated to this test-only diff, which touches only test/test_loop_watchdog.py. (1) All 8 Linux backend shards and Windows shard 1 fail only on test_chat_handlers_composition_contract.py (assert ['re'] == []), tracked as #16693 with fix #16695. (2) Windows shard 3 fails test_review_repair_routing_skill.py::test_slim_skills_do_not_regrow_duplicate_guidance_or_wire_model_ids[prepare-pr-68000] (68077 <= 68000), a skill-file size budget this diff does not touch; it passes on Linux at the same base. (3) The Coverage Gate fails closed because backend-test failed. All AI lanes report no BLOCK on this head, and the Issue Gate is green after #16687 was triaged.

@bolichen97
bolichen97 merged commit b2060d6 into main Oct 4, 2026
85 of 99 checks passed
@bolichen97
bolichen97 deleted the fix/watchdog-child-parks-poll branch October 4, 2026 07:04
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 4, 2026
bolichen97 added a commit that referenced this pull request Oct 4, 2026
…, #16688) (#16694)

* test: give the shell-probe cancel test its own probe caches (#16139)

(cherry picked from commit 5583d36)

* test(watchdog): park the alarm children's daemon thread

The real-alarm child processes polled every 0.05 s, so the alarm often
landed while the loop-stall-watchdog daemon thread was running Python.
faulthandler walks every thread's frames from the signal handler with
no GIL, a frame changed under the walk, and the child died by SIGSEGV
part way through its dump instead of by SIGALRM after it. Under CI load
this failed test_alarm_dump_survives_a_temporary_sigalrm_owner_that_
restores_sig_dfl with (-11, ...) != -14.

stall_after (30 s) outlives every child, so no poll in these children
can act. A 60 s poll keeps the daemon thread blocked in Event.wait for
the child's whole life, and the dump only walks stable frames.

Repro, the test's child script run directly, 48 in parallel with 16
busy loops: poll 0.05 -> 8 of 6000 SIGSEGV (dump cut off in the
daemon thread's frames); poll 60.0 -> 0 of 6000, every one SIGALRM
with a full dump.

(cherry picked from commit 4d79309)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_loop_watchdog alarm children die by SIGSEGV under load (daemon thread running during the dump)

1 participant