Repository navigation
test: stop two release-candidate test failures on release/0.8.0 (#16139, #16688) - #16694
Conversation
(cherry picked from commit 5583d36)
The real-alarm child processes polled every 0.05 s, so the alarm often landed while the loop-stall-watchdog daemon thread was running Python. faulthandler walks every thread's frames from the signal handler with no GIL, a frame changed under the walk, and the child died by SIGSEGV part way through its dump instead of by SIGALRM after it. Under CI load this failed test_alarm_dump_survives_a_temporary_sigalrm_owner_that_ restores_sig_dfl with (-11, ...) != -14. stall_after (30 s) outlives every child, so no poll in these children can act. A 60 s poll keeps the daemon thread blocked in Event.wait for the child's whole life, and the dump only walks stable frames. Repro, the test's child script run directly, 48 in parallel with 16 busy loops: poll 0.05 -> 8 of 6000 SIGSEGV (dump cut off in the daemon thread's frames); poll 60.0 -> 0 of 6000, every one SIGALRM with a full dump. (cherry picked from commit 4d79309)
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] ea4a727 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Problem / Motivation
Goal: The release/0.8.0 release-candidate test job stops failing on two load- and order-dependent tests, so the next insider tag can publish.
The
v0.8.0-insider.10release run failed its Test Release Candidate SHA job twice (the original attempt and one rerun), each time on tests and not on product code:test_sandbox_interpreter_enoent_retry.py::TestACancelDuringTheShellProbeIsNotLost::test_the_cancelled_command_is_never_launched, failing withassert [] == [True]on both attempts. The cause is the one test: give the shell-probe cancel test its own probe caches #16139 fixed on main and never backported:cron_script._POSIX_STRICT_CACHE/_BRACE_OFF_SHELLSare process-wide, so an earlier test in the same xdist worker that ran the real shell probe leaves/bin/shdecided. The cancel test's probe then never runs, and the cancel has nowhere to land.test_loop_watchdog.py::test_alarm_dump_survives_a_temporary_sigalrm_owner_that_restores_sig_dfl(attempt 2): the child dies by SIGSEGV (-11) instead of SIGALRM (-14). The real-alarm children poll every 0.05 s, and faulthandler crashes while walking the daemon thread's frames from the signal handler (test_loop_watchdog alarm children die by SIGSEGV under load (daemon thread running during the dump) #16687, main fix test(watchdog): park the alarm children's daemon thread #16688).Why it matters
A red Test Release Candidate SHA job skips Record Immutable Promotion Bundle, so no insider candidate on this branch can be promoted.
Not a goal
cron_scriptorloop_watchdog; both are test-isolation or test-timing faults.What changed (motivation → approach → change)
5583d3624a), cleancherry-pick -x_probe_bliprebinds both probe caches to fresh dicts for the test4d79309744), cleancherry-pick -x_child_scriptpolls every 60 s instead of 0.05 s, so the daemon thread is parked inEvent.waitwhen the alarm dumpsBackwards compatibility
Compatible: test-only change.
Removes nothing: only two test helpers change.
Tests
/bin/shin both caches (standing in for an earlier real-probe test in the same worker) makes the test fail on the release tip with the exact CI assertionassert [] == [True], and pass on this branch.--timeout=120, and all passed.pytest -n 8 --timeout=120 test/test_loop_watchdog.py test/test_sandbox_interpreter_enoent_retry.py: 98 passed.Manual verification
N/A: test-only.
Related Issues
Fixes #16687 on release/0.8.0. Backports #16139 and #16688.
Pattern harvest
Rule candidate: review-prompt
Pattern: a test that patches a seam behind a process-wide cache must own that cache, and a child that ends by an all-threads faulthandler dump must not have another thread running Python when the signal lands.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)