Skip to content

test: stop two release-candidate test failures on release/0.8.0 (#16139, #16688) - #16694

Merged
bolichen97 merged 2 commits into
release/0.8.0from
fix/bp-rc-test-flakes-0.8
Oct 4, 2026
Merged

bolichen97 merged 2 commits into
release/0.8.0from
fix/bp-rc-test-flakes-0.8

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The release/0.8.0 release-candidate test job stops failing on two load- and order-dependent tests, so the next insider tag can publish.

The v0.8.0-insider.10 release run failed its Test Release Candidate SHA job twice (the original attempt and one rerun), each time on tests and not on product code:

  1. test_sandbox_interpreter_enoent_retry.py::TestACancelDuringTheShellProbeIsNotLost::test_the_cancelled_command_is_never_launched, failing with assert [] == [True] on both attempts. The cause is the one test: give the shell-probe cancel test its own probe caches #16139 fixed on main and never backported: cron_script._POSIX_STRICT_CACHE / _BRACE_OFF_SHELLS are process-wide, so an earlier test in the same xdist worker that ran the real shell probe leaves /bin/sh decided. The cancel test's probe then never runs, and the cancel has nowhere to land.
  2. test_loop_watchdog.py::test_alarm_dump_survives_a_temporary_sigalrm_owner_that_restores_sig_dfl (attempt 2): the child dies by SIGSEGV (-11) instead of SIGALRM (-14). The real-alarm children poll every 0.05 s, and faulthandler crashes while walking the daemon thread's frames from the signal handler (test_loop_watchdog alarm children die by SIGSEGV under load (daemon thread running during the dump) #16687, main fix test(watchdog): park the alarm children's daemon thread #16688).

Why it matters

A red Test Release Candidate SHA job skips Record Immutable Promotion Bundle, so no insider candidate on this branch can be promoted.

Not a goal

  • No product code change. Neither defect is in cron_script or loop_watchdog; both are test-isolation or test-timing faults.

What changed (motivation → approach → change)

Commit Source What
1 #16139 (5583d3624a), clean cherry-pick -x _probe_blip rebinds both probe caches to fresh dicts for the test
2 #16688 (4d79309744), clean cherry-pick -x _child_script polls every 60 s instead of 0.05 s, so the daemon thread is parked in Event.wait when the alarm dumps

Backwards compatibility

Compatible: test-only change.

Removes nothing: only two test helpers change.

Tests

  • Test 1: a plugin that pre-decides /bin/sh in both caches (standing in for an earlier real-probe test in the same worker) makes the test fail on the release tip with the exact CI assertion assert [] == [True], and pass on this branch.
  • Test 2: the test's child script run directly, 48 in parallel beside 16 busy loops. At poll 0.05 (the release tip), 8 of 6000 runs die by SIGSEGV with a truncated dump. At poll 60.0, 0 of 6000 do. On this branch, the three real-alarm tests ran 200 times each under the same load with --timeout=120, and all passed.
  • pytest -n 8 --timeout=120 test/test_loop_watchdog.py test/test_sandbox_interpreter_enoent_retry.py: 98 passed.
  • black, isort, flake8 and comment-history pass on both files.

Manual verification

N/A: test-only.

Related Issues

Fixes #16687 on release/0.8.0. Backports #16139 and #16688.

Pattern harvest

Rule candidate: review-prompt
Pattern: a test that patches a seam behind a process-wide cache must own that cache, and a child that ends by an all-threads faulthandler dump must not have another thread running Python when the signal lands.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

The real-alarm child processes polled every 0.05 s, so the alarm often
landed while the loop-stall-watchdog daemon thread was running Python.
faulthandler walks every thread's frames from the signal handler with
no GIL, a frame changed under the walk, and the child died by SIGSEGV
part way through its dump instead of by SIGALRM after it. Under CI load
this failed test_alarm_dump_survives_a_temporary_sigalrm_owner_that_
restores_sig_dfl with (-11, ...) != -14.

stall_after (30 s) outlives every child, so no poll in these children
can act. A 60 s poll keeps the daemon thread blocked in Event.wait for
the child's whole life, and the dump only walks stable frames.

Repro, the test's child script run directly, 48 in parallel with 16
busy loops: poll 0.05 -> 8 of 6000 SIGSEGV (dump cut off in the
daemon thread's frames); poll 60.0 -> 0 of 6000, every one SIGALRM
with a full dump.

(cherry picked from commit 4d79309)
@bolichen97
bolichen97 requested a review from a team as a code owner October 4, 2026 06:40
@bolichen97
bolichen97 requested review from iamwhatever and removed request for a team October 4, 2026 06:40
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 4, 2026
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of ea4a72794d2c45660e2c95737646481a70b3ce69 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] ea4a727

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of ea4a72794d2c45660e2c95737646481a70b3ce69 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ea4a727

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ea4a72794d2c45660e2c95737646481a70b3ce69: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed ea4a72794d2c45660e2c95737646481a70b3ce69 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ea4a727

Verdict parsed from the review's SHA-scoped output markers for commit ea4a72794d2c45660e2c95737646481a70b3ce69.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ea4a72794d2c45660e2c95737646481a70b3ce69: <one-sentence reason>

@bolichen97
bolichen97 merged commit 04b1c83 into release/0.8.0 Oct 4, 2026
21 of 22 checks passed
@bolichen97
bolichen97 deleted the fix/bp-rc-test-flakes-0.8 branch October 4, 2026 07:04
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant