Skip to content

fix(test): Keep the installer stand-in's step out of the TERM group - #38268

Merged
bolichen97 merged 1 commit into
mainfrom
macflaky/update-installer-cancel-race
Nov 17, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
macflaky/update-installer-cancel-race

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop test_update_installer_graceful_stop.py from going red at random on macOS.

TestCommandProviderRollsBack::test_a_cancel_while_the_child_is_starting_lets_the_apply_command_restore failed on #38209 (job) with FileNotFoundError: .../case/outcome: the stand-in installer's TERM-trap rollback never ran. #38209 touches no update or kill code.

Why it matters

A flaky macOS leg reds unrelated pull requests. Under load the same flake hits TestTerminateAndReap::test_a_child_that_honours_term_is_never_killed too.

Not a goal

  • No change to how the gateway stops a real installer (same signals, same 4 s grace in production).
  • No change to cli.sh: its _run_step already keeps the step in its own group.

What changed (motivation → approach → change)

Symptom → cause, both reproduced on a loaded Mac (30 × yes, -n 6, --count 300):

  1. bash 3.2 drops the TERM trap. The stand-in ran its long step (sleep 30 &) inside the process group the stop SIGTERMs. The same signal killed the waited step, and macOS /bin/sh (bash 3.2.57) sometimes left wait with 143 and exited without running the trap. A set -x trace ends at wait <pid>; exit status 143; venv.pre-rebuild left behind. About 1 in 150 runs.
    → Change: the step runs under set -m (a group of its own, as _run_step does on macOS); its pid goes to steppid, which the existing reap_groups fixture reaps.
  2. A product deadline used as a test boundary. The CommandProvider tests stopped the stand-in with the production 4 s TERM grace. A ~7 s runner stall SIGKILLed the rollback mid-trap. _stop_installer bound that grace as a def-time default, so no fixture could raise it.
    → Change: _stop_installer(grace=None) reads UPDATE_INSTALLER_TERM_GRACE_SECS at call time (as _start_installer already does). TestCommandProviderRollsBack raises it to FALLBACK_TIMEOUT_SECS through _floor_monkeypatch; the settle bound uses the same net.

Docs: testing-conventions.md § Shell stand-ins that trap a group signal; AUTOSDE.yaml tests-are-deterministic item 15.

Backwards compatibility

Compatible: _stop_installer is private; grace=None resolves to the same 4 s constant, and every caller that passes a grace is unchanged.

Removes nothing: only a default's binding time and test code change.

Tests

  • test_update_installer_graceful_stop.py: stand-in step in its own group; generous grace for the CommandProvider class.
  • Before (macOS, bash 3.2, under load): 1–4 failures per 900 runs.
  • After: 5400/5400 under load; 30/30 single runs; whole file × 10 (140 passed); test_update_provider.py + test_coder_shutdown_budget.py (105 passed); test_flake_pattern_ratchet.py (71 passed).

Proof: not provable -- other: the flake needs a loaded macOS runner; reverting the production hunk passes on an idle host

Manual verification

N/A — reproduced and verified by the load loops above.

Related Issues

Seen on #38209.

Sibling sweep

Swept: git grep -lE "trap [a-z_'"]+.*TERM" -- test | xargs grep -lE '& $|&\swait|wait "$'
Sibling: test/test_installer_distro.py -- not affected -- it reads cli.sh text and signals no stand-in
Sibling: test/test_mcp_gateway_backend_record.py -- not affected -- its child ignores TERM and relies on no trap

Pattern harvest

Rule candidate: review-prompt
Pattern: shell stand-in relies on a TERM trap while the step it waits on stays in the signalled process group

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

test_a_cancel_while_the_child_is_starting_lets_the_apply_command_restore
went red on macOS with no `outcome` file. Two causes, both reproduced
on a loaded Mac:

1. The cli.sh stand-in ran its long step (`sleep 30 &`) inside the
   process group the stop signals. The group SIGTERM killed the waited
   step at the same moment, and bash 3.2 (macOS /bin/sh) then returned
   from `wait` with 143 and exited without running the TERM trap
   (~1 in 150 runs under load). The step now runs in a group of its
   own (`set -m`), as cli.sh's _run_step does on macOS; teardown reaps
   it through `steppid`.
2. The CommandProvider tests stopped the stand-in with the production
   4 s TERM grace; a ~7 s stall SIGKILLed the rollback mid-trap. The
   class now raises the grace to FALLBACK_TIMEOUT_SECS, which needs
   _stop_installer to read UPDATE_INSTALLER_TERM_GRACE_SECS at call
   time instead of binding it as a default at def time. Product
   behaviour is unchanged.

Adds the rule to testing-conventions and AUTOSDE
tests-are-deterministic (item 15).
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 17, 2026 05:22
@bolichen97
bolichen97 requested a review from buluoray November 17, 2026 05:22
@bolichen97
bolichen97 enabled auto-merge (squash) November 17, 2026 05:22
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b83cd3d6c9bc1cf4dff46161404836c86752cd2c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b83cd3d6c9bc1cf4dff46161404836c86752cd2c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix removes both causes at the test seam. The one product edit, reading the grace when _stop_installer is called, keeps the same default value.

[DESIGN-REVIEWED] b83cd3d

Description read

[DESCRIPTION-READ] cec0f0a54fe0a58c0c3ccf11117b5940820be694f98e9605b3ea25557fabb27d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b83cd3d6c9bc1cf4dff46161404836c86752cd2c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Both causes are fixed where they start (the stand-in's process group and the grace default bound when the function is defined), and the same-file stand-in at test/test_update_installer_graceful_stop.py:77-82 already used that shape.

Notes

  • Item 5 — rides along: the new AUTOSDE.yaml item 15 is a permanent review rule for a flake in one file. The new testing-conventions section already records it ("test/test_update_installer_graceful_stop.py is the example"). Subtraction: delete item 15 and keep the conventions section as the only record.

What this change ships

Inventory (5 items) — 4 justified

Intent: stop a macOS flake where the installer stand-in's rollback is sometimes never recorded. This is a FIX.

  1. The stand-in's long step runs in its own process group, and teardown reaps it through steppid — justified
  2. _stop_installer's default grace is read when it is called, not when it is defined; product behaviour is unchanged — justified
  3. The CommandProvider rollback tests get the suite's safety net as their TERM grace — justified
  4. The cancel-settle bound uses FALLBACK_TIMEOUT_SECS instead of a literal 15 — justified
  5. A new item 15 in AUTOSDE.yaml plus a new testing-conventions section — rides along

[FIRST-PRINCIPLES-REVIEWED] b83cd3d

Description read

[DESCRIPTION-READ] 5d78f2d9fa0eb8e9498fac59fdde7795a0dbe6053332a8724796c43fd8f70ecf

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b83cd3d6c9bc1cf4dff46161404836c86752cd2c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b83cd3d

Verdict parsed from the review's SHA-scoped output markers for commit b83cd3d6c9bc1cf4dff46161404836c86752cd2c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b83cd3d6c9bc1cf4dff46161404836c86752cd2c false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b83cd3d6c9bc1cf4dff46161404836c86752cd2c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b83cd3d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b83cd3d6c9bc1cf4dff46161404836c86752cd2c false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 merged commit 8d4129d into main Nov 17, 2026
101 of 103 checks passed
@bolichen97
bolichen97 deleted the macflaky/update-installer-cancel-race branch November 17, 2026 06:18
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant