flaky: 'a stopped process turn (real processes, real clock) > ignores its SIGTERM ... nothing survives' failed once on CI at 14.2 s #1766

Activity

1. gHashTag commented on Oct 10, 2026

   

   gHashTag

   on Oct 10, 2026

   OwnerAuthor

   More actions

   Seen again in the same real-process family, on the vault ship PR #1780, Tests / server-api, attempt 1 of 3:

   • the failing test was queen-turn-stop-bench "B: process turns that leave a grandchild";
   • it passed on attempt 2 and failed on attempt 3, whose only change from attempt 2 was a stderr print in an unrelated vault test;
   • locally it passed twice on each side.

   Treat it together with the SIGTERM test above. The fix should assert the guarantee (no surviving grandchild) with a bounded wait instead of a wall-clock margin, plus a negative control.
2. gHashTag commented on Oct 11, 2026

   

   gHashTag

   on Oct 11, 2026

   OwnerAuthor

   More actions

   Upgraded from "flake" to "likely real bug", 2026-10-11.

   Evidence. Run 38094454978 (the hosting ship PR #1796), attempt 2, tests/api/queen-turn-stop.test.ts:

   • Lines 337–340 pass: the actor is dead, the grandchild alive, held is 1 and escalated 0.
   • Line 341 fails: until(() => endedAt > 0, TURN_STOP_GRACE_MS + 10_000) returns false.

   A turn deaf to SIGTERM had still not ended 13 s after its kill, against a 3 s grace. Either the group escalation never fired, or its exit was never seen. The queen-turn-stop-bench "B" grandchild test failed in the same run.

   Production impact is bounded. TRIOS_QUEEN_TURN_STOP=on has been live since 20:38Z.

   • A model call hears its abort (2 ms in the same run). Model calls are the common hang, about 1 an hour in production.
   • With the flag off, a deaf process was abandoned anyway, so on is not worse than off on this path.
   • The flag stays on while a lane reproduces the failure under a CPU burner and fixes the root cause.

   The hosting ship #1796 waits for that fix.
3. dmitrii-f-t27 commented on Oct 11, 2026

   

   dmitrii-f-t27

   on Oct 11, 2026

   Collaborator

   More actions

   Fix in #1809, for the queen branch.

   Boundary.

   • In scope: stopTurn in queen-actors.ts (the grace timer) and one new test in tests/api/queen-turn-stop.test.ts.
   • Out of scope: no card, .t27, wasm or PIN change. processWork, the bench and the existing asserts are unchanged.

   User scenarios & testing.

   • A process turn deaf to SIGTERM is killed at its bound.
   • Its grace timer may come back before Date.now shows 3000 ms.
   • The runtime must still escalate at the grace and kill the whole group.

   Requirements.

   • stop_signal below the grace means "not yet", never "never".
   • No escalation before the grace.
   • Exactly one escalation at the grace.

   Success criteria.

   • The virtual-clock test with timers 1 ms early fails on the old code and passes on the fix.
   • The real-process SIGTERM test passes 20 of 20 under a CPU burner.
   • With the group kill replaced by a one-pid kill, the SIGTERM test and bench B fail.
   • Tests / server-api is green on the PR head.

   Dmitrii Fedorov (@dmitrii-f-t27)
4. dmitrii-f-t27

   added a commit that references this issue on Oct 11, 2026

   Merge pull request #1809 from gHashTag/sweep/trios-1766-turn-stop-bou…

   ...

   This issue will close once commit b30434c is merged into the 'main' branch.

   b30434c
5. dmitrii-f-t27 commented on Oct 11, 2026

   

   dmitrii-f-t27

   on Oct 11, 2026

   Collaborator

   More actions

   Fixed by #1809, merged into queen as b30434c.

   Cause. The grace timer could come back before Date.now showed 3000 ms, because Bun's setTimeout runs on its own monotonic clock. stop_signal then answered X_NONE, and the escalation was dropped for good.

   Fix. The runtime now asks the card again after the rest of the grace.

   Checks on PR head 996dd74.

   • Tests / server-api on Linux: 2009 pass, 0 fail.
     • The SIGTERM test escalated 3001 ms after the kill.
     • Bench B, after side: deaf turns freed at 3003 ms, 0 survivors.
   • New virtual-clock test: fails on the old code, passes on the fix.
   • Under a CPU burner (macOS): 20 of 20 for the SIGTERM test.
   • Negative control: with a one-pid kill instead of a group kill, both the SIGTERM test and bench B fail.

   Not checked. One green Linux run does not prove the flake is gone. The next queen runs are the evidence.

   CI. runner / Biome and runner / Typecheck are red with the same error counts as on base queen c1d8020 (run 38097729420). None of those errors is in the changed files.

   Dmitrii Fedorov (@dmitrii-f-t27)
6. dmitrii-f-t27

   closed this as completedon Oct 11, 2026
7. dmitrii-f-t27

   added a commit that references this issue on Oct 11, 2026

   fix(queen): a grace timer that comes back early asks the card again, …

   ...

   This issue will close once commit 996dd74 is merged into the 'main' branch.

   996dd74