Repository navigation
flaky pglive: 'PgLink at its edges > an acknowledgement the store refuses fences the node at once' races on CI #1778
Description
Part of #1729.
tests/pglive/queen-actors-pg-edges-live.test.ts:291 fails intermittently against the real PostgreSQL in Tests / server-pglive:
expect(said('Queen actor node fenced itself')[0][1]).toMatchObject({...})
- Expected - 1
+ Received + 2
Where it has been seen:
queen's run at 56c5df7 (2026-10-10, reported by the ship-2 lane);- the self-hosting ship PR queen(hosting): ship self-hosting - hosts join, full lab verdict, sandbox, sybil-proof credit, signed epoch statements (behind TRIOS_HOSTING, off) #1776, run 38086264626, attempt 1.
It passed on #1754 and on other runs. The two outcomes are a race between the old node's beat renewal and its acknowledgement reaching the store. #1746 made the sibling "a newer start fences the old one" test assert what holds either way: the node is fenced, the send goes nowhere, and FENCE_SQL admits no write. This test still pins one ordering of the fence log fields.
Fix: assert the guarantee (fenced, no write from the old incarnation, one fenced itself line) without pinning which of the two racing causes is logged. A negative control must still fail: let the old incarnation write after the fence.
Acceptance: 20 consecutive pglive runs of this file pass.
Activity
dmitrii-f-t27 commentedon Oct 11, 2026
Working on this in #1808 (base
queen).Boundary. Test only:
tests/pglive/queen-actors-pg-edges-live.test.ts. No change toqueen-actors-pg.tsor to the netlink rules in gHashTag/t27specs/queen/netlink.t27, so no.t27change.User scenarios and testing. The old node's acknowledgement is refused after another start claims its row. Its own-loop renewal may be refused first. Either way the node must fence once and write nothing.
Requirements.
fenced itselfline for node 13 and its incarnation;whyis the acknowledgement or the renewal.FENCE_SQLadmits no write of that incarnation.Success criteria. 20 consecutive runs of the file pass; the negative control fails;
Tests / server-pgliveis green on the PR head SHA.Dmitrii Fedorov (@dmitrii-f-t27)
test(queen): the heartbeat's refused expiry is a third racing cause (#…
Merge pull request #1808 from gHashTag/fix/1778-pglive-fence-race
dmitrii-f-t27 commentedon Oct 11, 2026
Fixed by #1808, merged into
queenas 8ea4b36 (head 2217287).The test now asserts the fence guarantee (mail handled, one fence line for node 13 and its incarnation, a refused write as the cause, the acknowledgement deleted nothing,
FENCE_SQLadmits no write of the old incarnation) and no longer pins acknowledgement vs. heartbeat renewal or expiry. A second case forces the heartbeat-first order every run. Negative control (unfenced acknowledgement) fails both cases.Acceptance: 20 consecutive local runs of the file at 2217287 with 0 test failures; 2 of the 20 ended in the Bun teardown panic tracked in #1813, after all tests had passed. CI
Tests / server-pgliveon 2217287: attempt 1 hit the same #1813 panic with all tests passed, attempt 2 passed (155 pass, 0 fail). Closing here; the panic stays open in #1813.Dmitrii Fedorov (@dmitrii-f-t27)
test(queen): pglive PgLink edge test asserts the fence guarantee, not…