Skip to content

test(qwp): deflake close-lifecycle interrupt tests on saturated CI agents - #72

Merged
bluestreak01 merged 1 commit into
mainfrom
fix/qwp-worker-quiescence-test-stability
Jul 23, 2026
Merged

test(qwp): deflake close-lifecycle interrupt tests on saturated CI agents#72
bluestreak01 merged 1 commit into
mainfrom
fix/qwp-worker-quiescence-test-stability

Conversation

@bluestreak01

@bluestreak01 bluestreak01 commented Jul 23, 2026

Copy link
Copy Markdown
Member

Tandem PRs: OSS server CI — questdb/questdb#7387 (pins this branch head for CI validation; move the pin to the merge commit once this lands) · Enterprise e2e — questdb/questdb-enterprise#1127

Summary

Test-only change. Make facadeCloseIsBoundedUnderRepeatedInterruptsDuringQueryCreation and facadeCloseIsBoundedUnderRepeatedInterruptsDuringSenderCreation robust on saturated CI agents: widen the close budget the tests configure from 100ms to 1s, and turn the "storm landed repeatedly" post-join count assert into a synchronized precondition proven while close() is still inside its bounded creation wait.

This addresses the intermittent failure in QuestDB build 254645 (on mac-other): test did not repeatedly interrupt query close, where the interrupt counter read 0/1 while every product-side assertion held.

Root cause

The tests assert, after closer.join(), that a free-running interrupter thread incremented its counter more than once. That is a liveness bet against the scheduler:

  • the 100ms creation-wait budget starts ticking when db.close() is called — before the interrupter thread even exists;
  • awaitCreationWaiter's poll latency and the interrupter's Thread.start()-to-first-quantum latency are paid out of that same window;
  • the interrupter yields after every iteration, sending itself to the back of the run queue on a busy box.

On a hosted 3-core mac agent running sibling surefire forks, the interrupter got at most one quantum inside the window. The failing run's log shows close honoring its deadline exactly (QueryClientPool.close(): 1 query client creation(s) still in flight after 100ms) and returning promptly — the product invariant the test protects (repeated interrupts must not restart the bounded deadline) was intact; only the test's own precondition failed.

Changes

  • Widen the creation-wait budget in the two interrupt tests from 100ms to 1s, making the window the storm must land in ~10x the worst observed starvation.
  • New awaitRepeatedInterrupts(count, closerStillWaiting, message) helper, called right after interrupter.start(): holds the test until at least two interrupts have landed while the closer is verifiably still inside its bounded wait. The deadline-restart property is now exercised on every run, and a pathologically starved interrupter fails with an explicit close left its creation wait before the interrupt storm landed twice; interrupts landed: N diagnosis instead of a misleading positive-control assert.
  • Count-first ordering inside the helper: two interrupts observed while polling count as success no matter how quickly the budget expires afterwards, so the precondition cannot introduce a new race at the boundary.

Validation

  • Whole QuestDBImplCloseLifecycleTest class green 3/3 on an idle machine (2.1s — the two tests now deliberately burn their full 1s budgets).
  • Both interrupt tests green 3/3 with 3 visible cores (-XX:ActiveProcessorCount=3) while all 12 physical cores were saturated by external spin load — the same starvation conditions as the failing agent.

…ents

facadeCloseIsBoundedUnderRepeatedInterruptsDuring{Query,Sender}Creation
raced the 100ms close budget against thread scheduling: the post-join
assert needs the interrupter to land at least two interrupts inside the
budget window, but the budget clock starts before the interrupter thread
even exists, and a freshly started, yielding thread is not guaranteed
two scheduler quanta within 100ms on a saturated agent. On a hosted
3-core mac agent (questdb macwin build 254645) the count read <= 1 and
the positive control failed while the product honored its bounded
deadline exactly.

- widen the creation-wait budget to 1s in the two interrupt tests
- hold the test until two interrupts have demonstrably landed while the
  closer is still inside its bounded wait (awaitRepeatedInterrupts), so
  the deadline-restart property is exercised on every run and a starved
  interrupter is reported as such instead of as a count-assert failure

Verified: whole class green 3/3 on an idle machine; both interrupt
tests green 3/3 with 3 visible cores under full external CPU load.
Test-only change.
@bluestreak01
bluestreak01 merged commit 1d54a25 into main Jul 23, 2026
14 checks passed
@bluestreak01
bluestreak01 deleted the fix/qwp-worker-quiescence-test-stability branch July 23, 2026 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant