Repository navigation
test: wait on observed state in five Linux-flaky tests - #17821
Conversation
Five backend tests failed intermittently on loaded Linux shards, or passed vacuously, because each bet on timing instead of observing the state it asserts: - design-tweak relay pump: a feeder thread paced with fixed sleeps raced the relay thread; a late relay read the handshake and the frame in one buffer and the client's handshake read took the frame. Each step now waits on bytes the other side has observed, with a named lost-run bound on every read and join. - crew container fake backend: the pidfile appeared before the SIGTERM handler, so a drain that signalled on it read -15 instead of 0. The pidfile is now written atomically after the disposition and after the forked child is recorded for --reap-on-term. - admission defer: twenty 5 ms turns, then an unconditional read of a lazily created attribute. The test now waits for the gated defer to be running, always releases the gate, and bounds the spawn await. - browser view free port: two calls must differ, which the kernel never promised. The test now proves each call binds port 0 and returns that bind's own kernel answer. - md-notebook locking: three "contender still waiting" negatives ran after a fixed sleep or yield count, so a contender that had not reached the lock passed them too. Each now waits until the contender is parked on the lock, through kiro_crew.testing.wait.until_parked. Test-only: no production code changes. Fixes #17287 Fixes #17071 Fixes #14453 Fixes #14011 Part of #16536
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that [DESIGN-REVIEWED] 446bf12 Description read[DESCRIPTION-READ] cf0c1eed74ae82d6af2aa8842dbf48c3eda9a3c6b0896edeeb06d0ab14ebf117 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
Merged-tree probe (no push):
The branch head is unchanged at |
Problem / Motivation
Goal: Five backend tests that went red on loaded Linux shards, or passed with the guard they name removed, now wait for the state they assert, so they stop failing unrelated PRs and fail when their code really breaks.
Four of these tests failed on PRs that did not touch them; the fifth passed for the wrong reason:
test_design_tweak_relay_paths.py::TestRelayWsPump::test_pump_forwards_and_exits_on_closeTimeoutError: timed outon the client's framerecv(#17287)test_supervisor_process.py::test_terminate_drains_a_cooperative_child_without_sigkillassert -15 == 0: the fake backend died on the default SIGTERM action (#17071)test_overload_integration_glue.py::test_admitting_id_is_held_until_the_posted_defer_landedAttributeError: _admitting_ids(#14453)test_browser_cli_view.py::test_free_port_is_not_hardcodedassert 52489 != 52489: the kernel handed back the port it had just released (#14011)test_md_notebook.pylock testsWhy it matters
Each red lands on whatever PR shares the shard, and the usual answer is a rerun, which spends shared runner time and hides real failures in the same job. The md-notebook checks are worse: they guard the save and sync locking that keeps one note edit from silently overwriting another, and under load they stop guarding it at all.
Not a goal
kiro_crew.testing.waitas it is onmain.test_browser_cli_view.py's bind-close_free_porthelper and its other short literal waits, the feeder sleeps in the other relay tests, and theelapsed < 4.0bound intest_supervisor_process.py.What changed (motivation → approach → change)
Symptom → cause. Each test bet on timing instead of observing what it asserts. The relay test's feeder thread paced itself with two 0.1 s sleeps; a relay thread that started late read the handshake and the frame in one buffer, and the client's handshake read took the frame with it. The fake backend wrote its pidfile before it installed its SIGTERM handler, so a drain that signalled as soon as the pidfile appeared hit the default action. The admission test gave the spawn twenty 5 ms loop turns and then read an attribute the spawn creates only after several off-loop hops. The free-port test assumed two ephemeral ports differ. The md-notebook tests slept 0.2 s, or yielded 100 times, before asserting that a second request had not finished, and a request that had not reached the lock yet passed that check too.
Approach. Wait on a signal, never a duration. Every wait the test itself must end is bounded by a named lost-run constant that fails by name. Each "still waiting" check first sees the contender parked on the lock.
Change.
_recv_untilreads each step under_PUMP_LOST_RUN_SECS; both directions are asserted, and the pump must end on the upstream hanguptest_supervisor_fakes.py)--reap-on-term, and the child's pidfile follows its own dispositionasync_wait_until; the gate is released in afinally; the spawn await is bounded by_DEFER_LOST_RUN_SECSview's ownsocketbinding proves each of two calls binds(LOOPBACK_HOST, 0)and returns that bind's kernel answer; the two numbers are never compared_parked_behindwaits until the Sync or the later save is parked on the vault or per-note lock (until_parked), failing by name if it finishes first; the parked first request is released in afinallyBackwards compatibility
Compatible: the diff only changes test files. No production module, fixture name or test node id changes.
Removes nothing: every assertion is kept or made stricter. The relay test also asserts the client-to-upstream frame it used to send unchecked, and the free-port test asserts each call's bind and answer instead of comparing two ports.
Tests
All runs: Linux, CPython 3.12.13, under the campaign gate slot. The fix was proven on base
cd0298429and rebased ontocb397379f(range-diff identical; upstream touched none of these tests or the code they drive), where the four changedtest/files pass again. The probes are out-of-tree pytest plugins and are not committed.Forced condition: parent red, head green.
cd0298429_relay_wsstarts 0.3 s late (also 1.0 s on the head)TimeoutError: timed outassert -15 == 0backend's drain should have reaped the escaped workerKC_FORCE=slow_thread, 0.2 sAttributeError: _admitting_idsassert 52489 != 52489Mutation proofs (negative controls). Each reverts the guard a fixed test pins, in a throwaway edit restored from a copy.
api_syncno longer takesvault_write_lockthe Sync finished without ever waiting on the lockslow_thread0.2 itsnot sync.done()check passes; it fails only on a later outcome checkthe later save finished without ever waiting on the lock;until_parked ... still falselocked()checks_free_portreturns a fixed 51515a call did not leave the port to the kernelRepeats (host 1-minute load 7-11).
-n0, the six changed backend tests-n0,test_supervisor_process.py+test_supervisor_backend.py(36 tests on the changed fake)-n 4 --dist loadgroup --max-worker-restart=2, the four changedtest/files plustest_design_tweak_backend.pyandtest_md_notebook_git_ops.py(780 tests)coarse_clock, the 3.12 Windows clock model (wall,tick64),slow_thread0.3 s,gil_hogover the six testsCREW_CONTAINER_TESTS_REQUIRED=1Also run: the flake-pattern ratchet (passes;
test_design_tweak_relay_paths.pydrops from 2 to 1 short literal wait), semgrep 1.78 with CI's rule set in baseline mode (0 findings; the custom-rule sleep hits in these files drop from 21 to 19), the comment-history, brand-name, harness-parity, loop-bound-lock, testpaths and working-directory gates, black, isort, flake8 andmypy src/kiro_crew. Local pre-push review: CI's GPT and Opus review contracts, extracted bylocal_review.py, were both run on Opus 5.5; the GPT lane therefore ran on Opus at reduced fidelity, because no GPT model is available here. Neither found anything. A separate three-lens adversarial review found two issues, both fixed here: comment-history markers, and a single-call free-port check that a cached answer would pass.local-gate.py --dry-runselects 9 backend files and 325 frontend cross-surface files; the backend set was run, the frontend set was not because the diff touches nowebsite/**path.Manual verification
N/A: unit coverage is the subject. The exact-SHA Linux (3.12, 1-8), Windows (1-8) and macOS lanes are the cross-platform evidence and are recorded on this PR.
Related Issues
Fixes #17287
Fixes #17071
Fixes #14453
Fixes #14011
Part of #16536
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)