Skip to content

test: wait on observed state in five Linux-flaky tests - #17821

Merged
iamwhatever merged 1 commit into
mainfrom
refactor/df-b3b-linux-races
Oct 7, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
refactor/df-b3b-linux-races

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Five backend tests that went red on loaded Linux shards, or passed with the guard they name removed, now wait for the state they assert, so they stop failing unrelated PRs and fail when their code really breaks.

Four of these tests failed on PRs that did not touch them; the fifth passed for the wrong reason:

Test Symptom
test_design_tweak_relay_paths.py::TestRelayWsPump::test_pump_forwards_and_exits_on_close TimeoutError: timed out on the client's frame recv (#17287)
crew container test_supervisor_process.py::test_terminate_drains_a_cooperative_child_without_sigkill assert -15 == 0: the fake backend died on the default SIGTERM action (#17071)
test_overload_integration_glue.py::test_admitting_id_is_held_until_the_posted_defer_landed AttributeError: _admitting_ids (#14453)
test_browser_cli_view.py::test_free_port_is_not_hardcoded assert 52489 != 52489: the kernel handed back the port it had just released (#14011)
three test_md_notebook.py lock tests their "the contender is still waiting" checks pass with the lock removed when the contender is slow to arrive

Why it matters

Each red lands on whatever PR shares the shard, and the usual answer is a rerun, which spends shared runner time and hides real failures in the same job. The md-notebook checks are worse: they guard the save and sync locking that keeps one note edit from silently overwriting another, and under load they stop guarding it at all.

Not a goal

  • No production change: every fix is in the five test files.
  • No new shared helper: the fixes use kiro_crew.testing.wait as it is on main.
  • Other timing shapes in these files that are not on the DF-B3b card stay for their own scopes: test_browser_cli_view.py's bind-close _free_port helper and its other short literal waits, the feeder sleeps in the other relay tests, and the elapsed < 4.0 bound in test_supervisor_process.py.
  • No CI or workflow change.

What changed (motivation → approach → change)

Symptom → cause. Each test bet on timing instead of observing what it asserts. The relay test's feeder thread paced itself with two 0.1 s sleeps; a relay thread that started late read the handshake and the frame in one buffer, and the client's handshake read took the frame with it. The fake backend wrote its pidfile before it installed its SIGTERM handler, so a drain that signalled as soon as the pidfile appeared hit the default action. The admission test gave the spawn twenty 5 ms loop turns and then read an attribute the spawn creates only after several off-loop hops. The free-port test assumed two ephemeral ports differ. The md-notebook tests slept 0.2 s, or yielded 100 times, before asserting that a second request had not finished, and a request that had not reached the lock yet passed that check too.

Approach. Wait on a signal, never a duration. Every wait the test itself must end is bounded by a named lost-run constant that fails by name. Each "still waiting" check first sees the contender parked on the lock.

Change.

Test Fix
relay pump no feeder thread: upstream answers the request it read, then sends its frame once the client holds the whole handshake; _recv_until reads each step under _PUMP_LOST_RUN_SECS; both directions are asserted, and the pump must end on the upstream hangup
fake backend (test_supervisor_fakes.py) SIGTERM disposition first, then the pidfile, written aside and renamed into place; the leader's pidfile also waits until a forked child is recorded for --reap-on-term, and the child's pidfile follows its own disposition
admission defer waits for the gated defer to be running (or the spawn to end) with async_wait_until; the gate is released in a finally; the spawn await is bounded by _DEFER_LOST_RUN_SECS
free port a recording socket on view's own socket binding proves each of two calls binds (LOOPBACK_HOST, 0) and returns that bind's kernel answer; the two numbers are never compared
md-notebook locks _parked_behind waits until the Sync or the later save is parked on the vault or per-note lock (until_parked), failing by name if it finishes first; the parked first request is released in a finally

Backwards compatibility

Compatible: the diff only changes test files. No production module, fixture name or test node id changes.

Removes nothing: every assertion is kept or made stricter. The relay test also asserts the client-to-upstream frame it used to send unchecked, and the free-port test asserts each call's bind and answer instead of comparing two ports.

Tests

All runs: Linux, CPython 3.12.13, under the campaign gate slot. The fix was proven on base cd0298429 and rebased onto cb397379f (range-diff identical; upstream touched none of these tests or the code they drive), where the four changed test/ files pass again. The probes are out-of-tree pytest plugins and are not committed.

Forced condition: parent red, head green.

Test Forced condition Parent cd0298429 Head
relay pump _relay_ws starts 0.3 s late (also 1.0 s on the head) TimeoutError: timed out passed
fake backend, cooperative drain the fake installs its SIGTERM handler 0.2 s late assert -15 == 0 36 passed (all supervisor tests)
fake backend, escaped-worker reap the leader records its forked child 0.3 s after the fork backend's drain should have reaped the escaped worker 36 passed
admission defer KC_FORCE=slow_thread, 0.2 s AttributeError: _admitting_ids passed
free port every port-0 bind reports the first port the kernel chose assert 52489 != 52489 passed

Mutation proofs (negative controls). Each reverts the guard a fixed test pins, in a throwaway edit restored from a copy.

Guard removed Head Parent
api_sync no longer takes vault_write_lock red: the Sync finished without ever waiting on the lock under slow_thread 0.2 its not sync.done() check passes; it fails only on a later outcome check
a tokenless save skips the per-note lock when another save holds it red: the later save finished without ever waiting on the lock; until_parked ... still false both tests pass (vacuous)
a tokenless save takes no per-note lock red on the existing locked() checks red on the same checks
_free_port returns a fixed 51515 red: a call did not leave the port to the kernel red

Repeats (host 1-minute load 7-11).

Run Result
20x -n0, the six changed backend tests 20/20 green, ~2.3 s each
20x -n0, test_supervisor_process.py + test_supervisor_backend.py (36 tests on the changed fake) 20/20 green, ~6 s each
10x -n 4 --dist loadgroup --max-worker-restart=2, the four changed test/ files plus test_design_tweak_backend.py and test_md_notebook_git_ops.py (780 tests) 10/10 green, ~20 s each
shuffled order, seeds 17287, 14453, 4011 (487 tests in the four files); container seeds 17071, 2026 all green
coarse_clock, the 3.12 Windows clock model (wall,tick64), slow_thread 0.3 s, gil_hog over the six tests all green
full crew container suite with CREW_CONTAINER_TESTS_REQUIRED=1 430 passed

Also run: the flake-pattern ratchet (passes; test_design_tweak_relay_paths.py drops from 2 to 1 short literal wait), semgrep 1.78 with CI's rule set in baseline mode (0 findings; the custom-rule sleep hits in these files drop from 21 to 19), the comment-history, brand-name, harness-parity, loop-bound-lock, testpaths and working-directory gates, black, isort, flake8 and mypy src/kiro_crew. Local pre-push review: CI's GPT and Opus review contracts, extracted by local_review.py, were both run on Opus 5.5; the GPT lane therefore ran on Opus at reduced fidelity, because no GPT model is available here. Neither found anything. A separate three-lens adversarial review found two issues, both fixed here: comment-history markers, and a single-call free-port check that a cached answer would pass.

local-gate.py --dry-run selects 9 backend files and 325 frontend cross-surface files; the backend set was run, the frontend set was not because the diff touches no website/** path.

Manual verification

N/A: unit coverage is the subject. The exact-SHA Linux (3.12, 1-8), Windows (1-8) and macOS lanes are the cross-platform evidence and are recorded on this PR.

Related Issues

Fixes #17287
Fixes #17071
Fixes #14453
Fixes #14011
Part of #16536

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Five backend tests failed intermittently on loaded Linux shards, or
passed vacuously, because each bet on timing instead of observing the
state it asserts:

- design-tweak relay pump: a feeder thread paced with fixed sleeps
  raced the relay thread; a late relay read the handshake and the
  frame in one buffer and the client's handshake read took the frame.
  Each step now waits on bytes the other side has observed, with a
  named lost-run bound on every read and join.
- crew container fake backend: the pidfile appeared before the SIGTERM
  handler, so a drain that signalled on it read -15 instead of 0. The
  pidfile is now written atomically after the disposition and after
  the forked child is recorded for --reap-on-term.
- admission defer: twenty 5 ms turns, then an unconditional read of a
  lazily created attribute. The test now waits for the gated defer to
  be running, always releases the gate, and bounds the spawn await.
- browser view free port: two calls must differ, which the kernel
  never promised. The test now proves each call binds port 0 and
  returns that bind's own kernel answer.
- md-notebook locking: three "contender still waiting" negatives ran
  after a fixed sleep or yield count, so a contender that had not
  reached the lock passed them too. Each now waits until the contender
  is parked on the lock, through kiro_crew.testing.wait.until_parked.

Test-only: no production code changes.

Fixes #17287
Fixes #17071
Fixes #14453
Fixes #14011
Part of #16536
@bolichen97
bolichen97 requested a review from a team as a code owner October 7, 2026 19:08
@bolichen97
bolichen97 requested a review from buluoray October 7, 2026 19:08
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 446bf128fbbaf08ae97a20f19d832f9318db776a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that _free_port builds its socket through the module-level socket name; the recording-socket test only sees binds made that way.

[DESIGN-REVIEWED] 446bf12

Description read

[DESCRIPTION-READ] cf0c1eed74ae82d6af2aa8842dbf48c3eda9a3c6b0896edeeb06d0ab14ebf117

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 446bf128fbbaf08ae97a20f19d832f9318db776a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 446bf12

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 446bf128fbbaf08ae97a20f19d832f9318db776a: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 446bf128fbbaf08ae97a20f19d832f9318db776a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 446bf12

Verdict parsed from the review's SHA-scoped output markers for commit 446bf128fbbaf08ae97a20f19d832f9318db776a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 446bf128fbbaf08ae97a20f19d832f9318db776a: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Merged-tree probe (no push): green_age.py reports this head stale against main bf946c04, which is 20 commits newer than its base cb397379. None of those commits touches the five files this PR changes, and git merge-tree is clean. Three of them change code these tests drive: the crew-container supervisor and conftest (#13374), subagent.py (#17606), and md-notebook git_ops.py (#16930). So this PR's commit was cherry-picked onto bf946c04 in a throwaway detached checkout and tested there:

Run Result
the four changed test/ files + test_md_notebook_git_ops.py + test_ci_surface_tests.py, -n 4 --dist loadgroup 561 passed
test_flake_pattern_ratchet.py 53 passed
full crew container suite, CREW_CONTAINER_TESTS_REQUIRED=1, -n0 (main's new test_supervisor_startup_order.py and test_supervisor_exit_code.py included) 713 passed

The branch head is unchanged at 446bf128f.

@iamwhatever
iamwhatever merged commit 32e36a0 into main Oct 7, 2026
106 checks passed
@iamwhatever
iamwhatever deleted the refactor/df-b3b-linux-races branch October 7, 2026 20:18
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants