Skip to content

test(chat): wait for the queued-prompt writer to settle, not its file - #20433

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/queued-prompt-writer-test-barrier
Oct 20, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/queued-prompt-writer-test-barrier

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The queued-prompt one-writer tests wait for the writer to settle, so they pass on every runner and still fail when a second writer starts.

test_a_send_during_a_write_does_not_start_a_second_writer failed on Backend Tests (Windows) (7) with assert True is False on slot.queue_persist_pending, on a pull request that does not touch the queued-prompt writer (#20430 has the run). Only its pending-flag assertion failed.

Why it matters

An unrelated red shard costs every open pull request a rerun or a fresh head. The test also guards a real property: a prompt typed during a turn reaches disk with one writer at a time.

Not a goal

  • Any change to the queued-prompt writer, the save or the queue witness. The production order is correct (below).
  • The other short fixed waits in the tree. This only touches the one file.

What changed (motivation → approach → change)

Symptom: the test polled the file until both prompts were on disk, then asserted queue_persist_pending is False, which on Windows was still true.

Cause, test-only: the file is not the barrier. The save commits the file and only then records the queue witness (_queue_persisted_sig) under the transcript lock. The test reads the file without that lock, so a read between the two sees both prompts on disk while the witness still holds the old value. The forced interleaving under Tests reproduces the CI assertion.

Why it is not a production race: the writer's done callback runs on the loop after the executor thread returns, so it always reads queue_persist_pending after the witness is recorded, and a second immediate writer cannot start from this window. Recording the witness after the commit is the intended order: the witness must never claim a write that has not landed.

Four other loop-side readers can see queue_persist_pending still true there, and each errs to "still owed":

  • chat_persistence.py:3023, the resumed-slot skip guard in the save: it does not skip, so one extra save runs.
  • dashboard_persistence.py:137, the periodic flush gate: one redundant flush, and the stale-snapshot guard orders the commits.
  • chat_api/handover_drain.py:258, the hand-over "nothing unsaved" check: it takes the save path instead of the no-write path.
  • channel_handoff.py:316, the close classification: it reports REFUSED_UNSAVED_CLOSE instead of QUEUED_BY_CLOSE, the documented fail-closed choice.

None starts an immediate writer: the window costs at most a redundant save or a refused close.

Change, in test/test_queued_prompt_durability.py:

  • A _queue_writer_settled(slot) helper waits, through async_wait_until, for slot._queue_persist_inflight to clear. The done callback clears it on the loop after the thread returns, and a follow-up pass sets it again in that callback, so a clear flag means the whole chain is done.
  • The three TestOneWriterPerSlot tests and the three async TestEnqueueStartsTheWriteImmediately tests use it in place of fixed-count polls and a sleep(0.1) barrier.
  • The one-writer test waits for the held write to start with async_wait_until(entered.is_set), not by parking entered.wait, 5 on the executor the write itself needs. release is set in a finally.
  • The one-writer test counts writer starts as well as overlap: on a one-thread executor a second writer queues behind the first and never overlaps it.
  • A new test, test_a_send_after_the_snapshot_gets_a_follow_up_pass, covers the follow-up pass: the held writer commits "first" and blocks, "second" arrives, and only the follow-up can land it before the periodic flush.
  • The held writers record a release timeout in never_released, asserted on the loop side (raised on the executor thread, it reached only the done callback's log). The hold waits half the settle's budget, so that assertion fails first, and the settle helper prints the list too.

Why not hold the transcript lock in the test's read: that couples the test to the save's locking, and the property under test is the slot's own settlement, which the single-flight flag signals.

Backwards compatibility

Compatible: test-only change. No production module is touched.

Tests

  • Forced interleaving (scratch repro, not committed): with the witness assignment held until the loop has read the file, main's barrier fails with assert True is False on queue_persist_pending. The new barrier passes with the same hold.
  • Mutation runs on the production writer, each red:
    • Start a second writer while the first is in flight and still record the debt: max(concurrent) == 1 fails on the default executor, and on a one-thread executor, where the old overlap-only check would pass, the start counts fail it (see below).
    • Drop the in-flight check: red.
    • Clear the single-flight flag at submit time instead of in the done callback: red (5 tests on the default executor, 6 on one thread).
    • Delete the follow-up pass in _finish_queue_persist: the new test fails with assert 1 == 2 on both executors. Before it, this mutation passed the whole file.
    • On a one-thread executor the second-writer mutation is caught by the new test on every run (assert 3 == 2). The one-writer test's start count catches it in most runs but not all (7 of 10 in the eight-test selection), because a writer queued behind the first may not have started yet when the flag clears.
  • Release withheld (the finally that sets release removed in both held-writer tests): both fail with AssertionError: the held write timed out waiting for release, 6 of 6 runs.
  • Determinism: on main 3d09514 the 8 tests in both classes passed 30 of 30 on the default executor and on a one-thread executor, and the same again under 32 CPU-spinner processes on a 32-core host. On main b4135f1: 30 of 30 at -n 4, and 30 of 30 serially with the two classes in reverse order.
  • test/test_queued_prompt_durability.py in full plus test/test_wait_discipline_ratchet.py and test/test_flake_pattern_ratchet.py: 216 passed with -n 4 on main b4135f1. --write-baseline changes nothing. Every test/test_*ratchet*.py plus test/test_agent_refactor_facade.py: 303 passed.
  • On main ed975d7: the file passed 119 of 119, four runs at -n 2; test/test_flake_pattern_ratchet.py plus test/test_wait_discipline*.py: 99 passed.
  • black, isort, flake8, the subprocess-encoding gate, the comment-history gate against the merge-base, and Semgrep 1.78.0 with the repo's semgrep/ rules against the merge-base: 0 findings on this diff.

Manual verification

N/A — unit coverage sufficient: test-only change, proven by the forced interleaving and the mutation runs.

Related Issues

Closes #20430

Pattern harvest

Rule candidate: review
Pattern: a test waits on a durable side effect (a file on disk) and then asserts in-memory settlement that the writer records after that effect. The barrier should be the writer's own completion signal.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 14, 2026 18:54
@jeeshofone
jeeshofone requested a review from mbriones98 October 14, 2026 18:54
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of e9056dc2486945352e6db090308117b2b523c9b7 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] e9056dc

Description read

[DESCRIPTION-READ] 2506ad0da0488679fe7bfa6b012f592e402ef3419758ec54b8643696ed71bd47

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision e9056dc2486945352e6db090308117b2b523c9b7 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision e9056dc2486945352e6db090308117b2b523c9b7 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed e9056dc2486945352e6db090308117b2b523c9b7 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e9056dc

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed e9056dc2486945352e6db090308117b2b523c9b7 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e9056dc

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@jeeshofone
jeeshofone force-pushed the fix/queued-prompt-writer-test-barrier branch from a9fa746 to a251fba Compare October 14, 2026 22:16
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for the independent review of a9fa746. New head: a251fba. Test-only; production is unchanged.

S1, no test exercised the owed follow-up pass: fixed. test_a_send_after_the_snapshot_gets_a_follow_up_pass (in TestOneWriterPerSlot): the held writer runs the real flush, commits "first", then blocks; "second" is sent while the single-flight is held; the writer is released. It asserts two writer starts, ["first", "second"] on disk and queue_persist_pending is False. Deleting the follow-up in _finish_queue_persist (if owed and slot.queue_persist_pending: start_queue_persist(...)) fails it with assert 1 == 2 on the default executor and on a one-thread executor; source restored and verified clean by diff.

N1, release timeout raised on the executor thread: fixed. Both held writers now append to a never_released list instead of asserting, and each test asserts the list is empty on the loop side, before any disk check.

N2, the body implied nothing observes the commit-to-witness window: fixed. "Why it is not a production race" now names the four loop-side readers that can see queue_persist_pending true there (the resumed-slot skip guard in the save, the periodic flush gate, the hand-over "nothing unsaved" check, the close classification) and why each errs to "still owed": a redundant save or the documented fail-closed REFUSED_UNSAVED_CLOSE, never a second immediate writer.

Mutation re-runs on the new head (8 tests in the two classes):

  • second writer started while one is in flight: default red (2 tests); one thread red on every run via the new test (assert 3 == 2). The one-writer test's own start count caught it in 7 of 10 one-thread runs, because a writer queued behind the first may not have started when the flag clears; the body says so.
  • in-flight check dropped: red (2 tests) on both executors.
  • flag cleared at submit: red (5 default, 6 one thread).

Determinism: 30/30 default and 30/30 one-thread, and the same under 32 CPU spinners on a 32-core host (spinners killed and confirmed gone). Whole file plus the wait-discipline and flake-pattern ratchets: 211 passed, also on the merge with current main; --write-baseline changes nothing. black, isort, flake8, subprocess-encoding and comment-history gates clean; Semgrep 1.78.0 with the repo rules against the merge-base: 0 new findings (the one hit is the unchanged _config_dir fixture).

The Backend Tests (3.12, 3) red on a9fa746 (and the Coverage Gate and PR Readiness that follow it) is test_skill_view_relaunch_harness, outside this diff: tracked in #20475, fix in #20485.

First-pass check: R1-R8 OK.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 14, 2026
@jeeshofone
jeeshofone force-pushed the fix/queued-prompt-writer-test-barrier branch from a251fba to 86befed Compare October 15, 2026 04:52
@jeeshofone

Copy link
Copy Markdown
Contributor Author

New head 86befed (rebased onto main 3d09514).

Light-check NIT (named release-timeout assertion unreachable): fixed. The held write now waits half the settle's budget, so it gives up first and assert never_released == [] is the failure reported, and _queue_writer_settled prints never_released in its timeout message as a second net. With the release.set() in both finally blocks removed, both held-writer tests fail with the held write timed out waiting for release, 6 of 6 runs.

Rebased onto main 3d09514, which carries the fix for the Backend Tests (3.12, 3) flake (#20475). Still one test-only commit; production unchanged. Re-proved on the new head: deleting the follow-up pass fails only the follow-up test (assert 1 == 2) on the default and one-thread executors; the 8 tests passed 30 of 30 in each of default / one-thread, with and without 32 CPU spinners; the file plus both ratchets 214 passed at -n 8, --write-baseline unchanged; black, isort, flake8, comment-history and Semgrep 1.78.0 (0 findings on the diff) clean.

Rulings on this PR: light-check NIT DONE (test/test_queued_prompt_durability.py:110, :795, :849); rebase DONE; earlier review S1 DONE (:828), N1 DONE (:795-797, :816, :864), N2 DONE (body). Body edited before the push and passes the description check and the Pattern harvest line check.

First-pass check: R1-R8 OK

@jeeshofone
jeeshofone force-pushed the fix/queued-prompt-writer-test-barrier branch from 86befed to 98ef278 Compare October 15, 2026 12:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026
@jeeshofone
jeeshofone force-pushed the fix/queued-prompt-writer-test-barrier branch from 98ef278 to c81ec89 Compare October 15, 2026 18:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 15, 2026
@jeeshofone
jeeshofone force-pushed the fix/queued-prompt-writer-test-barrier branch from c81ec89 to a5372b0 Compare October 16, 2026 12:37
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 16, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

New head a5372b0 (rebased onto main b4135f1). Still one test-only commit; git range-diff shows the patch unchanged.

Reds on the previous head c81ec89, none from this diff:

Rulings on this PR, checked on the new head:

  • The settle helper's failure names never_released: DONE, the describe of _queue_writer_settled (test/test_queued_prompt_durability.py:110).
  • The held write uses a shorter bound than the settle: DONE, release.wait(default_timeout() / 2) at :795 and :849, while the settle takes async_wait_until's default, default_timeout().
  • Rebased onto a main that carries test(acp): settle lease releases before the skill-view relaunch census #20485: DONE, b4135f1 contains its merge commit a1830f7.
  • The body names the four other readers that can see queue_persist_pending true, and why each errs safe: DONE. Re-read on b4135f1 (chat_persistence.py:3023, dashboard_persistence.py:137, chat_api/handover_drain.py:258, channel_handoff.py:316): same lines, same behaviour.

Body, edited before the push: one inaccurate clause fixed (the held write waits half of default_timeout(), not all of it), What changed cut to 494 words, the stale Windows (1) line removed, test counts updated.

Re-proved on the new head:

  • Deleting the follow-up pass in _finish_queue_persist fails only test_a_send_after_the_snapshot_gets_a_follow_up_pass (assert 1 == 2). Source restored.
  • The 8 tests in the two classes: 30 of 30 at -n 4, and 30 of 30 serially with the classes in reverse order.
  • The file plus the wait-discipline and flake-pattern ratchets: 216 passed, and --write-baseline changes nothing. Every test/test_*ratchet*.py plus test/test_agent_refactor_facade.py: 303 passed.
  • black (baselined checker), isort, flake8, subprocess-encoding and comment-history gates clean. Semgrep 1.78.0 with the repo's semgrep/ rules against the merge-base: 0 findings on this diff (the one hit in the file is the unchanged _config_dir fixture). Description check and Pattern harvest line pass; body 7344 B.

AUTOSDE rules that match the file:

  • tests-are-deterministic, a-stopwatch-is-not-a-barrier: the barrier is the writer's single-flight flag through async_wait_until. No sleep and no wall-clock bound; the held writes are released in a finally.
  • no-test-side-effects: state lives under tmp_path, and the test double replaces flush_slot_now on that test's own state object only.
  • a-ratchet-may-only-tighten: one baseline line removed; --write-baseline unchanged.
  • an-invariant-line-needs-a-mutation-pin: the mutation above, plus the earlier runs in this thread.
  • an-async-test-respects-the-on-loop-lock: the metadata read is a plain file read with no file_lock.

First-pass check: R1-R8 OK.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 16, 2026
mbriones98
mbriones98 previously approved these changes Oct 18, 2026

@mbriones98 mbriones98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green and readiness passed at a5372b0; no new feature.

…file

TestOneWriterPerSlot polled the transcript until both queued prompts were on
disk, then asserted queue_persist_pending was false. The save commits the file
and only then records the queue witness, so a read in between sees the new
queue while pending is still true. On a slow Windows runner the poll landed in
that window and the last assertion failed.

The tests in this class and in TestEnqueueStartsTheWriteImmediately now wait
for the single-flight flag to clear. The writer's done callback clears it on the
loop after the executor thread returned, and a follow-up pass sets it again in
the same callback, so a clear flag means the whole chain is done. The writer
tests count starts as well as overlap, so a second writer queued behind the
first on a one-thread executor still fails the class.

A new test covers the follow-up pass itself: the held writer commits "first"
and then blocks, "second" arrives, and only the follow-up can land it. The held
writers record a release timeout for the test to assert, instead of raising it
on the executor thread where only the done callback's log would see it.

Closes kirodotdev#20430
@bolichen97

Copy link
Copy Markdown
Contributor

[operator: bolichen97] Rebased onto main ed975d7 to clear the merge conflict: a5372b0 -> e9056dc. Still one test-only commit.

  • Conflict: main (test(small-files-1): replace fixed waits with signal waits #21557-style signal-wait sweep) had already swapped this file's fixed-count polls for async_wait_until. Every main-side hunk sat inside a site this PR rewrites, so the file is byte-identical to the PR's version at a5372b0.
  • Main has already dropped test_queued_prompt_durability.py from test/wait_discipline_baseline.txt, so that one-line change fell out of the diff; the body and commit message no longer claim it.
  • Local: the file 119/119 passed four times at -n 2; flake-pattern and wait-discipline ratchets 99 passed; black, isort, flake8, Semgrep clean.

This push dismissed @mbriones98's approval of a5372b0. A re-approval on e9056dc is welcome once CI is green.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

@mbriones98 I rebased this onto current main to clear the merge conflict (mechanical, no behaviour change). That dismissed your approval on a5372b0. The new head e9056dc is green with readiness: passed. Could you re-approve when you have a moment?

[operator: bolichen97]

@iamwhatever
iamwhatever merged commit 4731325 into kirodotdev:main Oct 20, 2026
101 of 102 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: one-writer queued-prompt test reads the file before the witness

4 participants