Skip to content

test: touch the crew log off the event loop in every async test - #16102

Merged
chenmingwei23 merged 2 commits into
mainfrom
test/deflake-issue-radar-on-loop-writes
Oct 2, 2026
Merged

chenmingwei23 merged 2 commits into
mainfrom
test/deflake-issue-radar-on-loop-writes

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: no test fails at random because it touched a crew log from the event-loop thread.

TestSweep::test_marks_for_finished_items_are_pruned failed on Backend Tests (Windows) (1) of #16096, a PR that does not touch issue radar, with CrewLedgerNotRecorded: ... the crew log did not take the append. The captured log shows the cause: could not acquire exclusive file lock: still held after waiting 0.00s (one attempt: an acquire on the event-loop thread never waits ...).

Why it matters

Any PR can be reddened by it, and the error text blames the crew log rather than the test. Each hit costs a shard rerun and an hour of someone proving it is not theirs.

Not a goal

  • Any change to file_lock's on-loop policy or to the crew store. Product callers already run these writes off the loop (asyncio.to_thread); only the test helper did not.
  • Other flakes; each gets its own fix.

What changed (motivation ? approach ? change)

Root cause: a crew log has two background holders of its per-unit lock -- the writer thread that lands every append and the eager folder that reads the unit right after each entry -- and file_lock makes ONE attempt on the event-loop thread. Any crew-log call made straight from an async def is refused whenever one of them holds the lock. Writes raise CrewLedgerNotRecorded; reads raise OSError or, for a best-effort reader, answer an empty record. Product handlers make these calls through asyncio.to_thread; these tests did not. A bounded sample of the newest 40 failed Windows backend shards (01:57 PDT) found the class on several unrelated PRs: issue-radar seeds, test_discord.py (3 PRs, OSError: could not acquire exclusive file lock ... event-loop thread never waits) and test_session_ledger.py (3 PRs, assert '' == 'fold me').

  • Writes: the three helpers that write the crew log (_item in the runtime tests, _work in the fabric tests, _seed_item in the deps tests) go through the existing off_loop from test/off_loop_helpers.py.
  • Reads: test_discord.py reads the session log through a _read_session_log helper called via off_loop (the named test plus four TestDispatcher siblings with the same on-loop read); test_session_ledger.py::test_route_write_lands_under_ledger_key reads back via off_loop(sl.read_state, ...), as api_session_ledger_get does; one latent on-loop read in test_work_ledger_projection.py (found by the pin) does the same.
  • TestSeedingSurvivesABriefLogLock holds the crew log lock from another thread during the read-back: the seed from an async test succeeds, and the negative control proves the same write made on the loop is refused. The holder lets go on an event, never a timer: the read-back thread's own non-waiting probe must be refused first (asserted); off the loop that probe releases the holder, on the loop nothing does until the read-back gives up. Only the read-back's thread counts, so the crew log's own reader threads cannot release it early. 15/15 local repeats.
  • test/test_crew_log_off_loop_pin.py scans every test module and fails an async def that writes or reads a crew log (commit_work_progress, record_crew_checkpoint, open_session_log, session_ledger.read_state) directly or through a same-module sync helper; a nested fake or lambda handed to monkeypatch does not count. A synthetic negative control proves it sees direct, helper and read calls and spares a hop.
  • docs/system-specs/common/testing-conventions.md: one paragraph next to the "unawaited executor job holding a file_lock" entry naming both background holders, the symptoms, the off_loop hop, the pin, and the rule not to patch _on_event_loop.

The discord and session-ledger root causes were each reproduced deterministically on origin/main with an event-released held-lock probe (exact CI message on the old code, pass on the new); 10 consecutive local runs of each file pass.

Backwards compatibility

Compatible: test and doc changes only.

Tests

On Windows: the pin plus the runtime, fabric, deps, discord, session-ledger and work-ledger files pass (744). The race was 0 failures in 400 local stress runs, so the stochastic path is not a usable proof; the held-lock probe reproduces the exact CI error on the old helper and passes on the new one, deterministically. flake8, black gate and the comment-history gate are clean.

Manual verification

N/A — unit coverage sufficient; nothing user-visible changes.

Related Issues

Seen on #16096 (Backend Tests (Windows) (1), Backend Tests (3.12, 1)) and #16102 (Backend Tests (Windows) (7)).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 2, 2026 07:43
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 588c33e1dad85a6aace8a05774bcc670a615ab4f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 588c33e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 588c33e1dad85a6aace8a05774bcc670a615ab4f: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 588c33e1dad85a6aace8a05774bcc670a615ab4f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the pin's deliberate bare-read_state exclusion: from kiro_crew.session_ledger import read_state in an async test escapes _CREW_LOG_CALLS silently.

Suggestions

  • Have the pin resolve module import aliases (one pass over ast.ImportFrom) so read_state imported bare matches session_ledger.read_state, closing the only spelling the synthetic control doesn't cover.

[DESIGN-REVIEWED] 588c33e

Description read

[DESCRIPTION-READ] 0a1eead14483c1609e16553744498f292235e1dc7229fc12bd9e2deb9223cab7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 588c33e1dad85a6aace8a05774bcc670a615ab4f — this comment is updated in place on each push.

Review details

Nothing blocks; the new off-loop pin fences less than the spec paragraph it ships with claims.

FINDING — test/test_crew_log_off_loop_pin.py:27 — _CREW_LOG_CALLS omits iter_from (and CrewLog.open, the dominant opener in these suites), so the half-hop shape — handle = off_loop(projection.open_session_log, unit) followed by handle.iter_from(1, known=...) inside the same async def — is reported clean even though iter_from → _iter_segments → _durable_end → store._open_lock → file_lock makes a single, never-waiting acquire on the event-loop thread, while docs/system-specs/common/testing-conventions.md:4005 names iter_from among the calls and :4015 states the pin "fails any async def that makes such a call" → Fix: add "iter_from" (and the CrewLog.open / CrewLog.create spellings) to _CREW_LOG_CALLS, or narrow the spec sentence to the four spellings actually pinned.

[OPUS-REVIEWED] 588c33e

Verdict parsed from the review's SHA-scoped output markers for commit 588c33e1dad85a6aace8a05774bcc670a615ab4f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 588c33e1dad85a6aace8a05774bcc670a615ab4f: <one-sentence reason>

@bolichen97 bolichen97 changed the title test: seed issue-radar work items off the event loop test: write the crew log off the event loop in every async test Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision 588c33e1dad85a6aace8a05774bcc670a615ab4f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ⏭️ skipped

Revision 588c33e1dad85a6aace8a05774bcc670a615ab4f ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@bolichen97
bolichen97 force-pushed the test/deflake-issue-radar-on-loop-writes branch 2 times, most recently from 71f4730 to 3e8ac00 Compare October 2, 2026 08:56
_item wrote through the crew log on the event-loop thread whenever an async test called it. file_lock takes one attempt there, so a crew-log writer thread still inside its critical section made the read-back refuse and the seed raise CrewLedgerNotRecorded (Windows shard flake). Run the write on a worker thread, add a held-lock probe plus its on-loop negative control, and document the shape.
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97 bolichen97 changed the title test: write the crew log off the event loop in every async test test: touch the crew log off the event loop in every async test Oct 2, 2026
@bolichen97
bolichen97 force-pushed the test/deflake-issue-radar-on-loop-writes branch from 3e8ac00 to 71822c0 Compare October 2, 2026 09:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 2, 2026
The crew log's writer thread and eager folder hold the unit lock on their own schedule, and file_lock never waits on the event loop. Route issue-radar seeds and the discord, session-ledger and work-ledger read-backs through off_loop_helpers.off_loop, and pin the class with a whole-tree AST scan over crew-log writes and reads.
@bolichen97
bolichen97 force-pushed the test/deflake-issue-radar-on-loop-writes branch from 71822c0 to 588c33e Compare October 2, 2026 09:39
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@chenmingwei23
chenmingwei23 merged commit a34299d into main Oct 2, 2026
97 checks passed
@chenmingwei23
chenmingwei23 deleted the test/deflake-issue-radar-on-loop-writes branch October 2, 2026 15:08
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants