Repository navigation
test: touch the crew log off the event loop in every async test - #16102
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the pin's deliberate bare- Suggestions
[DESIGN-REVIEWED] 588c33e Description read[DESCRIPTION-READ] 0a1eead14483c1609e16553744498f292235e1dc7229fc12bd9e2deb9223cab7 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsNothing blocks; the new off-loop pin fences less than the spec paragraph it ships with claims. FINDING — test/test_crew_log_off_loop_pin.py:27 — [OPUS-REVIEWED] 588c33e Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
UX Review (Fable 5) — ⏭️ skippedRevision |
First Principles Review (Fable 5) — ⏭️ skippedRevision |
71f4730 to
3e8ac00
Compare
_item wrote through the crew log on the event-loop thread whenever an async test called it. file_lock takes one attempt there, so a crew-log writer thread still inside its critical section made the read-back refuse and the seed raise CrewLedgerNotRecorded (Windows shard flake). Run the write on a worker thread, add a held-lock probe plus its on-loop negative control, and document the shape.
3e8ac00 to
71822c0
Compare
The crew log's writer thread and eager folder hold the unit lock on their own schedule, and file_lock never waits on the event loop. Route issue-radar seeds and the discord, session-ledger and work-ledger read-backs through off_loop_helpers.off_loop, and pin the class with a whole-tree AST scan over crew-log writes and reads.
71822c0 to
588c33e
Compare
Problem / Motivation
Goal: no test fails at random because it touched a crew log from the event-loop thread.
TestSweep::test_marks_for_finished_items_are_prunedfailed onBackend Tests (Windows) (1)of #16096, a PR that does not touch issue radar, withCrewLedgerNotRecorded: ... the crew log did not take the append. The captured log shows the cause:could not acquire exclusive file lock: still held after waiting 0.00s (one attempt: an acquire on the event-loop thread never waits ...).Why it matters
Any PR can be reddened by it, and the error text blames the crew log rather than the test. Each hit costs a shard rerun and an hour of someone proving it is not theirs.
Not a goal
file_lock's on-loop policy or to the crew store. Product callers already run these writes off the loop (asyncio.to_thread); only the test helper did not.What changed (motivation ? approach ? change)
Root cause: a crew log has two background holders of its per-unit lock -- the writer thread that lands every append and the eager folder that reads the unit right after each entry -- and
file_lockmakes ONE attempt on the event-loop thread. Any crew-log call made straight from anasync defis refused whenever one of them holds the lock. Writes raiseCrewLedgerNotRecorded; reads raiseOSErroror, for a best-effort reader, answer an empty record. Product handlers make these calls throughasyncio.to_thread; these tests did not. A bounded sample of the newest 40 failed Windows backend shards (01:57 PDT) found the class on several unrelated PRs: issue-radar seeds,test_discord.py(3 PRs,OSError: could not acquire exclusive file lock ... event-loop thread never waits) andtest_session_ledger.py(3 PRs,assert '' == 'fold me')._itemin the runtime tests,_workin the fabric tests,_seed_itemin the deps tests) go through the existingoff_loopfromtest/off_loop_helpers.py.test_discord.pyreads the session log through a_read_session_loghelper called viaoff_loop(the named test plus fourTestDispatchersiblings with the same on-loop read);test_session_ledger.py::test_route_write_lands_under_ledger_keyreads back viaoff_loop(sl.read_state, ...), asapi_session_ledger_getdoes; one latent on-loop read intest_work_ledger_projection.py(found by the pin) does the same.TestSeedingSurvivesABriefLogLockholds the crew log lock from another thread during the read-back: the seed from an async test succeeds, and the negative control proves the same write made on the loop is refused. The holder lets go on an event, never a timer: the read-back thread's own non-waiting probe must be refused first (asserted); off the loop that probe releases the holder, on the loop nothing does until the read-back gives up. Only the read-back's thread counts, so the crew log's own reader threads cannot release it early. 15/15 local repeats.test/test_crew_log_off_loop_pin.pyscans every test module and fails anasync defthat writes or reads a crew log (commit_work_progress,record_crew_checkpoint,open_session_log,session_ledger.read_state) directly or through a same-module sync helper; a nested fake or lambda handed tomonkeypatchdoes not count. A synthetic negative control proves it sees direct, helper and read calls and spares a hop.docs/system-specs/common/testing-conventions.md: one paragraph next to the "unawaited executor job holding afile_lock" entry naming both background holders, the symptoms, theoff_loophop, the pin, and the rule not to patch_on_event_loop.The discord and session-ledger root causes were each reproduced deterministically on origin/main with an event-released held-lock probe (exact CI message on the old code, pass on the new); 10 consecutive local runs of each file pass.
Backwards compatibility
Compatible: test and doc changes only.
Tests
On Windows: the pin plus the runtime, fabric, deps, discord, session-ledger and work-ledger files pass (744). The race was 0 failures in 400 local stress runs, so the stochastic path is not a usable proof; the held-lock probe reproduces the exact CI error on the old helper and passes on the new one, deterministically. flake8, black gate and the comment-history gate are clean.
Manual verification
N/A — unit coverage sufficient; nothing user-visible changes.
Related Issues
Seen on #16096 (
Backend Tests (Windows) (1),Backend Tests (3.12, 1)) and #16102 (Backend Tests (Windows) (7)).Checklist