Skip to content

test(issue-radar): seed the crew fabric route tests off the event loop - #16103

Closed
jeeshofone wants to merge 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/issue-radar-crew-ledger-loop-lock
Closed

jeeshofone wants to merge 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/issue-radar-crew-ledger-loop-lock

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop test_issue_radar_crew_fabric.py::RouteTest failing on Windows CI when the crew log writer still holds a unit's lock while the test seeds the ledger.

RouteTest::test_populated_repo_returns_folded_items failed on an unrelated fork PR's Windows shard with CrewLedgerNotRecorded. Its captured log shows the read-back in crew_store._landed_since refused twice with could not acquire exclusive file lock: still held after waiting 0.00s (one attempt: an acquire on the event-loop thread never waits ...).

Why it matters

The flake turns unrelated PRs red on Backend Tests (Windows) and fails Coverage Gate and PR Readiness closed with them. The product code is correct, so every such red costs a contributor a re-push for nothing.

Not a goal

  • Changing crew_store._append, _landed_since or platform_lock_compat.file_lock. The write route already runs off-loop, so the one-attempt rule never meets this read-back in production.
  • Auditing other test files for on-loop store calls. The other Issue Radar crew test files (test_issue_radar_crew_routes.py, test_issue_radar_crew_runtime.py, test_issue_radar_pipeline_routes.py) have no async test that calls commit_work_progress or record_crew_checkpoint directly.

What changed (motivation → approach → change)

The two async RouteTest tests seeded the ledger by calling _step (so commit_work_progress) synchronously on the event-loop thread. On that thread file_lock gives up after one attempt. Right after flush() reports drained, the crew log writer thread can still hold the unit's lock, so the read-back loses and _append raises CrewLedgerNotRecorded. A loaded Windows runner widens that window.

POST /crew/work reaches commit_work_progress through routes._st, which is asyncio.to_thread. GET /crew/fabric runs fold_fabric through asyncio.to_thread too. So the fix belongs in the test: a new _seed_step helper runs _step through asyncio.to_thread, the same way the route does, and both tests use it.

A new regression, test_seeding_survives_a_briefly_held_unit_lock, holds the unit's crew log lock from another thread for 0.3s while it seeds. It then checks that the write came back durable and the route serves the item.

Backwards compatibility

Compatible: test-only change; no product code, config or contract is touched.

Tests

  • test_seeding_survives_a_briefly_held_unit_lock (new): seeding while another thread holds the unit lock records a durable entry, and GET /crew/fabric returns it. With _seed_step reverted to an on-loop _step call it fails with the CI OSError (still held after waiting 0.00s).
  • test_populated_repo_returns_folded_items and test_non_github_provider_answers_200_empty now seed through _seed_step; assertions unchanged.
  • Ran test/test_issue_radar_crew_fabric.py (28 passed), RouteTest three times (6 passed each), and test/test_sync_io_in_async_gate.py (93 passed). isort, flake8, black (file-level and scripts/check_black_formatting.py) clean. mypy reports one existing error at the _registered helper (r.resource may be None), which is also on origin/main.

Manual verification

N/A — unit coverage sufficient: the change is test-only and the regression reproduces the CI failure deterministically.

Related Issues

Closes #16093

Pattern harvest

Not generalizable: the defect is this test driving a sync store write on the event loop that its route always runs off-loop; test_sync_io_in_async_gate.py already guards the product side.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested a review from a team as a code owner October 2, 2026 07:44
@jeeshofone
jeeshofone requested a review from bolichen97 October 2, 2026 07:44
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 399905e7ef9e188106dff3c382b45262f0614afc via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 399905e

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 399905e7ef9e188106dff3c382b45262f0614afc via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 399905e

Description read

[DESCRIPTION-READ] fb82bd3850cd493ce0ee5afb432f0ba537152da2cfb9042294fe07cdd647f966

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 399905e7ef9e188106dff3c382b45262f0614afc via the fork AI-review pipeline; updated in place on each push.

Review details

Nothing blocks — but the new contention test can pass without ever contending, and its prose names the wrong route and the wrong failure.

FINDING — test/test_issue_radar_crew_fabric.py:615 — timer = threading.Timer(0.3, release.set) is armed before await _seed_step(...), so when the to_thread hop plus _require_crew_log/_units_for_write/_unit_marks take longer than 300 ms (loaded Windows shard) the holder has exited _open_lock before the write's first _settle_tail acquire, the write is uncontended, out["durable"] and the fold assertion both still hold, and the regression pin passes vacuously with nothing reporting it — the shape testing-conventions.md:3996 names ("a holder on a fixed timer: it races the waiter's own start") → Fix: release on the waiter's observed contention (poll file_lock(..., wait=False) on a second descriptor from _hold until it sees the lock taken) or assert the write actually waited, so a vacuous run goes red.

FINDING — test/test_issue_radar_crew_fabric.py:502 — the helper's rationale says POST /crew/work reaches commit_work_progress, but register_crew_routes registers only _add("PUT", "/crew/work", _handle_crew_work) (crew_routes.py:1343) and _AGENT_REACHABLE holds ("PUT", "/crew/work"); no POST /crew/work exists, so a reader auditing which writes run off-loop is sent to a nonexistent entry point → Fix: write PUT /crew/work.

FINDING — test/test_issue_radar_crew_fabric.py:601 — the comment claims the on-loop variant "raised CrewLedgerNotRecorded", but the holder owns the lock before the write starts, so _prepare_write → _fold_checkpoint → fold_slot_warm → _SlotStream.__iter__ → open_session_log → CrewLog.open → _settle_tail → _open_lock is the contended acquire, and it is unguarded (_units_for_write is the only try), so _step leaves with the bare OSError from file_lock before _append/_landed_since ever runs — the post-append read-back window the helper's docstring blames is never opened by this arrangement → Fix: say the contended acquisition is _prepare_write's strict fold and the on-loop symptom is file_lock's refusal, or move the hold to start only once the entry is queued.

[OPUS-REVIEWED] 399905e

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 2, 2026
RouteTest seeded the crew ledger by calling commit_work_progress on the
event-loop thread, where file_lock makes one attempt. When the crew log
writer still held the unit lock, the read-back failed and the write
raised CrewLedgerNotRecorded. The write route runs the same call through
asyncio.to_thread, so the tests now do too, and a regression holds the
unit lock briefly from another thread while seeding.

Closes kirodotdev#16093
@jeeshofone
jeeshofone force-pushed the fix/issue-radar-crew-ledger-loop-lock branch from 355046d to 399905e Compare October 2, 2026 09:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 2, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for head 399905e7e: this PR is superseded by #16102 and should be closed without merging.

#16102 (merged 2 Oct 15:07 UTC) fixes the whole class this PR was opened for, not only the seed path here:

  • test_issue_radar_crew_fabric.py: _work now runs commit_work_progress through off_loop_helpers.off_loop, so both RouteTest seeds already run off the event loop on main. This PR's _seed_step would only wrap that worker-thread call in a second asyncio.to_thread.
  • test_issue_radar_deps.py::SweepDepUnblockTest (the Windows (6) red on this head): _seed_item goes through off_loop on main.
  • test_discord.py::TestDispatcher::test_a_dm_turn_opens_the_crew_log_the_work_ledger_writes_into: its crew log read-backs go through off_loop on main.
  • test_issue_radar_crew_runtime.py adds a held-lock probe with an on-loop negative control, which covers what test_seeding_survives_a_briefly_held_unit_lock adds here.
  • test_crew_log_off_loop_pin.py adds a whole-tree AST scan that fails any async test calling a crew log write or read on the loop.

Checked locally: this branch rebases cleanly onto current main, but its diff is then redundant. On current main, test_crew_log_off_loop_pin.py, test_issue_radar_crew_fabric.py, test_issue_radar_deps.py and the discord test pass (118 passed).

The other reds on this head come from main or predate the rebase:

No new head will be pushed.

@jeeshofone jeeshofone closed this Oct 3, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests: crew fabric RouteTest seeds the ledger on the event loop, so a held crew log lock fails the read-back (Windows)

1 participant