Repository navigation
test(ratchet): count session metadata writes made on the event loop - #31736
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: a test-only ratchet class for a known Windows flake, added to the per-file gate that already exists, and nothing outside the patch reads the moved JS detectors. Notes
[DESIGN-REVIEWED] d07f360 Description read[DESCRIPTION-READ] f06d063bfdea43b74fb7125a843f06ee5dba2dd4f647d5f05f2c143cf7ad389b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
An async test that seeds a transcript with update_metadata on the loop thread gets one rename attempt from replace_with_retry, so a brief Windows sharing violation fails the test's setup. The flake-pattern ratchet gains K20, which counts that call in a coroutine's own frame, so a change can no longer add one; the existing sites stay for a burn-down. The frontend detectors move verbatim to a sibling module so the ratchet stays under the file-line cap.
01dc1e8 to
d07f360
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved for the on-call. Review lanes pass on d07f360, no conflict, no open change request.
Problem / Motivation
Goal: Stop new async tests from seeding session metadata on the event loop, the shape that fails Windows test setup with a refused rename.
An
async deftest that callsConversationLog.update_metadataon the loop thread gets one rename attempt fromreplace_with_retry, so a brief Windows sharing violation on the transcript fails the test's setup withPermissionError(WinError 5). These flakes are fixed one file at a time, and nothing stops a new test from adding the same shape.Why it matters
Each site is a red Windows backend shard on some later PR that did not cause it. Without a gate, the class grows as fast as it is burned down.
Not a goal
_seed_loghelper): the detector sees only a call in the coroutine's own frame.replace_with_retry's on-loop behaviour, which is the intended contract.What changed (motivation → approach → change)
Seeding metadata on the loop is a test-side mistake with a known fix, so it belongs in the gate that already counts flaky test shapes per file and fails a change that adds one.
K20-on-loop-session-write: anupdate_metadataorupdate_metadata_ifcall whose innermost frame is a coroutine. A call in a nesteddeforlambda, or the method handed toasyncio.to_thread, is not counted. A file's count may not rise, so a new site fails the change;# flake-ok: <reason>keeps one on purpose.test/flake_pattern_js_detectors.py, sotest/test_flake_pattern_ratchet.pystays under the 2,000-line cap.asyncio.to_threadshape.A semgrep rule in
semgrep/test-determinism.yamlwas tried first. The pinned engine (1.78) cannot express this shape: adefpattern also matches anasync def, so a nested sync helper cannot be excluded, and anasync defpattern does not match a decorated function, which every@pytest.mark.asynciotest is.Backwards compatibility
Compatible: test tooling only. The ratchet compares each changed file with its merge-base, so no existing site fails a change.
Removes nothing: the frontend detectors move unchanged and keep their keys in
JS_DETECTORS.Tests
test_each_python_detector_counts_its_shape_and_not_the_near_miss[K20-on-loop-session-write](new case): two on-loop shapes are counted once each (a module-level test, and a method under awith); a near miss holding a sync test, the method handed toto_thread, alambdaand a nesteddefcounts zero. The ratchet's mutation and registry tests cover K20 through the same tables.update_metadatacall intest/test_dashboard_chat_history.pymadetest_no_changed_file_gains_a_flaky_shapefail withK20-on-loop-session-write 6 > 5.python test/test_flake_pattern_ratchet.py) counts 138 K20 sites in 41 files, the same as an independent AST scan of the tree.Proof: not provable -- test-only change
Repeats: the K20 cases 20/20 at
-n0; the ratchet file 10/10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest/test_wait_discipline_ratchet.pyandtest/test_ratchet_scope.py; the whole file in a shuffled order (seed 24746), 68 passed.Local gates: black, isort, flake8, subprocess-encoding, comment-history, brand-name, file-line-cap and docs-lint passed. Semgrep on the diff found nothing and
semgrep --testpassed. The backend related set fromscripts/run_scoped_tests.pypassed (446 passed, 8 skipped). The frontend cross-surface set was not run locally; the diff touches no frontend file.Manual verification
N/A: test tooling only. The forced check above shows the gate failing on a new site.
Screenshots / video
N/A: no user-visible change.
Related Issues
Part of #24746
Sibling sweep
Swept: the ratchet's burn-down view over every coroutine in
test/andsrc/kiro_crew/apps/builtins/**/tests/for a directupdate_metadataorupdate_metadata_ifcallSibling: test/test_history_consolidation_retry_accounting.py -- tracked in #24746 (15 sites)
Sibling: test/test_history_consolidation_bounds.py -- tracked in #24746 (14 sites)
Sibling: test/test_members_dm_thread_resume_and_reservation.py -- tracked in #24746 (11 sites)
Sibling: test/test_resume_publishes_hydrated_slot.py -- tracked in #24746 (11 sites)
Sibling: test/test_history_consolidation_retry.py -- tracked in #24746 (9 sites; the other 36 files hold 1 to 6 each)
Pattern harvest
Rule candidate: lint
Pattern: an async test calls a session-metadata writer that ends in a rename on the loop thread instead of through
asyncio.to_thread. This PR adds it as flake-pattern ratchet class K20.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.