Skip to content

test(ratchet): count session metadata writes made on the event loop - #31736

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/24746-rest-ratchet-on-loop-seed
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/24746-rest-ratchet-on-loop-seed

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop new async tests from seeding session metadata on the event loop, the shape that fails Windows test setup with a refused rename.

An async def test that calls ConversationLog.update_metadata on the loop thread gets one rename attempt from replace_with_retry, so a brief Windows sharing violation on the transcript fails the test's setup with PermissionError (WinError 5). These flakes are fixed one file at a time, and nothing stops a new test from adding the same shape.

Why it matters

Each site is a red Windows backend shard on some later PR that did not cause it. Without a gate, the class grows as fast as it is burned down.

Not a goal

What changed (motivation → approach → change)

Seeding metadata on the loop is a test-side mistake with a known fix, so it belongs in the gate that already counts flaky test shapes per file and fails a change that adds one.

Piece Change
Flake-pattern ratchet New class K20-on-loop-session-write: an update_metadata or update_metadata_if call whose innermost frame is a coroutine. A call in a nested def or lambda, or the method handed to asyncio.to_thread, is not counted. A file's count may not rise, so a new site fails the change; # flake-ok: <reason> keeps one on purpose.
Frontend detectors Moved verbatim to test/flake_pattern_js_detectors.py, so test/test_flake_pattern_ratchet.py stays under the 2,000-line cap.
testing-conventions The ratchet's class list and the Async tests section name K20 and the asyncio.to_thread shape.

A semgrep rule in semgrep/test-determinism.yaml was tried first. The pinned engine (1.78) cannot express this shape: a def pattern also matches an async def, so a nested sync helper cannot be excluded, and an async def pattern does not match a decorated function, which every @pytest.mark.asyncio test is.

Backwards compatibility

Compatible: test tooling only. The ratchet compares each changed file with its merge-base, so no existing site fails a change.

Removes nothing: the frontend detectors move unchanged and keep their keys in JS_DETECTORS.

Tests

  • test_each_python_detector_counts_its_shape_and_not_the_near_miss[K20-on-loop-session-write] (new case): two on-loop shapes are counted once each (a module-level test, and a method under a with); a near miss holding a sync test, the method handed to to_thread, a lambda and a nested def counts zero. The ratchet's mutation and registry tests cover K20 through the same tables.
  • Forced check, not committed: one added on-loop update_metadata call in test/test_dashboard_chat_history.py made test_no_changed_file_gains_a_flaky_shape fail with K20-on-loop-session-write 6 > 5.
  • The burn-down view (python test/test_flake_pattern_ratchet.py) counts 138 K20 sites in 41 files, the same as an independent AST scan of the tree.

Proof: not provable -- test-only change

Repeats: the K20 cases 20/20 at -n0; the ratchet file 10/10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test/test_wait_discipline_ratchet.py and test/test_ratchet_scope.py; the whole file in a shuffled order (seed 24746), 68 passed.

Local gates: black, isort, flake8, subprocess-encoding, comment-history, brand-name, file-line-cap and docs-lint passed. Semgrep on the diff found nothing and semgrep --test passed. The backend related set from scripts/run_scoped_tests.py passed (446 passed, 8 skipped). The frontend cross-surface set was not run locally; the diff touches no frontend file.

Manual verification

N/A: test tooling only. The forced check above shows the gate failing on a new site.

Screenshots / video

N/A: no user-visible change.

Related Issues

Part of #24746

Sibling sweep

Swept: the ratchet's burn-down view over every coroutine in test/ and src/kiro_crew/apps/builtins/**/tests/ for a direct update_metadata or update_metadata_if call
Sibling: test/test_history_consolidation_retry_accounting.py -- tracked in #24746 (15 sites)
Sibling: test/test_history_consolidation_bounds.py -- tracked in #24746 (14 sites)
Sibling: test/test_members_dm_thread_resume_and_reservation.py -- tracked in #24746 (11 sites)
Sibling: test/test_resume_publishes_hydrated_slot.py -- tracked in #24746 (11 sites)
Sibling: test/test_history_consolidation_retry.py -- tracked in #24746 (9 sites; the other 36 files hold 1 to 6 each)

Pattern harvest

Rule candidate: lint
Pattern: an async test calls a session-metadata writer that ends in a rename on the loop thread instead of through asyncio.to_thread. This PR adds it as flake-pattern ratchet class K20.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 19:53
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 19:53
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 20:13
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision d07f360cf26ac50b255107f80f2c558ac78936af ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d07f360cf26ac50b255107f80f2c558ac78936af touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d07f360cf26ac50b255107f80f2c558ac78936af — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: a test-only ratchet class for a known Windows flake, added to the per-file gate that already exists, and nothing outside the patch reads the moved JS detectors.

Notes

[DESIGN-REVIEWED] d07f360

Description read

[DESCRIPTION-READ] f06d063bfdea43b74fb7125a843f06ee5dba2dd4f647d5f05f2c143cf7ad389b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d07f360cf26ac50b255107f80f2c558ac78936af — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d07f360

Verdict parsed from the review's SHA-scoped output markers for commit d07f360cf26ac50b255107f80f2c558ac78936af.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d07f360cf26ac50b255107f80f2c558ac78936af false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d07f360cf26ac50b255107f80f2c558ac78936af and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d07f360

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d07f360cf26ac50b255107f80f2c558ac78936af false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
An async test that seeds a transcript with update_metadata on the loop
thread gets one rename attempt from replace_with_retry, so a brief
Windows sharing violation fails the test's setup. The flake-pattern
ratchet gains K20, which counts that call in a coroutine's own frame,
so a change can no longer add one; the existing sites stay for a
burn-down. The frontend detectors move verbatim to a sibling module so
the ratchet stays under the file-line cap.
@bolichen97
bolichen97 force-pushed the fix/24746-rest-ratchet-on-loop-seed branch from 01dc1e8 to d07f360 Compare November 6, 2026 21:53
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The red Backend Tests shards on the previous head were main's, not this change's: the security package line budget (fixed on main by #31625) and the pull_request_target allowlist (fixed on main by #31803). Rebased onto main with no change to the diff; CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on d07f360, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit fec3c60 into main Nov 7, 2026
133 of 139 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/24746-rest-ratchet-on-loop-seed branch November 7, 2026 02:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants