Repository navigation
test: share cached scans per worker and stop waiting out policy reads - #38693
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this changes tests only. Every check, size and bound stays, and the planted-bug runs show the assertions still catch real failures. [DESIGN-REVIEWED] 76cb3e5 Description read[DESCRIPTION-READ] b18bf37803a5c4c5bfa9e3a3711377a662254c94940719ad50314613794163ca That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Three tree-scan files cache their scan per process, but their tests spread over xdist workers and each worker paid the whole scan again; an xdist_group keeps each on one worker. The policy-cache cases waited out the product's 5 s read timeout on every busy-gateway case; a slow read now gets a short timeout and the held answer is released at teardown. The workflow-scope ratchets parse each workflow once while the module runs. Every check and size is unchanged.
c2cf38a to
76cb3e5
Compare
|
Fixed in 76cb3e5: the Backend Tests (3.12, 7) red on c2cf38a was this PR's own. Re-run locally: |
Code change approver recommendation: A person should decideWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier pending: being reviewed. Test-only speedups. The xdist_group marks follow the documented Tier: pending: being reviewed. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved. On head 76cb3e5 the review checked these points:
- The slow-read wrapper is installed through the mcp_shared facade. It reaches the product's one policy read call, and its signature matches that call.
- Teardown sets the release Event before the server shuts down, so no handler thread outlives its test.
- The workflow parse cache stays empty at collection time. Every caller of
_loadonly reads the parsed data. - The three
tree_scan_*groups follow the documented testing convention and overwrite no existing mark. - No assertion, case or size is removed.
Done well: a threading.Event that teardown releases replaces the old time.sleep(6.5), so the tests stop waiting out the read timeout. The planted-bug table was run on both trees, and it also reports the one plant that neither tree catches.
Optional:
- The
tree_scan_prefix also raises these tests' per-test timeout from 120 s to the group default of 300 s. One sentence in the body would make that visible. - With slow reads capped at 0.5 s, these tests no longer fail if the product's 5 s policy read timeout grows past 6.5 s. If you want that check back, record the timeout the wrapper receives and assert it is at most 5 in one slow case.
- A follow-up outside this PR could close the LRU-vs-FIFO gap you reported. For example, assert that
dashboard:in-useis still cached afterdashboard:one-moreis added.
Problem / Motivation
Goal: Cut repeated work from five slow backend test files on Linux shards 2, 3 and 7 without changing any check, size or bound.
Three tree-scan files cache their scan per process, but their tests land on different xdist workers, so each worker paid the whole scan again (four equal 13-18 s shares on CI). Three policy-cache cases each waited out the product's 5 s read timeout. The workflow-scope ratchets parsed every workflow YAML again in each test.
Why it matters
Shards 2, 3 and 7 are the slowest Linux shards (27-29 min). The survey of their top files estimates ~190 CPU-seconds of this repeated work per run.
Not a goal
src/ortest/conftest.py.test_security_single_storage.py(3,205 per-name items): turning them into loops loses the per-item failure report, which needs a reviewer's call.test_crew_log_slot_cold_prime.py: another worker holds that file.What changed (motivation → approach → change)
test_taskq_starting_claim_readers.py_scanpaid on each workerxdist_grouptest_vector_memory_owners.pyxdist_grouptest_security_posture.pyxdist_grouponTestGateSideLogRedactorSpellingtest_mcp_policy_cache_lru.pytest_workflow_secret_and_cache_scope.pyA group keeps a file's tests on one worker, so the cache each one already has is filled once. In the policy-cache file, only a read made while the gateway is slow gets the short timeout; fast reads keep the product's 5 s. The slow handler waits on an event that teardown sets, so no handler thread outlives its test. The workflow cache exists only while the module's tests run, so a collection-time read keeps no copy on a worker that never runs the module.
Backwards compatibility
Compatible: test-only change.
Removes nothing: no test, assertion, case or size is dropped.
Tests
Proof: not provable -- test-only change
Time, same host, the five files,
-n 4 --dist loadgroup --cov=kiro_crew, two runs each, summed test seconds: main 119.0 / 101.6, branch 58.4 / 58.7 (wall 53.4 / 47.5 s to 38.1 / 38.3 s). Per file (main to branch): vector_memory_owners 16.4 to 8.8, taskq_starting_claim_readers 12.7 to 6.5, mcp_policy_cache_lru 26.6 to 12.2, workflow_secret_and_cache_scope 18.0 to 3.4, security_posture 27.9-45.4 to 27.9 (main varies with how the census tests split over workers).Planted bugs, each run on this branch and on main (
ce21fa1f01):mcp_shared_policy: never serve the last-good policytest_evicted_session_is_served_its_last_good_policy_on_a_busy_gatewaymcp_shared_policy: serve an empty policy to a never-resolved sessiontest_explicit_refusal_drops_the_last_good_policy,test_session_that_never_resolved_is_still_refused_on_a_busy_gatewayrelease.yml:secrets: inherittest_no_caller_inherits_the_whole_secret_storetaskq: a new function that reads_agentsand callsself.canceltest_every_stopping_liveness_reader_is_accounted_forvector_memory_member: a barenpread in a functiontest_no_out_owner_reads_a_seam_as_a_bare_globalvoice_reply: a newlogger.warning(..., redact_credentials(...))test_no_new_gate_side_log_line_reads_the_baseline_redactorOne plant passes on both trees: FIFO instead of LRU eviction in
mcp_shared_policy, because the last-good fallback answers the evicted in-use session either way. That gap is on main too; this PR does not change it.Coverage, the five files at
-n 0 -p no:randomly: TOTAL 866367 statements / 754540 missed on both trees, every per-file row equal.Determinism: 3 shuffled orders of the 115 items,
test_mcp_policy_cache_lru.py20 times andtest_workflow_secret_and_cache_scope.py10 times at-n 0, and the five files 3 times at-n 4 --dist loadgroup: all pass. Flake-pattern ratchet passes.Open PRs touching these files: #23591 and #27626 (
test_vector_memory_owners.py, two lines inBOUND_MEMBERS), #25450, #10599 and #21633 (test_security_posture.py, table rows). None touches the lines this PR changes.Manual verification
N/A — test-only change; the runs above are the evidence.
Related Issues
N/A — test-cost work from the Linux shard survey.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)