Skip to content

test: share cached scans per worker and stop waiting out policy reads - #38693

Merged
iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/test-cost-8
Nov 17, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/test-cost-8

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Cut repeated work from five slow backend test files on Linux shards 2, 3 and 7 without changing any check, size or bound.

Three tree-scan files cache their scan per process, but their tests land on different xdist workers, so each worker paid the whole scan again (four equal 13-18 s shares on CI). Three policy-cache cases each waited out the product's 5 s read timeout. The workflow-scope ratchets parsed every workflow YAML again in each test.

Why it matters

Shards 2, 3 and 7 are the slowest Linux shards (27-29 min). The survey of their top files estimates ~190 CPU-seconds of this repeated work per run.

Not a goal

  • Changing src/ or test/conftest.py.
  • test_security_single_storage.py (3,205 per-name items): turning them into loops loses the per-item failure report, which needs a reviewer's call.
  • test_crew_log_slot_cold_prime.py: another worker holds that file.

What changed (motivation → approach → change)

File Before Change
test_taskq_starting_claim_readers.py cached _scan paid on each worker module xdist_group
test_vector_memory_owners.py cached corpus scan paid on each worker module xdist_group
test_security_posture.py cached log census paid on each worker xdist_group on TestGateSideLogRedactorSpelling
test_mcp_policy_cache_lru.py each busy-gateway case waited 5 s slow reads get a 0.5 s timeout
test_workflow_secret_and_cache_scope.py YAML parsed in every test parsed once while the module runs

A group keeps a file's tests on one worker, so the cache each one already has is filled once. In the policy-cache file, only a read made while the gateway is slow gets the short timeout; fast reads keep the product's 5 s. The slow handler waits on an event that teardown sets, so no handler thread outlives its test. The workflow cache exists only while the module's tests run, so a collection-time read keeps no copy on a worker that never runs the module.

Backwards compatibility

Compatible: test-only change.

Removes nothing: no test, assertion, case or size is dropped.

Tests

Proof: not provable -- test-only change

Time, same host, the five files, -n 4 --dist loadgroup --cov=kiro_crew, two runs each, summed test seconds: main 119.0 / 101.6, branch 58.4 / 58.7 (wall 53.4 / 47.5 s to 38.1 / 38.3 s). Per file (main to branch): vector_memory_owners 16.4 to 8.8, taskq_starting_claim_readers 12.7 to 6.5, mcp_policy_cache_lru 26.6 to 12.2, workflow_secret_and_cache_scope 18.0 to 3.4, security_posture 27.9-45.4 to 27.9 (main varies with how the census tests split over workers).

Planted bugs, each run on this branch and on main (ce21fa1f01):

Plant Fails on both
mcp_shared_policy: never serve the last-good policy test_evicted_session_is_served_its_last_good_policy_on_a_busy_gateway
mcp_shared_policy: serve an empty policy to a never-resolved session test_explicit_refusal_drops_the_last_good_policy, test_session_that_never_resolved_is_still_refused_on_a_busy_gateway
release.yml: secrets: inherit test_no_caller_inherits_the_whole_secret_store
taskq: a new function that reads _agents and calls self.cancel test_every_stopping_liveness_reader_is_accounted_for
vector_memory_member: a bare np read in a function test_no_out_owner_reads_a_seam_as_a_bare_global
voice_reply: a new logger.warning(..., redact_credentials(...)) test_no_new_gate_side_log_line_reads_the_baseline_redactor

One plant passes on both trees: FIFO instead of LRU eviction in mcp_shared_policy, because the last-good fallback answers the evicted in-use session either way. That gap is on main too; this PR does not change it.

Coverage, the five files at -n 0 -p no:randomly: TOTAL 866367 statements / 754540 missed on both trees, every per-file row equal.

Determinism: 3 shuffled orders of the 115 items, test_mcp_policy_cache_lru.py 20 times and test_workflow_secret_and_cache_scope.py 10 times at -n 0, and the five files 3 times at -n 4 --dist loadgroup: all pass. Flake-pattern ratchet passes.

Open PRs touching these files: #23591 and #27626 (test_vector_memory_owners.py, two lines in BOUND_MEMBERS), #25450, #10599 and #21633 (test_security_posture.py, table rows). None touches the lines this PR changes.

Manual verification

N/A — test-only change; the runs above are the evidence.

Related Issues

N/A — test-cost work from the Linux shard survey.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 17, 2026 18:21
@iamwhatever
iamwhatever enabled auto-merge (squash) November 17, 2026 18:21
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 76cb3e5449a037c13d09fefb5953e54dbacc2712 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 76cb3e5449a037c13d09fefb5953e54dbacc2712 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 76cb3e5449a037c13d09fefb5953e54dbacc2712 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this changes tests only. Every check, size and bound stays, and the planted-bug runs show the assertions still catch real failures.

[DESIGN-REVIEWED] 76cb3e5

Description read

[DESCRIPTION-READ] b18bf37803a5c4c5bfa9e3a3711377a662254c94940719ad50314613794163ca

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 76cb3e5449a037c13d09fefb5953e54dbacc2712 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 76cb3e5

Verdict parsed from the review's SHA-scoped output markers for commit 76cb3e5449a037c13d09fefb5953e54dbacc2712.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 76cb3e5449a037c13d09fefb5953e54dbacc2712 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 76cb3e5449a037c13d09fefb5953e54dbacc2712 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 76cb3e5

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 76cb3e5449a037c13d09fefb5953e54dbacc2712 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever disabled auto-merge November 17, 2026 19:01
Three tree-scan files cache their scan per process, but their tests
spread over xdist workers and each worker paid the whole scan again; an
xdist_group keeps each on one worker. The policy-cache cases waited out
the product's 5 s read timeout on every busy-gateway case; a slow read
now gets a short timeout and the held answer is released at teardown.
The workflow-scope ratchets parse each workflow once while the module
runs. Every check and size is unchanged.
@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/test-cost-8 branch from c2cf38a to 76cb3e5 Compare November 17, 2026 19:02
@iamwhatever
iamwhatever enabled auto-merge (squash) November 17, 2026 19:02
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Fixed in 76cb3e5: the Backend Tests (3.12, 7) red on c2cf38a was this PR's own. test_facade_mirror_guards.py::test_no_test_patches_a_shared_name_on_an_owner_directly flagged the short-timeout wrapper in test_mcp_policy_cache_lru.py, which patched loopback_urlopen on the mcp_shared_policy owner. That name is shared by members of the mcp_shared family. The wrapper is now installed through the mcp_shared facade, which forwards it to every holder.

Re-run locally: test_facade_mirror_guards.py + test_facade_shared_checks.py (376 passed), test_mcp_policy_cache_lru.py 10 times at -n 0, and both policy plants (never serve the last-good policy; serve an empty policy to a never-resolved session) still fail the same tests on this branch and on main.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 17, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier pending: being reviewed.

Test-only speedups. The xdist_group marks follow the documented tree_scan_<module> convention, and CI uses --dist loadgroup, which honours them. In the policy-cache test, loopback_urlopen is swapped by setting it on the mcp_shared module, which passes the change to every module holding the name. The replacement's arguments match the one product call (timeout=5), and the 0.5 s timeout applies only when the slow-answer delay is set, so the busy-gateway cases still exercise the same timeout path. The release event is set at teardown before the server shuts down. The workflow YAML cache is safe because no caller of _load changes the parsed result, and the collection-time read keeps no copy. Both cached census readers in test_security_posture.py are inside the grouped class. CI is green and every automated review lane reported no blocking findings.

Tier: pending: being reviewed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 76cb3e5449 · 73 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On head 76cb3e5 the review checked these points:

  • The slow-read wrapper is installed through the mcp_shared facade. It reaches the product's one policy read call, and its signature matches that call.
  • Teardown sets the release Event before the server shuts down, so no handler thread outlives its test.
  • The workflow parse cache stays empty at collection time. Every caller of _load only reads the parsed data.
  • The three tree_scan_* groups follow the documented testing convention and overwrite no existing mark.
  • No assertion, case or size is removed.

Done well: a threading.Event that teardown releases replaces the old time.sleep(6.5), so the tests stop waiting out the read timeout. The planted-bug table was run on both trees, and it also reports the one plant that neither tree catches.

Optional:

  • The tree_scan_ prefix also raises these tests' per-test timeout from 120 s to the group default of 300 s. One sentence in the body would make that visible.
  • With slow reads capped at 0.5 s, these tests no longer fail if the product's 5 s policy read timeout grows past 6.5 s. If you want that check back, record the timeout the wrapper receives and assert it is at most 5 in one slow case.
  • A follow-up outside this PR could close the LRU-vs-FIFO gap you reported. For example, assert that dashboard:in-use is still cached after dashboard:one-more is added.

@iamwhatever
iamwhatever merged commit eac971b into main Nov 17, 2026
100 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/test-cost-8 branch November 17, 2026 20:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants