Repository navigation
test: run the lenient-read ratchet's tree scan on one worker - #36653
bolichen97 wants to merge 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. It is a one-line, test-only xdist grouping, which is the fix testing-conventions prescribes for a module-cached tree walk paid once per worker. [DESIGN-REVIEWED] 8ac30da Description read[DESCRIPTION-READ] e450916c0104ec0c98a81416415e4d175cdb5d3dd2130353e394bfa8b15db2a2 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
This PR and #36648 fix the same shard-7 memory problem from #36261 in test/test_lenient_read_rewrite_ratchet.py. Both add the same |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Test-only memory fix to the lenient-read ratchet. release() drops a module's tree and functions but keeps readers. Cross-module lookups (_imports, is_reader) read only readers, and _scan_module uses its own module's functions only between parse() and release(), so the findings are unchanged. Re-parsing in pass 2 gives identical readers. _src_scan's new return shape has one consumer, which was updated; nothing else imports the module. The new test fails without either release() call, and its expected line number is correct. The xdist_group follows the testing-conventions per-file tree_scan rule, and the conftest hook gives these tests the 300 s budget. Shard 7 is green; the only failing check (Windows shard 3, mate_conductor_parity goal_plan) is unrelated. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
buluoray
left a comment
There was a problem hiding this comment.
🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:
Approve. This goes at the cause of the Linux shard 7 losses that keep reddening main and unrelated PRs (#36261): three red main runs today on that shard alone, as lost runners or test_ssrf_guard_fail_closed.py node timeouts.
The change is test-only and keeps the ratchet's answers the same. The scan now drops each module's syntax tree once its pass is done, _src_scan caches only the per-module reader names and the findings instead of every _Module, and the xdist group puts the three tree-wide tests on one worker so the scan is paid once per run.
Measured on current main, one process, test/test_lenient_read_rewrite_ratchet.py alone (peak RSS from resource.getrusage):
| Peak RSS | Result | |
|---|---|---|
| main | 1440 MiB | all pass |
| main + this diff | 437 MiB | all pass |
test_the_scan_keeps_no_syntax_tree_once_it_answers pins the release, so a later edit that keeps trees again goes red.
The one red check here, Backend Tests (Windows) (3), is the goal_plan mapping that main fixed in #36652 after this branch was cut. It is not this diff.
#36655 makes the same change (plus re-reading each file instead of holding the text). Only one of them should land; I am approving this one because it also groups the three tests onto one worker.
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray]: @bolichen97 this is approved with auto-merge on, and the only red is Backend Tests (Windows) (3): |
The lenient-read rewrite ratchet caches its whole-tree scan of src/ per worker, but the module has no xdist group, so its three tree-wide tests go to three different workers and each one pays the scan and keeps its cached result to the end of the run. With each parsed tree released after it is read, that is still about 280 MiB per worker on top of the worker's floor, three times over, on Linux Backend Tests shard 7, which runs within a gigabyte of the runner's memory. Put the module in its own tree_scan_ group, the testing-conventions fix for a module-cached walk that is still paid once per worker: the scan runs once per run on one worker, and the tests get the tree_scan 300 s budget. Measured at -n 3 on this file alone: the three workers peaked at 436/434/437 MiB before and 424/152/153 MiB after, and the module ran in 22 s instead of 28 s. Fixes #36261
ee498e5 to
8ac30da
Compare
|
Rebased onto main 39fd360 (head ee498e5 -> 8ac30da) to clear the conflict in
|
buluoray
left a comment
There was a problem hiding this comment.
🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:
Approve again on 8ac30da. The rebase reset my approval of ee498e5, so this is a read of what moved.
The tree release, the smaller _src_scan cache and the no-tree test from my earlier approval are now on main through #36731. After the rebase this PR carries only the module-level pytestmark = pytest.mark.xdist_group(name="tree_scan_lenient_read_rewrite_ratchet"). I compared the two patches line by line: every other added line is gone because main already has it, and nothing new was added.
The group is still worth having. Without it the three tree-wide tests can land on three workers, and each one reads all of src/ again. With it, one worker pays for the scan once per run. That matches the testing conventions' "a module-cached walk that is still paid once per worker" rule. The marker changes no test result.
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray]: @bolichen97 this PR now conflicts with main, and main already carries its one remaining change: #36648 (merged 20:13 UTC as 1c17fbd) added the same |
|
Confirmed: nothing is left for this PR to change. Rebasing 8ac30da onto main 6d61070 conflicts only in This PR can be closed as superseded by #36648. I do not have authority to close it, so a maintainer needs to. Auto-merge is still armed, but the PR cannot merge while it conflicts. #36261 stays open: its acceptance (shard 7 stops running out of memory) should be judged on main's Backend Tests (3.12, 7) runs, which now include both fixes. |
|
Not rebased: this PR's change is already on
For a maintainer:
|
Pull request was closed
Problem / Motivation
Goal:
Backend Tests (3.12, 7)stops running out of memory, so the design_critique node tests and the other tests that run late in that shard stop timing out or getting killed on unrelated PRs.Shard 7 peaks at 15.0-15.2 GB of a 15.7 GB runner, about 3.5-4 GB above every other shard at the same 6 workers (#36261). Near the end of the run the host stalls: faulthandler fires on several workers within seconds of each other, the four
test_ssrf_guard_fail_closed.pynode children either sit inselectors.pollpast the 120 s timeout or exit-9, andissue_radar/spec_builder/work_ledger_capstests in the same window time out too.A local run of shard 7 (same file-shard plugin and pins,
-n 6, coverage on) with the per-test RSS probe from #36635 shows where the extra memory comes from.test/test_lenient_read_rewrite_ratchet.py's three tree-wide tests each added 1,091-1,225 MiB to the worker that ran them, and that memory stayed held to the end of the run. The file has no xdist group, so the three tests went to three different workers:The ratchet's
functools.cached_src_scan()kept every module's source text and every parsed syntax tree ofsrc/for the whole worker lifetime, and it parsed them all before it scanned any.Why it matters
Shard 7 has gone red on PRs that do not touch the failing files many times in one day (#36202, #36345, #36362, #36187, #36562, #36511, #36436, #36347, #31518 in the ledger alone). The same memory ceiling probably explains the lost runners and the xdist worker crashes on that shard (#36441, #36406, #36364). A rerun does not fix it, and raising
NODE_TIMEOUT_SECSwould not either.Not a goal
NODE_TIMEOUT_SECS. They are victims of the host stall, not its cause.What changed (motivation → approach → change)
The ratchet's three tree-wide tests now run on one worker, so
src/is scanned once per run instead of once on each of three workers.#36731 landed on main while this PR was open and already releases each parsed tree once it is read, which took the scan's high-water mark from ~1.27 GiB to ~0.26 GiB per worker. What it left is the spread: the module has no xdist group, so
--dist loadgroupstill hands its three tree-wide tests to three workers, and each one runs the scan and keeps its cached result (the module texts and reader sets) to the end of its run.-n 3, this file alone (MiB)The change is one line at module scope in
test/test_lenient_read_rewrite_ratchet.py:pytestmark = pytest.mark.xdist_group(name="tree_scan_lenient_read_rewrite_ratchet"). This is the testing-conventions fix for "A module-cached walk that is still paid once per worker", and thetree_scan_prefix gives the scan the 300 sTREE_SCAN_TIMEOUT_SECSbudget. On shard 7 that is about 560 MiB less held across the workers, on a shard that runs within a gigabyte of the runner's memory.Backwards compatibility
Compatible: test-only change. The scan, its findings and every assertion are unchanged; only which worker runs the tests moves.
Removes nothing: no product code, path, record, validator or key is touched.
Tests
gw0at-n 3).-n 4 --dist loadgroup --max-worker-restart=2withtest_tree_scan_budget.pyas a co-tenant; two shuffled orders of the module's ten tests at-n0(seeds 7, 19), all green.test_flake_pattern_ratchet.pyandtest_tree_scan_budget.pypass. The black, subprocess-encoding, isort, flake8, comment-history, brand and file-line-cap gates pass.Proof: not provable -- test-only change
Manual verification
N/A: the CI shard's peak memory is printed by "Report peak memory" on this PR's own
Backend Tests (3.12, 7)run.Related Issues
Fixes #36261
Related: #36731 (released each parsed tree; this PR builds on it), #36441, #36406, #36364, #36635 (diagnostic probe whose plugin was used for the original measurement).
Sibling sweep
Swept: per-test RSS probe (
scripts/ci_rss_probe.pyfrom #36635) over a full local shard-7 run, listing every test that grew a worker by at least 32 MiB, plusgit grep -n "^@functools.cache\|^@functools.lru_cache" -- test/for module-cached whole-tree scans with noxdist_groupSibling: test/test_facade_mirror_guards.py -- not affected -- already one tree_scan group and the conftest clears its text caches per module, so one worker pays once
Sibling: test/test_backend_cards_advertised.py -- not affected -- adds about 100-220 MiB on one worker, an order of magnitude below the ratchet
Pattern harvest
Rule candidate: review-prompt
Pattern: a test module whose tests share a module-level
functools.cacheover a whole tree, with noxdist_group, pays that scan and keeps its result on every worker that draws one of its tests; put the module in its owntree_scan_<module>group.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.