Repository navigation
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The scan only reads [DESIGN-REVIEWED] 11a60b7 Description read[DESCRIPTION-READ] e7ff6a71a748a88a1a8acdcf80c6e2da4aeb0a1c74da93d06501bf69b02eb614 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Test-only memory fix. The src scan now reads each module's text on demand, drops its syntax tree and function index after each pass, and caches only each module's readers plus the flagged sites. The scan's results are unchanged: modules released after pass 1 are re-parsed in pass 2 and get the same readers. Lookups into other modules (_imports, is_reader) only use readers and package, which release() keeps. Nothing else calls _src_scan or the changed parse(). The new test fails on main because the old code never dropped the tree. CI is green, including shard 7. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
The tree-wide scan cached every src module's text and syntax tree for the life of the worker. Each of the three tree-wide tests lands on its own xdist worker, so Backend Tests shard 7 held about 1 GiB more per worker (967-1239 MiB measured in CI) and peaked within 700 MiB of the runner memory. The scan now reads each module, keeps only its lenient reader names, and drops the text and tree; the cache holds the readers and the flagged sites. Same result; peak scan memory 1277 MiB to 40 MiB.
8159514 to
11a60b7
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray]: Heads-up, @CrysisDeu: #36653 makes the same fix to |
Problem / Motivation
Goal: Stop
Backend Tests (3.12, 7)from running out of runner memory, so it no longer loses its runner or crashes an xdist worker.Shard 7 loses its CodeBuild runner several times a day ("The self-hosted runner lost communication with the server", no log). On 11-12 this happened on runs created at 18:08Z, 19:16Z, 19:49Z and 20:17Z, then four more times after 22:00Z. On 11-13 it happened on most of the lost-runner jobs, including three main runs. Shard-7 jobs that survive peak at 14,877–15,076 MiB of 15,754 MiB. The other seven shards peak at 10,076–12,287 MiB with the same 6 workers.
A per-test RSS probe (diagnostic draft #36635, shard 7 job) shows where shard 7's extra memory goes:
test_lenient_read_rewrite_ratchet.pyRSS growthtest_every_known_site_still_matches)test_no_new_rewrite_starts_from_a_lenient_read)test_the_scan_still_sees_the_real_lenient_readers)All three tests started at the same second on three different workers. Host memory rose from 9,981 to 13,766 MiB in the next three minutes and never came back down. At the host peak (14,934 MiB) the three workers that ran the scan held 2.8–3.0 GB each; the other three held 1.5–1.7 GB. No child process held more than 10 MB.
Why it matters
Shard 7 goes red on unrelated PRs and on main. A lost runner leaves no log, so each one costs a rerun of the whole run plus a manual check.
Not a goal
test/test_facade_mirror_guards.pyholds another ~684 MiB on one worker; tracked in test_facade_mirror_guards.py holds ~684 MiB in its xdist worker for the rest of the run #36654.What changed (motivation → approach → change)
Symptom: shard 7 runs about 3.5 GiB hotter than the other shards and hits the runner's memory limit.
Root cause:
_src_scan()isfunctools.cached and returned the full_Modulemap. That map held the source text of all ~4,260src/modules (108 MB on disk) and the syntax tree of every module it parsed, for the life of the worker. The file has no xdist group, so each of its three tree-wide tests computes the cache on its own worker.Change, in
test/test_lenient_read_rewrite_ratchet.pyonly:_Moduletakes a path and reads its text when a pass needs it (source()), instead of holding it.release()drops a module's tree and function index once its readers are known (pass 1) or once it is scanned (pass 2). Thereadersnames stay._src_scan()caches only{module: frozenset(readers)}plus the flagged sites.Result, measured with a script outside pytest on this checkout: the scan's peak RSS drops from 1,277 MiB to 40 MiB, and retained memory from 1,262 MiB to 22 MiB. The flagged sites (25) and the readers of all 4,261 modules are identical to the old scan's output. Scan time is about the same (21 s before, 18 s after).
Backwards compatibility
Compatible: test-only change; the ratchet flags the same sites and sees the same readers.
Removes nothing: no product code, path, record or key changes.
Tests
test_the_scan_keeps_no_syntax_tree_once_done: a planted scan still flags its rewrite and finds both readers, and leaves no module holding a tree or function index.readers[module].Proof: not provable -- test-only change
Full-shard effect in CI. A test-only diff runs this PR's own shards in leaf mode, so this commit was cherry-picked onto the diagnostic draft #36635. Both runs use the same base, 1ca713c, with the per-test RSS probe:
Backend Tests (3.12, 7)test_lenient_read_rewrite_ratchet.pygrowthAt 11,429 MiB, shard 7 now peaks in the same range as the other seven shards (10,076–12,287 MiB).
Manual verification
Ran the old and new
_src_scan()from a script withPYTHONPATH=src: same 25 flagged sites, same readers for every module, peak RSS 1,277 MiB → 40 MiB.Related Issues
These worker crashes on shard 7 all happened with the host at its memory limit:
Each crash was
node down: Not properly terminatedinsidetest_transcribe.py::TestPcmFromWav. Those tests allocate tens of MiB of WAV data late in the shard, after the lenient scan has already run. This is consistent with the kernel killing a worker once the host is full. The owner can close them once shard 7 runs green with this fix.Main shows the same pressure. Shard 7 on main 5275a5d / cae338a hit three failures on one host: gw5
node downinTestPcmFromWav, a 60 s subprocess timeout intest_testing_determinism_helpers.py, and a 150 s pytest-timeout in the design_critiquetest_ssrf_guard_fail_closed.pytest.Sibling sweep
Swept: per-test RSS growth of every test on shard 7 in the diagnostic job above (tests that grew a worker by 32 MiB or more)
Sibling: test/test_facade_mirror_guards.py -- tracked in #36654
Sibling: test/test_backend_cards_advertised.py -- not affected -- +188 MiB on one worker, under a fifth of the ratchet's growth
Pattern harvest
Rule candidate: review-prompt
Pattern: a
functools.cached tree-wide scan that returns whole-tree source texts or syntax trees, so every xdist worker that runs it keeps them for the rest of the runChecklist
Contribution License Agreement
Contributed under the project's Apache-2.0 license.