Skip to content

test(lenient-read-ratchet): hold one module at a time in the src scan - #36655

Closed
CrysisDeu wants to merge 1 commit into
mainfrom
crew/unflaky/lenient-scan-memory
Closed

CrysisDeu wants to merge 1 commit into
mainfrom
crew/unflaky/lenient-scan-memory

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop Backend Tests (3.12, 7) from running out of runner memory, so it no longer loses its runner or crashes an xdist worker.

Shard 7 loses its CodeBuild runner several times a day ("The self-hosted runner lost communication with the server", no log). On 11-12 this happened on runs created at 18:08Z, 19:16Z, 19:49Z and 20:17Z, then four more times after 22:00Z. On 11-13 it happened on most of the lost-runner jobs, including three main runs. Shard-7 jobs that survive peak at 14,877–15,076 MiB of 15,754 MiB. The other seven shards peak at 10,076–12,287 MiB with the same 6 workers.

A per-test RSS probe (diagnostic draft #36635, shard 7 job) shows where shard 7's extra memory goes:

Worker test_lenient_read_rewrite_ratchet.py RSS growth Worker RSS at end
gw3 +967 MiB (test_every_known_site_still_matches) 2,922 MiB
gw4 +1,239 MiB (test_no_new_rewrite_starts_from_a_lenient_read) 2,741 MiB
gw5 +1,099 MiB (test_the_scan_still_sees_the_real_lenient_readers) 2,809 MiB
gw0–gw2 (not run there) 1,486–1,675 MiB

All three tests started at the same second on three different workers. Host memory rose from 9,981 to 13,766 MiB in the next three minutes and never came back down. At the host peak (14,934 MiB) the three workers that ran the scan held 2.8–3.0 GB each; the other three held 1.5–1.7 GB. No child process held more than 10 MB.

Why it matters

Shard 7 goes red on unrelated PRs and on main. A lost runner leaves no log, so each one costs a rerun of the whole run plus a manual check.

Not a goal

What changed (motivation → approach → change)

Symptom: shard 7 runs about 3.5 GiB hotter than the other shards and hits the runner's memory limit.
Root cause: _src_scan() is functools.cached and returned the full _Module map. That map held the source text of all ~4,260 src/ modules (108 MB on disk) and the syntax tree of every module it parsed, for the life of the worker. The file has no xdist group, so each of its three tree-wide tests computes the cache on its own worker.
Change, in test/test_lenient_read_rewrite_ratchet.py only:

  • _Module takes a path and reads its text when a pass needs it (source()), instead of holding it.
  • release() drops a module's tree and function index once its readers are known (pass 1) or once it is scanned (pass 2). The readers names stay.
  • _src_scan() caches only {module: frozenset(readers)} plus the flagged sites.

Result, measured with a script outside pytest on this checkout: the scan's peak RSS drops from 1,277 MiB to 40 MiB, and retained memory from 1,262 MiB to 22 MiB. The flagged sites (25) and the readers of all 4,261 modules are identical to the old scan's output. Scan time is about the same (21 s before, 18 s after).

Backwards compatibility

Compatible: test-only change; the ratchet flags the same sites and sees the same readers.
Removes nothing: no product code, path, record or key changes.

Tests

  • New test_the_scan_keeps_no_syntax_tree_once_done: a planted scan still flags its rewrite and finds both readers, and leaves no module holding a tree or function index.
  • The three tree-wide tests and the planted tests are unchanged apart from reading readers[module].

Proof: not provable -- test-only change

Full-shard effect in CI. A test-only diff runs this PR's own shards in leaf mode, so this commit was cherry-picked onto the diagnostic draft #36635. Both runs use the same base, 1ca713c, with the per-test RSS probe:

Backend Tests (3.12, 7) Without the fix With the fix
Job (both green) 131923786845 131946546299
Host peak 14,934 MiB of 15,754 11,429 MiB of 15,754
Largest worker at peak 2,966 MB 1,934 MB
Worker RSS at session end 1,486–2,922 MiB 1,446–1,888 MiB
test_lenient_read_rewrite_ratchet.py growth +967 / +1,239 / +1,099 MiB on gw3/gw4/gw5 +5 MiB on gw5, no row in the top-growth tables

At 11,429 MiB, shard 7 now peaks in the same range as the other seven shards (10,076–12,287 MiB).

Manual verification

Ran the old and new _src_scan() from a script with PYTHONPATH=src: same 25 flagged sites, same readers for every module, peak RSS 1,277 MiB → 40 MiB.

Related Issues

These worker crashes on shard 7 all happened with the host at its memory limit:

Each crash was node down: Not properly terminated inside test_transcribe.py::TestPcmFromWav. Those tests allocate tens of MiB of WAV data late in the shard, after the lenient scan has already run. This is consistent with the kernel killing a worker once the host is full. The owner can close them once shard 7 runs green with this fix.

Main shows the same pressure. Shard 7 on main 5275a5d / cae338a hit three failures on one host: gw5 node down in TestPcmFromWav, a 60 s subprocess timeout in test_testing_determinism_helpers.py, and a 150 s pytest-timeout in the design_critique test_ssrf_guard_fail_closed.py test.

Sibling sweep

Swept: per-test RSS growth of every test on shard 7 in the diagnostic job above (tests that grew a worker by 32 MiB or more)
Sibling: test/test_facade_mirror_guards.py -- tracked in #36654
Sibling: test/test_backend_cards_advertised.py -- not affected -- +188 MiB on one worker, under a fifth of the ratchet's growth

Pattern harvest

Rule candidate: review-prompt
Pattern: a functools.cached tree-wide scan that returns whole-tree source texts or syntax trees, so every xdist worker that runs it keeps them for the rest of the run

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners November 13, 2026 14:22
@CrysisDeu
CrysisDeu requested a review from mbriones98 November 13, 2026 14:22
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 11a60b771bd6e1999114fdc2dcd4bae025e004c6 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 11a60b771bd6e1999114fdc2dcd4bae025e004c6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 11a60b771bd6e1999114fdc2dcd4bae025e004c6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The scan only reads .readers from other modules, so dropping each module's tree after use keeps the results the same and limits what each worker holds after the scan to what the tests read.

[DESIGN-REVIEWED] 11a60b7

Description read

[DESCRIPTION-READ] e7ff6a71a748a88a1a8acdcf80c6e2da4aeb0a1c74da93d06501bf69b02eb614

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 11a60b771bd6e1999114fdc2dcd4bae025e004c6 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 11a60b7

Verdict parsed from the review's SHA-scoped output markers for commit 11a60b771bd6e1999114fdc2dcd4bae025e004c6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 11a60b771bd6e1999114fdc2dcd4bae025e004c6 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 11a60b771bd6e1999114fdc2dcd4bae025e004c6 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 11a60b7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 11a60b771bd6e1999114fdc2dcd4bae025e004c6 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only memory fix. The src scan now reads each module's text on demand, drops its syntax tree and function index after each pass, and caches only each module's readers plus the flagged sites. The scan's results are unchanged: modules released after pass 1 are re-parsed in pass 2 and get the same readers. Lookups into other modules (_imports, is_reader) only use readers and package, which release() keeps. Nothing else calls _src_scan or the changed parse(). The new test fails on main because the old code never dropped the tree. CI is green, including shard 7.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 11a60b771b · 75 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
The tree-wide scan cached every src module's text and syntax tree for the life of the worker. Each of the three tree-wide tests lands on its own xdist worker, so Backend Tests shard 7 held about 1 GiB more per worker (967-1239 MiB measured in CI) and peaked within 700 MiB of the runner memory. The scan now reads each module, keeps only its lenient reader names, and drops the text and tree; the cache holds the readers and the flagged sites. Same result; peak scan memory 1277 MiB to 40 MiB.
@buluoray

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]: Heads-up, @CrysisDeu: #36653 makes the same fix to test/test_lenient_read_rewrite_ratchet.py (release each module's tree after its pass, cache only reader names and findings in _src_scan), and adds an xdist group so the three tree-wide tests share one worker. I approved #36653 and it has auto-merge on. On current main it takes this file's peak RSS from 1440 MiB to 437 MiB. The two will conflict, so only one should land. If you think re-reading each file instead of holding its text is worth keeping on top, a small follow-up after #36653 merges would carry it.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@CrysisDeu

CrysisDeu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Correction: closed too early by mistake and reopened. #36653 is not merged yet, and the merge SHA in the first version of this comment was wrong. This PR is superseded by #36653 and will be closed once it merges.

@CrysisDeu CrysisDeu closed this Nov 13, 2026
@CrysisDeu CrysisDeu reopened this Nov 13, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 13, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Superseded by #36731 (c347b0c). Measured shard-7 peak 14,934 -> 12,179 MiB on main 39fd360 (job 132112269089). The text-on-demand part here saves a further ~700 MiB (lenient_read 91-225 MiB per worker -> 5 MiB); branch crew/unflaky/lenient-scan-memory stays available if anyone wants it.

@CrysisDeu CrysisDeu closed this Nov 13, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants