Skip to content

test: run the lenient-read ratchet's tree scan on one worker - #36653

Closed
bolichen97 wants to merge 1 commit into
mainfrom
fix/36261-ssrf-guard-shard7-flake
Closed

bolichen97 wants to merge 1 commit into
mainfrom
fix/36261-ssrf-guard-shard7-flake

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Backend Tests (3.12, 7) stops running out of memory, so the design_critique node tests and the other tests that run late in that shard stop timing out or getting killed on unrelated PRs.

Shard 7 peaks at 15.0-15.2 GB of a 15.7 GB runner, about 3.5-4 GB above every other shard at the same 6 workers (#36261). Near the end of the run the host stalls: faulthandler fires on several workers within seconds of each other, the four test_ssrf_guard_fail_closed.py node children either sit in selectors.poll past the 120 s timeout or exit -9, and issue_radar / spec_builder / work_ledger_caps tests in the same window time out too.

A local run of shard 7 (same file-shard plugin and pins, -n 6, coverage on) with the per-test RSS probe from #36635 shows where the extra memory comes from. test/test_lenient_read_rewrite_ratchet.py's three tree-wide tests each added 1,091-1,225 MiB to the worker that ran them, and that memory stayed held to the end of the run. The file has no xdist group, so the three tests went to three different workers:

worker final RSS before (MiB) largest file
gw5 2851 test_lenient_read_rewrite_ratchet.py +1225
gw4 2817 test_lenient_read_rewrite_ratchet.py +1220
gw1 2758 test_lenient_read_rewrite_ratchet.py +1091
gw0, gw2, gw3 1442-1754

The ratchet's functools.cached _src_scan() kept every module's source text and every parsed syntax tree of src/ for the whole worker lifetime, and it parsed them all before it scanned any.

Why it matters

Shard 7 has gone red on PRs that do not touch the failing files many times in one day (#36202, #36345, #36362, #36187, #36562, #36511, #36436, #36347, #31518 in the ledger alone). The same memory ceiling probably explains the lost runners and the xdist worker crashes on that shard (#36441, #36406, #36364). A rerun does not fix it, and raising NODE_TIMEOUT_SECS would not either.

Not a goal

  • Changing the design_critique tests or NODE_TIMEOUT_SECS. They are victims of the host stall, not its cause.
  • Changing CI worker counts, shard pins or runner sizes.
  • Narrowing what the ratchet scans or what it flags.

What changed (motivation → approach → change)

The ratchet's three tree-wide tests now run on one worker, so src/ is scanned once per run instead of once on each of three workers.

#36731 landed on main while this PR was open and already releases each parsed tree once it is read, which took the scan's high-water mark from ~1.27 GiB to ~0.26 GiB per worker. What it left is the spread: the module has no xdist group, so --dist loadgroup still hands its three tree-wide tests to three workers, and each one runs the scan and keeps its cached result (the module texts and reader sets) to the end of its run.

before (main) after
workers that run the scan 3 1
per-worker peak, -n 3, this file alone (MiB) 436 / 434 / 437 424 / 152 / 153
module wall time 28 s 22 s

The change is one line at module scope in test/test_lenient_read_rewrite_ratchet.py: pytestmark = pytest.mark.xdist_group(name="tree_scan_lenient_read_rewrite_ratchet"). This is the testing-conventions fix for "A module-cached walk that is still paid once per worker", and the tree_scan_ prefix gives the scan the 300 s TREE_SCAN_TIMEOUT_SECS budget. On shard 7 that is about 560 MiB less held across the workers, on a shard that runs within a gigabyte of the runner's memory.

Backwards compatibility

Compatible: test-only change. The scan, its findings and every assertion are unchanged; only which worker runs the tests moves.

Removes nothing: no product code, path, record, validator or key is touched.

Tests

  • No test is added or changed in behaviour; the module's ten tests run as before, now on one worker (checked: all ten land on gw0 at -n 3).
  • Determinism: 3/3 at -n 4 --dist loadgroup --max-worker-restart=2 with test_tree_scan_budget.py as a co-tenant; two shuffled orders of the module's ten tests at -n0 (seeds 7, 19), all green.
  • test_flake_pattern_ratchet.py and test_tree_scan_budget.py pass. The black, subprocess-encoding, isort, flake8, comment-history, brand and file-line-cap gates pass.

Proof: not provable -- test-only change

Manual verification

N/A: the CI shard's peak memory is printed by "Report peak memory" on this PR's own Backend Tests (3.12, 7) run.

Related Issues

Fixes #36261

Related: #36731 (released each parsed tree; this PR builds on it), #36441, #36406, #36364, #36635 (diagnostic probe whose plugin was used for the original measurement).

Sibling sweep

Swept: per-test RSS probe (scripts/ci_rss_probe.py from #36635) over a full local shard-7 run, listing every test that grew a worker by at least 32 MiB, plus git grep -n "^@functools.cache\|^@functools.lru_cache" -- test/ for module-cached whole-tree scans with no xdist_group
Sibling: test/test_facade_mirror_guards.py -- not affected -- already one tree_scan group and the conftest clears its text caches per module, so one worker pays once
Sibling: test/test_backend_cards_advertised.py -- not affected -- adds about 100-220 MiB on one worker, an order of magnitude below the ratchet

Pattern harvest

Rule candidate: review-prompt
Pattern: a test module whose tests share a module-level functools.cache over a whole tree, with no xdist_group, pays that scan and keeps its result on every worker that draws one of its tests; put the module in its own tree_scan_<module> group.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 13, 2026 14:20
@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 14:20
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8ac30da4b29409a24cb0f90f7ac5c0a371b0a091 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 8ac30da4b29409a24cb0f90f7ac5c0a371b0a091 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8ac30da4b29409a24cb0f90f7ac5c0a371b0a091 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing to check. It is a one-line, test-only xdist grouping, which is the fix testing-conventions prescribes for a module-cached tree walk paid once per worker.

[DESIGN-REVIEWED] 8ac30da

Description read

[DESCRIPTION-READ] e450916c0104ec0c98a81416415e4d175cdb5d3dd2130353e394bfa8b15db2a2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8ac30da4b29409a24cb0f90f7ac5c0a371b0a091 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8ac30da

Verdict parsed from the review's SHA-scoped output markers for commit 8ac30da4b29409a24cb0f90f7ac5c0a371b0a091.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8ac30da4b29409a24cb0f90f7ac5c0a371b0a091 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8ac30da4b29409a24cb0f90f7ac5c0a371b0a091 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8ac30da

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8ac30da4b29409a24cb0f90f7ac5c0a371b0a091 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This PR and #36648 fix the same shard-7 memory problem from #36261 in test/test_lenient_read_rewrite_ratchet.py. Both add the same xdist_group pytestmark, the same _Module.release() that clears tree and functions, and the same two release() calls in the scan loop. Please keep one of #36648 and #36653 and close the other, carrying over anything only this one has (for example the frozenset readers cache and test_the_scan_keeps_no_syntax_tree_once_it_answers).

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only memory fix to the lenient-read ratchet. release() drops a module's tree and functions but keeps readers. Cross-module lookups (_imports, is_reader) read only readers, and _scan_module uses its own module's functions only between parse() and release(), so the findings are unchanged. Re-parsing in pass 2 gives identical readers. _src_scan's new return shape has one consumer, which was updated; nothing else imports the module. The new test fails without either release() call, and its expected line number is correct. The xdist_group follows the testing-conventions per-file tree_scan rule, and the conftest hook gives these tests the 300 s budget. Shard 7 is green; the only failing check (Windows shard 3, mate_conductor_parity goal_plan) is unrelated.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit ee498e585c · 46 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

buluoray
buluoray previously approved these changes Nov 13, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:

Approve. This goes at the cause of the Linux shard 7 losses that keep reddening main and unrelated PRs (#36261): three red main runs today on that shard alone, as lost runners or test_ssrf_guard_fail_closed.py node timeouts.

The change is test-only and keeps the ratchet's answers the same. The scan now drops each module's syntax tree once its pass is done, _src_scan caches only the per-module reader names and the findings instead of every _Module, and the xdist group puts the three tree-wide tests on one worker so the scan is paid once per run.

Measured on current main, one process, test/test_lenient_read_rewrite_ratchet.py alone (peak RSS from resource.getrusage):

Peak RSS Result
main 1440 MiB all pass
main + this diff 437 MiB all pass

test_the_scan_keeps_no_syntax_tree_once_it_answers pins the release, so a later edit that keeps trees again goes red.

The one red check here, Backend Tests (Windows) (3), is the goal_plan mapping that main fixed in #36652 after this branch was cut. It is not this diff.

#36655 makes the same change (plus re-reading each file instead of holding the text). Only one of them should land; I am approving this one because it also groups the three tests onto one worker.

@buluoray

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]: @bolichen97 this is approved with auto-merge on, and the only red is Backend Tests (Windows) (3): test_mate_conductor_parity.py on goal_plan, which main fixed in #36652 after this branch was cut. A rebase onto main should clear it and let auto-merge land the shard 7 memory fix.

The lenient-read rewrite ratchet caches its whole-tree scan of src/ per
worker, but the module has no xdist group, so its three tree-wide tests
go to three different workers and each one pays the scan and keeps its
cached result to the end of the run. With each parsed tree released
after it is read, that is still about 280 MiB per worker on top of the
worker's floor, three times over, on Linux Backend Tests shard 7, which
runs within a gigabyte of the runner's memory.

Put the module in its own tree_scan_ group, the testing-conventions fix
for a module-cached walk that is still paid once per worker: the scan
runs once per run on one worker, and the tests get the tree_scan 300 s
budget. Measured at -n 3 on this file alone: the three workers peaked at
436/434/437 MiB before and 424/152/153 MiB after, and the module ran in
22 s instead of 28 s.

Fixes #36261
@bolichen97 bolichen97 changed the title test: keep the lenient-read ratchet from holding 1.2 GiB per worker test: run the lenient-read ratchet's tree scan on one worker Nov 13, 2026
@bolichen97
bolichen97 force-pushed the fix/36261-ssrf-guard-shard7-flake branch from ee498e5 to 8ac30da Compare November 13, 2026 19:24
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto main 39fd360 (head ee498e5 -> 8ac30da) to clear the conflict in test/test_lenient_read_rewrite_ratchet.py.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:

Approve again on 8ac30da. The rebase reset my approval of ee498e5, so this is a read of what moved.

The tree release, the smaller _src_scan cache and the no-tree test from my earlier approval are now on main through #36731. After the rebase this PR carries only the module-level pytestmark = pytest.mark.xdist_group(name="tree_scan_lenient_read_rewrite_ratchet"). I compared the two patches line by line: every other added line is gone because main already has it, and nothing new was added.

The group is still worth having. Without it the three tree-wide tests can land on three workers, and each one reads all of src/ again. With it, one worker pays for the scan once per run. That matches the testing conventions' "a module-cached walk that is still paid once per worker" rule. The marker changes no test result.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 13, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 13, 2026
@buluoray

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]: @bolichen97 this PR now conflicts with main, and main already carries its one remaining change: #36648 (merged 20:13 UTC as 1c17fbd) added the same pytestmark = pytest.mark.xdist_group(name="tree_scan_lenient_read_rewrite_ratchet") at test/test_lenient_read_rewrite_ratchet.py:63, and #36731 carried the tree release. Nothing is left for this PR to add, so it can be closed.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Confirmed: nothing is left for this PR to change. Rebasing 8ac30da onto main 6d61070 conflicts only in test/test_lenient_read_rewrite_ratchet.py. Main already carries the same xdist_group(name="tree_scan_lenient_read_rewrite_ratchet") pytestmark from #36648 (1c17fbd), and the tree release from #36731. With the conflict resolved to main's version, the diff is empty (the only difference left is comment wording), so I have not pushed.

This PR can be closed as superseded by #36648. I do not have authority to close it, so a maintainer needs to. Auto-merge is still armed, but the PR cannot merge while it conflicts. #36261 stays open: its acceptance (shard 7 stops running out of memory) should be judged on main's Backend Tests (3.12, 7) runs, which now include both fixes.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Not rebased: this PR's change is already on main.

main got the same line in #36648 (1c17fbd, "test: stream five slow tree-scan ratchets through source_corpus"): pytestmark = pytest.mark.xdist_group(name="tree_scan_lenient_read_rewrite_ratchet") in test/test_lenient_read_rewrite_ratchet.py. That PR also clears module.text from the cached scan. The conflict on this head is only in the comment above the marker. After resolving it, the diff against main (46f62de) would be that one comment line, so a rebase would only change a comment and re-run all of CI for it. I am not pushing it.

For a maintainer:

@bolichen97 bolichen97 closed this Nov 14, 2026
auto-merge was automatically disabled November 14, 2026 22:36

Pull request was closed

@bolichen97

Copy link
Copy Markdown
Contributor Author

Closing as superseded: #36648 (merged 2026-11-13, 1c17fbd) landed the same xdist group and the cached-scan release in test/test_lenient_read_rewrite_ratchet.py; the only remaining difference was a comment line. #36261 stays open until shard 7 is confirmed stable on main.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_ssrf_guard_fail_closed node subprocess tests time out on Linux shard 7

3 participants