Skip to content

test: stream five slow tree-scan ratchets through source_corpus - #36648

Merged
iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/ratchet-cache
Nov 13, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/ratchet-cache

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Five slow tree-scan ratchets run faster, and the largest one stops holding every parsed module in memory, while each still checks exactly what it checked before.

Some ratchets rglob + ast.parse every module under src/ even though only files that spell the names they look for can match. One of them does this at collection time on every xdist worker. The lenient-read ratchet also kept every parsed module alive in its cached scan, which made it the biggest memory step on its shard.

Why it matters

These files are among the slowest in the backend suite, so every CI run pays for them on every worker that collects or runs them. test_lenient_read_rewrite_ratchet.py is pinned to shard 7. That shard's runner already peaks at about 15.0 GB of 15.7 GB and is out-of-memory-killing node children there (#36261), so a 1 GB drop in one module's peak memory is real headroom.

Not a goal

  • No cache of parsed trees or file text. test/source_corpus.py and testing-conventions.md record why retaining trees is the wrong fix. Only small extracted results are cached.
  • No change to what any ratchet checks, its pins or its known-site lists.
  • No edit to test/conftest.py, source_corpus.py, any scripts/ gate, or security-held test files.
  • The other scanners measured (already filtered, cached and grouped, or showing no gain) stay as they are.

What changed (motivation → approach → change)

Each scan goes through source_corpus.parsed_candidates / iter_candidate_sources. These stream one file at a time and parse only the files whose NFKC-normalised text holds the literal the AST check needs. A file without that literal can never match, so the filter removes only non-matches.

File Change -n0 wall (s) before → after peak RSS (MiB) before → after
test_lenient_read_rewrite_ratchet.py main already drops each tree (#36731); the cached scan also drops module text, and the module runs as one xdist_group so the scan runs once per run, not once per worker 21.0 → 16.7 437 → 423
test_persistence_writer_inventory.py parses only files that spell a write primitive; result set cached once 39.5 → 5.3 199 → 195
test_turn_trigger_source.py parses only files that spell the persist call 29.6 → 5.8 211 → 194
test_slack_mirror_echo.py collection-time scan parses only files that spell _run_chat 27.5 → 14.3 263 → 257
test_decisions_model_route_app_actor.py queue sweep parses only files that spell a queue call 12.7 → 7.1 191 → 196
total 130.3 → 49.2

Times are each file alone at -n0 on one host (16 cores, idle), including about 2 s of pytest start-up per file; the lenient row is measured against current main, which already carries #36731.

Backwards compatibility

Compatible: test-only change. The ratchets scan the same population and assert the same things.

Removes nothing: no production code, pin, known-site list or assertion is removed. Only how the tests read the tree changes.

Tests

Proof: not provable -- test-only change

A violation was planted in the working tree for each ratchet, the ratchet was run, and the plant was restored from a byte copy. Every plant failed its ratchet and named the planted site:

Ratchet Planted violation Result
lenient read rewrite new module: lenient _load feeds _write in bump fails, names _planted_lenient.py::bump
persistence writer inventory new module calls store.write_lesson fails, names _planted_writer.py::sneaky
turn trigger source new persist_token_record_async call without trigger= fails, names _planted_turn.py
slack mirror echo both taskrunner system dispatches re-stamped as user origin test_the_source_scan_finds_every_known_system_producer fails
app-actor queue sweep new dashboard queue_append that reads request_app with no actor fails, names _planted_queue.py
  • The five files plus test/test_source_corpus.py, -n 2: 174 passed.
  • test/test_flake_pattern_ratchet.py and test/test_wait_discipline_ratchet.py against the merge base: 122 passed.
  • flake8, isort, black, comment history, brand name: clean.

Manual verification

N/A — unit coverage sufficient: test-only change, proven by the planted violations above.

Related Issues

Part of #25195
Refs #36261

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 13, 2026 14:08
@iamwhatever
iamwhatever enabled auto-merge (squash) November 13, 2026 14:08
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 13, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 7% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only change that routes five tree-scan ratchets through source_corpus.parsed_candidates/iter_candidate_sources with name literals each AST pattern must spell. Filters only drop files that could never match (the matching is NFKC-aware and every literal is an identifier or attribute name). Every rewrite keeps its old population, exclusions, syntax-error handling and relative paths. The lenient ratchet drops module text only after the scan is done, and the later tests read only readers and the released tree fields. Its new tree_scan_ xdist group follows testing-conventions.md. Nothing regresses and no assertion is weakened.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 2264d182d1 · 63 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 2264d182d1333e7a98c8bd9e4786abb69f569484 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 2264d182d1333e7a98c8bd9e4786abb69f569484 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2264d182d1333e7a98c8bd9e4786abb69f569484 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only change, and the text filter can only skip files that could never match, so each ratchet still checks what it checked before.

[DESIGN-REVIEWED] 2264d18

Description read

[DESCRIPTION-READ] a6671f0f53c7588236677bfd6adebafa2f2d71d49551edbdfa8f5f656b27ef84

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2264d182d1333e7a98c8bd9e4786abb69f569484 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2264d18

Verdict parsed from the review's SHA-scoped output markers for commit 2264d182d1333e7a98c8bd9e4786abb69f569484.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2264d182d1333e7a98c8bd9e4786abb69f569484 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2264d182d1333e7a98c8bd9e4786abb69f569484 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2264d18

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2264d182d1333e7a98c8bd9e4786abb69f569484 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@iamwhatever
iamwhatever disabled auto-merge November 13, 2026 17:54
Four ratchets parsed every module under src/ (one at collection time on
every worker) although only files that spell the names they look for
can match. They now go through source_corpus.parsed_candidates or
iter_candidate_sources, which read one file at a time and parse only
those candidates.

The lenient-read ratchet already drops each tree; its cached scan now
also drops the module text, and its tree-wide tests run as one xdist
group so the scan runs once per run instead of once per worker.

What each ratchet checks is unchanged; a planted violation still fails
each one.
@iamwhatever
iamwhatever enabled auto-merge (squash) November 13, 2026 17:59
@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/ratchet-cache branch from 603ea02 to 2264d18 Compare November 13, 2026 17:59
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #36731 (Bolin Chen) landed the same per-tree release in test_lenient_read_rewrite_ratchet.py and conflicted with this branch. That file now carries only what main lacks: the cached scan drops module text, and the module runs as one xdist_group so its three tree-wide tests share one scan per run. So the lenient-row memory saving under "Why it matters" now comes from #36731, not from this PR. Measured against current main, this file goes from 21.0 s / 437 MiB to 16.7 s / 423 MiB at -n0, and a planted lenient rewrite still fails it. The other four files are unchanged. The branch is squashed to one commit, head 2264d18.

@github-actions github-actions Bot removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 13, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On this head the review read all five changed test files, and each new pre-filter is a name the AST check itself must spell, so a filter can only drop files that could never match. Each scan keeps its old behaviour: the _vendor/ and /tests/ exclusions, the dashboard root restriction, sorted file order, and SyntaxError handling (skip_syntax_errors=False where the old code raised). Old and new scans run over the head tree found the same sites in every case (35, 12, 4, 7, 13), and no assertion, count, known list or timeout changed. Done well: each ratchet was proven to still fail by planting a violation, and the failure named its site.

Optional:

  • #36653 adds the same xdist_group line to test/test_lenient_read_rewrite_ratchet.py, and nothing else. This PR covers it, so only one of the two should land.
  • In _src_scan, nothing tests the module.text = None drop. Adding or m.text is not None to the held check in test_the_scan_keeps_no_parsed_tree would make the test fail if someone deletes that line.
  • Most of the lenient-row memory saving comes from #36731, which is already on main. This PR's own drop is 437 to 423 MiB, as your 17:59Z comment says.

@iamwhatever
iamwhatever merged commit 1c17fbd into main Nov 13, 2026
131 of 139 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/ratchet-cache branch November 13, 2026 20:13
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants