Skip to content

test: 0.13 backport, share each collected item's fixture bookkeeping to cut group 7 worker memory (#37814) - #37884

Merged
chenmingwei23 merged 2 commits into
release/0.13.0from
backport/0.13-test-memory-37814-36575
Nov 16, 2026
Merged

chenmingwei23 merged 2 commits into
release/0.13.0from
backport/0.13-test-memory-37814-36575

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Backend Tests (3.12, 7) passes on release/0.13.0 again, so 0.13 backport PRs are not red for a reason they did not cause.

On #37598 (head 302590dbfe), Backend Tests (3.12, 7) failed three runs in a row with the same two reds. In run 43811874378 the shard ends 2 failed, 31696 passed:

  • test/test_transcribe.py::TestPcmFromWav::test_truncated_header_returns_none_rather_than_raising[4]: worker 'gw5' crashed, then xdist: worker gw5 crashed and worker restarting disabled
  • test/test_testing_determinism_helpers.py::test_the_helper_modules_import_without_pytest: subprocess.TimeoutExpired on its pytest-free import child

Neither test is touched by #37598. Both come from per-worker memory pressure that main already cut.

Why it matters

Backend Tests is a blocking job. While this shard dies on the release branch, every 0.13 backport PR that lands in group 7 is red, and a real regression there cannot be told apart from this one.

Not a goal

  • No src/ change and no product behaviour change.
  • No change to the shard split, worker count or CI workflow.
  • No other test sync from main beyond the one pick and its follow-up fix.

What changed (motivation -> approach -> change)

Symptom: an xdist worker is killed and a subprocess times out in group 7. Root cause: per-worker memory during collection is high on the branch. Change: cherry-pick main's fix unchanged (git cherry-pick -x, no conflicts).

  • perf(test): share each collected item's fixture bookkeeping #37814 (433f545103), "perf(test): share each collected item's fixture bookkeeping": conftest.py shares fixture bookkeeping across collected items (main measured the collection peak per worker at 5,295 -> 2,957 MiB), with test/test_worker_memory_retention.py, a docs/system-specs/common/testing-conventions.md section and a writing-tests skill note.

The second commit closes a gap the Opus review found in the pick. When pytest-timeout's pytest.fail escapes pytest_runtest_protocol, the rootdir conftest's escape guard writes the missing reports and runs teardown, but runtestprotocol's last step (item._request = False; item.funcargs = None) never runs. That item keeps its rebuilt request, so test_no_collected_item_waits_with_a_request_built would fail on any worker where a neighbour timed out, and blame collection. The guard now takes that step itself, and test/test_xdist_escaped_failure_guard.py gains a test that drives each of the guard's three branches with an item holding a request (it fails 3 of 3 without the conftest change). main has the same gap; this fix is not on main yet.

The first version of this PR also picked #36575 (stop kiro_crew.testing importing its submodules at package import). release/0.13.0 now carries that commit itself, so the rebase onto 2fae50aae7 dropped it as already upstream.

Backwards compatibility

Compatible: test infrastructure, a test file and docs only; no src/ change, so nothing that works today stops working.

Removes nothing: the change shares identical per-item fixture bookkeeping, and after an escaped timeout it clears the same two fields pytest clears after every test; no fixture, test or product behaviour is taken away.

Tests

Proof: not provable -- test-only change

Local, Python 3.12, -n 2, on the rebased tree:

  • test/test_worker_memory_retention.py, test/test_testing_determinism_helpers.py, test/test_transcribe.py: 283 passed, 1 skipped
  • test/test_harness.py, test/test_real_kiro_smoke_gating.py: 129 passed, 1 skipped
  • With the second commit: test/test_xdist_escaped_failure_guard.py, test/test_worker_memory_retention.py, test/test_testing_determinism_helpers.py, test/test_transcribe.py, test/test_conftest_parts_reach.py: 340 passed, 1 skipped; the new test alone fails 3 of 3 with the conftest change reverted
  • flake8, isort and black: clean on conftest.py and test/test_xdist_escaped_failure_guard.py
  • ./scripts/docs-lint.sh: All documentation checks passed

Manual verification

N/A -- the CI shard this PR fixes is the check.

Related Issues

Unblocks Backend Tests (3.12, 7) on #37598. Backports #37814.

Sibling sweep

Swept: the short test summary of Backend Tests (3.12, 7) on #37598 (run 43811874378), the only red shard of that run
Sibling: test/test_transcribe.py -- not changed (crash victim, fixed by the memory pick)
Sibling: conftest.py pytest_runtest_protocol escape guard -- changed here (left a request on an item whose timeout escaped)
Sibling: test/test_testing_determinism_helpers.py -- not changed (its import-cost fix, #36575, is already on the branch)

Pattern harvest

Not generalizable: release branches miss main's test-infrastructure fixes until they are picked; the shard failure on a backport PR is how this one was found.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@chenmingwei23
chenmingwei23 requested review from buluoray and removed request for a team November 16, 2026 08:04
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision eb5cafe33268a56dd485ccb93858f2b3d80317d6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of eb5cafe33268a56dd485ccb93858f2b3d80317d6 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. This is a straight cherry-pick of main's memory fix plus one guard step. That step is needed because a test in the pick asserts no other item holds a request, which fails once a neighbour's timeout escapes.

What this change ships

Inventory (6 items) — 5 justified

Intent: make Backend Tests (3.12, 7) pass on release/0.13.0 again, so backport PRs aren't red for a reason they didn't cause. This is a FIX.

  1. Each collected test now shares its identical fixture bookkeeping and drops its request until setup, so a worker's collection peak falls from 5,295 to 2,957 MiB. — justified
  2. After an escaped timeout, the conftest guard clears the item's request and fixture values, as pytest does after every test. — justified
  3. New tests check the sharing, that items holding different fixtures keep their own lookup, and that no idle item holds a request. — justified
  4. New guard test covers all three escape branches; it fails 3 of 3 without item 2. — justified
  5. The testing-conventions spec section is re-measured, with a "merge new guards into an existing autouse fixture" rule. — justified
  6. The writing-tests skill gains a note on the cost of an autouse fixture and a note on module-level lru_cache memos. — rides along

[FIRST-PRINCIPLES-REVIEWED] eb5cafe

Description read

[DESCRIPTION-READ] 9a08e9aa3ee3f7e3fd59003d931544be2d3032e7ba79a9aec69e06572223a27a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of eb5cafe33268a56dd485ccb93858f2b3d80317d6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: a test-infrastructure backport of main's fix, plus a guard step that does the same cleanup pytest's own protocol does at its end.

Notes

  • The escape-guard fix (item._request = False; item.funcargs = None) is not on main yet, so the release branch now differs from main. Land the same commit on main so the next pick doesn't drop it, and so main's test_no_collected_item_waits_with_a_request_built can't go red after a neighbouring test times out.

[DESIGN-REVIEWED] eb5cafe

Description read

[DESCRIPTION-READ] d77327914599a835051dd0a59c845ba7792dbe569be1c017aa234b87369a036a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed eb5cafe33268a56dd485ccb93858f2b3d80317d6 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] eb5cafe

Verdict parsed from the review's SHA-scoped output markers for commit eb5cafe33268a56dd485ccb93858f2b3d80317d6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable eb5cafe33268a56dd485ccb93858f2b3d80317d6 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of eb5cafe33268a56dd485ccb93858f2b3d80317d6 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] eb5cafe

False positive or not applicable? A repository writer can comment:
/ai-review override gpt eb5cafe33268a56dd485ccb93858f2b3d80317d6 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 16, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@chenmingwei23
chenmingwei23 force-pushed the backport/0.13-test-memory-37814-36575 branch from 8f1ec2f to 50a0b64 Compare November 16, 2026 09:21
@chenmingwei23 chenmingwei23 changed the title test: 0.13 backport, cut group 7 worker memory and the kiro_crew.testing import cost (#37814, #36575) test: 0.13 backport, share each collected item's fixture bookkeeping to cut group 7 worker memory (#37814) Nov 16, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 16, 2026
When pytest-timeout's pytest.fail escapes pytest_runtest_protocol, the
rootdir conftest's escape guard synthesizes the missing reports and runs
teardown, but runtestprotocol's last step never ran: the item keeps its
rebuilt TopRequest and funcargs. On the same xdist worker,
test_no_collected_item_waits_with_a_request_built then fails beside the
timed-out neighbour and blames collection.

The guard now takes that step itself (item._request = False,
item.funcargs = None), as runtestprotocol does, and a parametrized test
drives each of its three branches with an item holding a request.
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@chenmingwei23
chenmingwei23 merged commit 061d2ff into release/0.13.0 Nov 16, 2026
95 of 98 checks passed
@chenmingwei23
chenmingwei23 deleted the backport/0.13-test-memory-37814-36575 branch November 16, 2026 12:21
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants