Repository navigation
fix: stop a pinned test throwing away its host-memory pin - #12341
Conversation
The 2026-09-20 Nightly Build failed on one job of 25 — macOS backend
shard 3 — and every publish step skipped behind it. The only failure
text was `assert 'queued' == 'starting'` at
test_taskq_admission_integration.py:245. Nothing in the traceback named
memory.
Root cause: pytest hands a test function and every fixture it requests
the SAME MonkeyPatch instance. test_taskq_admission_integration.py
carries `pytestmark = usefixtures("healthy_host_memory")`, which pins
both host memory guards SubagentManager.spawn consults, but
test_memory_pressure_defers_instead_of_refusing called a bare
`monkeypatch.undo()` half way through to lift its own simulated
pressure. That reverted the fixture's two pins along with it, so the
drain that follows read the runner's real free memory. The shard had
2.58 GB available, under the 4.5 GB floor, so the pump deferred the row
a second time and it stayed `queued`. The file was pinned, read as
pinned, and was not pinned where it mattered.
The fix scopes the simulated pressure in a `monkeypatch.context()`
block, so leaving it restores the fixture's healthy readings rather than
the host's. The same shape in test_overload_integration_glue.py is
scoped too: its later `_dispatch_async` does not currently reach the
memory gate, so it was not red, but the pins were being dropped for the
rest of the test the same way.
test_subagent_spawn_host_pin.py already ratchets that a spawning module
is pinned or excluded with a reason. It gains a second ratchet for the
other half of the same class: a pinned module must not revert the pin
with a bare `undo()`. Proved non-vacuous by reinstating the `undo()` and
confirming it names the offending file.
Verified by patching the real macOS reader to the shard's 2.58 GB: the
failure reproduced byte-for-byte (`assert 'queued' == 'starting'`, same
line), and after the fix all 32 tests in that file pass under the same
simulated pressure. 109 tests across the three touched files pass;
black, isort, flake8 and docs-lint are clean.
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] afdab95 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review —
|
…#12341) The 2026-09-20 Nightly Build failed on one job of 25 — macOS backend shard 3 — and every publish step skipped behind it. The only failure text was `assert 'queued' == 'starting'` at test_taskq_admission_integration.py:245. Nothing in the traceback named memory. Root cause: pytest hands a test function and every fixture it requests the SAME MonkeyPatch instance. test_taskq_admission_integration.py carries `pytestmark = usefixtures("healthy_host_memory")`, which pins both host memory guards SubagentManager.spawn consults, but test_memory_pressure_defers_instead_of_refusing called a bare `monkeypatch.undo()` half way through to lift its own simulated pressure. That reverted the fixture's two pins along with it, so the drain that follows read the runner's real free memory. The shard had 2.58 GB available, under the 4.5 GB floor, so the pump deferred the row a second time and it stayed `queued`. The file was pinned, read as pinned, and was not pinned where it mattered. The fix scopes the simulated pressure in a `monkeypatch.context()` block, so leaving it restores the fixture's healthy readings rather than the host's. The same shape in test_overload_integration_glue.py is scoped too: its later `_dispatch_async` does not currently reach the memory gate, so it was not red, but the pins were being dropped for the rest of the test the same way. test_subagent_spawn_host_pin.py already ratchets that a spawning module is pinned or excluded with a reason. It gains a second ratchet for the other half of the same class: a pinned module must not revert the pin with a bare `undo()`. Proved non-vacuous by reinstating the `undo()` and confirming it names the offending file. Verified by patching the real macOS reader to the shard's 2.58 GB: the failure reproduced byte-for-byte (`assert 'queued' == 'starting'`, same line), and after the fix all 32 tests in that file pass under the same simulated pressure. 109 tests across the three touched files pass; black, isort, flake8 and docs-lint are clean.
Problem / Motivation
The Nightly Build failed. One job of 25 was red —
Platform Tests / Backend Tests (macOS) (3)— and every publish step (CLI, Docker, Windows, all six Linux package targets, macOS notarization) skipped behind it, so the nightly channel got no build.The only failure text was:
Nothing in the traceback named memory. The sole evidence was a WARNING in the captured log:
only 2.58 GB available (min 4.5 GB required).Failing run: https://github.com/kirodotdev/KiroCrew/actions/runs/35525206603
Why it matters
The nightly channel is how the whole publish chain gets exercised daily. One host-dependent test file takes all of it down, and the failure text points nowhere near the cause, so each occurrence costs a full diagnosis. It is intermittent by construction: it reds only when the runner happens to be under the memory floor at that moment, so it will keep recurring and keep looking new.
What changed (motivation → approach → change)
Symptom → the post-pressure drain left the row
queuedinstead ofstarting.Root cause → pytest hands a test function and every fixture it requests the same
MonkeyPatchinstance.test_taskq_admission_integration.pycarriespytestmark = usefixtures("healthy_host_memory"), which pins both host-memory guardsSubagentManager.spawnconsults (the absolutespawn_min_memory_gbfloor and thecached_admission_checkposture tier). The test sets its own simulated CRITICAL pressure, asserts the deferral, then calls a baremonkeypatch.undo()to lift it. That reverts the fixture's two pins along with the test's own patch, so everything after that line reads the runner's real free memory. The shard had 2.58 GB, under the 4.5 GB floor, so the pump deferred the row a second time. The file was pinned, read as pinned, and was not pinned where it mattered.Change → three parts.
Scope the simulated pressure in a
with monkeypatch.context() as pressure:block. Leaving it reverts only what the block set, restoring the fixture's healthy readings rather than the host's.The same shape in
test_overload_integration_glue.pyis scoped the same way. Its later_dispatch_asyncdoes not currently reach the memory gate, so it was not red — but the pins were being dropped for the rest of the test identically, and itsundo()also reverted everything_app_managerhad patched (which the old code worked around by re-applying one of them by hand).test_subagent_spawn_host_pin.pyalready ratchets that a module reachingSubagentManager.spawnis pinned or excluded with a reason. It gains a second ratchet for the other half of the class: a pinned module must not revert the pin with a bareundo(). The message names the offending file and the remedy.docs/system-specs/common/testing-conventions.mdrecords the hole in the host-memory section, next to the pin it undoes.Backwards compatibility
Compatible: test-only plus one docs section. No production code, no interface, no config, no schema.
Tests
test_memory_pressure_defers_instead_of_refusingnow holds its own pressure patch in a scoped context, so the assertion it makes after the pressure lifts is a statement about the test's input rather than about the runner.TestTheHostPinSurvivesTheTestsOwnPatches::test_no_pinned_spawning_module_reverts_the_pin(new) fails if any module that pinshealthy_host_memoryand reaches.spawn(calls a zero-argument.undo().Manual verification
Reproduced the nightly failure deterministically. With
kiro_crew.subagent._macos_available_memory_gbpinned to the shard's2.58(by plainsetattr, notmonkeypatch, so the test's ownundo()could not revert the probe), the failure came back byte-for-byte onorigin/main—AssertionError: assert 'queued' == 'starting', same file, same line 245. With the fix applied and the same probe still in place, all 32 tests in that file pass. The probe file was deleted; it is not in the diff.Negative control at 1.5 GB. With the real host probe pinned to 1.5 GB (plain
setattrin a throwaway pytest plugin, so nothing the test does can revert it; both guards bottom out insubagent._available_memory_gb), the fixed file passes all 32 tests, while under that same pincheck_memory_available(min_gb=4.5)answers(False, 1.5). Reverting only the test change and re-running the same node under the same pin fails with the CI signature —AssertionError: assert 'queued' == 'starting'plusgate.py:660 Subagent spawn deferred: only 1.50 GB available (min 4.5 GB required)— so the control is valid rather than vacuous. The plugin lives outside the repository and is not in the diff.Neighbours checked. The file's other two memory tests,
test_low_memory_floor_defers_tooandtest_without_store_pressure_still_refuses, pincheck_memory_availableto a fixed(False, 0.2)and assert the deferral/refusal, so their verdict is their own input and no host reading can move it; neither callsundo(). No live bareundo()is left anywhere in the file — the only occurrence of the word is the comment saying why it is not used — and the remaining 29 tests never touch either guard, taking the module'shealthy_host_memorypin as-is. The same sweep over every test module that pins that fixture found no other live bareundo(), which the new ratchet now keeps true.macOS runs. On a real macOS host,
pytest test/test_taskq_admission_integration.py -vis 32 passed, and the three touched test files under the macOS shard's own flags (-n auto --timeout=180 --no-cov --max-worker-restart=0 -o faulthandler_timeout=150) are 109 passed. The shard itself is the whole suite split three ways, so its breadth belongs to CI:ci:macosis applied to this PR so the lane that reds runs against this head.The new ratchet is not vacuous. Reinstating the
monkeypatch.undo()line turned it red namingtest_taskq_admission_integration.py; restoring the fix turned it green (3 passed).Gates. 40 of the profile's 52 gates run and all pass, including
mypy, the pinned-version black gate,isort,flake8,docs-lint,check_comment_history, and the scoped backend suite (9 related files, 349 passed / 49 skipped). The 12 not run are the frontend, electron, bundle-analysis and cfn-lint lanes, which this diff does not touch.prove.py --base origin/main --per-hunkreturnsPROVEN(the docs hunk is prose, reported unprovable); the whole-diff mode returnsNOT_PROVENbecause it looks for production hunks and there are none.Open-PR overlap. #11454 (+432/-1), #12325, #10637 and #10595 touch these files additively; none deletes or rewrites them.
Related Issues
The same host dependency was filed from the pull-request side as #12337, with a second failing run (https://github.com/kirodotdev/KiroCrew/actions/runs/35526884755). The failing nightly run is linked above.
Fixes #12337
Pattern harvest
Rule candidate:
review-prompt— and shipped as a ratchet in this PR (test_subagent_spawn_host_pin.py), which is the enforceable form for this repo.Pattern: a test reverts a fixture's pin by calling
monkeypatch.undo()on the instance it shares with that fixture, so the rest of the test silently reads the host instead of the fixture's value. It generalizes past memory: the same instance carries theKIROCREW_HOMEpin and every other fixture the test requested.testing-conventions.mdalready stated the rule in its data-home section; what was missing was enforcement for the host-memory pin, which this adds.Checklist