Skip to content

fix: stop a pinned test throwing away its host-memory pin - #12341

Merged
bolichen97 merged 1 commit into
mainfrom
fix/nightly-host-memory-pin-undo
Sep 20, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/nightly-host-memory-pin-undo

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

The Nightly Build failed. One job of 25 was red — Platform Tests / Backend Tests (macOS) (3) — and every publish step (CLI, Docker, Windows, all six Linux package targets, macOS notarization) skipped behind it, so the nightly channel got no build.

The only failure text was:

FAILED test/test_taskq_admission_integration.py::test_memory_pressure_defers_instead_of_refusing
  - AssertionError: assert 'queued' == 'starting'

Nothing in the traceback named memory. The sole evidence was a WARNING in the captured log: only 2.58 GB available (min 4.5 GB required).

Failing run: https://github.com/kirodotdev/KiroCrew/actions/runs/35525206603

Why it matters

The nightly channel is how the whole publish chain gets exercised daily. One host-dependent test file takes all of it down, and the failure text points nowhere near the cause, so each occurrence costs a full diagnosis. It is intermittent by construction: it reds only when the runner happens to be under the memory floor at that moment, so it will keep recurring and keep looking new.

What changed (motivation → approach → change)

Symptom → the post-pressure drain left the row queued instead of starting.

Root cause → pytest hands a test function and every fixture it requests the same MonkeyPatch instance. test_taskq_admission_integration.py carries pytestmark = usefixtures("healthy_host_memory"), which pins both host-memory guards SubagentManager.spawn consults (the absolute spawn_min_memory_gb floor and the cached_admission_check posture tier). The test sets its own simulated CRITICAL pressure, asserts the deferral, then calls a bare monkeypatch.undo() to lift it. That reverts the fixture's two pins along with the test's own patch, so everything after that line reads the runner's real free memory. The shard had 2.58 GB, under the 4.5 GB floor, so the pump deferred the row a second time. The file was pinned, read as pinned, and was not pinned where it mattered.

Change → three parts.

  1. Scope the simulated pressure in a with monkeypatch.context() as pressure: block. Leaving it reverts only what the block set, restoring the fixture's healthy readings rather than the host's.

  2. The same shape in test_overload_integration_glue.py is scoped the same way. Its later _dispatch_async does not currently reach the memory gate, so it was not red — but the pins were being dropped for the rest of the test identically, and its undo() also reverted everything _app_manager had patched (which the old code worked around by re-applying one of them by hand).

  3. test_subagent_spawn_host_pin.py already ratchets that a module reaching SubagentManager.spawn is pinned or excluded with a reason. It gains a second ratchet for the other half of the class: a pinned module must not revert the pin with a bare undo(). The message names the offending file and the remedy.

docs/system-specs/common/testing-conventions.md records the hole in the host-memory section, next to the pin it undoes.

Backwards compatibility

Compatible: test-only plus one docs section. No production code, no interface, no config, no schema.

Tests

  • test_memory_pressure_defers_instead_of_refusing now holds its own pressure patch in a scoped context, so the assertion it makes after the pressure lifts is a statement about the test's input rather than about the runner.
  • TestTheHostPinSurvivesTheTestsOwnPatches::test_no_pinned_spawning_module_reverts_the_pin (new) fails if any module that pins healthy_host_memory and reaches .spawn( calls a zero-argument .undo().

Manual verification

Reproduced the nightly failure deterministically. With kiro_crew.subagent._macos_available_memory_gb pinned to the shard's 2.58 (by plain setattr, not monkeypatch, so the test's own undo() could not revert the probe), the failure came back byte-for-byte on origin/main — AssertionError: assert 'queued' == 'starting', same file, same line 245. With the fix applied and the same probe still in place, all 32 tests in that file pass. The probe file was deleted; it is not in the diff.

Negative control at 1.5 GB. With the real host probe pinned to 1.5 GB (plain setattr in a throwaway pytest plugin, so nothing the test does can revert it; both guards bottom out in subagent._available_memory_gb), the fixed file passes all 32 tests, while under that same pin check_memory_available(min_gb=4.5) answers (False, 1.5). Reverting only the test change and re-running the same node under the same pin fails with the CI signature — AssertionError: assert 'queued' == 'starting' plus gate.py:660 Subagent spawn deferred: only 1.50 GB available (min 4.5 GB required) — so the control is valid rather than vacuous. The plugin lives outside the repository and is not in the diff.

Neighbours checked. The file's other two memory tests, test_low_memory_floor_defers_too and test_without_store_pressure_still_refuses, pin check_memory_available to a fixed (False, 0.2) and assert the deferral/refusal, so their verdict is their own input and no host reading can move it; neither calls undo(). No live bare undo() is left anywhere in the file — the only occurrence of the word is the comment saying why it is not used — and the remaining 29 tests never touch either guard, taking the module's healthy_host_memory pin as-is. The same sweep over every test module that pins that fixture found no other live bare undo(), which the new ratchet now keeps true.

macOS runs. On a real macOS host, pytest test/test_taskq_admission_integration.py -v is 32 passed, and the three touched test files under the macOS shard's own flags (-n auto --timeout=180 --no-cov --max-worker-restart=0 -o faulthandler_timeout=150) are 109 passed. The shard itself is the whole suite split three ways, so its breadth belongs to CI: ci:macos is applied to this PR so the lane that reds runs against this head.

The new ratchet is not vacuous. Reinstating the monkeypatch.undo() line turned it red naming test_taskq_admission_integration.py; restoring the fix turned it green (3 passed).

Gates. 40 of the profile's 52 gates run and all pass, including mypy, the pinned-version black gate, isort, flake8, docs-lint, check_comment_history, and the scoped backend suite (9 related files, 349 passed / 49 skipped). The 12 not run are the frontend, electron, bundle-analysis and cfn-lint lanes, which this diff does not touch. prove.py --base origin/main --per-hunk returns PROVEN (the docs hunk is prose, reported unprovable); the whole-diff mode returns NOT_PROVEN because it looks for production hunks and there are none.

Open-PR overlap. #11454 (+432/-1), #12325, #10637 and #10595 touch these files additively; none deletes or rewrites them.

Related Issues

The same host dependency was filed from the pull-request side as #12337, with a second failing run (https://github.com/kirodotdev/KiroCrew/actions/runs/35526884755). The failing nightly run is linked above.

Fixes #12337

Pattern harvest

Rule candidate: review-prompt — and shipped as a ratchet in this PR (test_subagent_spawn_host_pin.py), which is the enforceable form for this repo.

Pattern: a test reverts a fixture's pin by calling monkeypatch.undo() on the instance it shares with that fixture, so the rest of the test silently reads the host instead of the fixture's value. It generalizes past memory: the same instance carries the KIROCREW_HOME pin and every other fixture the test requested. testing-conventions.md already stated the rule in its data-home section; what was missing was enforcement for the host-memory pin, which this adds.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

The 2026-09-20 Nightly Build failed on one job of 25 — macOS backend
shard 3 — and every publish step skipped behind it. The only failure
text was `assert 'queued' == 'starting'` at
test_taskq_admission_integration.py:245. Nothing in the traceback named
memory.

Root cause: pytest hands a test function and every fixture it requests
the SAME MonkeyPatch instance. test_taskq_admission_integration.py
carries `pytestmark = usefixtures("healthy_host_memory")`, which pins
both host memory guards SubagentManager.spawn consults, but
test_memory_pressure_defers_instead_of_refusing called a bare
`monkeypatch.undo()` half way through to lift its own simulated
pressure. That reverted the fixture's two pins along with it, so the
drain that follows read the runner's real free memory. The shard had
2.58 GB available, under the 4.5 GB floor, so the pump deferred the row
a second time and it stayed `queued`. The file was pinned, read as
pinned, and was not pinned where it mattered.

The fix scopes the simulated pressure in a `monkeypatch.context()`
block, so leaving it restores the fixture's healthy readings rather than
the host's. The same shape in test_overload_integration_glue.py is
scoped too: its later `_dispatch_async` does not currently reach the
memory gate, so it was not red, but the pins were being dropped for the
rest of the test the same way.

test_subagent_spawn_host_pin.py already ratchets that a spawning module
is pinned or excluded with a reason. It gains a second ratchet for the
other half of the same class: a pinned module must not revert the pin
with a bare `undo()`. Proved non-vacuous by reinstating the `undo()` and
confirming it names the offending file.

Verified by patching the real macOS reader to the shard's 2.58 GB: the
failure reproduced byte-for-byte (`assert 'queued' == 'starting'`, same
line), and after the fix all 32 tests in that file pass under the same
simulated pressure. 109 tests across the three touched files pass;
black, isort, flake8 and docs-lint are clean.
@bolichen97
bolichen97 requested review from a team and smeyffret September 20, 2026 19:02
@bolichen97
bolichen97 enabled auto-merge (squash) September 20, 2026 19:02
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of afdab95b89cd834f69f92e60732847ba4d491150 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] afdab95

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of afdab95b89cd834f69f92e60732847ba4d491150 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] afdab95

False positive or not applicable? A repository writer can comment:
/ai-review override gpt afdab95b89cd834f69f92e60732847ba4d491150: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed afdab95b89cd834f69f92e60732847ba4d491150 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] afdab95

Verdict parsed from the review's SHA-scoped output markers for commit afdab95b89cd834f69f92e60732847ba4d491150.

False positive or not applicable? A repository writer can comment:
/ai-review override fable afdab95b89cd834f69f92e60732847ba4d491150: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@bolichen97
bolichen97 disabled auto-merge September 20, 2026 19:28
@bolichen97 bolichen97 added the ci:macos Run the on-demand macOS suite against this PR head label Sep 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for afdab95b89cd834f69f92e60732847ba4d491150; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt afdab95b89cd834f69f92e60732847ba4d491150: <one-sentence reason>

@bolichen97
bolichen97 enabled auto-merge (squash) September 20, 2026 19:38
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 20, 2026
@bolichen97
bolichen97 disabled auto-merge September 20, 2026 19:40
@bolichen97
bolichen97 merged commit f2ea126 into main Sep 20, 2026
109 of 113 checks passed
@bolichen97
bolichen97 deleted the fix/nightly-host-memory-pin-undo branch September 20, 2026 19:40
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 20, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…#12341)

The 2026-09-20 Nightly Build failed on one job of 25 — macOS backend
shard 3 — and every publish step skipped behind it. The only failure
text was `assert 'queued' == 'starting'` at
test_taskq_admission_integration.py:245. Nothing in the traceback named
memory.

Root cause: pytest hands a test function and every fixture it requests
the SAME MonkeyPatch instance. test_taskq_admission_integration.py
carries `pytestmark = usefixtures("healthy_host_memory")`, which pins
both host memory guards SubagentManager.spawn consults, but
test_memory_pressure_defers_instead_of_refusing called a bare
`monkeypatch.undo()` half way through to lift its own simulated
pressure. That reverted the fixture's two pins along with it, so the
drain that follows read the runner's real free memory. The shard had
2.58 GB available, under the 4.5 GB floor, so the pump deferred the row
a second time and it stayed `queued`. The file was pinned, read as
pinned, and was not pinned where it mattered.

The fix scopes the simulated pressure in a `monkeypatch.context()`
block, so leaving it restores the fixture's healthy readings rather than
the host's. The same shape in test_overload_integration_glue.py is
scoped too: its later `_dispatch_async` does not currently reach the
memory gate, so it was not red, but the pins were being dropped for the
rest of the test the same way.

test_subagent_spawn_host_pin.py already ratchets that a spawning module
is pinned or excluded with a reason. It gains a second ratchet for the
other half of the same class: a pinned module must not revert the pin
with a bare `undo()`. Proved non-vacuous by reinstating the `undo()` and
confirming it names the offending file.

Verified by patching the real macOS reader to the shard's 2.58 GB: the
failure reproduced byte-for-byte (`assert 'queued' == 'starting'`, same
line), and after the fix all 32 tests in that file pass under the same
simulated pressure. 109 tests across the three touched files pass;
black, isort, flake8 and docs-lint are clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests: test_memory_pressure_defers_instead_of_refusing reads the runner's real memory after monkeypatch.undo() (macOS shard red)

2 participants