Skip to content

test(mcp-gateway): keep the home pin while the port-chain env is swapped - #40263

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/mcp-policy-home-floor
Nov 21, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/mcp-policy-home-floor

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_a_stub_that_registers_first_waits_then_gets_a_bound_backend stops erroring at teardown with "resolved the operator's REAL default data home".

The test passes, then the rootdir teardown floor fails it on Linux Backend Tests shard 4:

ERROR test/test_mcp_policy_instance_binding.py::TestAcquisitionBeforeTheAnnounce::test_a_stub_that_registers_first_waits_then_gets_a_bound_backend - AssertionError: this test resolved the operator's REAL default data home (/home/runner/.kiro/crew): it dropped the KIROCREW_HOME pin and let config_dir() / data_home() fall through to ~/.kiro/crew, which CREATES that directory.
test/conftest_floor_home.py:630: AssertionError

Logs read (attempt 1 of each, plus the linux-pytest-progress-4 artifact for per-worker order):

Hit Job Worker Test before it on that worker
1 run 44808436348 job 136545717446 (PR #40067) gw3 TestAcquisitionBeforeTheAnnounce::test_a_third_party_server_never_waits
2 run 44605802926 job 135855601982 (PR #36451, reported in #39647) gw3 TestAcquisitionBeforeTheAnnounce::test_it_waits_with_keepalives_and_then_proceeds

The preceding tests differ and neither starts a thread, so neither is the offender.

Mechanism, traced on main cbffcd8fcb:

  • paths._resolved_home is written only by _resolve_default_home() (src/kiro_crew/config/paths.py:118-129). It is reached from config_dir() (paths.py:778-808) when _valid_override_home() (paths.py:323-342) finds no KIROCREW_HOME. One more caller, shared_kiro_agents_writable() (paths.py:775), is not reachable from this test.
  • The floor resets _resolved_home to None at setup (test/conftest_floor_home.py:514). The evidence is read in pytest_runtest_teardown before any fixture unwinds (conftest.py:1109-1111). So the real home was memoized during this test, while every pin was still set.
  • Nothing on the test's own thread resolves the home unpinned. The handler's resolver runs in to_thread (daemon/launch.py:1007-1008) and reads the process env, which holds the home fixture's KIROCREW_HOME. The resolver copies KIROCREW_HOME into the spawn env, because _scrub_sensitive_env only drops credential keys (mcp_gateway/manager.py:218) and the control-plane strip only drops LD_/DYLD_/PYTHON (env.py:1306-1310). resolve_client_port_src returns at the bound step (port_resolution.py:339-343) before it reads any home.
  • The one window in this test with no KIROCREW_HOME in the process is _resolve_in (test line 231): patch.dict(os.environ, env, clear=True). With clear=True it empties os.environ before it refills it. On exit _patch_dict._unpatch_dict ALWAYS clears and refills, even without clear=True. Any other thread that calls config_dir() in either gap finds no override and memoizes the real home. The floor then blames the test that is running.

I could not tell from the logs WHICH thread resolved: the floor records no stack. The fix does not depend on that. With the gap gone, no thread can see an unpinned home while this test runs. This is the same class #39297 fixed for test_install_receipt.py (process-wide KIROCREW_HOME removal plus a live thread).

Why it matters

Each hit fails a shard on an unrelated PR and costs a rerun. Two hits are counted for 11-19 and 11-20.

Not a goal

  • No production change, and no change to the teardown floor.
  • Not finding or fixing the thread that resolved the home during the gap.
  • Not converting the other patch.dict(os.environ, ...) sites in other test files.

What changed (motivation → approach → change)

Symptom: a real-home resolution memoized during the test. Root cause: the test empties the process environment for a moment, through patch.dict(os.environ), while other threads are live. Change: a small _environ context manager in the test module sets and restores keys one at a time through pytest.MonkeyPatch.context(), and never removes KIROCREW_HOME. _resolve_in uses it with exclusive=True, so the port chain still sees only the given env, minus that one pin. The instance-isolation test at the end of the file had the same exit-time gap, and now uses it too. The unused patch import is dropped.

Backwards compatibility

Compatible: test-only change. The tests assert the same values.

Removes nothing: only test helpers change; no product path, record or key is touched.

Tests

The existing tests in test/test_mcp_policy_instance_binding.py cover this. Their assertions are unchanged.

  • CI on head b3f4211754: Backend Tests (3.12, 4) passed. Its linux-pytest-progress-4 artifact shows all 116 tests of test/test_mcp_policy_instance_binding.py passed setup, call and teardown, including test_a_stub_that_registers_first_waits_then_gets_a_bound_backend. So every _resolve_in port verdict is unchanged.
  • Proof: not provable -- test-only change

Static gates run locally: flake8, isort, black 26.10.1 --check, comment-history and brand gates. All clean. No local pytest: CI is the test runner.

Manual verification

N/A -- test-only change; CI runs the file.

Real-machine verification

N/A -- test-only change to an in-process helper; nothing a separate machine would run differently.

Related Issues

Fixes #39647

Sibling sweep

Swept: git grep -n "patch.dict(os.environ" -- test/test_mcp_policy_instance_binding.py
Sibling: test/test_mcp_policy_instance_binding.py:1912 -- changed here

The wider class is out of this PR's goal: 364 patch.dict(os.environ sites in 61 other test files. None of them has a hit for this floor in the logs read here.

Pattern harvest

Rule candidate: review-prompt
Pattern: patch.dict(os.environ, ...) clears the whole process env on exit (and on entry with clear=True); in a test with live threads, use monkeypatch per key so the floor pins never disappear

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

…me pin

_resolve_in and the instance-isolation test ran the port chain under
patch.dict(os.environ), which empties the whole process environment
before refilling it on exit (and on entry with clear=True). Any other
thread that called config_dir() in that gap found no KIROCREW_HOME and
memoized the operator's real default home, so the teardown floor failed
test_a_stub_that_registers_first_waits_then_gets_a_bound_backend.

The environment is now set and restored one key at a time through
MonkeyPatch.context(), and KIROCREW_HOME is never removed.
@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners November 21, 2026 02:30
@CrysisDeu
CrysisDeu requested a review from mbriones98 November 21, 2026 02:30
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b3f4211754026fd0f68bd5de05170b2a10341f37 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision b3f4211754026fd0f68bd5de05170b2a10341f37 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b3f4211754026fd0f68bd5de05170b2a10341f37 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only change. Setting and restoring keys one at a time removes the moment when the environment is empty, which is when another thread can memoize the real data home.

Notes

  • _resolve_in callers whose env leaves out KIROCREW_HOME now resolve with the fixture's home pin in place. If any port-chain step past bound reads the home, that test's premise has quietly changed. Check that no _resolve_in case relied on having no home.

[DESIGN-REVIEWED] b3f4211

Description read

[DESCRIPTION-READ] b2d8010825b7f9a7766e64fbfea02fde65b08a518dd0706157ac0a0f664937bf

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b3f4211754026fd0f68bd5de05170b2a10341f37 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b3f4211

Verdict parsed from the review's SHA-scoped output markers for commit b3f4211754026fd0f68bd5de05170b2a10341f37.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b3f4211754026fd0f68bd5de05170b2a10341f37 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b3f4211754026fd0f68bd5de05170b2a10341f37 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b3f4211

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b3f4211754026fd0f68bd5de05170b2a10341f37 false-positive: <reason, 20 to 500 characters>

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Checked whether keeping KIROCREW_HOME in _resolve_in can change a port verdict through _resolve_once_home() (src/kiro_crew/mcp_gateway/daemon/launch.py:863), which reads os.environ["KIROCREW_HOME"] directly.

  • resolve_client_port_src (src/kiro_crew/port_resolution.py:310-351) never reaches it. The chain is cli, KIROCREW_PORT, KIROCREW_BOUND_PORT, _config_url_port, _marker_port, then the default. port_resolution.py does not import the daemon.
  • _resolve_once_home has one caller, resolve_once_resolver (launch.py:894), which wraps a target resolver. These tests resolve through _recording_resolver(gw.env_target_resolver), and they do it before _resolve_in runs.
  • Each _resolve_in env these tests build sets KIROCREW_PORT or KIROCREW_BOUND_PORT. The chain therefore returns at the env or bound step, before it reads a home.

CI on head b3f4211754: Backend Tests (3.12, 4) passed. Its linux-pytest-progress-4 artifact shows all 116 tests of this file passed setup, call and teardown.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 21, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier pending: being reviewed.

Test-only change. In this file, patch.dict(os.environ) is replaced by a MonkeyPatch.context() helper that sets and restores one key at a time. In exclusive mode it never removes KIROCREW_HOME, so no other thread can see the environment with the home unpinned. All _resolve_in assertions expect "env" or "bound". resolve_client_port_src returns at those steps before the config and marker steps that could read the home, so keeping the KIROCREW_HOME pin does not change any verdict. contextlib is already imported and dropping patch from the imports is safe. CI is green, including shard 4, the Windows shards and all AI review lanes. Repo testing conventions endorse MonkeyPatch.context() for this.

Tier: pending: being reviewed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit b3f4211754 · 25 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 21, 2026 03:45

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review checked head b3f4211 and found no blocking issues. The diagnosis is right. patch.dict always clears os.environ and then refills it on exit, even without clear=True. That opened a window with no KIROCREW_HOME, and another thread could resolve the real home in it. The new _environ helper replaces both patch.dict(os.environ, ...) sites in the file. It sets and restores one key at a time and never removes the home pin. All nine _resolve_in assertions expect the env or bound source, and resolve_client_port_src returns at those steps before it reads any home. So keeping the pin cannot change a verdict. The change adds no sleeps, ports or clocks, and the removed patch import has no other use.

Done well: the PR fixes the root cause the way the testing conventions prescribe. It also converts the sibling site that only had the gap on exit, and leaves the teardown floor and its retries unchanged.

Optional:

  • Add a small pin test. Enter _environ({}, exclusive=True), assert KIROCREW_HOME is still set, and record that it is never removed on entry or exit. Today nothing turns red if the exclusion at line 241 is dropped.
  • exclusive=True still removes the other floor pins (KIROCREW_WORKSPACE, KIROCREW_POD_ROOT, KIROCREW_POD_ENV_DIR). It did that before this PR too. Either keep them as well, or change the docstring and harvest line to say only the home pin is kept.

@KiroCrewCaptain
KiroCrewCaptain merged commit bc62fe8 into main Nov 21, 2026
107 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/mcp-policy-home-floor branch November 21, 2026 03:45
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_stub_that_registers_first_waits_then_gets_a_bound_backend resolves the real default data home at teardown on Linux

2 participants