Skip to content

test(mcp): give weak tests real outcome checks - #24772

Merged
chenmingwei23 merged 1 commit into
mainfrom
test-cleanup/weak-mcp
Oct 27, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
test-cleanup/weak-mcp

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: MCP tests check what a tool call leaves behind (the reply text the agent reads, the stored audit row, the policy left in force, the record or fingerprint on disk) instead of only checking that a mock was called.

88 tests under test/test_mcp*.py had no check except a mock's call record (assert_called*, call_args, call_args_list, call_count), or no check the scanner could see. Such a test passes as long as the call happens, even when what the agent reads or what is stored is wrong.

Why it matters

These tests cover the strict identity gate on the panel, crew-log and debug servers, channel governance on send_message and update_message, gateway audit rows, OAuth discovery auditing, confirm-tool reloads, the computer-use audit, the tool-policy cache, credential rotation and backend reaping. A test that only reads a mock record does not catch a reply that reports the wrong thing, a duplicate audit row, or a reaped-backend record left on disk. Several planted bugs below pass the old checks and fail the new ones.

Not a goal

  • No change to production code. Only test files change.
  • No security or permission test is weakened. The identity, governance, audit and confirm-tool tests here only gain checks.
  • Tests whose only outcome is a call across a mocked OS or transport boundary are left as they are. See the list under Tests.

What changed (motivation → approach → change)

Each fixed test keeps its old check and adds a check on a real result.

Area Real outcome the test now reads
Panel, crew-log and debug identity (test_mcp_panel_runtime.py, test_mcp_crew_log.py, test_mcp_debug.py) The tool's reply. The gateway double answers only the session key the gate vetted, and refuses any other key, so the reply shows which key reached it. debug_refusals gets a refusal unless the query is session=self.
Panel writes (test_mcp_panel_runtime.py) The exact "Wrote field (type)" reply, built from the gateway's answer.
Core tools (test_mcp_core*.py, test_mcp_artifacts.py) The reply text of spawn_run, workflow_run, workflow_rerun_subtree, learn_remove, spawn_status, ops_mission_control_api, artifact_get and artifact_get_comments; the spawn_run description schemas() builds from the registry.
Channel governance (test_mcp_messaging_discord.py, test_mcp_messaging_update_message.py, test_mcp_send_message_routing.py) A channels policy double that allows one transport. A Discord send is delivered under a Slack-denying policy, a Slack send is refused with the policy's reason and posts nothing, an edit passes only as update_message, and the gateway double delivers only when caller_session is the strict caller.
Audit rows (test_mcp_gatewayd_coverage.py, test_mcp_oauth_discovery.py, test_mcp_computer.py) Rows stored by a real synchronous SEL through the sel_private_root fixture: the operation, the unknown caller, one row for three re-recordings, the resolved app label and never the agent's claim.
Confirm tools (test_mcp_confirm_tools.py) The decisions after a reload: the kept pattern still asks, the rest of the server still auto-approves.
Stored state (test_mcp_session_report_wiring.py, test_mcp_oauth_banner.py, test_mcp_app_absence_marked.py, test_mcp_remote_header_secret_refs.py) The slot's stored MCP report, the banner's stored meta, the tool rows' meta, the report's header-ref list.
Policy cache (test_mcp_shared_cache.py) The policy the resolver returns. After the short window expires, a gateway that now answers proves the retry.
Files on disk (test_mcp_gateway_backend_record.py, test_mcp_gateway_target_args_codec.py) The reap count and the removed record file; the fingerprint's schema before and after the rebuild.
Other (test_mcp_gateway_bluegreen.py, test_mcp_discovery_agent_config.py, test_mcp_sync_agent.py, test_mcp_core.py) Credential rotation fires exactly once with the new bytes; both specs are repaired from the cached invocation; sessions_reset in the sync reply; the spawn_run reply.

Formatting the touched files with black 26.10.1 made four of them black-clean, so their four lines leave .github/black-baseline.txt (test_mcp_artifacts.py, test_mcp_core.py, test_mcp_oauth_banner.py, test_mcp_shared_cache.py). That file is the only non-test change.

One test is removed: test_mcp_shared_cache.py::TestLongCacheFailures::test_long_cache_short_circuits_repeated_calls ran the same input (a 500, then a second call) as test_long_cache_hit_reports_the_refusing_reason in the same class, which checks the same call_count == 0 and also the cached reason.

Backwards compatibility

Compatible: test-only change. No production code, config or public contract changes.

Reader: test/test_mcp_shared_cache.py:TestLongCacheFailures::test_long_cache_hit_reports_the_refusing_reason -- release -- same 500 input, same two calls, checks call_count == 0 and unresolved == "resolution_failed"; same class and file, so the same platforms (no skip or xfail on either, neither is in test/windows-expected-failures.txt).

Tests

Proof: not provable -- test-only change

Weak-test counts from goal_scan.py weak, at merge-base 82dbd99ad5 and on this branch:

Rule Merge-base Branch Cut
strict (the real count) 2,178 2,124 −54
--loose 1,532 1,520 −12

The strict rule counts a plain assert on a mock record as weak, so it is the true cut. MCP hits go from 88 to 34 (53 strengthened, 1 removed).

Planted bugs: 12 caught out of 12 tried. Each one edited the source under test, ran only the changed tests, then was reverted with git checkout: the credential baseline not advanced after a fire, every OAuth re-record audited, the empty caller not normalised, the computer-use audit recording the claim, the short policy window never expiring, the backend record left on disk, a cached invocation not applied, the overlay fingerprint not stored, caller_session not forwarded, every learn_remove reported as no match, the panel reply ignoring the written type, and debug_refusals defaulting to the wide view. The credwatch, OAuth, audit-row, record-file, invocation-cache and panel-reply plants pass the old checks.

Coverage (the 24 touched files, 3 runs per side with --cov=kiro_crew --cov-branch, combined per side): base 120,718 lines and 8,595 arcs, head 120,722 lines and 8,599 arcs. The one base-only line is mcp_gateway/daemon/sweepers.py:108, the sleep(3600) after the backend-temp sweep's first pass. It is background timing: base hit it in 2 of 3 runs, head in 0 of 3, and the gateway files alone hit it 0 of 6 times on either side.

Determinism: the touched files passed 3 times with -n 4. test_flake_pattern_ratchet.py and test_wait_discipline_ratchet.py pass. No new timed waits. The collected ids in the touched files are identical except the one removal (1,795 → 1,794). Two tests in test_mcp_messaging_discord.py that this PR does not touch (test_an_addressed_send_is_refused_without_a_verifiable_identity, test_a_bare_send_still_works_without_a_verifiable_identity) fail on this host on the merge-base too, because the host resolves the enterprise profile.

Left as they are (34 hits, no real outcome beyond the mock boundary, or already checked another way):

  • Checks the scanner does not see: test_mcp_gateway_stub_keepalive.py (8), whose asserts run inside the nested _inner coroutine; test_mcp_shareability.py (2), which raise AssertionError from a loop.
  • The outcome is a call on a mocked OS or process seam: test_mcp_discovery_probe_stdio.py (3: start_new_session, killpg group reap, non-int pid), test_mcp_discovery_probe_spawn_policy.py::test_spec_declared_temp_suppresses_probe_containment, test_mcp_gateway_backend_coverage_call_metrics_logging.py::test_write_metric_line_returns_early_when_unset.
  • Sweeper and loop exits whose only effect is the call they skip: test_mcp_gatewayd_coverage.py (5 sweeper tests), test_mcp_gateway_adopted_daemon_repair.py::test_the_watchdog_reconciles_on_the_liveness_reply, test_mcp_gateway_backend_coverage_stdio_pipes.py::test_reader_error_ends_the_pump.
  • The outcome is a frame or metric sent to a mocked sink: test_mcp_gateway_backend_coverage_stdio_pipes.py::test_serialises_one_newline_terminated_frame, test_mcp_gatewayd_coverage.py::test_backend_acquire_metric_carries_the_warm_attribute, test_mcp_oauth_banner.py (3 broadcast tests where the stored row is unchanged in both the pass and the fail case), test_mcp_gateway_inflight_report.py::test_unknown_notification_still_records_a_hazard.
  • Wiring to a mocked collaborator with no state of its own: test_mcp_stub_apply.py (3 refresh_defaults tests), test_mcp_hot_reload.py::test_the_mark_is_handed_to_the_inner_provider, test_mcp_core_spawn_sub_agents.py (2: no commit after abandon, keepalive ping during the poll).

Manual verification

N/A — test-only change. The planted-bug run and the coverage comparison above are the verification.

Related Issues

Part of the weak-test cleanup tracked by TestOptimizer.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a652ebcf39538f0b7650cb338d7cb8159282189f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision a652ebcf39538f0b7650cb338d7cb8159282189f ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a652ebcf39538f0b7650cb338d7cb8159282189f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This change only touches tests and adds outcome checks; nothing blocks it. The one removed test is covered by test_long_cache_hit_reports_the_refusing_reason, which runs the same 500 input and also checks the cached reason.

[DESIGN-REVIEWED] a652ebc

Description read

[DESCRIPTION-READ] 06a5f873d9e05729247d217c5a7c0f5e39663abc2be83cce48fffa7ed88f96e9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a652ebcf39538f0b7650cb338d7cb8159282189f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a652ebc

Verdict parsed from the review's SHA-scoped output markers for commit a652ebcf39538f0b7650cb338d7cb8159282189f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a652ebcf39538f0b7650cb338d7cb8159282189f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a652ebcf39538f0b7650cb338d7cb8159282189f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a652ebc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a652ebcf39538f0b7650cb338d7cb8159282189f false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-mcp branch from e5fb60b to aa5df0d Compare October 27, 2026 07:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-mcp branch from aa5df0d to 327c925 Compare October 27, 2026 07:50
@iamwhatever

Copy link
Copy Markdown
Contributor Author

CI disposition for head 327c925 (run 40469603518):

@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-mcp branch from 327c925 to 72fcdc7 Compare October 27, 2026 12:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026
@iamwhatever
iamwhatever force-pushed the test-cleanup/weak-mcp branch from 72fcdc7 to a652ebc Compare October 27, 2026 12:58
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main d069596, which carries #24774 (drops the stale strict xfail in test_crew_log_fullscan_guard.py). New head a652ebc. Locally the touched mcp files give 1784 passed, 2 failed: test_mcp_messaging_discord.py::test_an_addressed_send_is_refused_without_a_verifiable_identity and ::test_a_bare_send_still_works_without_a_verifiable_identity raise PlatformCompositionError (profile resolved to enterprise). Both fail the same way on clean main a62012b on this host, so this is a local host profile issue, not this PR. CI re-running.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 27, 2026 18:20
@chenmingwei23
chenmingwei23 merged commit 29c7213 into main Oct 27, 2026
133 of 159 checks passed
@chenmingwei23
chenmingwei23 deleted the test-cleanup/weak-mcp branch October 27, 2026 23:10
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants