Repository navigation
test(mcp): give weak tests real outcome checks - #24772
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This change only touches tests and adds outcome checks; nothing blocks it. The one removed test is covered by [DESIGN-REVIEWED] a652ebc Description read[DESCRIPTION-READ] 06a5f873d9e05729247d217c5a7c0f5e39663abc2be83cce48fffa7ed88f96e9 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
e5fb60b to
aa5df0d
Compare
aa5df0d to
327c925
Compare
|
CI disposition for head 327c925 (run 40469603518):
|
327c925 to
72fcdc7
Compare
72fcdc7 to
a652ebc
Compare
|
Rebased onto main d069596, which carries #24774 (drops the stale strict xfail in test_crew_log_fullscan_guard.py). New head a652ebc. Locally the touched mcp files give 1784 passed, 2 failed: test_mcp_messaging_discord.py::test_an_addressed_send_is_refused_without_a_verifiable_identity and ::test_a_bare_send_still_works_without_a_verifiable_identity raise PlatformCompositionError (profile resolved to enterprise). Both fail the same way on clean main a62012b on this host, so this is a local host profile issue, not this PR. CI re-running. |
Problem / Motivation
Goal: MCP tests check what a tool call leaves behind (the reply text the agent reads, the stored audit row, the policy left in force, the record or fingerprint on disk) instead of only checking that a mock was called.
88 tests under
test/test_mcp*.pyhad no check except a mock's call record (assert_called*,call_args,call_args_list,call_count), or no check the scanner could see. Such a test passes as long as the call happens, even when what the agent reads or what is stored is wrong.Why it matters
These tests cover the strict identity gate on the panel, crew-log and debug servers, channel governance on
send_messageandupdate_message, gateway audit rows, OAuth discovery auditing, confirm-tool reloads, the computer-use audit, the tool-policy cache, credential rotation and backend reaping. A test that only reads a mock record does not catch a reply that reports the wrong thing, a duplicate audit row, or a reaped-backend record left on disk. Several planted bugs below pass the old checks and fail the new ones.Not a goal
What changed (motivation → approach → change)
Each fixed test keeps its old check and adds a check on a real result.
test_mcp_panel_runtime.py,test_mcp_crew_log.py,test_mcp_debug.py)debug_refusalsgets a refusal unless the query issession=self.test_mcp_panel_runtime.py)field(type)" reply, built from the gateway's answer.test_mcp_core*.py,test_mcp_artifacts.py)spawn_run,workflow_run,workflow_rerun_subtree,learn_remove,spawn_status,ops_mission_control_api,artifact_getandartifact_get_comments; thespawn_rundescriptionschemas()builds from the registry.test_mcp_messaging_discord.py,test_mcp_messaging_update_message.py,test_mcp_send_message_routing.py)channelspolicy double that allows one transport. A Discord send is delivered under a Slack-denying policy, a Slack send is refused with the policy's reason and posts nothing, an edit passes only asupdate_message, and the gateway double delivers only whencaller_sessionis the strict caller.test_mcp_gatewayd_coverage.py,test_mcp_oauth_discovery.py,test_mcp_computer.py)sel_private_rootfixture: theoperation, theunknowncaller, one row for three re-recordings, the resolved app label and never the agent's claim.test_mcp_confirm_tools.py)test_mcp_session_report_wiring.py,test_mcp_oauth_banner.py,test_mcp_app_absence_marked.py,test_mcp_remote_header_secret_refs.py)test_mcp_shared_cache.py)test_mcp_gateway_backend_record.py,test_mcp_gateway_target_args_codec.py)test_mcp_gateway_bluegreen.py,test_mcp_discovery_agent_config.py,test_mcp_sync_agent.py,test_mcp_core.py)sessions_resetin the sync reply; thespawn_runreply.Formatting the touched files with black 26.10.1 made four of them black-clean, so their four lines leave
.github/black-baseline.txt(test_mcp_artifacts.py,test_mcp_core.py,test_mcp_oauth_banner.py,test_mcp_shared_cache.py). That file is the only non-test change.One test is removed:
test_mcp_shared_cache.py::TestLongCacheFailures::test_long_cache_short_circuits_repeated_callsran the same input (a 500, then a second call) astest_long_cache_hit_reports_the_refusing_reasonin the same class, which checks the samecall_count == 0and also the cached reason.Backwards compatibility
Compatible: test-only change. No production code, config or public contract changes.
Reader: test/test_mcp_shared_cache.py:TestLongCacheFailures::test_long_cache_hit_reports_the_refusing_reason -- release -- same 500 input, same two calls, checks
call_count == 0andunresolved == "resolution_failed"; same class and file, so the same platforms (no skip or xfail on either, neither is intest/windows-expected-failures.txt).Tests
Proof: not provable -- test-only change
Weak-test counts from
goal_scan.py weak, at merge-base82dbd99ad5and on this branch:--looseThe strict rule counts a plain
asserton a mock record as weak, so it is the true cut. MCP hits go from 88 to 34 (53 strengthened, 1 removed).Planted bugs: 12 caught out of 12 tried. Each one edited the source under test, ran only the changed tests, then was reverted with
git checkout: the credential baseline not advanced after a fire, every OAuth re-record audited, the empty caller not normalised, the computer-use audit recording the claim, the short policy window never expiring, the backend record left on disk, a cached invocation not applied, the overlay fingerprint not stored,caller_sessionnot forwarded, everylearn_removereported as no match, the panel reply ignoring the written type, anddebug_refusalsdefaulting to the wide view. The credwatch, OAuth, audit-row, record-file, invocation-cache and panel-reply plants pass the old checks.Coverage (the 24 touched files, 3 runs per side with
--cov=kiro_crew --cov-branch, combined per side): base 120,718 lines and 8,595 arcs, head 120,722 lines and 8,599 arcs. The one base-only line ismcp_gateway/daemon/sweepers.py:108, thesleep(3600)after the backend-temp sweep's first pass. It is background timing: base hit it in 2 of 3 runs, head in 0 of 3, and the gateway files alone hit it 0 of 6 times on either side.Determinism: the touched files passed 3 times with
-n 4.test_flake_pattern_ratchet.pyandtest_wait_discipline_ratchet.pypass. No new timed waits. The collected ids in the touched files are identical except the one removal (1,795 → 1,794). Two tests intest_mcp_messaging_discord.pythat this PR does not touch (test_an_addressed_send_is_refused_without_a_verifiable_identity,test_a_bare_send_still_works_without_a_verifiable_identity) fail on this host on the merge-base too, because the host resolves the enterprise profile.Left as they are (34 hits, no real outcome beyond the mock boundary, or already checked another way):
test_mcp_gateway_stub_keepalive.py(8), whose asserts run inside the nested_innercoroutine;test_mcp_shareability.py(2), which raiseAssertionErrorfrom a loop.test_mcp_discovery_probe_stdio.py(3:start_new_session,killpggroup reap, non-int pid),test_mcp_discovery_probe_spawn_policy.py::test_spec_declared_temp_suppresses_probe_containment,test_mcp_gateway_backend_coverage_call_metrics_logging.py::test_write_metric_line_returns_early_when_unset.test_mcp_gatewayd_coverage.py(5 sweeper tests),test_mcp_gateway_adopted_daemon_repair.py::test_the_watchdog_reconciles_on_the_liveness_reply,test_mcp_gateway_backend_coverage_stdio_pipes.py::test_reader_error_ends_the_pump.test_mcp_gateway_backend_coverage_stdio_pipes.py::test_serialises_one_newline_terminated_frame,test_mcp_gatewayd_coverage.py::test_backend_acquire_metric_carries_the_warm_attribute,test_mcp_oauth_banner.py(3 broadcast tests where the stored row is unchanged in both the pass and the fail case),test_mcp_gateway_inflight_report.py::test_unknown_notification_still_records_a_hazard.test_mcp_stub_apply.py(3refresh_defaultstests),test_mcp_hot_reload.py::test_the_mark_is_handed_to_the_inner_provider,test_mcp_core_spawn_sub_agents.py(2: no commit after abandon, keepalive ping during the poll).Manual verification
N/A — test-only change. The planted-bug run and the coverage comparison above are the verification.
Related Issues
Part of the weak-test cleanup tracked by TestOptimizer.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)