Repository navigation
test(slack): give weak tests real outcome checks - #24495
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Test-only change that adds real outcome checks to 19 Slack tests. It removes no assertion that pinned the opposite behaviour, so no design-level concern applies. Notes
[DESIGN-REVIEWED] 0c216dc Description read[DESCRIPTION-READ] 1bf829cc57e32d50155e0dc0173a9e8dec20511aa1e76e4cd21b2c7c74496252 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
848013a to
db16063
Compare
db16063 to
64a6c1b
Compare
|
The red Backend Tests shards on head 64a6c1b are main's, not this PR's: |
64a6c1b to
bb52310
Compare
|
Reworked to bb52310 (one commit, rebased on main 20c46e9). The earlier revision counted 121 fixed tests, but 102 of them only rewrote one mock check into another form, for example Those 102 tests are back to their exact main form. The 19 tests that remain each check an outcome: an ack's envelope id, a log line, an audit record, a return value, state left behind or text the user sees. Under the strict rule the repo-wide weak count drops by 18 (2,313 to 2,295); with |
|
Disposition: |
bb52310 to
0c216dc
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only changes that add real outcome assertions to existing Slack tests.
Problem / Motivation
Goal: Make 19 weak Slack tests check a real outcome, so a broken Slack ack, audit, log or reply path fails a test instead of passing it.
These tests in
test/test_slack*.pyeither checked nothing or checked only that a mock was called. One of them,test_empty_allowed_users_rejects_all, clicked a message ts that nothing was waiting on, so it passed whether the gate refused the click or not.Why it matters
A test that only checks "the mock was called" stays green when the code skips the log line, audit record, return value or reply that the test name promises. The Slack layer owns the interactive approval gate, the socket-mode ack and governance denials, where those slips go unnoticed.
Not a goal
m.assert_called_once_with(x)andassert m.call_args_list == [call(x)]check the same thing, so this PR leaves such tests as main has them.test/test_slack*.py.What changed (motivation → approach → change)
An earlier revision of this branch reported 121 fixed tests, but 102 of them only swapped a mock check for another form of mock check (
assert_called_once_withtocall_args_list == [call(...)],assert_not_calledtocall_count == 0). Those tests are back to their exact main form. The branch now changes 19 tests, each gaining a check on an outcome the code produces:test_slack_events_coverage.py:test_unknown_envelope_type_ignored,test_home_tab_other_tab_ignored,test_non_message_event_ignored,test_unhandled_subtype_ignored,test_trusted_bot_other_subtype_still_droppedtest_persist_turn_row_swallows_failure(also checks the return value),test_a_stop_signalled_during_the_restart_is_not_swallowed,test_update_failure_is_swallowed,test_slack_none_logs_warning,test_post_failure_skips_ack,test_governance_deny_blocks_transcript_publication,test_session_closing_does_not_spool_restricted_slack_turntest_empty_allowed_users_rejects_all(rebuilt, see below),test_dm_mode_opens_dm_and_links(also checks the DM header text)test_reject_orphaned_tool_swallows_failure: a failed reject reportsFalsetest_second_call_is_a_no_op: the thread is marked hydratedtest_title_updates_conversation_log: the Slack thread title is set;test_handler_stop_reason_cancelled_skips_consolidation: the partial text still reaches the thread;test_cron_ack_calls_service: the button shows "✅ Acknowledged"The envelope-id check reads the response the client sent through a small helper,
_acked_envelopes. The id is the content of the ack Slack receives, so a dropped or blank id fails. The cron-ack text check reads theupdate_messagecall'stext, and the scan below still counts that test as weak.test_empty_allowed_users_rejects_allnow runs three clicks with the owner pinned throughmonkeypatch: no owner set, a non-owner click on a pending approval, and an owner click on a ts with no pending entry. The pending entry survives all three, and each click is auditeddenied.Files touched: 11 test files under
test/..github/black-baseline.txtis unchanged: no baselined file was reformatted.Backwards compatibility
Compatible: test-only change.
Removes nothing: no test, fixture or product code is removed.
Tests
Proof: not provable -- test-only change
goal_scan.py weak), whole repo, on this branch's base 20c46e9 and on this branch:--loose(an assert on a mock's call record counts as a check): 1,665 on base, 1,646 here, a cut of 19. The one-test gap istest_cron_ack_calls_service.git checkout: a blank socket ack envelope id; the home-tab warning text; the usage-row debug text; the shutdown restart-skip log text; the closing-dispatch log text; the "✅ Acknowledged" update text; the new-session failure text; the resume audit set todenied; the governance-drop log text; the approval gate audit set toallowed;_reject_orphaned_toolreturningTrueon failure; the_safe_updatefailure log removed; the!titlethread title cut to 3 characters.--cov=kiro_crewon the 11 touched files, union of two runs per side: base 139,919 lines, head 139,920, none lost. Two head runs differ from each other by 2 lines (config/loader_snapshots.py1087 and 1158, a timing branch these tests do not drive).-n 4.test_flake_pattern_ratchet.pyandtest_wait_discipline_ratchet.py: 113 passed. black (non-baselined touched files), isort, flake8,check_file_line_cap.py,check_comment_history.py: clean.Determinism: no new sleep, clock or port. caplog reads filter on the logger name. The owner id is set through
monkeypatch, notset_owner_id.Left as main has them (102): the earlier revision only rewrote a mock check in each test below into another form. They are back to their main form and still count as weak. Most drive code that returns
Noneand whose only effect is a call on a mocked collaborator, so an outcome check needs a fake in place of the mock; that is left for a later pass.test_no_module_converts_slack_markdown_directlyis in the list too: it already fails throughraise AssertionError, so theassertrewrite changed nothing.Tests left as main has them
test_slack_cron_remove_audit.py::TestGatewayDoneRemovalAudit::test_already_removed_job_is_not_auditedtest_slack_cron_remove_audit.py::TestGatewayDoneRemovalAudit::test_busy_store_defers_without_gateway_audittest_slack_cron_remove_audit.py::TestGatewayDoneRemovalAudit::test_unreadable_store_defers_without_gateway_audittest_slack_cron_remove_audit.py::TestSlackRemoveAllAudit::test_remove_all_nothing_deleted_audits_failedtest_slack_cron_remove_audit.py::TestSlackSingleRemoveAudit::test_caller_falls_back_to_surface_when_no_usertest_slack_events_coverage.py::TestInitSocketMode::test_dangerously_skip_permissions_enables_yolotest_slack_events_coverage.py::TestOnEventDispatch::test_interactive_is_dispatchedtest_slack_events_coverage.py::TestOnEventDispatch::test_member_joined_channel_routed_to_prompttest_slack_events_coverage.py::TestOnEventDispatch::test_message_deleted_is_delegatedtest_slack_events_coverage.py::TestOnEventDispatch::test_slash_command_is_dispatchedtest_slack_events_coverage.py::TestOnEventDispatch::test_update_pause_refuses_before_interaction_acktest_slack_events_coverage_route_flow.py::TestRouteMessageEnterpriseAndObserve::test_observe_skips_history_for_unauthorized_sendertest_slack_events_coverage_route_flow.py::TestRouteMessageQueueing::test_busy_session_enqueues_and_reactstest_slack_events_coverage_route_flow.py::TestRouteMessageStopCommand::test_idle_stop_dismisses_stale_ephemeraltest_slack_events_coverage_route_guards.py::TestRouteMessageGuards::test_pure_stop_is_exempt_from_governance_denialtest_slack_events_coverage_route_guards.py::TestRouteMessageGuards::test_stop_is_recorded_against_the_threads_owning_sessiontest_slack_events_coverage_route_guards.py::TestRouteMessageGuards::test_stop_is_recorded_before_the_liveness_checkstest_slack_gateway.py::TestDeliverResult::test_dashboard_new_slottest_slack_gateway.py::TestDeliverResult::test_dashboard_specific_slottest_slack_gateway.py::TestDeliverResult::test_dashboard_target_slot_not_foundtest_slack_gateway.py::TestShutdown::test_shutdown_awaits_autonudge_draintest_slack_gateway.py::TestShutdown::test_shutdown_stops_crontest_slack_gateway_channels.py::TestChannelTransportStartGate::test_no_policy_starts_every_transport_as_todaytest_slack_gateway_channels.py::TestUnreadyChannelBadge::test_no_dashboard_state_is_toleratedtest_slack_gateway_coverage_notifications.py::TestTaskNotify::test_denied_title_also_dms_the_ownertest_slack_gateway_coverage_nudge_routing.py::TestAutonudgeRouterAndObserver::test_observer_does_not_repeat_a_gated_legacy_terminal_notificationtest_slack_gateway_coverage_nudge_routing.py::TestAutonudgeRouterAndObserver::test_observer_expired_event_also_notifiestest_slack_gateway_coverage_nudge_routing.py::TestAutonudgeRouterAndObserver::test_observer_marks_delivery_only_after_notification_persistencetest_slack_gateway_coverage_nudge_routing.py::TestAutonudgeRouterAndObserver::test_observer_notifies_one_structured_terminal_transitiontest_slack_gateway_coverage_nudge_routing.py::TestAutonudgeRouterAndObserver::test_observer_without_loop_broadcasts_nothingtest_slack_gateway_coverage_nudge_routing.py::TestAutonudgeRouterAndObserver::test_restart_deduplicates_terminal_notice_with_delivery_recordtest_slack_gateway_coverage_nudge_routing.py::TestAutonudgeRouterAndObserver::test_terminal_transition_during_start_is_not_losttest_slack_gateway_cron_exec_coverage_delivery.py::TestTaskNotifyChannelRouting::test_channel_wins_over_the_owner_dmtest_slack_gateway_cron_exec_coverage_delivery.py::TestTaskNotifyChannelRouting::test_no_session_key_never_touches_the_channel_laddertest_slack_gateway_cron_exec_coverage_message_arm.py::TestCronMessageDenialAndWiring::test_reaper_not_started_without_sessionstest_slack_gateway_memory_boot.py::TestAutoMigrateMemory::test_model_absent_awaits_download_then_sweepstest_slack_gateway_memory_boot.py::TestAutoMigrateMemory::test_skips_migrate_when_already_migratedtest_slack_gateway_memory_boot.py::TestAutoMigrateMemory::test_sweep_deferred_when_model_not_readytest_slack_gateway_more_coverage.py::TestAutoMigrateMemory::test_set_memory_migrated_delegates_to_handlertest_slack_gateway_more_coverage.py::TestDeliverResultRouting::test_dashboard_arm_without_dashboard_statetest_slack_gateway_more_coverage.py::TestDeliverResultRouting::test_default_arm_notifies_even_when_dm_unavailabletest_slack_gateway_more_coverage.py::TestDeliverResultRouting::test_slack_only_arm_survives_dm_failuretest_slack_gateway_more_coverage.py::TestPersistSlotTitle::test_missing_conversation_log_skips_writetest_slack_gateway_more_coverage.py::TestShutdownExtras::test_slot_save_failure_does_not_abort_shutdowntest_slack_gateway_more_coverage.py::TestWarnIfKiroCliOutdated::test_transport_error_falls_back_to_plain_killtest_slack_gateway_run.py::TestBgSessionDashboardBranch::test_bg_session_prints_dashboard_urltest_slack_gateway_run.py::TestBgSessionDashboardBranch::test_failing_url_announcement_does_not_abort_boottest_slack_gateway_run.py::TestRunMethod::test_run_raises_on_shutdowntest_slack_gateway_run.py::TestRunSignalAndBgSession::test_run_with_ollama_configtest_slack_gateway_subagent_delivery.py::TestInjectWithRetry::test_acp_process_died_during_injectiontest_slack_gateway_subagent_delivery.py::TestInjectWithRetry::test_prompt_busy_exhaustedtest_slack_gateway_subagent_delivery.py::TestSubagentChannelTransportDelivery::test_mirror_link_delivers_into_forum_topictest_slack_gateway_subagent_delivery.py::TestSubagentChannelTransportDelivery::test_origin_link_wins_over_stored_channeltest_slack_gateway_subagent_delivery.py::TestSubagentChannelTransportDelivery::test_stored_peer_id_is_resolved_to_a_postable_conversationtest_slack_gateway_subagent_delivery.py::TestSubagentChannelTransportDelivery::test_target_snapshotted_before_injection_survives_session_resettest_slack_gateway_subagent_delivery.py::TestSubagentChannelTransportDelivery::test_telegram_parent_reply_reaches_registered_transporttest_slack_gateway_subagent_delivery.py::TestSubagentChannelTransportDelivery::test_unified_parent_uses_stored_channel_valuetest_slack_gateway_subagent_delivery.py::TestSubagentSlackInjection::test_cron_injection_timeouttest_slack_gateway_update_apply.py::TestAutoApplyUpdate::test_failure_path_resolves_the_restart_hint_off_the_event_looptest_slack_gateway_update_apply.py::TestAutoApplyUpdate::test_no_project_dir_returns_earlytest_slack_gateway_update_apply.py::TestCheckMissingDepsPip::test_version_probe_transport_error_does_not_abort_boottest_slack_gateway_update_reset.py::TestAutoApplyUpdateResetPath::test_reset_then_frontend_then_piptest_slack_handler.py::TestHookIntegration::test_auto_reply_records_agent_in_first_turntest_slack_handler.py::TestHookIntegration::test_spawn_command_records_agent_in_first_turntest_slack_handler_coverage_branches.py::TestCompactCommandFailureArms::test_compact_failure_tears_down_even_when_every_cleanup_failstest_slack_handler_coverage_branches.py::TestCompactCommandFailureArms::test_reporting_and_audit_failures_are_swallowedtest_slack_inline_stop.py::TestHandleInlineStop::test_allowed_soft_stoptest_slack_inline_stop.py::TestHandleInlineStop::test_hard_stop_callbacktest_slack_inline_stop.py::TestHandleSessionEnd::test_end_resolves_keytest_slack_inline_stop.py::TestHandleSessionEnd::test_end_with_consolidatortest_slack_interactions_coverage.py::TestAckHandlers::test_cron_ack_marks_matching_notificationtest_slack_interactions_coverage.py::TestAckHandlers::test_cron_ack_without_job_id_is_a_nooptest_slack_interactions_coverage.py::TestAckHandlers::test_subagent_ack_marks_matching_notificationtest_slack_interactions_coverage.py::TestAgentSelect::test_empty_selection_is_a_nooptest_slack_interactions_coverage.py::TestAgentSelect::test_non_owner_rejectedtest_slack_interactions_coverage.py::TestAgentSelect::test_reset_rejected_by_setter_abortstest_slack_interactions_coverage.py::TestAgentSelect::test_switch_rejected_by_setter_abortstest_slack_interactions_coverage.py::TestAgentSelect::test_unresolvable_agent_abortstest_slack_interactions_coverage.py::TestChannelsModal::test_refresh_without_orch_is_a_nooptest_slack_interactions_coverage.py::TestDispatchPayloadParsing::test_message_action_routed_to_shortcuttest_slack_interactions_coverage.py::TestDispatchPayloadParsing::test_payload_without_actions_returns_before_authtest_slack_interactions_coverage.py::TestDispatchPayloadParsing::test_unknown_action_without_channel_does_nothingtest_slack_interactions_coverage.py::TestDispatchPayloadParsing::test_view_payload_routedtest_slack_interactions_coverage.py::TestDispatchRouting::test_dashboard_copy_without_url_posts_nothingtest_slack_interactions_coverage.py::TestDispatchRouting::test_review_cancel_is_exempt_from_governancetest_slack_interactions_coverage.py::TestDispatchRouting::test_spent_options_marker_is_a_nooptest_slack_interactions_coverage.py::TestDispatchTransportToolApproval::test_governance_deny_resolves_approval_as_deniedtest_slack_interactions_coverage.py::TestResumeChoice::test_dashboard_state_link_is_notifiedtest_slack_interactions_coverage.py::TestResumeChoice::test_dm_unavailabletest_slack_interactions_coverage.py::TestResumeChoice::test_malformed_json_value_is_rejectedtest_slack_interactions_coverage.py::TestResumeChoice::test_missing_session_key_is_rejectedtest_slack_interactions_coverage.py::TestResumeChoice::test_thread_mode_links_sessiontest_slack_interactions_coverage.py::TestResumeChoice::test_unknown_mode_returns_without_linkingtest_slack_interactions_coverage.py::TestResumeChoice::test_without_sessions_returnstest_slack_interactions_coverage.py::TestSessionResume::test_empty_value_is_rejectedtest_slack_interactions_coverage.py::TestSessionResume::test_without_sessions_returnstest_slack_interactions_coverage.py::TestStopCancel::test_api_delete_failure_is_swallowedtest_slack_interactions_coverage.py::TestStopCancel::test_without_response_url_deletes_via_apitest_slack_interactions_coverage.py::TestStopConfirm::test_response_url_failure_is_swallowedtest_slack_interactions_coverage.py::TestStopConfirm::test_soft_stop_notifies_threadtest_slack_interactions_coverage.py::TestViewRegistry::test_submission_falls_back_to_forward_handlertest_slack_render_pipeline.py::test_no_module_converts_slack_markdown_directlyManual verification
N/A — test-only change; the runs above are the evidence.
Related Issues
N/A — part of the ongoing weak-test cleanup; no tracking issue.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.