Repository navigation
test: fold security twin test bodies into parametrized tests - #24804
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this only changes tests, and the description shows that every case, mark and fixture still maps one to one. [DESIGN-REVIEWED] f8e64cb Description read[DESCRIPTION-READ] ee74def7d6010f832701e6e5c46afbd18efab385a917724a2685e0f02170be94 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
CI disposition: Backend Tests (Windows) (2) is red because of main, not this PR.
|
|
CI disposition: Backend Tests (3.12) shards 5-8 and Coverage Gate are a runner outage, not this PR. All four shards stopped at about 09:48 UTC during The Windows shard 2 red is unchanged: it is main's strict XPASS, fixed by #24774 (see the earlier comment). |
246ae9e to
62ab1ad
Compare
|
Rebased onto main Main did not touch any of this PR's 22 files in between. The re-run checks on the new head are all green:
CI is re-running. |
Twenty groups of tests repeat one body word for word: eighteen security, sandbox or permission groups, plus two whose names only matched a security keyword. Each group becomes one parametrized test; named_cases keeps every case under its old name, and a moved case carries its own file's marks, so all 326 changed ids map one to one with the same marks and fixtures. No assert changed. Duplicate body groups drop from 210 to 190.
62ab1ad to
f8e64cb
Compare
|
CI disposition: Backend Tests (3.12) shards 5–8 were a runner outage, not test failures. In run 40508658391 attempt 2, all four shards ran about 24–26 min and then stopped within one minute of each other (05:17:45–05:18:34 UTC, Oct 28), mid Rebased onto main
|
buluoray
left a comment
There was a problem hiding this comment.
🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:
Approving at f8e64cb.
I checked the fold mechanically rather than by reading. On the merge-base and on this head I
collected the 22 touched files and dumped, for every item, its parameter values, its marks with
arguments, and its full fixture closure:
- 2908 items on both sides; 2582 ids unchanged, 326 renamed.
- The (parameter values, marks) multiset over the 326 old items equals the one over the 326 new
items exactly. No case was dropped or added, and no mark was lost, including the module-level
usefixtures("fresh_darwin_libc_handles")on the cases that moved from the mint module into
the kill module, theskipif(IS_WINDOWS)case now intest_name_grant_windows.py, and the
posix_onlycase now intest_terminal_ctty_shim.py. - Fixture usage over the old and new items is identical overall; per-file differences are only
the cross-module moves. - Running the 22 files: 2902 passed / 6 skipped on the base and 2902 passed / 6 skipped on this
head, so no moved body depends on a helper its new module lacks.
The eight unchanged ids in TestSyncLimitedSpawns whose parameters look different only differ
in a function's memory address.
Line cap check passes and CI is green. Mutation checks were not run; for a merge-only fold the
collection identity plus the pass/skip parity is the evidence.
Problem / Motivation
Goal: Fold security, sandbox and permission tests whose bodies are identical into one parametrized test each, without weakening any check or dropping any case or platform, so
scripts/find_duplicate_test_bodies.pydrops from 210 to 190 groups.Eighteen groups of security tests, and two groups whose names only looked like security, repeat one body word for word. Each copy just feeds the body different cases. The same check lives in several places and drifts when one copy is edited.
Why it matters
Security checks are the ones that must not drift. One test per check, holding every case, is easier to read and to keep correct. This is batch 3 of the twin-body cleanup, after #24626 and #24648.
Not a goal
src/kiro_crew.What changed (motivation → approach → change)
Same body → copies differ only in their cases → one test per group whose parametrize list joins every copy's cases, built with
named_cases. Each case keeps the id<old test name>-<old param id>. The body is the original body; no assert changed.test_name_grant.pycase keeps itsskipif(IS_WINDOWS), and the movedtest_spawn_exec_shim.pycase keeps its ownposix_onlyreason, beside the ctty case's different reason.test_cloud_iam.pythe two copies checked the same IAM statement, so their two rationales are combined into one comment above the merged body.test_security_bash_gate.py: the merged test sits inTestTraversalSimulationIsGoneand reads the other class's data asTestTheBashGateMatchesNoPaths.UNMATCHED.Hand check of the security label. The scan tagged 31 groups as security by keyword. Checked by hand:
test_kas_mcp_reattempt.py"credential" in a name;test_proof_line_check.py"author")The look-alikes left alone:
test_argv_floor_inline_and_brace_scope.pyvstest_inline_payload_scoped_regressions.py(4 groups): only the first file has the autouse_own_host_seed_connects_nothingstub, so moving a case changes whether the own-host seed opens real sockets.test_governance_list_{decisions,feature_video_hosts,keys}.py(4 groups): each file feeds its own_CASES, a per-list contract.test_denied_commands_security_rows.pyvs..._ssh_self.py: each class has its own autouse own-host fixture.test_sel_bindings.pygroup andtest_cli_doctor_parts.py/test_sandbox_launcher_program_parts.py: each file checks its own copy of a helper.Backwards compatibility
Compatible: test-only change; every collected case still runs with the same marks and fixtures.
Reader: test/windows-expected-failures.txt:node ids -- release -- lists none of the 22 touched files
Reader: docs, setup.cfg and .github/*:test names -- release --
git grep -wover all 42 old names outsidetest/test_*.pyfinds nothingTests
Proof: not provable -- test-only change
Evidence on the 22 touched files, base = main
aeb5c02c4b6d:skipifreasons count) and itsfixturenames. Old id vs mapped new id for all 326: 0 differences. Unchanged ids: 0 differences.-n 4: 2,888 passed and 6 skipped each time. The base gives the same.kiro_crew, union of 2 runs per side: lines 107,410 vs 107,411, 0 lost. Branches 6,444 vs 6,445, 0 lost.is_deniedreturningNonefails all 79 cases of the two merged denied-command tests, the cases moved in fromtest_denied_commands_security_mint.pyincluded.is_sensitive_bash_commandrefusing everything fails all 18 cases of the merged bash-gate test, from both former tests.check_comment_history,check_brand_name,check_file_line_cap,check_subprocess_encoding,test_flake_pattern_ratchet.py,test_wait_discipline_ratchet.py,test_denied_commands_security.py,test_conftest_parts_reach.py,test_leaf_test_scope.py, and the scoped runner's related tests: all green. Semgrep is not installed locally; CI's SAST job runs it.No timed waits, caplog reads or shared state were added.
Manual verification
N/A — unit coverage sufficient: test-only refactor, verified by the id, mark, fixture and coverage comparisons above.
Related Issues
Follows #24626 and #24648.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.
Old tests → new test, per group (20 groups)
test/test_argv_floor_inline_and_brace_scope.py::TestBraceExpansionMirrorsBash::test_a_quoted_brace_program_next_to_a_product_path_is_allowedtest/test_argv_floor_inline_and_brace_scope.py::TestAGlobArgumentIsNotAKillProgram::test_glob_argument_next_to_a_product_path_is_allowedtest/test_argv_floor_inline_and_brace_scope.py::TestAGlobArgumentIsNotAKillProgram::test_a_prefixed_substitution_glob_argument_is_a_filenametest/test_argv_floor_inline_and_brace_scope.py::TestAnOptionDashEIsNotSedsExecuteCommand::test_a_wildcard_path_next_to_a_product_path_is_allowedtest/test_argv_floor_inline_and_brace_scope.py::TestBraceExpansionMirrorsBash::test_a_glob_or_brace_next_to_a_product_path_is_allowedtest/test_argv_floor_inline_and_brace_scope.py::TestAGlobArgumentIsNotAKillProgram::test_a_prefixed_substitution_whose_body_kills_is_deniedtest/test_argv_floor_inline_and_brace_scope.py::TestAnOptionDashEIsNotSedsExecuteCommand::test_a_kill_through_a_wildcard_or_an_executor_is_still_deniedtest/test_argv_floor_inline_and_brace_scope.py::TestAGlobArgumentIsNotAKillProgram::test_a_kill_hidden_in_a_glob_or_substitution_is_deniedtest/test_agentcore_schema.py::TestNormalizeAgentcoreGatewayUrl::test_rejects_insecure_or_credentialed_urlstest/test_agentcore_schema.py::TestNormalizeAgentcoreGatewayUrl::test_rejects_internal_whitespacetest/test_agentcore_schema.py::TestNormalizeAgentcoreGatewayUrl::test_rejects_an_unsafe_gateway_urltest/test_cloud_iam.py::TestPolicyDocument::test_put_role_policy_is_tag_scopedtest/test_cloud_iam.py::TestPolicyDocument::test_tag_role_gated_on_existing_managed_tagtest/test_cloud_iam.py::TestPolicyDocument::test_the_role_policy_statement_is_tag_scopedtest/test_commit_permalink_redaction.py::test_a_key_beside_a_commit_is_still_redactedtest/test_commit_permalink_redaction.py::test_a_key_in_a_plural_route_link_is_still_redactedtest/test_commit_permalink_redaction.py::test_a_key_beside_a_link_is_still_redactedtest/test_connections_oauth_relay.py::test_return_address_validation_rejects_non_loopback_or_incomplete_urlstest/test_connections_oauth_relay.py::test_scheme_default_does_not_widen_containmenttest/test_connections_oauth_relay.py::test_return_address_validation_keeps_loopback_containmenttest/test_managed_tool_policy_include.py::TestPerToolGrants::test_a_whole_server_grant_names_no_includetest/test_managed_tool_policy_include.py::TestPerToolGrants::test_no_ref_to_the_server_names_no_includetest/test_managed_tool_policy_include.py::TestPerToolGrants::test_no_include_is_namedtest/test_mcp_cron_security.py::test_vet_shell_command_allows_benigntest/test_mcp_cron_security.py::test_an_assignment_nothing_reads_is_ordinarytest/test_mcp_cron_security.py::test_an_ordinary_command_is_allowedtest/test_name_grant_windows.py::TestPowerShellBlockComments::test_controls_are_unaffectedtest/test_name_grant_windows.py::TestPowerShellExpressionInvocation::test_controls_still_collect_the_programtest/test_name_grant_windows.py::TestPowerShellBlockComments::test_controls_collect_the_programtest/test_plantuml_link_redaction.py::test_text_that_is_not_a_whole_diagram_is_not_onetest/test_plantuml_link_redaction.py::test_a_diagram_cut_mid_group_or_with_a_further_group_is_not_kepttest/test_plantuml_link_redaction.py::test_a_partial_diagram_is_not_kepttest/test_workflow_security.py::TestExtractToolInputStrings::test_empty_stringtest/test_workflow_security.py::TestExtractToolInputStrings::test_none_coercedtest/test_workflow_security.py::TestExtractToolInputStrings::test_an_empty_string_yields_no_stringstest/test_computer_use_skylight.py::TestSkyClickIsLeftButtonOnly::test_the_gate_permits_sky_click_with_the_left_buttontest/test_computer_use_policy_cov80.py::TestArgumentShapeRefusals::test_sky_click_accepts_the_left_buttontest/test_computer_use_skylight.py::TestSkyClickIsLeftButtonOnly::test_sky_click_with_the_left_button_is_permittedtest/test_dashboard_approval.py::TestBuildRefusalRecoveryPrompt::test_empty_returns_emptytest/test_deny_guidance.py::TestNoticeIntegration::test_empty_refusals_still_yield_nothingtest/test_dashboard_approval.py::TestBuildRefusalRecoveryPrompt::test_no_refusals_yield_an_empty_prompttest/test_denied_commands_security_kill.py::TestSelfProtectionKillTargetScoping::test_resolved_program_text_is_judged_by_its_ruletest/test_denied_commands_security_mint.py::TestInterpreterArgvLiteralMint::test_resolved_payload_is_judged_by_its_ruletest/test_denied_commands_security_kill.py::TestSelfProtectionKillTargetScoping::test_resolved_text_is_judged_by_its_ruletest/test_denied_commands_security_kill.py::TestSelfProtectionKillTargetScoping::test_text_fed_to_a_shell_still_blockedtest/test_denied_commands_security_mint.py::TestInterpreterArgvLiteralMint::test_interpreter_payload_spellings_still_blockedtest/test_denied_commands_security_kill.py::TestSelfProtectionKillTargetScoping::test_text_fed_to_an_executor_is_still_blockedtest/test_name_grant_windows.py::TestSystemAndWitness::test_the_windows_dispatcher_table_stays_disjoint_from_the_shared_onetest/test_name_grant.py::TestDispatchers::test_the_windows_dispatcher_table_stays_out_of_the_shared_onetest/test_name_grant_windows.py::TestSystemAndWitness::test_the_windows_dispatcher_table_is_disjoint_from_the_shared_onetest/test_kas_mcp_reattempt.py::test_any_other_failure_is_left_alonetest/test_kas_mcp_reattempt.py::test_a_failure_that_only_mentions_a_credential_is_left_alonetest/test_kas_mcp_reattempt.py::test_a_non_auth_failure_is_left_alonetest/test_proof_line_check.py::test_well_formed_lines_passtest/test_proof_line_check.py::test_forms_an_author_types_passtest/test_proof_line_check.py::test_a_well_formed_proof_line_passestest/test_security_bash_gate.py::TestTheBashGateMatchesNoPaths::test_a_path_in_command_text_is_not_a_verdicttest/test_security_bash_gate.py::TestTraversalSimulationIsGone::test_read_only_traversals_are_not_refusedtest/test_security_bash_gate.py::TestTraversalSimulationIsGone::test_a_command_naming_a_path_is_not_refusedtest/test_terminal_ctty_shim.py::TestCttyArgvContract::test_no_limits_profile_alone_still_skips_the_interpreter_hoptest/test_spawn_exec_shim.py::TestSpawnShimArgv::test_policy_free_profile_skips_the_interpreter_hoptest/test_terminal_ctty_shim.py::TestCttyArgvContract::test_a_policy_free_profile_skips_the_interpreter_hop