Skip to content

test: fold security twin test bodies into parametrized tests - #24804

Merged
iamwhatever merged 1 commit into
mainfrom
test-cleanup/twins-security
Oct 30, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
test-cleanup/twins-security

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Fold security, sandbox and permission tests whose bodies are identical into one parametrized test each, without weakening any check or dropping any case or platform, so scripts/find_duplicate_test_bodies.py drops from 210 to 190 groups.

Eighteen groups of security tests, and two groups whose names only looked like security, repeat one body word for word. Each copy just feeds the body different cases. The same check lives in several places and drifts when one copy is edited.

Why it matters

Security checks are the ones that must not drift. One test per check, holding every case, is easier to read and to keep correct. This is batch 3 of the twin-body cleanup, after #24626 and #24648.

Not a goal

  • Changing what any security test asserts, or adding cases.
  • Look-alike groups, whose copies read different data or run under different setup (listed below).
  • Folding per-module contract tests into shared helpers.
  • Any change under src/kiro_crew.

What changed (motivation → approach → change)

Same body → copies differ only in their cases → one test per group whose parametrize list joins every copy's cases, built with named_cases. Each case keeps the id <old test name>-<old param id>. The body is the original body; no assert changed.

  • Merge-only. Every case of every copy is still collected, with the same marks and the same active fixtures.
  • Platforms kept per case. A copy's module or class marks travel with its cases. The test_name_grant.py case keeps its skipif(IS_WINDOWS), and the moved test_spawn_exec_shim.py case keeps its own posix_only reason, beside the ctty case's different reason.
  • Comments kept. Each "why" comment from a removed copy sits above that copy's cases. In test_cloud_iam.py the two copies checked the same IAM statement, so their two rationales are combined into one comment above the merged body.
  • test_security_bash_gate.py: the merged test sits in TestTraversalSimulationIsGone and reads the other class's data as TestTheBashGateMatchesNoPaths.UNMATCHED.

Hand check of the security label. The scan tagged 31 groups as security by keyword. Checked by hand:

Verdict Groups
Security, merged 18
Keyword false hit, merged as plain 2 (test_kas_mcp_reattempt.py "credential" in a name; test_proof_line_check.py "author")
Look-alike, left alone 11

The look-alikes left alone:

  • test_argv_floor_inline_and_brace_scope.py vs test_inline_payload_scoped_regressions.py (4 groups): only the first file has the autouse _own_host_seed_connects_nothing stub, so moving a case changes whether the own-host seed opens real sockets.
  • test_governance_list_{decisions,feature_video_hosts,keys}.py (4 groups): each file feeds its own _CASES, a per-list contract.
  • test_denied_commands_security_rows.py vs ..._ssh_self.py: each class has its own autouse own-host fixture.
  • test_sel_bindings.py group and test_cli_doctor_parts.py / test_sandbox_launcher_program_parts.py: each file checks its own copy of a helper.

Backwards compatibility

Compatible: test-only change; every collected case still runs with the same marks and fixtures.

Reader: test/windows-expected-failures.txt:node ids -- release -- lists none of the 22 touched files
Reader: docs, setup.cfg and .github/*:test names -- release -- git grep -w over all 42 old names outside test/test_*.py finds nothing

Tests

Proof: not provable -- test-only change

Evidence on the 22 touched files, base = main aeb5c02c4b6d:

  • Collected ids: 2,894 before and 2,894 after. 326 ids changed, and all map one to one: 0 unmapped, 0 extra.
  • Marks and active fixtures per id: a collection plugin dumped every item's marks (with arguments, so skipif reasons count) and its fixturenames. Old id vs mapped new id for all 326: 0 differences. Unchanged ids: 0 differences.
  • Runs: 3 runs with -n 4: 2,888 passed and 6 skipped each time. The base gives the same.
  • Coverage of kiro_crew, union of 2 runs per side: lines 107,410 vs 107,411, 0 lost. Branches 6,444 vs 6,445, 0 lost.
  • Planted failures:
    • is_denied returning None fails all 79 cases of the two merged denied-command tests, the cases moved in from test_denied_commands_security_mint.py included.
    • is_sensitive_bash_command refusing everything fails all 18 cases of the merged bash-gate test, from both former tests.
  • Twin groups: 210 → 190 (tests in groups 598 → 556).
  • Gates: flake8, isort, black, check_comment_history, check_brand_name, check_file_line_cap, check_subprocess_encoding, test_flake_pattern_ratchet.py, test_wait_discipline_ratchet.py, test_denied_commands_security.py, test_conftest_parts_reach.py, test_leaf_test_scope.py, and the scoped runner's related tests: all green. Semgrep is not installed locally; CI's SAST job runs it.

No timed waits, caplog reads or shared state were added.

Manual verification

N/A — unit coverage sufficient: test-only refactor, verified by the id, mark, fixture and coverage comparisons above.

Related Issues

Follows #24626 and #24648.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

Old tests → new test, per group (20 groups)
Kind Old tests New test
security test/test_argv_floor_inline_and_brace_scope.py::TestBraceExpansionMirrorsBash::test_a_quoted_brace_program_next_to_a_product_path_is_allowed
test/test_argv_floor_inline_and_brace_scope.py::TestAGlobArgumentIsNotAKillProgram::test_glob_argument_next_to_a_product_path_is_allowed
test/test_argv_floor_inline_and_brace_scope.py::TestAGlobArgumentIsNotAKillProgram::test_a_prefixed_substitution_glob_argument_is_a_filename
test/test_argv_floor_inline_and_brace_scope.py::TestAnOptionDashEIsNotSedsExecuteCommand::test_a_wildcard_path_next_to_a_product_path_is_allowed
test/test_argv_floor_inline_and_brace_scope.py::TestBraceExpansionMirrorsBash::test_a_glob_or_brace_next_to_a_product_path_is_allowed
security test/test_argv_floor_inline_and_brace_scope.py::TestAGlobArgumentIsNotAKillProgram::test_a_prefixed_substitution_whose_body_kills_is_denied
test/test_argv_floor_inline_and_brace_scope.py::TestAnOptionDashEIsNotSedsExecuteCommand::test_a_kill_through_a_wildcard_or_an_executor_is_still_denied
test/test_argv_floor_inline_and_brace_scope.py::TestAGlobArgumentIsNotAKillProgram::test_a_kill_hidden_in_a_glob_or_substitution_is_denied
security test/test_agentcore_schema.py::TestNormalizeAgentcoreGatewayUrl::test_rejects_insecure_or_credentialed_urls
test/test_agentcore_schema.py::TestNormalizeAgentcoreGatewayUrl::test_rejects_internal_whitespace
test/test_agentcore_schema.py::TestNormalizeAgentcoreGatewayUrl::test_rejects_an_unsafe_gateway_url
security test/test_cloud_iam.py::TestPolicyDocument::test_put_role_policy_is_tag_scoped
test/test_cloud_iam.py::TestPolicyDocument::test_tag_role_gated_on_existing_managed_tag
test/test_cloud_iam.py::TestPolicyDocument::test_the_role_policy_statement_is_tag_scoped
security test/test_commit_permalink_redaction.py::test_a_key_beside_a_commit_is_still_redacted
test/test_commit_permalink_redaction.py::test_a_key_in_a_plural_route_link_is_still_redacted
test/test_commit_permalink_redaction.py::test_a_key_beside_a_link_is_still_redacted
security test/test_connections_oauth_relay.py::test_return_address_validation_rejects_non_loopback_or_incomplete_urls
test/test_connections_oauth_relay.py::test_scheme_default_does_not_widen_containment
test/test_connections_oauth_relay.py::test_return_address_validation_keeps_loopback_containment
security test/test_managed_tool_policy_include.py::TestPerToolGrants::test_a_whole_server_grant_names_no_include
test/test_managed_tool_policy_include.py::TestPerToolGrants::test_no_ref_to_the_server_names_no_include
test/test_managed_tool_policy_include.py::TestPerToolGrants::test_no_include_is_named
security test/test_mcp_cron_security.py::test_vet_shell_command_allows_benign
test/test_mcp_cron_security.py::test_an_assignment_nothing_reads_is_ordinary
test/test_mcp_cron_security.py::test_an_ordinary_command_is_allowed
security test/test_name_grant_windows.py::TestPowerShellBlockComments::test_controls_are_unaffected
test/test_name_grant_windows.py::TestPowerShellExpressionInvocation::test_controls_still_collect_the_program
test/test_name_grant_windows.py::TestPowerShellBlockComments::test_controls_collect_the_program
security test/test_plantuml_link_redaction.py::test_text_that_is_not_a_whole_diagram_is_not_one
test/test_plantuml_link_redaction.py::test_a_diagram_cut_mid_group_or_with_a_further_group_is_not_kept
test/test_plantuml_link_redaction.py::test_a_partial_diagram_is_not_kept
security test/test_workflow_security.py::TestExtractToolInputStrings::test_empty_string
test/test_workflow_security.py::TestExtractToolInputStrings::test_none_coerced
test/test_workflow_security.py::TestExtractToolInputStrings::test_an_empty_string_yields_no_strings
security test/test_computer_use_skylight.py::TestSkyClickIsLeftButtonOnly::test_the_gate_permits_sky_click_with_the_left_button
test/test_computer_use_policy_cov80.py::TestArgumentShapeRefusals::test_sky_click_accepts_the_left_button
test/test_computer_use_skylight.py::TestSkyClickIsLeftButtonOnly::test_sky_click_with_the_left_button_is_permitted
security test/test_dashboard_approval.py::TestBuildRefusalRecoveryPrompt::test_empty_returns_empty
test/test_deny_guidance.py::TestNoticeIntegration::test_empty_refusals_still_yield_nothing
test/test_dashboard_approval.py::TestBuildRefusalRecoveryPrompt::test_no_refusals_yield_an_empty_prompt
security test/test_denied_commands_security_kill.py::TestSelfProtectionKillTargetScoping::test_resolved_program_text_is_judged_by_its_rule
test/test_denied_commands_security_mint.py::TestInterpreterArgvLiteralMint::test_resolved_payload_is_judged_by_its_rule
test/test_denied_commands_security_kill.py::TestSelfProtectionKillTargetScoping::test_resolved_text_is_judged_by_its_rule
security test/test_denied_commands_security_kill.py::TestSelfProtectionKillTargetScoping::test_text_fed_to_a_shell_still_blocked
test/test_denied_commands_security_mint.py::TestInterpreterArgvLiteralMint::test_interpreter_payload_spellings_still_blocked
test/test_denied_commands_security_kill.py::TestSelfProtectionKillTargetScoping::test_text_fed_to_an_executor_is_still_blocked
security test/test_name_grant_windows.py::TestSystemAndWitness::test_the_windows_dispatcher_table_stays_disjoint_from_the_shared_one
test/test_name_grant.py::TestDispatchers::test_the_windows_dispatcher_table_stays_out_of_the_shared_one
test/test_name_grant_windows.py::TestSystemAndWitness::test_the_windows_dispatcher_table_is_disjoint_from_the_shared_one
plain test/test_kas_mcp_reattempt.py::test_any_other_failure_is_left_alone
test/test_kas_mcp_reattempt.py::test_a_failure_that_only_mentions_a_credential_is_left_alone
test/test_kas_mcp_reattempt.py::test_a_non_auth_failure_is_left_alone
plain test/test_proof_line_check.py::test_well_formed_lines_pass
test/test_proof_line_check.py::test_forms_an_author_types_pass
test/test_proof_line_check.py::test_a_well_formed_proof_line_passes
security test/test_security_bash_gate.py::TestTheBashGateMatchesNoPaths::test_a_path_in_command_text_is_not_a_verdict
test/test_security_bash_gate.py::TestTraversalSimulationIsGone::test_read_only_traversals_are_not_refused
test/test_security_bash_gate.py::TestTraversalSimulationIsGone::test_a_command_naming_a_path_is_not_refused
security test/test_terminal_ctty_shim.py::TestCttyArgvContract::test_no_limits_profile_alone_still_skips_the_interpreter_hop
test/test_spawn_exec_shim.py::TestSpawnShimArgv::test_policy_free_profile_skips_the_interpreter_hop
test/test_terminal_ctty_shim.py::TestCttyArgvContract::test_a_policy_free_profile_skips_the_interpreter_hop

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f8e64cb50ce5a046a61c30eb1b2cd78550a58345 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision f8e64cb50ce5a046a61c30eb1b2cd78550a58345 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f8e64cb50ce5a046a61c30eb1b2cd78550a58345 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this only changes tests, and the description shows that every case, mark and fixture still maps one to one.

[DESIGN-REVIEWED] f8e64cb

Description read

[DESCRIPTION-READ] ee74def7d6010f832701e6e5c46afbd18efab385a917724a2685e0f02170be94

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f8e64cb50ce5a046a61c30eb1b2cd78550a58345 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f8e64cb

Verdict parsed from the review's SHA-scoped output markers for commit f8e64cb50ce5a046a61c30eb1b2cd78550a58345.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f8e64cb50ce5a046a61c30eb1b2cd78550a58345 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f8e64cb50ce5a046a61c30eb1b2cd78550a58345 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f8e64cb

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f8e64cb50ce5a046a61c30eb1b2cd78550a58345 false-positive: <reason, 20 to 500 characters>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

CI disposition: Backend Tests (Windows) (2) is red because of main, not this PR.

test/test_crew_log_fullscan_guard.py::test_a_warm_session_read_that_earns_a_savepoint_reads_only_the_gap reports XPASS(strict) (job log). This PR does not touch that file. The bug the strict xfail tracks has been fixed on main, and #24774 drops the stale xfail. I will rebase onto main only if this PR conflicts; otherwise the shard goes green once #24774 lands and CI reruns.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

CI disposition: Backend Tests (3.12) shards 5-8 and Coverage Gate are a runner outage, not this PR.

All four shards stopped at about 09:48 UTC during Run tests, with the annotation "The self-hosted runner lost communication with the server." No test reported a failure, and the CodeBuild kirocrew-gha-linux build behind them went red at the same time. Coverage Gate fails only because those shards uploaded no coverage. Shards 1-4 of the same run passed. I reran the failed jobs once.

The Windows shard 2 red is unchanged: it is main's strict XPASS, fixed by #24774 (see the earlier comment).

@iamwhatever
iamwhatever force-pushed the test-cleanup/twins-security branch from 246ae9e to 62ab1ad Compare October 27, 2026 13:02
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main d0695966ae, which carries #24774 (the stale strict xfail behind the Windows shard 2 red). New head 62ab1ad8e2.

Main did not touch any of this PR's 22 files in between. The re-run checks on the new head are all green:

  • Id and mark check: 2,894 ids → 2,894; 326 changed ids map one to one; marks and fixtures differ 0 times.
  • Touched files: 2,888 passed, 6 skipped.
  • Flake-pattern ratchet: green.

CI is re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 29, 2026 07:13
@iamwhatever
iamwhatever disabled auto-merge October 30, 2026 04:41
Twenty groups of tests repeat one body word for word: eighteen security,
sandbox or permission groups, plus two whose names only matched a
security keyword. Each group becomes one parametrized test; named_cases
keeps every case under its old name, and a moved case carries its own
file's marks, so all 326 changed ids map one to one with the same marks
and fixtures. No assert changed. Duplicate body groups drop from 210 to
190.
@iamwhatever
iamwhatever force-pushed the test-cleanup/twins-security branch from 62ab1ad to f8e64cb Compare October 30, 2026 04:48
@iamwhatever
iamwhatever enabled auto-merge (squash) October 30, 2026 04:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 30, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

CI disposition: Backend Tests (3.12) shards 5–8 were a runner outage, not test failures.

In run 40508658391 attempt 2, all four shards ran about 24–26 min and then stopped within one minute of each other (05:17:45–05:18:34 UTC, Oct 28), mid Run tests. Each has the annotation "The self-hosted runner lost communication with the server". No test failed and no Timeout > line appears. Coverage Gate and PR Readiness failed only because those shards uploaded nothing. Shards 1–4 of the same attempt passed.

Rebased onto main f369225a20, with no conflicts. Main added new tests to test_managed_tool_policy_include.py, test_name_grant_windows.py and test_spawn_exec_shim.py, but none of them touch a merged or moved test. New head f8e64cb50c. Re-checked on the new head:

  • Ids: 2,908 → 2,908, and all 326 changed ids map one to one. Marks and fixtures differ 0 times.
  • Touched files, 3 runs at -n 4: 2,902 passed and 6 skipped each time.
  • Flake-pattern ratchet, wait-discipline ratchet, line cap, comment-history, brand, black and lint: all green.
  • Twin groups: main 223 → 203 with this PR.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 30, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Kiro Crew Auto-Pipeline [operator: buluoray]:

Approving at f8e64cb.

I checked the fold mechanically rather than by reading. On the merge-base and on this head I
collected the 22 touched files and dumped, for every item, its parameter values, its marks with
arguments, and its full fixture closure:

  • 2908 items on both sides; 2582 ids unchanged, 326 renamed.
  • The (parameter values, marks) multiset over the 326 old items equals the one over the 326 new
    items exactly. No case was dropped or added, and no mark was lost, including the module-level
    usefixtures("fresh_darwin_libc_handles") on the cases that moved from the mint module into
    the kill module, the skipif(IS_WINDOWS) case now in test_name_grant_windows.py, and the
    posix_only case now in test_terminal_ctty_shim.py.
  • Fixture usage over the old and new items is identical overall; per-file differences are only
    the cross-module moves.
  • Running the 22 files: 2902 passed / 6 skipped on the base and 2902 passed / 6 skipped on this
    head, so no moved body depends on a helper its new module lacks.

The eight unchanged ids in TestSyncLimitedSpawns whose parameters look different only differ
in a function's memory address.

Line cap check passes and CI is green. Mutation checks were not run; for a merge-only fold the
collection identity plus the pass/skip parity is the evidence.

@iamwhatever
iamwhatever merged commit fd3dd92 into main Oct 30, 2026
93 checks passed
@iamwhatever
iamwhatever deleted the test-cleanup/twins-security branch October 30, 2026 16:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants