Skip to content

test: fix two Windows-red tests and write down why - #16083

Merged
iamwhatever merged 1 commit into
mainfrom
test/deflake-two-windows-tests
Oct 2, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
test/deflake-two-windows-tests

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Two backend tests are red on the Windows shards of every open pull request (seen on #15823, shards 1 and 5). Neither is a flake: both fail the same way every run, locally too.

Test Why it is red
test_app_script_cron_sibling_import.py::TestACommandCronRunningBundleCode::test_the_call_site_leaves_the_bundle_readable Windows refuses every command cron before it spawns (no POSIX shell, by design). The recorder sees no spawn, so assert len(spawns) == 1 fails with []. The file claimed this case runs "on every host".
test_context_composition_contract.py::TestSurface::test_builder_members_keep_their_kind_and_signature It pins build_message's full signature byte for byte. #15998 recorded the list; #15761, cut from an older base, added the keyword thread_replies_text. Each was green alone; on main together the pin is red on every platform.

What changed

  • Command-cron cases now gate on the product's own refusal probe, cron_script._resolve_command_shell() is None, not on the platform name. A host that runs command crons is still held to the pin.
  • The two signature pins compare through _frozen_view, which drops only a NEW keyword-only parameter that has a default (no caller can feel it). A removed, renamed, reordered or retyped parameter, a new positional or required one, or a changed return annotation still reds. test_only_a_compatible_addition_is_tolerated is the negative control for each.
  • docs/system-specs/common/testing-conventions.md: two entries under "A gate that reds on someone else's pull request" -- gate on the product's refusal probe and run an "every host" test on Windows before merge; pin what callers depend on, not every byte, for a surface feature work keeps growing.

#16078 bumps the frozen string for the same red. The two do not touch the same lines and either can merge first.

Testing

On Windows (this host): both files 125 passed, 3 skipped; --noconftest on the cron file passes; flake8, black and the comment-history gate are clean. The doc-reading tests (test_docs_mock_pid_convention, test_black_gate_contract, test_symbols_manifest_contract, test_harness) pass. Linux/macOS rely on CI.

Pattern harvest

Rule candidate: a test whose name or docstring says "every host" must gate on the product's own refusal probe when the product refuses that feature on some platform, and a byte-exact snapshot of a surface other branches routinely grow should tolerate compatible additions.

test_app_script_cron_sibling_import: the command-cron recorder case claimed to run on every host, but Windows refuses every command cron before it spawns (no POSIX shell), so the recorder saw nothing. Gate the command cases on the product's own probe, cron_script._resolve_command_shell().

test_context_composition_contract: the byte-exact build_message signature pin went red on main when a branch added the keyword thread_replies_text beside the branch that froze the list. Compare through _frozen_view, which tolerates only a new keyword-only parameter with a default, and add a negative control proving every caller-visible change still reds.

testing-conventions.md: two new entries under 'A gate that reds on someone else's pull request'.
@bolichen97
bolichen97 requested a review from a team as a code owner October 2, 2026 06:49
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 49e1d34a46825db69f0df5255ee599b0df7dcac5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the Linux/macOS shards of this round are green — the author tested only Windows locally and the probe-gated skip must not fire there.

[DESIGN-REVIEWED] 49e1d34

Description read

[DESCRIPTION-READ] 845857546763a11cde7c8f7ae985d73dfcf65972b6306ac3ae400473f3b6b152

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 49e1d34a46825db69f0df5255ee599b0df7dcac5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 49e1d34

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 49e1d34a46825db69f0df5255ee599b0df7dcac5: <one-sentence reason>

@iamwhatever
iamwhatever merged commit 5a70ce7 into main Oct 2, 2026
90 of 93 checks passed
@iamwhatever
iamwhatever deleted the test/deflake-two-windows-tests branch October 2, 2026 06:57
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Opus 5 Review — 🔴 changes requested (blocking)

Reviewed 49e1d34a46825db69f0df5255ee599b0df7dcac5 — this comment is updated in place on each push.

Two blocking issues: both changed test files loosen a ratchet rather than re-pin it — the context-surface signature pin stops catching the exact class of drift already present on main, and the new cron-test skip stands the pin down on hosts that do run command crons.

[BLOCK-MERGE] 49e1d34

BLOCKING — test/test_context_composition_contract.py:1568

or re.search(rf"(?:\(|, ){re.escape(p.name)}(?=[:=,)])", frozen)

ContextBuilder.build_message (src/kiro_crew/context.py:2776) already carries thread_replies_text: str | None = None, which _BUILDER_MEMBERS["build_message"] (line 1240) does not record → _frozen_view classifies it as keyword-only-with-default, drops it from the live side, and assert current == _BUILDER_MEMBERS goes green with the snapshot permanently misdescribing the surface → every future keyword-only addition to build_message, build_session_context or any _PUBLIC_SIGNATURES entry (including a context-gating switch) lands with no signal, which is the one change class the exact-set pin existed to catch. This is the a-ratchet-may-only-tighten path the rule names explicitly: "LOOSENING it so the diff fits is not [correct] … the count MOVES to the new true value".
Fix: re-record _BUILDER_MEMBERS["build_message"] with thread_replies_text: 'str | None' = None in its declared position and drop _frozen_view (and test_only_a_compatible_addition_is_tolerated), restoring byte-exact comparison.

BLOCKING — test/test_app_script_cron_sibling_import.py:155

if cron_script._resolve_command_shell() is None:
    pytest.skip("this host refuses command crons: no POSIX shell to run them")

On a POSIX host with /bin/sh but no OS sandbox backend — the host class this same file already recognises via _NO_BACKEND — the autouse fixture runs the real probe: _resolve_command_shell → _shell_is_posix_strict → _probe_one_form → unpatched wrap_argv(..., mode="strict") raises SandboxUnavailableError (sandbox.py:10611), which _probe_one_form swallows for both forms → _POSIX_STRICT_CACHE["/bin/sh"] = False → None → skip. test_the_call_site_leaves_the_bundle_readable is written to run on exactly that host (it replaces wrap_argv with a recording passthrough and forces credential_mask_applies True), and its sibling is already _NO_BACKEND-skipped, so the command-cron half of the #13273/#16072 apps-tree-mask pin runs nowhere there — under a skip reason that is false, since the host has a POSIX shell. A transient probe failure caches False the same way and silently stands the pin down on any host.
Fix: gate on the permanent product refusal the docstring actually names — platform_compat.IS_WINDOWS — or evaluate the probe with wrap_argv patched as the test itself patches it, so the skip tracks "this host refuses command crons" rather than "this host has no sandbox backend".

[OPUS-REVIEWED] 49e1d34

Verdict parsed from the review's SHA-scoped output markers for commit 49e1d34a46825db69f0df5255ee599b0df7dcac5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 49e1d34a46825db69f0df5255ee599b0df7dcac5: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

Re the Opus review (posted after merge), both findings handled in #16096:

  • Command-cron skip (test_app_script_cron_sibling_import.py:155): agreed, the probe ran before wrap_argv was patched, so a sandbox-less POSIX host read as shell-less. test: keep the command-cron pin live on sandbox-less POSIX hosts #16096 runs the probe after the recorder patch, with _POSIX_STRICT_CACHE cleared.
  • Signature pin (test_context_composition_contract.py:1568): thread_replies_text is now recorded, so the table describes main exactly. The tolerance itself is kept on purpose: a byte-exact pin over build_message reds main whenever a feature branch adds a defaulted keyword beside a branch that recorded the list, which is the red test: fix two Windows-red tests and write down why #16083 fixed. Every change a caller can feel (removed, renamed, reordered, retyped, new positional or required parameter, return type) still reds, proved by test_only_a_compatible_addition_is_tolerated. If maintainers prefer byte-exact, that is a one-line revert of _frozen_view.

bolichen97 added a commit that referenced this pull request Oct 2, 2026
)

* test: keep the command-cron pin live on sandbox-less POSIX hosts

Address Opus review of #16083: probe the command shell after the recorder patches wrap_argv (unpatched, the probe raises on a POSIX host with no sandbox backend and wrongly skips), and re-record thread_replies_text in the frozen build_message signature so the snapshot describes main exactly.

* test: gate the real-sandbox command-cron case on the shell probe too
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants