Skip to content

test: keep the near-ceiling shell title test off the slow regex path - #25952

Merged
chenmingwei23 merged 1 commit into
mainfrom
crew/issue-triage/fix-23346-size-ceiling-test
Oct 29, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
crew/issue-triage/fix-23346-size-ceiling-test

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_a_command_just_under_the_size_ceiling_is_not_pushed_over_it stops crashing its xdist worker on Windows.

On Windows shard 1/8 of PR #23334 the test ran 224s and its worker died ([gw1] node down: Not properly terminated). On Linux the same test takes about 14s, by far the slowest in its file.

Why it matters

A crashed worker turns an unrelated PR's Windows shard red. Every author then has to rerun or explain it.

Not a goal

  • Speeding up the deny rules themselves. Two built-in rules are quadratic on one very long word; that is a production question, raised separately, not changed here.
  • Changing what the test checks. The size boundary it pins stays exactly the same.

What changed (motivation → approach → change)

The test built its command from one 20 KiB word ("a" * N). A profile showed 13.7s of the run inside re.Pattern.search, all called from denied_rules._DenyMatcher.match. Two rules with a \w+ run (self-protection-kill-interpreter, credential-exfil-kirocrew-token-argv) cost about 0.84s each per pass on that word, and the gate makes several passes. A slower Windows runner multiplied that past the 120s timeout.

The test only cares about length: a command that fits the ceiling must not be refused because the Running: prefix pushes the title over it. So the filler is now short words ("a a a ...") of the same total length. The deny rules then run in about 0.03s per pass and the test takes about 1s. A new assert pins the command at exactly MAX_SCANNABLE_COMMAND_CHARS - 5, as before.

Backwards compatibility

Compatible: test-only change; no product code is touched.

Removes nothing: the test keeps the same length and the same assertions, plus one new length assert.

Tests

test/test_llm_helpers_tool_input_offload.py::TestTruncatedShellTitleTier::test_a_command_just_under_the_size_ceiling_is_not_pushed_over_it still checks that a command just under the ceiling is approved and nothing is rejected.

Proof: not provable -- test-only change

Determinism proof (the forced condition is a slow runner, modelled as a 5s per-test timeout):

  • Parent (origin/main), --timeout=5: FAILED, Timeout >5.0s.
  • Fix, --timeout=5, 20 runs at -n0: 20/20 passed (call 1.07s).
  • Fix, whole file at -n 4 --dist loadgroup, --timeout=5: 53 passed.
  • Fix, whole file in three shuffled orders (seeds 11, 22, 33), --timeout=5: 53 passed each time.
  • test/test_flake_pattern_ratchet.py: 64 passed.

Manual verification

N/A — unit coverage sufficient; the change is one test's input.

Related Issues

Closes #23346

Related: #25954 (the two slow deny rules; not changed here)

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8f9d22d8924f78fc6ca555d98f4df4c46a9a6a8e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 8f9d22d8924f78fc6ca555d98f4df4c46a9a6a8e ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8f9d22d8924f78fc6ca555d98f4df4c46a9a6a8e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only input change; the size boundary under test is unchanged, and the slow regex behind it is tracked separately in #25954.

Notes

[DESIGN-REVIEWED] 8f9d22d

Description read

[DESCRIPTION-READ] ea2b3aefabbf4553b054e3f72f9f224049a2a70d4653385f74802f3f275dae21

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8f9d22d8924f78fc6ca555d98f4df4c46a9a6a8e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8f9d22d

Verdict parsed from the review's SHA-scoped output markers for commit 8f9d22d8924f78fc6ca555d98f4df4c46a9a6a8e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8f9d22d8924f78fc6ca555d98f4df4c46a9a6a8e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8f9d22d8924f78fc6ca555d98f4df4c46a9a6a8e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8f9d22d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8f9d22d8924f78fc6ca555d98f4df4c46a9a6a8e false-positive: <reason, 20 to 500 characters>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

The only red on head 34c4d03 is Backend Tests (Windows) (2). It fails on the 3 tests that are red on main and tracked in #25937 (test_no_lock_file_is_created_with_a_nonexclusive_open, test_the_package_stays_within_its_line_budget, test_hook_time_is_not_charged_to_the_first_prompt_watchdog). This PR changes only test/test_llm_helpers_tool_input_offload.py, and that test passed on this shard. No push, rerun or override for it: I will rebase once after the fix, #25947, merges.

🤖 AI-assisted comment 🤖

The test fed one 20 KiB word to the deny rules. Two rules with a \w+
run are quadratic on such a word, so the call took about 14s on Linux
and passed the 120s timeout on a Windows runner, crashing the worker.
Use short words of the same total length: the size boundary under
test is unchanged and the call takes about 1s.

Closes #23346
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased once onto main now that #25947 (fix for the #25937 reds) has merged. New head 8f9d22d; the changed test file passes locally on it (53 passed). CI re-running.

🤖 AI-assisted comment 🤖

@iamwhatever
iamwhatever force-pushed the crew/issue-triage/fix-23346-size-ceiling-test branch from 34c4d03 to 8f9d22d Compare October 29, 2026 21:53
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 29, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready on head 8f9d22d: PR Readiness passed (label readiness: passed), 92 checks with 0 failing, GPT and Opus reviews fresh on this head, 0 unresolved threads, and green_age fresh (main moved in nothing this PR touches). The change is test-only. It needs a maintainer approval to merge; I have not merged, approved or enabled auto-merge.

🤖 AI-assisted comment 🤖

@chenmingwei23 chenmingwei23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved as part of a bulk pass over readiness: passed PRs.

@chenmingwei23
chenmingwei23 merged commit 4c69800 into main Oct 29, 2026
93 checks passed
@chenmingwei23
chenmingwei23 deleted the crew/issue-triage/fix-23346-size-ceiling-test branch October 29, 2026 23:34
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_command_just_under_the_size_ceiling_is_not_pushed_over_it (test_llm_helpers_tool_input_offload.py) crashes its xdist worker on Windows

2 participants