Skip to content

test: assert the wait or the signal, not seconds, in 23 long bounds - #37906

Merged
bolichen97 merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/timing-budgets-ten-second
Nov 16, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/timing-budgets-ten-second

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The last 23 tests that hold their claim to a 10-120 s wall clock stop reading the host's speed, and each still fails, by name and promptly, when the regression it guards comes back.

After #37600, #37630, #37675, #37715, #37782 and #37853 these are the K8 hits left on main: every literal duration bound of 10 s or more, in 18 files. The looser bounds rarely red a healthy run, but under coverage, a busy CPU or a fast clock the tighter ones do, and most of the regressions they exist to catch either pass under them or hang the worker until pytest's timeout.

Why it matters

A bound that a regression sits under or hangs past gives no signal, and one that follows the runner blocks unrelated PRs. Clearing them leaves the K8 ratchet at zero in these files.

Not a goal

  • Any change under src/.
  • Bounds whose limit is a variable or an expression (elapsed < budget); the K8 detector does not see them yet, and a later PR takes them with the detector.
  • New assertions beyond what each test already claimed.

What changed (motivation → approach → change)

Each bound is replaced by what it stood for:

Claim the bound stood for Tests Now asserts
the turn ends at its budget stream command (2) the timeout each armed queue poll is handed (<= _QUEUE_POLL_CEILING_SECS), at most one park after the budget, and the stop reason
a reconnect gives up inside its budget stub broker a backoff clock that only the code's own waits advance; budget - 1e-6 <= spent <= budget + max backoff
shutdown does not wait on a held usage row subagent stop reason the timeout handed to the drain asyncio.wait (<= _REPORT_DRAIN_TIMEOUT)
a timed-out driver is reaped with its group codex and opencode session MCP the pipe drains the driver runner makes, each with its bound
a cancel ends the command cron cancel the command exits by the cancel's SIGTERM
teardown sends SIGKILL after its grace oneshot supervisor the signal rounds sent (SIGTERM, then SIGKILL after the grace on a virtual clock); the helper reports readiness after installing its handler
a bounded call gives up at its cap pod video guard the setitimer calls it arms and disarms
a hanging child is stopped at its timeout preflight (2), decisions installer the wait it is handed, then the stop / kill, then a bounded reap
handle mode answers without polling pod harness paths a sleep shim on PATH is never called
a deep self-link is judged without walking it apps registry the copy keeps B -> . as one link in the checkout and every preview
the notarize watchdog fires inside its budget notarize (3) the events a fake xcrun/sleep record: every hang stopped, no wait past the budget; a stop before the hang counts as an early cancel
the caller is bounded by its own deadline subprocess pool (2) the select / reader waits, each within the caller's budget, on a manual clock so a slow lease cannot expire it first
linear scans chat slot source URLs, display split redaction interpreter work or redactor calls at three sizes (_linear_work)
the scan runs off the loop tool-input offload the loop keeps ticking during the scan (the 120 s ceiling is dropped; the test's own timeout mark is the lost-run bound)
the spawn stamp is projected onto the wall clock session memory both clocks pinned through the kiro_crew.session_allocation facade; created == wall - 90.0

False positives, left unchanged: test_turn_harness.py:371 (record.elapsed is the harness's virtual loop time), test_mcp_quarantine.py:757 (arm_at - flag_at are character offsets into source text), test_bench_retrieval_stats.py:762 (the sign of a delta between two fixed samples).

test_subagent_stop_reason_consistency.py would have crossed the 2,000-line cap, so its task-runner section moves unchanged to test/test_subagent_stop_reason_taskrunner.py, and the shared _STALL_EVIDENCE text moves into test/subagent_stop_reason_helpers.py.

Backwards compatibility

Compatible: test files and test helpers only; no product behaviour changes.

Removes nothing: every test keeps its claim; one test section and one constant move to sibling files unchanged.

Tests

Proof: not provable -- test-only change

Planted regressions as in-memory monkeypatches (scratch pytest plugins; src/ untouched), the same plant against main's original test and this branch's test. "Lost run" means pytest's timeout killed the test.

Sites Plant main branch
stream command, budget budget acted on one park late red after ~10 s red by name
stream command, co-tenant fanout own-frame check always true red after ~32 s red by name
stream command, queue poll poll timeout 5 s → 12 s red after ~12 s red by name
stub broker retry forever lost run (120 s) red at once
subagent drain drain wait 20 s red red by name
codex / opencode driver kill only the driver pid red after 67 s (IndexError) red by name
oneshot supervisor 30 s grace red after ~30 s red by name
pod video guard cap ×40 red after 22 s red by name
preflight unbounded wait / stop never reaps / stop waits out red after 32 s / passes / red after 32 s red / red / red
preflight default bound default GIT_TIMEOUT_S dropped red after 32 s red by name
pod harness paths handle mode keeps polling red after 62 s red at once
decisions installer ignore stop / no kill / unbounded reap red after 62 s / passes / passes red / red / red
apps registry copy follows links / walks one level red / passes at depth 60 red / red
notarize (3) watchdog 40 s late / backoff +30 s red after 33-45 s red in 2-3 s
subprocess pool (2) select / reader wait +60 s / deadline dropped red after 63 s / lost run red at once
chat slot unbounded next-occurrence scan red after ~10 s red: work not linear
display split interior run cap removed lost run (120 s) red: calls not linear
tool-input offload scan inline on the loop red red
session memory raw monotonic stamp kept red red

Slow-host repros (real pauses injected in memory before the spawn, signal, poll or lease, up to 6 s) pass on the branch, including the cases the local reviews raised: a helper slow to install its handler, a watchdog SIGTERM before the fake child records its hang, and a >1 s pause after the pool lease.

Forced-condition repeat runs on an 8-core DevSpace, the 23 converted tests (25 ids), -n0, main (274647852d) against this branch (da76e9e5ad; the later head changed only the source-pin test, outside this set). "Busy" is 8 busy loops; the two trees ran one after the other. These finished after the merge.

Condition main this branch
plain, 10 runs 10/10 10/10
coverage, 10 runs 10/10 10/10
coverage + busy, 10 runs 3/10 10/10
every clock 6x fast, 10 runs 0/10 0/10
clock 6x + busy, 10 runs 0/10 2/10
coverage + busy + clock 6x, 10 runs 0/10 10/10
shuffled order (seeds 37576, 1, 2), coverage 3/3 3/3

Main's reds: test_severs_a_credential_on_2451_pieces_finishes_within_budget under load, test_failed_compaction_then_no_response_ends_the_turn with the clock 6x fast, and with the clock 6x fast plus busy also the oneshot supervisor, apps registry, notarize, tool-input offload and subprocess pool tests. This branch's reds are one test only: test_failed_compaction_then_no_response_ends_the_turn, which still ran on the real clock: the 5 s queue poll waits real seconds while the 30 s turn deadline reads the scaled clock, so the deadline wins. #37946 drives that test's dispatch loop on a manual clock.

Changed test files and the new sibling: -n 4, 901 passed, 14 skipped. isort, flake8, black gate, file line cap, comment-history and brand gates, test_flake_pattern_ratchet.py, test_facade_mirror_guards.py, and semgrep --config semgrep/ (no finding on an added line): clean.

Manual verification

N/A — test-only change; the planted regressions and forced-condition runs above are the check.

Related Issues

Follows #37782 and #37853 (no issue).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 16, 2026 09:10
@bolichen97
bolichen97 enabled auto-merge (squash) November 16, 2026 09:10
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 9afafc3f7ab8dcde394aaf32d4f184bd610e0092 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9afafc3f7ab8dcde394aaf32d4f184bd610e0092 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9afafc3f7ab8dcde394aaf32d4f184bd610e0092 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change is test-only. Each wall-clock bound becomes the wait or signal it stood for, and the planted-regression table shows every test still fails by name when its regression returns.

Notes

  • test_source_pin_budget.py drops the test_subagent_stop_reason_consistency.py row, but the new sibling test_subagent_stop_reason_taskrunner.py gets no row. Check that the moved task-runner section has no source pin, or the budget test will flag the new file.
  • The description says the forced-condition table is still pending. Fill it in before merge so the description doesn't claim proof that isn't there.

[DESIGN-REVIEWED] 9afafc3

Description read

[DESCRIPTION-READ] 83eff39d170766efaeb2d14b831f5cc026de39a51cce7aa4a9d16e67b891f9f0

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9afafc3f7ab8dcde394aaf32d4f184bd610e0092 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9afafc3

Verdict parsed from the review's SHA-scoped output markers for commit 9afafc3f7ab8dcde394aaf32d4f184bd610e0092.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9afafc3f7ab8dcde394aaf32d4f184bd610e0092 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9afafc3f7ab8dcde394aaf32d4f184bd610e0092 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9afafc3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9afafc3f7ab8dcde394aaf32d4f184bd610e0092 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-ten-second branch from 986b715 to da76e9e Compare November 16, 2026 09:36
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 16, 2026
Twenty-three tests bounded a measured duration at 10 to 120 seconds.
Under coverage, a busy CPU or a fast clock the tighter ones go red with
no bug, and several regressions they exist to catch hang the worker
until pytest's timeout instead of failing by name. Each now asserts the
claim the bound stood for: the timeout handed to a wait, the signals
and kills sent, the timer a bounded call arms, the sleeps a fake child
never takes, the drain waits a shutdown makes, a backoff clock that
only the code's own waits advance, or interpreter work that grows
linearly.
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-ten-second branch from da76e9e to 9afafc3 Compare November 16, 2026 10:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 16, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@bolichen97
bolichen97 merged commit 6d8d9f8 into main Nov 16, 2026
97 checks passed
@bolichen97
bolichen97 deleted the crew/LinuxFlackyGuard/timing-budgets-ten-second branch November 16, 2026 12:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant