Skip to content

test: assert the wait or the work, not seconds, in 28 timing bounds - #37715

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/timing-budgets-three-second
Nov 16, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/timing-budgets-three-second

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Twenty-eight tests that hold their claim to a 1.5-3 s wall clock stop reading the host's speed, and each still fails when the regression it guards comes back.

After #37600, #37630 and #37675 these are the next-tightest duration bounds the K8 ratchet lists: three at < 1.5 s, eighteen at < 2.0/< 2 s, three at < 2.5 s and four at < 3.0 s, in eighteen files. A clock cannot tell a slow or coverage-instrumented shard from slow code, so these bounds red PRs that never touched the code, while a short wait or a small regression slips under them.

Why it matters

A red that follows the runner blocks unrelated PRs and teaches people to rerun. Some of the planted regressions below passed the old bounds (for example a spawn verdict held back two reaper ticks).

Not a goal

  • Any change under src/.
  • The K8 hits at 5 s and looser.
  • New assertions beyond what each test already claimed.

What changed (motivation → approach → change)

Each bound is replaced by what it stood for, chosen by the claim the test makes:

Claim the bound stood for Tests Now asserts
linear work glued option markers, coder session browse (3) _linear_work on interpreter instructions at three sizes; for the browse diffs, one git diff --no-index call builds the hunks and the work around it grows at most 2.5x per doubling
regex / C-level scan does not backtrack brand gate (2), leaked tool-call quoting, ADF whitespace fold, say locale group test/conftest.py::assert_rejected_without_backtracking at its default 2.0 s budget, the constant each bound carried (no work counter sees regex or C cost)
never waits provider and runtime reader on a wedged writer, SEL loop audit (2), FIFO record, knowledge hand-off, tool-interrupted entry points, Windows reader arm the blocking call (write on the loop, blocking lock acquire, condition wait, os.open without O_NONBLOCK, a pull past the marker, an unbounded reader wait) is refused or recorded on the thread under test
waits at most its budget stalled path resolution, GIL-hog resolution in the child, history flock poll, cache-fill acquire, log-queue stop and hard-exit drain the timeout the call hands the wait ((True, 0.1), [0.2], the granted allowance), or a clock that moves only by the poll's own sleeps
teardown does not burn the grace window terminate_pgid graceful exit, wait hook, default grace the signals sent (SIGTERM, probes, no SIGKILL); the default grace read off a virtual clock on the test thread
the failure reaches the caller at once unspawnable host FileNotFoundError, not TimeoutError, and the marker is queued on the first note after the failed spawn
refused before scanning image openers past the bound iter_local_refs is never called

A spy that refuses or replaces a blocking primitive is installed for the thread under test only; a spy that only records arguments passes the call through. Globals are patched through monkeypatch or monkeypatch.context(). Three tests whose names promised speed are renamed to their claim (..._get_their_hunks_from_git ×2, ..._left_alone_unscanned).

test/test_harness.py would have crossed the 2,000-line cap, so its two self-contained Windows-teardown routing tests move unchanged to test/test_harness_windows_teardown.py. testing-conventions.md gains the three value-assertion shapes this batch adds.

False positives skipped: none in this set.

Backwards compatibility

Compatible: test files and docs only; no product behaviour changes.

Removes nothing: every test keeps its claim; three are renamed to it and two move to a sibling file unchanged.

Tests

Proof: not provable -- test-only change

Planted regressions (in-memory monkeypatches of the product function, or a scratch src/ edit reverted afterwards), one or more per site. Every new assertion fails by name:

Site Plant Branch
stalled resolution the allowance granted ×10 red: granted past budget+grace
GIL-hog resolution child path returns None / resolves in-process red / red
Windows reader arm extra result(timeout=1.0) / timeout + 1.0 red / red
unspawnable host marker held to the third tick red (the old 2.5 s bound passed)
terminate_pgid graceful / hook / default early return removed / leader_exited = False / grace hard-coded 5.0 red / red / red
tool-interrupted entry points marker never recognised red on all three
provider and runtime reader inline write_frame / blocking lock.acquire() red / red
SEL critical audit / loop audit _on_event_loop → False / reversed lock order red / red
history flock poll / cache fill deadline ×10, no deadline / timeout −1, 10.0 red / red
knowledge hand-off no admission flag red
FIFO record no O_NONBLOCK / S_ISREG → True red / red
brand gate lookbehind removed / per-run rescan red / red (79x per 10x)
coder browse (3) difflib instead of git / quadratic rescan red / red
leaked tool-call quoting backreference regex red
glued option markers per-candidate _in_open_fence red
image openers opener bound 10**9 red
log-queue stop / hard-exit drain unbounded join, close while wedged / stop with None red / red
ADF fold \s*\n\s* regex red (100x per 10x)
say locale [-_]\w+ red

Forced-condition repeat runs on an 8-core DevSpace, the 28 converted tests (55 ids with parameters), -n0, main (686d00fc11) against this branch. "Busy" is 8 busy loops; the two trees ran one after the other.

Branch rows are on the head, 8625ef8ec0.

Condition main this branch
plain, 10 runs 10/10 10/10
coverage, 10 runs 10/10 10/10
coverage + busy, 10 runs 10/10 10/10
every clock 6x fast, 10 runs 0/10 10/10
clock 6x + busy, 10 runs 0/10 10/10
coverage + busy + clock 6x, 10 runs 0/10 10/10
shuffled order (seeds 37576, 1, 2), coverage 3/3 3/3

Main's reds: test_a_stalled_resolution_is_refused_within_the_budget in every forced run, test_a_bounded_read_that_misses_its_deadline_retires_the_slot_without_waiting (10/10 with the clock 6x fast, fewer under load), and test_reordered_repeated_lines_get_their_hunks_fast under load (6/10 with clock 6x + busy, 10/10 with all three).

After the first push, CI found two defects in this PR's own tests, both fixed on the head: the Windows reader-arm wait read 0.3000000000001819 against <= 0.3 (an absolute deadline minus a coarse clock read; the bound now allows float error), and the history flock poll patched kiro_crew.history_locks._time on one owner instead of through the kiro_crew.history facade (test_facade_mirror_guards.py).

local-gate related set: the 18 changed test files and the new sibling (1398 passed, 3 skipped), isort, flake8, black gate, file line cap, comment-history and brand gates, docs lint, test_flake_pattern_ratchet.py (68 passed): clean.

Manual verification

N/A — test-only change; the planted regressions and forced-condition runs above are the check.

Related Issues

Follows #37600, #37630 and #37675 (no issue).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 22:56
@bolichen97
bolichen97 requested a review from buluoray November 15, 2026 22:56
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8625ef8ec06bd9737d4a9c21dc12f2a681aca9e5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 8625ef8ec06bd9737d4a9c21dc12f2a681aca9e5 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8625ef8ec06bd9737d4a9c21dc12f2a681aca9e5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only change with no product code touched; each wall-clock bound becomes an assertion on the wait or the work it stood for, and the description backs each one with a planted regression.

Notes

  • The description still says "The runs are still going; their table lands here when both trees finish." If those forced-condition repeat runs are part of the proof that the new assertions don't flake, add the results before merging.

[DESIGN-REVIEWED] 8625ef8

Description read

[DESCRIPTION-READ] 7c6f89263c7e8eaa4e38ccdc11ada12a6d297f543ea630cdda98608da23cb0d1

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8625ef8ec06bd9737d4a9c21dc12f2a681aca9e5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8625ef8

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8625ef8ec06bd9737d4a9c21dc12f2a681aca9e5 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8625ef8ec06bd9737d4a9c21dc12f2a681aca9e5 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8625ef8

Verdict parsed from the review's SHA-scoped output markers for commit 8625ef8ec06bd9737d4a9c21dc12f2a681aca9e5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8625ef8ec06bd9737d4a9c21dc12f2a681aca9e5 false-positive: <reason, 20 to 500 characters>

Twenty-eight tests bounded a measured duration between 1.5 and 3
seconds. Under coverage, a busy CPU or a fast clock they go red with no
bug. Each now asserts the claim the bound stood for: the wait the call
never takes, the timeout it hands a wait, the signals a teardown sends,
or interpreter work that grows linearly. Regex guards with no work
counter keep the shared ramped thread-CPU helper at the original 2 s.
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-three-second branch from a575d2d to 8625ef8 Compare November 15, 2026 23:42
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 1113 lines, more than the 400 it may approve on its own.

Test-only PR that replaces 28 wall-clock bounds (1.5-3 s) with assertions on the wait or work they stood for. I checked each spy/patch seam against product code: history/_time and security.paths patches propagate via _facade_mirror; frame-record guard runs on the same loop thread as the runtime reader task; cli_logging joins listener._thread with the caller timeout; SEL critical path skips flush on loop; subprocess pool _note_spawn_errors/_failed_queued match; harness terminate_pgid only uses time.monotonic/sleep; breadcrumb uses os.open/os.read directly; chat_attachments and session_browse import the patched names at module scope. Moved Windows tests have no fixture dependencies. CI fully green including new-test stability lanes. Non-blocking: proof table for repeat/forced-condition runs is still pending in the description; ReDoS pumps are smaller than the original inputs but backed by ratio checks.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 8625ef8ec0 · 1113 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 16, 2026 01:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (20 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change that replaces wall-clock timing bounds with assertions on the wait or work requested; no product code changes. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/testing-conventions.md.

@KiroCrewCaptain
KiroCrewCaptain merged commit 742cebb into main Nov 16, 2026
99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/LinuxFlackyGuard/timing-budgets-three-second branch November 16, 2026 01:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This merged while my review of 8625ef8 was still running, so here is its one finding as a follow-up item. It is a latent flake on main now, not a red yet.

test/test_subprocess_pool.py:942-973 (test_a_host_that_cannot_spawn_raises_the_oserror_inside_the_budget) patches _Child._spawn and SubprocessPoolExecutor._note_spawn_errors on the class, so every live pool in the xdist worker appends to the same events list. The process-wide path_resolve_executor pool can outlive earlier tests, and its reaper calls _note_spawn_errors every 0.5 s with live children, appending "no marker". If that tick lands before this executor's first failed spawn, first_note > 0 fails; if it lands between this reaper's failed spawns and its note, events[first_note] == "marker queued" fails.

Smallest fix: record only this executor's events, as the sibling test at line 1041 already does: append in _recorded_note only when self is executor, and in _unspawnable only when self in executor._children (bind executor before either fake can run, or tag each event with id(self) and filter after construction).

bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
bolichen97 added a commit that referenced this pull request Nov 16, 2026
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants