Skip to content

test: count and remove bounds on a real span against any limit - #37946

Merged
bolichen97 merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/timing-budgets-nonliteral
Nov 16, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/timing-budgets-nonliteral

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A test can no longer bound a real wall-clock span from above in any spelling without the flake ratchet counting it, and the bounds of that kind still on main stop reading the host's speed.

The K8 ratchet counted only assert <span> < <number literal>. A real-clock span against a name or an expression (elapsed < budget, time.monotonic() - started < _CEILING), the reversed and age forms (budget > elapsed, stamp > time.monotonic() - 5), unittest assertLess / assertGreater, and a span held in a name such as waited or cpu all landed unseen. A scan of main found 30 such sites besides the literal ones #37906 converts; the extended detector found 15 more.

Why it matters

These are the same flakes the earlier batches removed (#37600 through #37906): they red PRs on a slow or instrumented shard, and the regressions they guard often slip under them or hang. Without a detector that sees them, the next one lands the same way.

Not a goal

What changed (motivation → approach → change)

Detector. _k8_duration_bound moves into test/flake_pattern_k8.py (the ratchet file stays under the line cap) and counts an upper bound on a span read off a real clock: the stdlib clocks and their aliases (followed through assignments, cycle-guarded, scoped to the function), a running loop's time(), named spans whose value is opaque (elapsed = run_timed()), non-literal limits, the reversed forms, and assertLess / assertLessEqual / assertGreater / assertGreaterEqual. It exempts clocks the test drives (a ManualClock, a fake or virtual clock), remaining-time checks (deadline - time.monotonic() <= cap, which only shrink on a slow host), cutoffs taken before the call (t0 = time.monotonic() - 5.0; stamp > t0), lower bounds and flake-ok lines. Each shape has a flagged or clean fixture. The ratchet stays per file against the merge base, counting the base with the same detector, so concurrent merges cannot race it. testing-conventions.md (rule D7) says what K8 counts.

Sites. Of the 45, 27 were real host-speed bounds and are converted:

Claim the bound stood for Tests Now asserts
a healthy child is not waited out app backend survival check, failure path the survival sleeps taken / the probe charged once, on virtual time
a non-stream response is still cut app proxy SSE the asyncio.timeout armed and when it expires
a stop wakes the parked worker crew-log eager fold the parked get returns on the stop sentinel, not the poll
the barrier never waits on the loop governance profiles the reload-lock acquires taken on the loop thread
a killed tree is reaped installer helpers (2) a readiness handshake proves the grandchild holds the pipes, then the post-kill drain reaches EOF
one aggregate claim budget MCP gateway claim the wait_for budgets of the test's own task: [0.3]
JSON coercion is memoized notebook coercion visits counted, not seconds
the guard honours its stall push guard index the poll timeouts it is handed, with margin for a coarse clock
a lock acquire is single-shot / bounded POSIX lock ceiling (5) one non-blocking flock, sleeps on a pinned clock that sum to the ceiling, kernel-blocking flock refused on the test thread
the pump arms the coordinator deadline run loop the call_later delay armed
a tight budget stays tight path resolution the allowance granted (<= ceiling + 1e-6)
release refreshes liveness session last_used equals the module's frozen clock
one ping wait per miss stub bridge liveness the waits the liveness task makes, per cycle
an on-loop write never blocks subagent state no blocking acquire on the loop; a lost-run release, and the holder still held when the call returns
the stop does not wait twice wheel apply the stop's waits: one grace
the caller waits its own budget decisions points / integration the wait on the oracle future is handed the point's budget, on a ManualClock
the rebuild child is bounded embedding rebuild (3) the poll / communicate timeouts handed, after the child reports it printed
only the first caller pays the build skill catalog first call waits == [0.4] on a ManualClock, second call []

The other 18 are not host-speed bounds. 17 are left unchanged: test-driven clocks, remaining-time checks, string indices, constants, before/after brackets and stamp-ordering checks; the extended detector does not count them. The image-ref scrub's thread-CPU budget, kept at its original constant (D7), carries flake-ok with a reason, as do two literal-limit false positives the old detector counted: the benchmark's fixed metric deltas and the turn harness's virtual clock (the third, test_mcp_quarantine.py:757's string offsets, the new detector no longer counts). test_governance_profiles.py would have crossed the 2,000-line cap, so five unchanged tests move to a sibling file.

Over the current tree the detector counts 22 unexcused sites, all of them the literal bounds #37906 converts.

Backwards compatibility

Compatible: test files, test helpers and docs only; no product behaviour changes.

Removes nothing: every test keeps its claim; five tests move unchanged to a sibling file.

Tests

Proof: not provable -- test-only change

Planted regressions (in-memory monkeypatches or in-memory recompiles of the product function; src/ untouched), the same plant on main's original test and this branch's test:

Site Plant main branch
app backend survival no ownership early exit red after ~1.6 s red by name
app backend failure path count-bounded loop red red
SSE cut 5x timeout armed red after 2 s red by name
crew-log stop no stop sentinel red red by name
governance barrier waits on the loop red red by name
installer (2) no-op kill / direct-child kill red red by name
gateway claim one wait_for per phase red red by name
notebook no memo red after 18.6 s red in 4.6 s
push guard stall honoured 20x late red after the ceiling red by name
POSIX lock absorbed / dropped LOCK_NB / loop polls / +40 s overshoot red after 1-62 s red at once
stub bridge double wait per miss (2 spellings) red after 3 s red by name
subagent state blocking acquire on the loop / offload then join red red by name
wheel apply waits the grace again red red by name
decisions (2) wait budget 30 s red after 34-36 s red in ~4.5 s
embedding rebuild poll 4 s instead of the budget passes red by name
embedding rebuild poll 6 s / handshake given the ceiling red after 7-10 s red by name
skill catalog second caller waits again red red by name

Slow-host repros (real pauses injected in memory, 1-3 s, before a spawn, a ping write, a poll or the first wait) pass on the branch. On main several of them go red with no bug: the decisions tests after 125 s, the embedding child killed before it prints, the stub bridge at 4.5 s, the subagent write at 1.2 s, the 1.5 s flock pause.

Forced-condition repeat runs on an 8-core DevSpace, -n0, main (cada1cf842) against this branch. "Busy" is 8 busy loops; the trees ran one after the other. The 28 converted tests (28 ids) ran on 4ebd855bbe (the later heads change only the detector and test_harness.py); the two follow-up tests, test_failed_compaction_then_no_response_ends_the_turn and test_terminate_pgid_graceful_exit_returns_fast, ran again on the head 5636d6becd.

Condition main, 28 tests branch, 28 tests main, 2 follow-ups head, 2 follow-ups
plain, 10 runs 10/10 10/10 10/10 10/10
coverage, 10 runs 10/10 10/10 10/10 10/10
coverage + busy, 10 runs 10/10 10/10 10/10 10/10
every clock 6x fast, 10 runs 0/10 10/10 0/10 10/10
clock 6x + busy, 10 runs 0/10 10/10 1/10 10/10
coverage + busy + clock 6x, 10 runs 0/10 10/10 8/10 10/10
shuffled order (seeds 37576, 1, 2), coverage 3/3 3/3 3/3 3/3

Main's reds with the clock 6x fast, 10/10 each: the stub bridge liveness, path-resolution grace, gateway claim, installer tree reap, embedding communicate and SSE cut tests, plus the push guard with the CPU busy. On the follow-ups main reds test_failed_compaction_then_no_response_ends_the_turn (its 5 s queue poll waits real seconds while the turn deadline reads the scaled clock).

Review fixes since the PR opened: the detector follows stamp aliases with a cycle guard (no recursion on run = run.__wrapped__), counts elapsed = stop - start and reversed named spans, scopes clock aliases to their function, does not count a floor or ratio between two spans (elapsed >= min_duration), and counts a real loop's time() again; the installer recorder watches only the child under test (a Windows taskkill passes through); the graceful terminate_pgid test accepts macOS's EPERM for an unreaped zombie group only on macOS, with the False that path returns.

Changed test files: -n 4, 1173 passed; the 4 failures are TestBackendLifecycle in test_app_backend.py, which fail the same way on main on this host (no user namespaces). isort, flake8, black gate, file line cap, comment-history and brand gates, docs lint, test_flake_pattern_ratchet.py, test_facade_mirror_guards.py and semgrep --config semgrep/ (no finding on an added line): clean.

Manual verification

N/A — test-only change; the planted regressions, slow-host repros and forced-condition runs above are the check.

Related Issues

Follows #37906 (no issue).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 16, 2026 11:16
@bolichen97
bolichen97 requested a review from buluoray November 16, 2026 11:16
@bolichen97
bolichen97 enabled auto-merge (squash) November 16, 2026 11:16
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 5636d6becd8ef6636beeefd73d3560485116e885 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 5636d6becd8ef6636beeefd73d3560485116e885 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 5636d6becd8ef6636beeefd73d3560485116e885 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this changes only tests and docs, and each converted bound keeps its claim in a form that doesn't depend on host speed.

Notes

  • The 500-line K8 heuristic has to keep a list of exemptions (driven clocks, cutoffs, ratios, delta-named attributes) that will only grow. If false positives keep showing up, prefer # flake-ok at the site over adding more detector special cases.

[DESIGN-REVIEWED] 5636d6b

Description read

[DESCRIPTION-READ] e3b01805eae2f332940a1d137d153764b9f6a9240671682a46c03f5c152fbb59

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 5636d6becd8ef6636beeefd73d3560485116e885 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5636d6b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5636d6becd8ef6636beeefd73d3560485116e885 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5636d6becd8ef6636beeefd73d3560485116e885 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5636d6b

Verdict parsed from the review's SHA-scoped output markers for commit 5636d6becd8ef6636beeefd73d3560485116e885.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5636d6becd8ef6636beeefd73d3560485116e885 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-nonliteral branch from 093d5a9 to 26b2cac Compare November 16, 2026 12:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 16, 2026
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-nonliteral branch from 26b2cac to 4ebd855 Compare November 16, 2026 13:41
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-nonliteral branch 3 times, most recently from 4ea83f0 to 3757817 Compare November 16, 2026 15:19
K8 counted only `assert <span> < <number literal>`, so a measured
real-clock span against a name or an expression (`elapsed < budget`),
the reversed and age forms (`budget > elapsed`, `stamp >
time.monotonic() - 5`), unittest assertLess and assertGreater, and
spans held in names like `waited` or `cpu` all landed unseen. The
detector, now in test/flake_pattern_k8.py, counts them on real clocks
(the stdlib clocks, their aliases, and a running loop's time) and
exempts clocks the test drives, remaining-time checks, cutoffs taken
before the call and lower bounds. The ratchet stays per file against
the merge base, so concurrent merges cannot race it.

The real host-speed bounds it finds are converted to assert the wait,
the timeout handed to it, or a clock only the code's own waits advance.
A kept complexity budget and two bounds that read no host clock carry
flake-ok with a reason. test_governance_profiles.py splits a section
into a sibling to stay under the line cap.

Two follow-ups for tests #37715 and #37906 converted: the graceful
terminate_pgid test accepts the EPERM a macOS zombie group answers, and
the failed-compaction turn test drives the dispatch loop on a manual
clock, so a fast or busy host can no longer let the turn deadline win.
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-nonliteral branch from 3757817 to 5636d6b Compare November 16, 2026 15:19
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@bolichen97
bolichen97 merged commit 44eea1f into main Nov 16, 2026
111 checks passed
@bolichen97
bolichen97 deleted the crew/LinuxFlackyGuard/timing-budgets-nonliteral branch November 16, 2026 17:00
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant