Skip to content

test: assert the work or the wait, not seconds, in twenty timing bounds - #37675

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/timing-budgets-one-second
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/timing-budgets-one-second

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Twenty tests that hold their claim to a 1 s (one of them 5 s) wall clock stop reading the host's speed, and each still fails when the regression it guards comes back.

After #37600 and #37630 these are the next-tightest duration bounds the K8 ratchet lists: nineteen at < 1.0 s and test_many_keys_collapsing_to_one_spelling_cost_linear_work at < 5.0 s, which reads about 10 s on main when every clock runs 6x fast. A clock cannot tell a slow or coverage-instrumented shard from slow code, so these bounds red PRs that never touched the code, while a short wait or a small regression slips under them.

Why it matters

A red that follows the runner blocks unrelated PRs and teaches people to rerun. Several of the planted regressions below passed the old one-second bounds.

Not a goal

  • Any change under src/.
  • The K8 hits looser than 1 s (other than the 5 s frame-record test named above).
  • New assertions beyond what each test already claimed.

What changed (motivation → approach → change)

Each bound is replaced by what it stood for, chosen by the claim the test makes: a complexity claim counts work or reads the regex's parse, a "does not wait" claim proves the wait was never taken.

Claim the bound stood for Tests Now asserts
linear work frame-record suffixing, read-only classifier _linear_work on interpreter instructions at three sizes
regex does not backtrack context markers, read-only classifier, screencast base64 the pattern's parse (run_cost_unproven, unbounded_repeat_bodies)
regex does not backtrack, multiline ^ preview text assert_rejected_without_backtracking, the original 1.0 s constant
refused within its budget stalled root anchor, breadcrumb flush the budget the call is granted / the one bounded wait it makes
never waits steer, crew-log producer, cron reads, embedding handshake, fake ACP, governance store, history locks (3), mirror audit, subprocess pool the blocking call (lock, sleep, condition, queue, inline sel()) is refused on the test's thread
fits a deadline session drain no sleep timer is armed; the drain budget is 3600 s
stamps the current time session last_used equals the module's frozen clock

Every spy records and refuses on the test's own thread only and passes other threads through, patches the defining module's binding, and is scoped with monkeypatch.context(). The regex parse readers live in the new test/regex_structure_helpers.py; testing-conventions.md names them.

False positives skipped: none in this set. test/test_bench_retrieval_stats.py:762 (two deltas, not a duration) was listed in #37630 and stays.

Backwards compatibility

Compatible: test files, one test helper and docs only; no product behaviour changes.

Removes nothing: every test keeps its name and its claim; only how the claim is checked changes.

Tests

Proof: not provable -- test-only change

Planted regressions in a scratch copy of src/ (run with -o pythonpath=<copy>), one per test. Each new test fails by name; the main column is main's original test on the same plant.

Plant main branch
frame suffixing rescans from #2 red after 36.9 s red: work [2629233, 10295433, 23001633]
marker regex CONTEXT\s*\s*\] red (3.96 s) red: two repeats split one run
blockquote regex back to ^\s*>\s? MULTILINE red (32.7 s) red: 95x at 10x pump
${ scan via \$\{[^=]*= red red at once: a match can start at every character
root anchor granted budget=12.0 red after 16 s red at once: [16.0]
_B64_RE body {4}|{2} lost run (120 s timeout) red: body spans 2..4 chars
steer polls its grace window once (0.5 s) passes red: polled the grace window
crew-log _hand_over sleeps after submit lost run (120 s) red: the producer waited
cron list_jobs takes the store lock red (10 s) red: called open_lock_file
wait_until sleeps 0.5 s before its first check passes red: slept [0.5]
ungated permission waits for an answer red after 32 s red: waited for an answer
warm caller waits on the reload lock red (2.0 s) red: blocked on the reload lock
on-loop flock busy-polls / sleeps red after 32 s red: started the deadline poll / slept
warm hit / on-loop fill take the writer lock red after 7 s red: blocked on the writer lock
flush waits max(2.0, timeout) red (2.0 s) red: waited [2.0]
close_all sleeps 0.5 s passes red: close_all slept: [0.5]
release() stops stamping last_used passes red: must stamp last_used
mirror audit runs inline red after 30 s red at once: sel() on the caller's thread
pool refusal sleeps 1.5 s red (1.5 s) red: the caller slept

Forced-condition repeat runs on an 8-core DevSpace, the twenty converted tests, -n0, main (25860b22c6) against this branch. "Busy" is 8 busy loops per tree; both trees ran at the same time. Branch rows are on 641fbe1999, except the two marked rows, re-run on the head after the fix below.

Condition main this branch
plain, 10 runs 10/10 10/10
coverage, 10 runs 10/10 10/10
coverage + busy, 10 runs 10/10 10/10
every clock 6x fast, 10 runs 0/10 10/10
clock 6x + busy, 10 runs (head) 0/10 10/10
coverage + busy + clock 6x, 10 runs (head) 0/10 10/10
shuffled order (seeds 37576, 1, 2), coverage 3/3 3/3

Main's reds: test_a_stalled_root_anchor_refuses_within_the_budget in every forced run, plus test_long_command_is_classified_in_linear_time, test_blank_line_runs_do_not_backtrack and test_many_keys_collapsing_to_one_spelling_cost_linear_work under load. On 641fbe1999, clock 6x + busy was 7/10: the stalled-anchor test's warm-up was a real resolution under the fixture's 0.2 s budget, so a slow host turned the setup itself into a stall. The head gives that warm-up the lost-run bound; the stall under test keeps the 0.2 s budget.

The eighteen whole files at -n 8 under coverage + busy + clock 6x are red on both trees (0/3 each), on tests this PR does not change (for example test_shutdown_is_not_reported_drained_while_a_batch_is_still_writing, test_a_stalled_resolution_is_refused_within_the_budget, test_communicate_timeout_reports_the_child_instead_of_raising_bare): those are the looser K8 hits left for later. Of the twenty converted tests, main reds test_many_keys_collapsing_to_one_spelling_cost_linear_work, test_long_command_is_classified_in_linear_time and test_a_stalled_root_anchor_refuses_within_the_budget there (3/3 each); 641fbe1999 reds only the stalled-anchor test, once, the warm-up fixed on the head (whole-file runs were not repeated on the head).

local-gate backend related set and the 18 changed files (984 passed), isort, flake8, black gate, test_flake_pattern_ratchet.py (68 passed) and the repository semgrep rules: clean.

Manual verification

N/A — test-only change; the planted regressions and forced-condition runs above are the check.

Related Issues

Part of #37599

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 19:30
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f13b01162e64fcc88e030a6a4e59e2abf59e1eca touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision f13b01162e64fcc88e030a6a4e59e2abf59e1eca ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f13b01162e64fcc88e030a6a4e59e2abf59e1eca — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this changes tests only, each one checks the same claim it did before, and any part of it can be reverted cleanly.

Notes

  • regex_structure_helpers.run_cost_unproven relies on the private re._parser/re._compiler, and its idea of a safe pattern is narrower than the pump test it replaces. It reports any shape outside that idea as unproven, so it fails closed. The cost is that a Python upgrade or a new regex shape can turn these tests red with a false alarm. When that happens, fall back to assert_rejected_without_backtracking; don't widen the helper.

[DESIGN-REVIEWED] f13b011

Description read

[DESCRIPTION-READ] ebd95d9765e9c418b5ac63e058316aeb64cac255cc52bc7559e96ed6149c5e0b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f13b01162e64fcc88e030a6a4e59e2abf59e1eca and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f13b011

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f13b01162e64fcc88e030a6a4e59e2abf59e1eca false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f13b01162e64fcc88e030a6a4e59e2abf59e1eca — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • test/test_history_locking_remediation.py:638 patches _LoopTime onto history_mod._time, but the poll reads history_locks._time, so its monotonic/sleep refusal never runs. attempts == [1] still catches the regression, only after the 10 s budget instead of immediately; patch locks_mod instead.

[OPUS-REVIEWED] f13b011

Verdict parsed from the review's SHA-scoped output markers for commit f13b01162e64fcc88e030a6a4e59e2abf59e1eca.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f13b01162e64fcc88e030a6a4e59e2abf59e1eca false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-one-second branch from 6e19340 to e1cd746 Compare November 15, 2026 20:14
Nineteen tests held their claim to a one-second wall clock and one to five
seconds of it. Each now asserts what the bound stood for: instructions
that grow linearly (frame suffixing, the read-only classifier), a regex's
parse (no two run-consuming repeats, one fixed-width repeat body), a
budget the call is granted, or that the wait was never taken (a lock that
refuses a blocking acquire, a sleep or condition wait that raises, a timer
that must not fire, a frozen clock). The regex readers move into
test/regex_structure_helpers.py, and testing-conventions.md names them.
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/timing-budgets-one-second branch from e1cd746 to f13b011 Compare November 15, 2026 21:03
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 741 lines, more than the 400 it may approve on its own.

Test-only change that swaps 20 wall-clock bounds for checks on the wait or the work. I verified that each spy patches the binding the source actually reads: cron store uses platform_compat module attributes, the subprocess pool uses executor.time and _free.get, wait_until sleeps through _sleep, and the history facade passes the _time write on to history_locks (so the Opus lane's note is wrong). The stalled-anchor grant matches paths_resolver's formula under the fixture's 0.2 s budgets. close_all with an immediately-acking provider arms no sleep timer. The regex parse helper fails closed on the planted shapes. CI is green across Linux and Windows backend shards and the flake ratchet. No src change, no regression path found.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit f13b01162e · 741 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 15, 2026 22:42

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on head f13b011. The diff changes only tests: 18 test files, one new test helper and a short note in testing-conventions.md. Nothing under src/ or website/ changes. The review checked each new spy against the binding the code under test actually reads. It also ran all 18 changed test files plus the flake-pattern ratchet and facade-mirror guards on this head (1073 passed), probed the new regex structure helper against regressed patterns, and spot-checked the history-lock planted regression. No assertion was loosened, and no new wall-clock bound or sleep was added.

Done well: each test now asserts the signal its old bound stood for, such as the exact deadline granted, an armed sleep timer, or the regex parse. The body's planted-regression table shows several regressions that slipped past the old one-second bounds and now fail by name.

Optional:

  • test/test_session_drain.py:372 records only asyncio.sleep timers. A planted asyncio.wait(..., timeout=2.0) in close_all passes the new test but failed the old one. Recording every timer that fires during close_all, except the 3600 s drain wait, would close that gap. The docstring still says "small drain_timeout".
  • test/regex_structure_helpers.py:87 reports x?\s*END and (?<=\s)\s*END as linear, but both are quadratic. No current caller is affected. Skipping leading empty-width items, or reporting them as unproven, would keep the helper safe for future guards.
  • test/test_history_locking_remediation.py:166 patches _LoopTime on the history facade. Consider patching locks_mod._time directly, so the refusal reaches history_locks' own binding in every case.

@KiroCrewCaptain
KiroCrewCaptain merged commit e3e8b1f into main Nov 15, 2026
99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/LinuxFlackyGuard/timing-budgets-one-second branch November 15, 2026 22:42
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants