Repository navigation
test: assert the wait or the work, not seconds, in 39 five-second bounds - #37782
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This change only touches tests and docs. Each wall-clock bound becomes an assertion on the wait it was given or the work it does, using helpers the earlier PRs in this series already set up. Notes
[DESIGN-REVIEWED] 11d8330 Description read[DESCRIPTION-READ] e395b3f5f0459f0455375ab8d0a366d335a915c25c9e236798154904fe36b7dd That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — 🔴 changes requested (blocking)GPT 6.1 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- test/test_security_regex_linearity.py:204 -- Replacement complexity guards retain absolute CPU budgets
BLOCKING -- website/src/pages/settings/RemoteCrewPanel.tsx:1301 -- Shared Crews navigation is absent from the feature map Trigger: In the desktop dashboard with relay viewing enabled, Settings → Remote Crew’s Crews tab mounts [BLOCK-MERGE] 11d8330 Adjudication (Opus 5.5) — is blocking on each finding proportionate?F2. The rule it cites, F1 (fenced). The ceiling is still absolute: False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Thirty-nine tests bounded a measured duration at 5 seconds. Under coverage, a busy CPU or a fast clock they go red with no bug, and several regressions they exist to catch slipped under them. Each now asserts the claim the bound stood for: the timeout handed to the wait, the wait never taken on the thread under test, the signals or kill sent, a manual clock that only the code's own reads advance, or interpreter work that grows linearly. Regex scans with no work counter use the shared ramped thread-CPU helper at its default budget.
9d2b6c7 to
11d8330
Compare
Code change approver recommendation: A person should decideWhy: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 1661 lines, more than the 400 it may approve on its own. This PR touches only tests and docs. It replaces 39 five-second wall-clock bounds with checks on the wait the code is handed, on refused blocking calls, on interpreter work at three input sizes, or on the shared regex-backtracking helper. I checked the asserts against the product code: the knowledge maintenance-window loop, the ManualClock install scope and the PDF deadline path all match. The helper and test-class moves are complete, all callers are updated, and the helper's autouse fixture does not leak. Every spy is limited to the thread or task under test and is undone properly, and every remaining wait has a lost-run bound. All CI checks are green and the automated review lanes had no findings. I found no regressions or gaps in the goal. The only open item is the missing results table for the repeated busy-host runs in the description, which is non-blocking. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I reviewed head 11d8330. I read each replaced five second bound against the product code it guards: the manual clock installs, the recorded waits and timeouts, the per-thread sleep refusals, and the linear-work and backtracking checks. Every changed test file passes at this head. The converted tests also stayed green across five repeated runs on a heavily loaded host. I found no wrong assertion, vacuous check or leaked patch, and the change touches only tests, one test helper and the testing conventions doc. Nothing blocks.
Problem / Motivation
Goal: Thirty-nine tests that hold their claim to a 5 s wall clock stop reading the host's speed, and each still fails when the regression it guards comes back.
After #37600, #37630, #37675 and #37715 these are the next K8 hits: every
elapsed < 5/time.monotonic() - started < 5.0bound left on main, in 26 files (test_acp_frame_record.py:602, the fortieth, was converted in #37675). A clock cannot tell a slow or coverage-instrumented shard from slow code, so these bounds red PRs that never touched the code, while several of the regressions they guard slip under 5 s or hang the worker instead of failing.Why it matters
A red that follows the runner blocks unrelated PRs and teaches people to rerun. On the planted regressions below, main's original tests passed on 3, hit the pytest timeout on 9, and were red only after 8 to 62 s on most of the rest.
Not a goal
src/.What changed (motivation → approach → change)
Each bound is replaced by what it stood for, chosen by the claim the test makes:
ManualClockinstalled through theacp_clientfacade; each silent read advances it by exactly the timeout it was handed, so0.2 < elapsed <= 0.22is exact[0.5],== _PATTERN_TIMEOUT_SECS,<= budget + grace + 1e-6), or a per-thread clock that moves only by the code's own sleepssys.setprofilehook,os.openwithoutO_NONBLOCK, an inline pipe read, a wedged flush returning first) is refused or recorded on the thread under test_linear_work);_PyEqNamenames keep comparisons in Python;_RecordedFramerefuses anystrmethod outsidefind/rfind/slicing and bounds the characters scannedtest/conftest.py::assert_rejected_without_backtrackingat its default budgetkiro_crew.llm_helpersfacade) are never handed a string over the cap_assert_scan_is_linearmoves fromtest_denied_commands_security.pyintotest/denied_commands_security_helpers.py, so the push gate test imports it from a helper module.test/test_file_grep.pywould have crossed the 2,000-line cap, soTestWorkbookExpansionmoves unchanged totest/test_file_grep_workbook.py.testing-conventions.mdgains the two traps found here: refuse a sleep with asys.setprofilehook (the semgrep rule refuses rebindingtime.sleep, K3 refusesasyncio.sleep), and add anos.openspy toos.supports_dir_fd.False positives skipped: none in this set.
Backwards compatibility
Compatible: test files, one test helper and docs only; no product behaviour changes.
Removes nothing: every test keeps its name and its claim; one helper and one test class move to sibling files unchanged.
Tests
Proof: not provable -- test-only change
Planted regressions as in-memory monkeypatches of the product function (a scratch pytest plugin;
src/untouched), the same plant against main's original test and this branch's test. "Lost run" means the pytest timeout killed the worker.timeout=Noneto the turn+?wait_foronreaduntilwait_foron that phasewait_foron the flush\s.*\s.*zzfuture.result()timeout=60communicateO_NONBLOCKThe local GPT 6.1 and Opus 5.5 reviews found slow-host cases where a correct run could still go red: the grep sentinel sampled before the poll, the knowledge and PDF deadlines on the real clock, and the MCP round trip requiring all four phases when connect can legitimately time out. Each is fixed on the head (manual clocks, lost-run waits for prerequisite phases, the sentinel read after the poll), and each slow-host pause the reviewer used now passes while the plants stay red.
Forced-condition repeat runs on an 8-core DevSpace, the 39 converted tests (40 ids),
-n0, main (cb72f2c214) against this branch. "Busy" is 8 busy loops; the two trees ran one after the other.Branch rows are on the head,
11d8330068.Main's reds:
test_pattern_redos_is_bounded_fail_closedandtest_a_flush_that_never_gets_a_worker_cannot_hold_the_exitin every forced run, plustest_redos_safe_on_pathological_inputandtest_the_guard_bounds_the_scan_far_under_the_watchdogwith all three. This branch's reds are three ramped CPU checks only:test_a_long_message_of_path_fragments_scans_in_linear_time(image-ref strip 4/10, prompt blocks 3/10 and 2/10) andtest_redos_safe_on_pathological_input(10/10 with all three), each "a 20000-unit pump cost 2.05-5.44s of CPU" against the 2 s budget. These finished after the merge; #37853 gives the three long pumps headroom.local-gaterelated set: the changed test files and the new sibling (2611 passed, 18 skipped), isort, flake8, black gate, file line cap, comment-history and brand gates, docs lint,test_flake_pattern_ratchet.py(71 passed),test_facade_mirror_guards.py: clean.Manual verification
N/A — test-only change; the planted regressions and forced-condition runs above are the check.
Related Issues
Follows #37600, #37630, #37675 and #37715 (no issue).
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.