Skip to content

test: pin the clock in three tests that raced the real one on Windows - #15708

Merged
bolichen97 merged 1 commit into
mainfrom
test/pin-real-clock-in-windows-flakes
Oct 1, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
test/pin-real-clock-in-windows-flakes

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop three tests from failing on Windows CI when the real clock happens to tick between two reads they assume are simultaneous.

Backend Tests (Windows) fails first-attempt runs that then pass on rerun with the same SHA. Three of the failures sampled over the last five days come from tests that read the real clock and assume no time passed:

Test Shard Failing run (attempt 1) Signature
test_default_context_admission.py::test_shared_admission_exact_fit_does_not_reserve_unused_footer 2/8 36511810630 two builds differ only in 09-29 02:22 UTC vs 02:23 UTC
test_snapshot_absent_vs_wrong_type.py::…::test_a_second_allocation_in_the_same_second_gets_its_own_directory 6/8 36353914167 the suffixed name should stay recognisable: pre-restore-20260927T221341Z
test_diag_procs.py::test_a_read_refused_as_too_soon_does_not_restart_the_window 4/8 36296498527 assert 0.0 is None — 0.8s apart is under the floor

Why it matters

We are about to turn on the merge queue (#15200). In the queue every flaky first attempt ejects an innocent PR and rebuilds every group behind it, so each of these costs a queue cycle rather than a rerun click.

Not a goal

What changed (motivation → approach → change)

Each test reads the real clock at least twice and asserts as if it read it once.

  • Context admission — context.py renders [CURRENT DATE] %A, %Y-%m-%d %H:%M %Z from datetime.now() on every build. The test sizes an exact-fit cap from one build and compares a second build to it byte for byte, so a minute boundary between them fails it (and across midnight the weekday name changes length, so the cap no longer fits). → Freeze ctx.datetime in the shared rig fixture, the same _FrozenClock fix(test): freeze the clock in the subagent context identity test #15051 used for test_subagent_context_groups. An AST scan of the suite for other two-build equality checks found this was the last unpinned one.
  • Snapshot rollback directory — _allocate_rollback_dir names directories by one-second UTC stamp and suffixes -2 on collision. The test assumes two calls land in one second; on a slow runner they straddle it, the second gets a fresh unsuffixed name, and the collision path under test never runs. → Freeze snap.datetime (the pattern test_snapshot_replace_tree_fidelity.py already uses) and assert the exact names, which is stronger than the old startswith.
  • diag rate floor — the second read's gap is the hand-aged 0.8s plus the real time the runner spent between the two scans' time.monotonic() stamps. On a loaded Windows runner that pushed the gap past the 1s floor and the refusal under test was admitted. → Pin all three reads with monotonic=SCAN_MONOTONIC, as every other rate test in the file already does. The windows are now exactly 0.8s and 1.4s, so the third read's assertion goes back from a band (20 < x <= 50) to the exact figure (50.0, 10.0).

Backwards compatibility

Compatible: test-only change; no product code is touched.

Tests

  • test_shared_admission_exact_fit_does_not_reserve_unused_footer (and every other test using rig) now builds under one fixed instant.
  • test_a_second_allocation_in_the_same_second_gets_its_own_directory now proves the collision path itself: pre-restore-20260101T120000Z then …-2.
  • test_a_read_refused_as_too_soon_does_not_restart_the_window asserts the exact recovered rates instead of a band.

Manual verification

Looped on Windows Server (16 vCPU, Python 3.12.10, same pins as CI: pytest 9.0.3, xdist 3.5.0), pytest -p no:randomly --count=N -n 16, before = origin/main, after = this branch:

Test Before After
diag rate floor (natural) 32 / 900 failed (3.6%) 0 / 600
context admission, clock forced across a minute boundary between reads 100 / 100 failed 0 / 100
context admission (natural) 0 / 300 0 / 300
snapshot, clock forced across a second boundary between reads 100 / 100 failed 0 / 100
snapshot (natural) 0 / 300 0 / 300

The two boundary races are too rare to hit in a few hundred natural runs on an idle host, so they were reproduced by a local-only plugin (not in this diff) that makes the first now() read hh:mm:59.999 and later reads one second on — exactly the CI signatures. The fixed tests pin their own clock after that plugin runs, which is why they pass under it.

Related Issues

Merge-queue readiness: #15200.

Pattern harvest

Rule candidate: review-prompt
Pattern: a test that compares two results which embed datetime.now() / time.monotonic() (or assumes two calls share a second) without pinning the clock the module reads.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines (black gate, flake8, isort clean)
  • Documentation updated (if applicable) — N/A, test-only
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97
bolichen97 requested a review from a team as a code owner October 1, 2026 03:50
@bolichen97
bolichen97 requested a review from kyleseaman October 1, 2026 03:50
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of cdfd858951a5a18d3a4584ca6ca2254e877fe13a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] cdfd858

False positive or not applicable? A repository writer can comment:
/ai-review override gpt cdfd858951a5a18d3a4584ca6ca2254e877fe13a: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of cdfd858951a5a18d3a4584ca6ca2254e877fe13a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] cdfd858

Description read

[DESCRIPTION-READ] c610e9d44da4646de25f52d66e2fb24e60486e44df79a5d6ddcc772dbc908883

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@bolichen97
bolichen97 force-pushed the test/pin-real-clock-in-windows-flakes branch from c5f4add to cdfd858 Compare October 1, 2026 03:58
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed cdfd858951a5a18d3a4584ca6ca2254e877fe13a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] cdfd858

Verdict parsed from the review's SHA-scoped output markers for commit cdfd858951a5a18d3a4584ca6ca2254e877fe13a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable cdfd858951a5a18d3a4584ca6ca2254e877fe13a: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 1, 2026 05:11
@bolichen97
bolichen97 merged commit fac6dd9 into main Oct 1, 2026
142 of 143 checks passed
@bolichen97
bolichen97 deleted the test/pin-real-clock-in-windows-flakes branch October 1, 2026 05:14
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 1, 2026
chenmingwei23 pushed a commit that referenced this pull request Oct 7, 2026
Windows shard 6 was ours. The four new routes under
`/api/agent-panel/dashboard/` are registered through the deferred binder, and
three pins in the composition contract count those registrations: the MCP route
table's rows and digest, and the deferred run in each boot chain. All three now
carry the four. They were missed locally because the earlier round ran
`test_dashboard_route_table.py` and not the composition contract.

Windows shard 4 was not ours, and is fixed rather than reported because the fix
is the one this repo already made for its siblings.
`test_two_reads_too_close_together_decline_the_rate` left the clock unpinned and
relied on two back-to-back reads being under MIN_RATE_GAP_SECS of REAL time --
but it writes a fake /proc tree to disk between them, so a slow runner crosses
the floor, a rate is computed, and the test fails for where it ran. #15708
pinned three siblings for exactly this and missed this one. The gap is now set
by hand at 0.5s, so the floor is the subject.

Three findings Opus left on the previous head, all real and all small:

* an empty record with no default was still composed. `manifest={}` does not
  parse, so the read logged a warning WITH a traceback for a crewmate that has
  adopted nothing -- on every poll and every projection frame, which is the
  ordinary state of a build whose registry ships no default page. Nothing is
  attempted now; the frame's empty state is the answer and needs no html.
* the 400 body an agent reads was the stored 240-character row. The cap bounds
  the crew-log entry, but the cut takes the END of the sentence, and the end is
  the remedy: valid field names first, "you have done this before; 'x' worked"
  last. A manifest with about five fields pushed exactly the useful half past
  the cap. `refusal_sentence` now hands the body back whole and the stored row
  is a prefix of it -- one wording, one scrub, one cut.
* workstream boards sorted on `last_activity_at`, which `_work_iso` renders with
  a LOCAL offset, so two were compared as wall-clock text. Across an autumn DST
  hour a later board reads as older, and at WORKSTREAMS_BOARD_LIMIT the newest
  board is the one moved into `omitted`. Sorted on `last_activity_ms` now, which
  was already kept beside it.

Five mutations proven: cutting the 400 body, sorting on the rendered text,
composing the empty record, dropping one deferred route, and aging the rate
baseline past the floor.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants