Skip to content

test: retire two shared-runner flakes in reconcile and lease-scan tests - #14940

Merged
bolichen97 merged 1 commit into
mainfrom
test/flaky-sweep-0929
Sep 29, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
test/flaky-sweep-0929

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Two backend tests stop failing on unrelated PRs when a shared CI runner happens to hold pid 5500 or reads the clock from another thread.

Both tests fail and then pass on a rerun of the same commit. In the last three days test_a_descendant_only_record_is_retracted_from_the_descendant_file failed twice on Backend Tests (Windows) (1) and test_lease_scan_stops_on_its_deadline_and_answers_uncertain failed once on Backend Tests (3.12, 2), each time red on a PR that touched neither file.

Why it matters

A false red on a backend shard costs every PR that hashes into it a rerun, and the failure text points at the reconciler and the lease scan rather than at the test, so the first reading is always "this PR broke process cleanup".

Not a goal

  • No change to runtime_reconcile.py or skill_projection.py; both behave correctly when the tests fire.
  • No sweep of other call-count fake clocks in the test tree; only the two tests that failed in CI change.

What changed (motivation → approach → change)

test what made it flaky change
test_runtime_reconcile.py::test_a_descendant_only_record_is_retracted_from_the_descendant_file Windows OpenProcess(5502) opens pid 5500, so the real start-time probe answers a stranger's identity and the live sibling's row is retracted as recycled pin session_pid._pid_start_token to the tokens the rows record, like every other reconciler test in the file
test_native_skill_projection.py::test_lease_scan_stops_on_its_deadline_and_answers_uncertain the call-count fake clock replaced the global time.monotonic, so one read from a neighbour thread spent the schedule before the first entry replace projection.time with a module-scoped shim, the same shape #14535 gave the wait-tool keepalive test

Both tests keep their assertions. Only the seam each one patches moves.

Backwards compatibility

Compatible: test-only change, no production code touched.

Tests

The two changed tests are the tests. Each failure was reproduced on Linux by injecting its CI condition (a stranger start token; a thread spinning on time.monotonic()): the old tests fail 3/3, the changed tests pass 3/3.

Manual verification

N/A — the reproduction above is the verification.

Related Issues

Fixes #14937
Fixes #14938

Pattern harvest

Rule candidate: review-prompt
Pattern: a test that installs a fake clock keyed on call count on the global time module, or that records a pid identity token and leaves the recycle probe live.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

test_a_descendant_only_record_is_retracted_from_the_descendant_file pins the live sibling's start token instead of asking the real probe, which on Windows opens pid 5500 for pid 5502 and calls the row recycled.

test_lease_scan_stops_on_its_deadline_and_answers_uncertain scopes its call-count fake clock to the skill_projection module so a neighbour thread reading time.monotonic() cannot consume the schedule.

Fixes #14937
Fixes #14938
@bolichen97
bolichen97 requested a review from a team as a code owner September 29, 2026 05:43
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of e75c1cb931801a33384de080d808cde15da06cb4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e75c1cb

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e75c1cb931801a33384de080d808cde15da06cb4: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of e75c1cb931801a33384de080d808cde15da06cb4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] e75c1cb

Description read

[DESCRIPTION-READ] bc76da9bbab7f98163df90cc8156414e888bc2065f1fae3edac177f5c2fd0b70

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed e75c1cb931801a33384de080d808cde15da06cb4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e75c1cb

Verdict parsed from the review's SHA-scoped output markers for commit e75c1cb931801a33384de080d808cde15da06cb4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e75c1cb931801a33384de080d808cde15da06cb4: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 29, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 29, 2026 06:20
@bolichen97
bolichen97 merged commit 82fe8f9 into main Sep 29, 2026
96 checks passed
@bolichen97
bolichen97 deleted the test/flaky-sweep-0929 branch September 29, 2026 06:21
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants