Skip to content

test(skills): keep the queued re-walk out of the drop's busy-timeout spy - #39375

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix-skillcat-drop-flake
Nov 19, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix-skillcat-drop-flake

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make TestInvalidation::test_a_mutation_s_drop_waits_no_longer_than_the_index_s_own_timeout deterministic.

The test fails intermittently on main with the drop waited with busy_timeout=[0, 0] (expected [0]), seen on docs-only PR #33463, Backend Tests (3.12, 8), run 44421446325.

Why it matters

A red Backend Tests shard and Coverage Gate on PRs that change nothing near the skill catalog cost every author a re-run and a triage.

Not a goal

  • Changing the drop, the store or the invalidation's re-walk in product code.
  • Testing the re-walk itself (test_a_later_successful_drop_restores_trust and the drop_first tests cover it).

What changed (motivation → approach → change)

Symptom: the busy-timeout spy recorded two BEGIN IMMEDIATE reads where the test pins one.

Root cause: _invalidate_iter_cache ends by queueing a re-walk through loader._request_catalog_refresh(scope). With the drop refused by the held lock, the re-walk's store_catalog(drop_first=True) opens its own BEGIN IMMEDIATE through the same spied _db(). When the worker thread gets there before the assertion, the spy holds [0, 0]. The same re-walk could also land the drop after held.rollback() and restore trust, flipping the not _trusted(loader) assertion. The product is right: that read belongs to a different transaction, at the base timeout.

Change: the test stubs loader._request_catalog_refresh to a no-op, so the spy sees only the drop. The docstring says why. Two locals (conn, holder) are folded so the file stays at the 2,000-line cap.

Backwards compatibility

Compatible: test-only change.

Removes nothing: no product code, key or record changes.

Tests

  • test_a_mutation_s_drop_waits_no_longer_than_the_index_s_own_timeout: still pins that the drop's BEGIN IMMEDIATE runs at the base busy timeout and that a refused drop is not trusted, now without the background re-walk racing either assertion.

Proof: not provable -- test-only change

Determinism proof:

  • Forced condition on main f75e80b (wait for the queued refresh before asserting): red, busy_timeout=[0, 0]. With the fix the re-walk is never started, so that condition cannot arise.
  • Head f5b34a7e86f5: 50/50 alone; the whole file 20/20 under -n 4 (xdist order).
  • Mutation: the drop setting PRAGMA busy_timeout = 5000 before its BEGIN IMMEDIATE turns the test red (busy_timeout=[5000]).

Manual verification

N/A — unit coverage sufficient; the change is test-only.

Real-machine verification

Machine: Linux cloud desktop, Amazon Linux 2023 (kernel 6.12), aarch64, 64 vCPU
Before (main f75e80b): forced condition (wait for the queued re-walk) -> AssertionError: the drop waited with busy_timeout=[0, 0]
After (head f5b34a7e86f5): pytest -q test/test_skill_catalog_snapshot.py::TestInvalidation::test_a_mutation_s_drop_waits_no_longer_than_the_index_s_own_timeout -> 50/50 pass; file under -n 4 20/20 pass
Cleanup: own computer

Related Issues

Fixes #39374

Sibling sweep

Swept: git grep -n "_BusyTimeoutAtBegin(" -- test/
Siblings: none -- the busy-timeout spy is used by this one test only

Pattern harvest

Not generalizable: the race came from this test spying on a connection that a background re-walk shares; no lint shape separates that from a legitimate spy.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners November 19, 2026 03:22
@jeeshofone
jeeshofone requested a review from CrysisDeu November 19, 2026 03:22
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision d54a0c98dbf200f2e1443488206100f51cf1cc40 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d54a0c98dbf200f2e1443488206100f51cf1cc40 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d54a0c98dbf200f2e1443488206100f51cf1cc40 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d54a0c9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d54a0c98dbf200f2e1443488206100f51cf1cc40 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d54a0c98dbf200f2e1443488206100f51cf1cc40 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

The change only touches tests: it stubs the background re-walk so the spy sees only the drop's own transaction. The invalidation path calls loader._request_catalog_refresh(scope) through the instance, so the stub takes effect. Nothing blocks.

[DESIGN-REVIEWED] d54a0c9

Description read

[DESCRIPTION-READ] bb6f5989836a23335e9351e33f67335c0a95e5de24f38a3c664cc0e29b4f6fba

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d54a0c98dbf200f2e1443488206100f51cf1cc40 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d54a0c9

Verdict parsed from the review's SHA-scoped output markers for commit d54a0c98dbf200f2e1443488206100f51cf1cc40.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d54a0c98dbf200f2e1443488206100f51cf1cc40 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 4% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only fix for a flaky test. The analysis is correct: _invalidate_iter_cache queues _request_catalog_refresh(scope) after the drop (catalog.py:1496-1497), and that background re-walk's store opens a second BEGIN IMMEDIATE on the spied _db() and can restore trust. The re-walk is stubbed on the instance, the same pattern other tests already use, while _land_invalidation_drop still runs against the spy. Shortening two lines is safe: the opened fixture keeps its own reference to the holder index until teardown, and a None connection would still fail the spy assertion. No product, state, limit or trust changes; backend shards and the flake ratchet are green.

Tier: 1, every tier gate passed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit d54a0c98db · 12 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 19, 2026 10:24

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved automatically by the code change approver, not by a person. The decider estimates a 4% chance that this change introduces a defect, within the 12% limit for approving on its own, and the change has at most 400 changed lines.

Test-only fix for a flaky test. The analysis is correct: _invalidate_iter_cache queues _request_catalog_refresh(scope) after the drop (catalog.py:1496-1497), and that background re-walk's store opens a second BEGIN IMMEDIATE on the spied _db() and can restore trust. The re-walk is stubbed on the instance, the same pattern other tests already use, while _land_invalidation_drop still runs against the spy. Shortening two lines is safe: the opened fixture keeps its own reference to the holder index until teardown, and a None connection would still fail the spy assertion. No product, state, limit or trust changes; backend shards and the flake ratchet are green.

Tier: 1, every tier gate passed.
Criteria: no merge conflict, no owner block, no change request, the default branch, no security-sensitive path, no design document to agree on, the readiness status, the review lanes, the P0 history, the linked issues' tier, the review lessons, no concern label, the file count, a safe change category, not its own change, the decider, the security checklist, the change matches its title, no competing change.

Decider v3c (217fe607). To overrule it, request changes or dismiss this review.

@KiroCrewCaptain
KiroCrewCaptain merged commit 3c3dd60 into main Nov 19, 2026
101 of 161 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix-skillcat-drop-flake branch November 19, 2026 10:24
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_skill_catalog_snapshot: drop-waits test flakes with busy_timeout=[0, 0] when the queued re-walk's store hits the spy

2 participants