Skip to content

test(integration): background work a test starts is cancelled before the next (B-21) - #439

Merged
sshlg merged 2 commits into
mainfrom
test/b21-drain-background-work
Sep 26, 2026
Merged

sshlg merged 2 commits into
mainfrom
test/b21-drain-background-work

Conversation

@sshlg

@sshlg sshlg commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

B-21 — the test_learnings_api StaleDataError (CI on 2026-09-23 and 2026-09-26; passes alone and on rerun).

Mechanism. On SQLite every session shares one physical connection (StaticPool), and work started with spawn_tracked or an in-process task_queue fallback is never awaited by the request that started it. A leftover task that rolls back on that connection can undo the NEXT test's flushed-but-uncommitted insert; that test then commits nothing, finds the object in its identity map without a query, and its UPDATE matches zero rows. That is exactly UPDATE … expected to update 1 row(s); 0 were matched. Not reproduced deterministically — this is the mechanism the evidence fits, and the fix removes the precondition.

Fix. db_session teardown cancels and awaits every such task before the rollback and the truncate (_drain_background_work).

Guard. test_background_work_does_not_outlive_its_test.py: one test leaves a task running, the next asserts it did not inherit it — red without the drain.

Checks: integration 702/702 on SQLite locally (the PostgreSQL job runs here); ruff clean. Only tests/integration changed, so the unit suite is unaffected.

🤖 Generated with Claude Code

…the next (B-21)

On SQLite every session shares one physical connection (StaticPool), and
work started with spawn_tracked or an in-process task_queue fallback is
never awaited by the request that started it. A leftover task that rolls
back on that connection can undo the next test's flushed-but-uncommitted
insert; that test then commits nothing, finds the object in its identity
map, and its UPDATE matches zero rows — the StaleDataError
test_learnings_api hit on 2026-09-23 and 2026-09-26, passing alone and on
rerun.

db_session's teardown now cancels and awaits all such tasks before the
rollback and the truncate. test_background_work_does_not_outlive_its_test
leaves a task running in one test and asserts the next does not inherit
it — red without the drain. Integration suite 702/702 on SQLite.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… (B-21)

The first drain cancelled leftover tasks, and a task cancelled mid-statement
makes SQLAlchemy invalidate the connection — under StaticPool the
replacement is a new, EMPTY :memory: database, so every later test failed
with "no such table: users" (263 failed, 965 errors on CI).

Now the teardown waits up to 30 s and leaves anything still running alone,
naming it in a warning. `_assert_schema_survived` fails at the cause if the
shared database ever loses its tables again, instead of a cascade. The
guard test uses a task that finishes on its own and asserts the next test
sees it finished and not cancelled (red without the drain).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sshlg

sshlg commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator Author

First CI run: the drain cancelled leftover tasks, and a task cancelled mid-statement makes SQLAlchemy invalidate the connection — under StaticPool that swaps in a new empty :memory: DB, so everything after failed with no such table: users (263 failed, 965 errors). Pushed a wait-only drain (30 s, never cancels, names stragglers) plus _assert_schema_survived, which fails at the cause if the shared DB ever loses its tables. Local CI-like run under load average ~200 gave 700/702: test_performance_smoke (fails alone on this machine — timing) and one test_investigate_tenant_isolation FK failure that passes alone; this CI run is the real measurement.

@sshlg
sshlg merged commit 94d3826 into main Sep 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant