Skip to content

TST: Run All Tests Involving Redis on the Same xdist Worker - #1655

Closed
genematx wants to merge 5 commits into
bluesky:mainfrom
genematx:redis-tests-single-worker
Closed

genematx wants to merge 5 commits into
bluesky:mainfrom
genematx:redis-tests-single-worker

Conversation

@genematx

@genematx genematx commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes the intermittent failures in test_websockets.py / test_subscription.py seen in #1643 CI (wrong sequence counts, arrays off by one, occasional hangs).

Cause. Tests that use the redis_uri fixture share one Redis server, but each test builds its own Tiled server with a fresh in-memory catalog, so node IDs restart from 1 in every test. The streaming cache keys its Redis entries (data:{node_id}:{seq}) and pub/sub channels (notify:{node_id}) by node ID alone. When two such tests run concurrently on different xdist workers, each receives the other's sequences and notifications, and the per-test flushdb() in redis_uri deletes the other's data mid-test.

Fix. The redis_uri fixture now holds a file lock, shared by all xdist workers, for the duration of each test. Tests that use the Redis server therefore never overlap, whatever the scheduler does. No test files change, and new tests that use redis_uri are covered automatically. filelock was already installed transitively (via pre-commit); it is now declared as a test dependency.

Two alternatives were tried and rejected:

  • Separate Redis databases per worker: isolates keys but not pub/sub, which is global to the Redis server (verified directly).
  • --dist=loadgroup with an xdist_group for the Redis tests: silently broken with pytest 9.2, which derives Node.nodeid from a structured _id while pytest-xdist 3.8.0 still rewrites item._nodeid to append the group suffix. Scheduling falls back to per-test load, which is what the first CI run of this PR showed (Redis failures on Linux, and a module-scoped fixture split across workers on Windows).

Possible follow-up (not in this PR): a configurable key/channel prefix in StreamingCacheConfig would make this robust for any two Tiled instances sharing one Redis, including in production.

Assisted by: Claude Fable 5.1

Checklist

  • Add a Changelog entry
  • Add the ticket number which this PR closes to the comment section

Tests that use the redis_uri fixture share one Redis server, but each
test builds its own server with a fresh in-memory catalog, so node IDs
repeat across tests. The streaming cache keys its Redis entries and
pub/sub channels by node ID alone, so two such tests running at the same
time on different workers received each other's sequences and
notifications, and the per-test flushdb() deleted the other's data.

Switch to --dist=loadgroup and assign xdist groups in conftest.py: all
redis_uri users go to one group (serial on one worker); every other test
is grouped by file, which keeps one module per worker as before.

Separate Redis databases per worker would not have been enough, because
pub/sub channels are global to the Redis server.
@genematx genematx changed the title TST: Run All Tests involving Redis on the same xdist Worker TST: Run All Tests Involving Redis on the Same xdist Worker Nov 16, 2026
@genematx genematx added streaming WebSocket and streaming APIs maintenance Upkeep, compatibility and technical debt labels Nov 16, 2026
…groups

--dist=loadgroup does not work with pytest 9.2: pytest now derives
Node.nodeid from a structured _id, while pytest-xdist 3.8.0 still
rewrites item._nodeid to append the group suffix, so the suffix is lost
and scheduling silently falls back to per-test "load". In CI this
scattered the Redis tests across all workers (the original failures,
now worse) and split modules with module-scoped fixtures across
workers (409 Conflict in test_deep_search on Windows).

Go back to --dist=loadfile and instead have the redis_uri fixture hold a
file lock shared by all xdist workers (in the parent of the per-worker
basetemp) for the duration of each test. Tests that use the Redis
server therefore never overlap, whatever the scheduler does, and no
test files need to change.

filelock was already installed transitively (via pre-commit); declare
it as a test dependency.
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.01%. Comparing base (db14afc) to head (926d822).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1655      +/-   ##
==========================================
+ Coverage   80.97%   81.01%   +0.03%     
==========================================
  Files         198      198              
  Lines       20710    20756      +46     
==========================================
+ Hits        16770    16815      +45     
- Misses       3940     3941       +1     

see 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@danielballan

danielballan commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Thanks for addressing this. The analysis I ran in #1643 also caught this but didn't address it [edit: I see that referenced that in the first line of this PR 😁]

I set an LLM to explore the "configurable key/channel prefix" concept in #1657. It avoids some downsides I see in the approach in this PR:

  • One hung test blocks all others.
  • About 200 tests are serialized, a step back for the speed boost from xdist.
  • It introduces an assumption about the filesystem that could be broken by certain pytest flags and plugins.
  • Minor: It adds a test dep.

@genematx

Copy link
Copy Markdown
Collaborator Author

Thanks, @danielballan! I agree, a "configurable key" to select a channel is probably the most robust approach that we may want eventually anyway. I'll see how difficult it would be to implement it.

(Tests blocked by a single one is not something I've considered, but it's definitely bad.)

@genematx genematx closed this Nov 16, 2026
@genematx

Copy link
Copy Markdown
Collaborator Author

Closed in favor of #1657.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Upkeep, compatibility and technical debt streaming WebSocket and streaming APIs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants