Skip to content

feat(streaming): Add a Redis key prefix to the streaming cache - #1657

Open
danielballan wants to merge 2 commits into
bluesky:mainfrom
danielballan:redis-key-prefix
Open

danielballan wants to merge 2 commits into
bluesky:mainfrom
danielballan:redis-key-prefix

Conversation

@danielballan

@danielballan danielballan commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

This PR is an alternative to #1655. It de-couples tests that use the Redis by introducing an optional key prefix.

It's possible that there could be other use cases for this feature outside of unit test runs. For example, multiple staging instances sharing a Redis cluster. The 🤖 also touts the ability to use this production. I'm suspicious of that idea, but it could potentially come up for some facilities.

-- Dan Allan, Human Person

All of the following text is AI-generated.


The streaming cache names its Redis keys and pub/sub channels by node ID alone (sequence:{node_id}, data:{node_id}:{seq}, notify:{node_id}). Two Tiled deployments that share one Redis server therefore receive each other's streamed data. The test suite hits the same collision: each test builds a fresh catalog, so node IDs restart at 1, and tests running at the same time on different xdist workers read each other's sequences and notifications, while the per-test flushdb() deletes the other test's keys.

This is an alternative to #1655. That PR serializes the Redis tests with a file lock, which fixes the tests but leaves the collision in production, ties test correctness to a lock file at an xdist-specific path, and lets one hung test block every Redis test on the other workers. Here the collision is fixed at its source:

  • streaming_cache.key_prefix (default "", so existing keys are unchanged) is prepended to every key and channel in RedisStreamingDatastore. Replicas of one deployment share a prefix; separate deployments use different ones.
  • Each test that uses Redis gets a unique prefix (redis_key_prefix fixture) and on teardown deletes only its own keys, replacing flushdb(). Tests are isolated however xdist schedules them; no lock and no new dependency.

Testing: with local Redis and PostgreSQL, the Redis test files ran 3 times with -n 8 --dist=load (interleaving tests more than CI's loadfile): all Redis tests passed each time and no keys were left in Redis. The full suite passed twice with -n 8 (except test_metadata_index_is_used[postgresql_with_example_data_adapter], which needs the example database CI downloads).

Checklist

  • Add a Changelog entry
  • Add the ticket number which this PR closes to the comment section

Assisted-by: opencode:claude-opus-5-5

Redis keys and pub/sub channels were named by node ID alone, so two Tiled
deployments sharing a Redis server, or two tests running at the same time
on different xdist workers (each with a fresh catalog whose node IDs
start at 1), received each other's streamed data. The new
streaming_cache.key_prefix keeps them apart. Each test that uses Redis
gets a unique prefix and deletes only its own keys, replacing flushdb().

Alternative to bluesky#1655, which serializes the Redis tests with a file lock.

Assisted-by: opencode:claude-opus-5-5
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.00%. Comparing base (4e06620) to head (4e8e790).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1657      +/-   ##
==========================================
- Coverage   81.00%   81.00%   -0.01%     
==========================================
  Files         198      198              
  Lines       20756    20760       +4     
==========================================
+ Hits        16813    16816       +3     
- Misses       3943     3944       +1     
Files with missing lines Coverage Δ
tiled/config.py 98.52% <100.00%> (+<0.01%) ⬆️
tiled/server/streaming.py 90.52% <100.00%> (+0.10%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Eight hex characters are unique enough among tests running at once and
easier to type when inspecting Redis by hand. The test- prefix stays, to
mark keys left behind if teardown does not run.

Assisted-by: opencode:claude-opus-5-5
@genematx
genematx self-requested a review November 16, 2026 22:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants