Repository navigation
feat(streaming): Add a Redis key prefix to the streaming cache - #1657
Open
danielballan wants to merge 2 commits into
Open
danielballan wants to merge 2 commits into
danielballan wants to merge 2 commits into
Conversation
Redis keys and pub/sub channels were named by node ID alone, so two Tiled deployments sharing a Redis server, or two tests running at the same time on different xdist workers (each with a fresh catalog whose node IDs start at 1), received each other's streamed data. The new streaming_cache.key_prefix keeps them apart. Each test that uses Redis gets a unique prefix and deletes only its own keys, replacing flushdb(). Alternative to bluesky#1655, which serializes the Redis tests with a file lock. Assisted-by: opencode:claude-opus-5-5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1657 +/- ##
==========================================
- Coverage 81.00% 81.00% -0.01%
==========================================
Files 198 198
Lines 20756 20760 +4
==========================================
+ Hits 16813 16816 +3
- Misses 3943 3944 +1
🚀 New features to boost your workflow:
|
1 of 2 tasks
Eight hex characters are unique enough among tests running at once and easier to type when inspecting Redis by hand. The test- prefix stays, to mark keys left behind if teardown does not run. Assisted-by: opencode:claude-opus-5-5
genematx
self-requested a review
November 16, 2026 22:52
genematx
approved these changes
Nov 16, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is an alternative to #1655. It de-couples tests that use the Redis by introducing an optional key prefix.
It's possible that there could be other use cases for this feature outside of unit test runs. For example, multiple staging instances sharing a Redis cluster. The 🤖 also touts the ability to use this production. I'm suspicious of that idea, but it could potentially come up for some facilities.
-- Dan Allan, Human Person
All of the following text is AI-generated.
The streaming cache names its Redis keys and pub/sub channels by node ID alone (
sequence:{node_id},data:{node_id}:{seq},notify:{node_id}). Two Tiled deployments that share one Redis server therefore receive each other's streamed data. The test suite hits the same collision: each test builds a fresh catalog, so node IDs restart at 1, and tests running at the same time on different xdist workers read each other's sequences and notifications, while the per-testflushdb()deletes the other test's keys.This is an alternative to #1655. That PR serializes the Redis tests with a file lock, which fixes the tests but leaves the collision in production, ties test correctness to a lock file at an xdist-specific path, and lets one hung test block every Redis test on the other workers. Here the collision is fixed at its source:
streaming_cache.key_prefix(default"", so existing keys are unchanged) is prepended to every key and channel inRedisStreamingDatastore. Replicas of one deployment share a prefix; separate deployments use different ones.redis_key_prefixfixture) and on teardown deletes only its own keys, replacingflushdb(). Tests are isolated however xdist schedules them; no lock and no new dependency.Testing: with local Redis and PostgreSQL, the Redis test files ran 3 times with
-n 8 --dist=load(interleaving tests more than CI'sloadfile): all Redis tests passed each time and no keys were left in Redis. The full suite passed twice with-n 8(excepttest_metadata_index_is_used[postgresql_with_example_data_adapter], which needs the example database CI downloads).Checklist
Assisted-by: opencode:claude-opus-5-5