Add optional Robotty missed-message recovery - #47
Open
Alb11747 wants to merge 1 commit into
Open
Conversation
Recover messages missed during startup, reconnects, and runtime joins through a bounded Robotty backfill path. Validate, order, and deduplicate replayed messages before storage; configure the feature through the existing JSON configuration model.
This was referenced Aug 12, 2026
Alb11747
marked this pull request as ready for review
August 12, 2026 05:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
recentMessagesEnabled,recentMessagesUrl, andrecentMessagesLimittmi0.7.3; the parser fix is being developed in tmi-rs PR #20twitch-irc6 without changing this PR's released IRC dependenciesUpstream parser issue
Robotty's recent-messages API changed how it serializes empty IRC tags: explicit empty values such as
badges=;color=;room-id=42can now arrive as valueless tags such asbadges;color;room-id=42. This behavior is described in tmi-rs issue #18.The tag parser in
tmi0.7.3 expects each tag to contain=. When it encounters the;terminating a valueless tag while it is still parsing a key, that tag is not emitted correctly and subsequent tags can also become unavailable. In a representative Robotty message, this losesbadges,color, androom-id, even though later tags such astmi-sent-tsmay still be found. Losingroom-idand other metadata makes the response unsafe to validate and store as a complete Twitch message.tmi-rs PR #20: Fix IRC tag parsing and message accessor edge cases fixes both the scalar and SIMD parsers so
key,key=, and ordinarykey=valuetags are parsed consistently. Until that fix is accepted and released, this Rustlog PR performs a narrow conversion fromkey;tokey=;only on Robotty responses before passing them totmi. Live IRC messages are not rewritten.The recovery path is best-effort and isolated from live IRC ingestion. A failed persisted-ID lookup aborts that channel's replay instead of risking duplicate inserts.
Upstream reconnect issue
Internet loss also exposed a separate bug in
twitch-irc: its connection cooldown task was created only after transport initialization succeeded. A transport error or timeout returned early and dropped the semaphore permit immediately, so the pool could retry in a tight loop.twitch-irc PR #231: Rate limit failed connection attempts moves the existing cooldown to permit acquisition, covering successful, failed, and timed-out attempts. In the same isolated 45-second route-loss test, unpatched v6.1.1 emitted 295,890 log lines / 60,243,055 bytes; the patched fork emitted 54 lines / 10,711 bytes. Both recovered, stored the fixture exactly once, and completed clean SIGTERM stops, but the unpatched build generated about 5,480 times as many lines.
This PR continues to use released
twitch-irc5.0.1. The compatibility preparation selects rustls'sringprovider explicitly, accepts both the v5 and v6 IRC tag value shapes, and moves the container build to Rust 1.88 with a pinned cargo-chef. No fork or unreleased dependency is committed.Dependency compatibility
tmi0.7.3 andtwitch-irc5.0.1prometheusto 0.14 andmetrics-prometheusto 0.11.2 so Rustlog andtwitch-ircshare one Prometheus registrytmi, the local Robotty tag normalization is harmless but redundant and can be removed separately as cleanupValidation
cargo test --locked --all-targetson Rust 1.88, 25 passedgit diff --check: passed