feat(event): run native queue and event plugin simultaneously#6852
Open
h45hc47 wants to merge 1 commit into
Open
feat(event): run native queue and event plugin simultaneously#6852h45hc47 wants to merge 1 commit into
h45hc47 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Makes the native ZMQ message queue and the event plugin (e.g. the MongoDB plugin) two independent event sinks, so a node can publish triggers to both at the same time.
Previously they were mutually exclusive: when
useNativeQueue = true, the plugin'spath/server/dbconfigwere never loaded and only the native queue started; the plugin only ran whenuseNativeQueue = false.Key changes:
Args: always copypath/server/dbconfigintoEventPluginConfig, regardless ofuseNativeQueue.EventPluginLoader: add a separateuseEventPluginflag (derived from a non-empty plugin path).start()launches the plugin first — so the plugin's listeners exist before any topic is registered — and then the native queue. Eachpost*Triggeris dispatched to every active sink;setPluginTopicis null-guarded for the now-possible double trigger-config pass;isBusy()is gated on the plugin being active instead of on the native queue being off.HistoryEventService: honor the plugin's back-pressure (isBusy) during historical sync even when the native queue is enabled, so the plugin queue is not flooded. Native-only pacing is unchanged.config.conf/reference.conf: document thatuseNativeQueueand a pluginpathcan be combined.Why are these changes required?
Operators often want both transports at once: low-latency ZMQ delivery for real-time consumers AND durable persistence in MongoDB for querying and indexing. Today they must pick one. With a single
event.subscribeblock (native queue on + a plugin path set), both now receive every trigger.Example configuration enabling both:
This PR has been tested by:
:commonand:frameworkcompileJavasucceed.EventPluginLoaderTest#testIsBusy— updated for the newuseEventPluginsemantics — PASSED.HistoryEventServiceTest#test— covers the reordered back-pressure path — PASSED.useNativeQueue = trueand a MongoDB plugin path set starts cleanly, writes events to the MongoDB collections, and binds the native ZMQ queue on port 5555 simultaneously.Follow up
A future enhancement could add an explicit
useEventPluginconfig key instead of inferring plugin activation from a non-emptypath.Extra details
In combined mode the trigger-config pass runs once per launcher; this is idempotent (boolean flags and
setTopicoverwrite with the same values). Every trigger is now serialized and dispatched to both sinks, a modest overhead versus single-sink mode. Behavior is unchanged for the two existing single-sink configurations (native-only, or plugin-only).