Split out of the #3795 review at hubcio's suggestion.
With [state] enabled = true, elasticsearch_source persists its own cursor at close and applies it at open, overriding the state the runtime restored. A runtime-side guarantee therefore cannot cover this source: whatever position the runtime hands back, the plugin's own file wins.
That matters for the guarantee #3795 documents. After a dropped batch the runtime's contract is "resume from the last delivered batch's state", and this source can resume from somewhere else entirely.
It is opt-in and off by default, so it is not a live regression, but the two mechanisms should not both own the cursor. Either the plugin defers to the runtime state when one is present, or the plugin-side option is documented as replacing the runtime's guarantee rather than supplementing it.
Happy to take it once the direction is settled.
Split out of the #3795 review at hubcio's suggestion.
With
[state] enabled = true,elasticsearch_sourcepersists its own cursor at close and applies it at open, overriding the state the runtime restored. A runtime-side guarantee therefore cannot cover this source: whatever position the runtime hands back, the plugin's own file wins.That matters for the guarantee #3795 documents. After a dropped batch the runtime's contract is "resume from the last delivered batch's state", and this source can resume from somewhere else entirely.
It is opt-in and off by default, so it is not a live regression, but the two mechanisms should not both own the cursor. Either the plugin defers to the runtime state when one is present, or the plugin-side option is documented as replacing the runtime's guarantee rather than supplementing it.
Happy to take it once the direction is settled.