bug(Rust SDK): producer shutdown() error_callback is closed immediately, misses potential flush failures

[haubur]

Bug description

Situation
On shutdown() the IggyProducer calls the shutdown() method of the Dispatcher which broadcasts
a termination signal to it's workers:

iggy/core/sdk/src/clients/producer_dispatcher.rs

Line 187 in cc269ef

let _ = self.stop_tx.send(());

this immediately breaks the loop of the callback task:

iggy/core/sdk/src/clients/producer_dispatcher.rs

Lines 53 to 74 in cc269ef

	let handle = tokio::spawn(async move {
	loop {
	tokio::select! {
	maybe_message = err_rx.recv_async() => {
	match maybe_message {
	Ok(ctx) => {
	if let Err(panic) = std::panic::AssertUnwindSafe(err_callback.call(ctx))
	.catch_unwind()
	.await
	{
	tracing::error!("error_callback panicked: {:?}", panic);
	}
	}
	Err(_) => break
	}
	}
	_ = stop_rx.recv() => {
	tracing::debug!("error-callback worker finished");
	break
	}
	}
	}

The shutdown() of the Shards, however, is graceful and everything in buffer still gets flushed.
If the flush fails, messages that were supposed to be handled by a ErrorCallback are not, because nothing listens anymore on that channel and the error remains silent:

iggy/core/sdk/src/clients/producer_sharding.rs

Line 268 in cc269ef

let _ = err_sender.send_async(ctx).await;

Problem
Messages that were not send and come back at ErrorCtx.messages cannot be handled and are permanently lost.
On default, which logs the ErrorCtx the log does not trigger.

Affected area / component

Rust SDK

Deployment

None

Versions

No response

Hardware / environment

No response

Sample code

No response

Logs

No response

Iggy server config

No response

Reproduction

No response

Contribution

• I'm willing to submit a pull request to fix this bug

Good first issue

• I think this could be a good first issue for a new contributor