An endpoint is "force detached" by the broker when sending messages #994
Description
Describe the bug
Description
When sending/publishing messages (either using transactional session or inside a message handler), the default behaviour is to batch them and send them after the method calling the send/publish method has exited. (This behavior can be changed to dispatch immediately, but then transactionality is lost.)
Expected behavior
The messages are sent.
Actual behavior
The messages are not sent, and an exception is logged.
Versions
3.0.0 onwards. Introduced in #644
Steps to reproduce
In a message handler, send some messages, with the total size of all messages greater than the max allowed message size of the queue.
Relevant log output
> System.InvalidOperationException: The link 'G6:14740510:amqps://**namespace removed**.servicebus.windows.net/-dd910ad4;0:7:8' is force detached by the broker because publisher(link3082) received a batch message with no data in it. Detach origin: Publisher.
For troubleshooting information, see https://aka.ms/azsdk/net/servicebus/exceptions/troubleshoot.Additional Information
Workarounds
None identified so far.
Note that switching to immediate dispatch (actually sending the message when the send/publish method is called) is not a workaround, because that is very different behaviour which loses any guarantees around message sending/publishing transactionality.
Possible solutions
- Measure the size of the outgoing batch before sending it, and if it's too large, don't batch.
- Catch the exception, and retry in a non-batched manner. This probably requires some better behaviour from the Azure SDK client, see [BUG] Batch.TryAdd returns true on premium with large message support but sending the batch throws an InvalidOperationException Azure/azure-sdk-for-net#44261
- Do not fix the bug, but instead provide a configuration switch to switch off batching. This may not be sufficient to fix the bug in ServiceControl (see below).
Additional information
This also manifests in ServiceControl when it attempts to forward audit messages. See Particular/ServiceControl#4185.
Our resolution to this bug may well depend on changes to the Azure SDK client, see Azure/azure-sdk-for-net#44261