ServiceBus - Service side connection idle timeout is not surfaced, and can be compromised with improper values of autorenewer renew_period

[KieranBrantnerMagee]

• Package Name:
  azure-servicebus
• Package Version:
  0.50.3 (Same issue will occur in 7.0.* as well)
• Operating System:
  All
• Python Version:
  All

Describe the bug
The ServiceBus Service has a connection idle_timeout of 240000 ms (4 minutes). Not only is this unclear to users, but if renew_period for autolockrenewal is set to >4 minutes, (when max lock duration is theoretically 5 minutes) the connection will time out silently.

To Reproduce
Steps to reproduce the behavior:

1. Create a queue with lock duration of >4 minutes.
2. Send and receive a message, attaching an autolockrenewer with a period >4 minutes.
3. Observe that the message is now unable to be completed due to lock expiry. With all logging on, observe "the connection was inactive for more than the allowed 240000 milliseconds"

Expected behavior
There are a few options here of varying rigidity.

1. Disallow improperly large renew_period.
2. Warning/better docstring on renew period
3. "secretly" run our own keepalive in the 4 minute interval if a large renew period is specified. (should this be a function of the receiver potentially flag-enabled? I'm spitballing.)
4. Document this in our README (we should do this anyway)
5. Potentially coupled with any of the above strategies, find a more immediate way to surface connection idle expiry. (@annatisch I hate that I keep finding myself saying "I wish we had an on error handler"; although this could be an issue with pull mode as well. Consider this a note to see how/if the other languages addressed this issue)

Additional context
Initially noted with the user in this thread

A final note I'd append here: We noticed very odd symptoms when trying to use a 5 minute service side lock_duration, wherein the expires_at time on the message would appear to be >5 minutes in the future, resulting in guaranteed lock renewal failures if you trusted it. This should be investigated in parallel with the core issue to see if we're just dealing with clock skew, or if there's actually some odd edge case there.

[KieranBrantnerMagee]

This should be largely mitigated per #12307 , which among other things makes renew_period internal as it should not need to be adjusted for proper functionality, and adjustment only opens room for pitfalls as described.

This notably also adds on-failure callback capabilities to better alert if a lock renewal failure DOES occur nonetheless, as another component of the exchange that spawned this issue initially.

The 5 minute lock duration issues remain to be investigated, after which this can be closed.