System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: Resource temporarily unavailable

[saixiaohui]

I am running Azure Kubernetes container on VM ScaleSet with Public IP address for .net core 2.2. Each container will create 20k TCP connections between itself and Azure IoT hub. I am getting the following transient error message when trying to open connection to IoT Hub. We will definitely retry on this exception. However, this happens a lot and it's very annoying. I saw we have a similar issue opened long time ago but it has been closed. Here is the link for existing issue.
https://github.com/dotnet/corefx/issues/17434

Microsoft.Azure.IoT.Simulation.Common.Exceptions.WrapperSimulationException: Error connecting device car.000143949
---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: Resource temporarily unavailable
 at System.Net.Dns.InternalGetHostByName(String hostName)
 at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
 at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
 at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
 at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
 at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
 at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsyncInternal(CancellationToken cancellationToken)
 at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsync(CancellationToken cancellationToken)
 at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
 at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass22_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
 at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
 at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass32_0.<<OpenAsyncInternal>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
 at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
 at Microsoft.Azure.IoT.Simulation.DeviceClients.IoTHubSimulatedDeviceClient.ConnectAsync(CancellationToken cancellationToken) in /src/DeviceClients/IoTHub/IoTHubSimulatedDevice.cs:line 105
 --- End of inner exception stack trace ---
 at Microsoft.Azure.IoT.Simulation.DeviceClients.IoTHubSimulatedDeviceClient.ConnectAsync(CancellationToken cancellationToken) in /src/DeviceClients/IoTHub/IoTHubSimulatedDevice.cs:line 137
 at Microsoft.Azure.IoT.Simulation.Runner.DeviceConnection.DeviceConnectionActor.ConnectAsync(CancellationToken cancellationToken) in /src/Runner/DeviceConnection/DeviceConnectionActor.cs:line 123

[davidsh]

Can you please attach a small code sample of the .NET Core APIs you're using? It looks like you're using System.Net.Dns class. But we need to see more specifics on how you are calling the APIs.

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: Resource temporarily unavailable
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)

The exception you're getting is common when there are DNS resolution problems in your network. Perhaps Azure is having intermittent trouble. That would be outside of .NET Core API to control that.

[saixiaohui]

@davidsh, please find the code here:
https://github.com/Azure/azure-iot-sdk-csharp/blob/78f410cc0799a2949ad9e278821d0240f2339c33/iothub/device/src/Transport/Mqtt/MqttTransportHandler.cs#L527

[saixiaohui]

[saixiaohui]

Attached is a picture showing how many errors we received within an hour and it happens all the time.

[davidsh]

@davidsh, please find the code here:
https://github.com/Azure/azure-iot-sdk-csharp/blob/78f410cc0799a2949ad9e278821d0240f2339c33/iothub/device/src/Transport/Mqtt/MqttTransportHandler.cs#L527

this.serverAddresses = (await Dns.GetHostAddressesAsync(this.hostName).ConfigureAwait(true));

You should definitely change this to .ConfigureAwait(false); The default is already 'true' and isn't good to have that in library code such as this.

And the code should look like this:

this.serverAddresses = await Dns.GetHostAddressesAsync(this.hostName).ConfigureAwait(false);

I'm not sure that will solve your problem but it's something that should be done.

You might also try to use the sync API DnsGetHostAddresses() as well. I'm curious if it will work better for you. We've had some other issues with Dns.GetHostAddressesAsync() on Windows due to how the lower layer Win32 API GetAddrInfoExW() works. We use that Win32 API as part of Dns.GetHostAddressesAsync() since GetAddrInfoExW() is an "async" Win32 API.

[davidsh]

I am running Azure Kubernetes container on VM ScaleSet with Public IP address for .net core 2.2.

What OS is running in the VMs on Azure? Is it Linux or Windows? And since it is a container, it probably is Linux? If so, what container image is being used?

I still suggest using the sync version of the Dns.GetHost* APIs as a workaround to see if it works better for you.

[saixiaohui]

Yes, it's Linux OS. The base docker image is mcr.microsoft.com/dotnet/core/aspnet:2.2-stretch-slim. I am trying to see if there is anyway we can add a Dns Cache layer for docker.

If we can do that, that may help with the issue.

[karelz]

@saixiaohui are you still seeing this problem in production?
We would be interested in enabling some (new) logging and tracing to find out what is going on ...

[karelz]

Not enough info, closing.
We can reopen if there is more info available.

[JinsPeter]

I got same error trying to connect to and Amazon MQ Amqp queue. But I figured that the real issue was I lost internet.