Client error-ing out on long-running AWS Lambdas (`getaddrinfo EMFILE stream.launchdarkly.com`)

[alice-was-here]

Describe the bug

On long running lambda instances, the Launch Darkly client appears to get stuck in a loop while trying to establish a connection. This loop continues until the lambda function times out, or runs out of memory.

To reproduce

This error only occurs intermittently, but is most common in the following circumstances:

• Have a long running AWS Lambda instance (greater than 45 minutes).
• Try to initialise the Launch Darkly client within a function on the lambda
• When initialising, the client gets stuck in a loop and outputs the following error until the function times out or runs out of memory. ERROR warn: [LaunchDarkly] Received I/O error (getaddrinfo EMFILE stream.launchdarkly.com) for streaming request - will retry
• Subsequent calls to the function on the same lambda instance will successfully establish a Launch Darkly connection

Expected behavior

Ideally:

• The client is successfully able to establish a connection

However, an acceptable fall-back would be:

• If the client cannot establish a connection, a sensible error is thrown - rather than the function timing out.

Logs

The following log is repeated in the output until the Lambda function times out.

ERROR warn: [LaunchDarkly] Received I/O error (getaddrinfo EMFILE stream.launchdarkly.com) for streaming request - will retry

SDK version
launchdarkly-node-server-sdk v7.0.0

Language version, developer tools

AWS Lambda running nodejs:18.v4

OS/platform

AWS Lambda running nodejs:18.v4

[kinyoklion]

Hello @alice-was-here,

Thank you for your report. Are your logs decorated with timestamps, I am curious the frequency these are logged. The first one should not be for a second, and then they have a jitter and backoff algorithm until 30 seconds, but it does never stop retrying. The initialization is async and the retry itself is timer based so it shouldn't delaying a caller. But there could be some condition possibly causing it to be quicker.

Are you initializing the SDK within each operation done on the lambda? Or creating a single client and then using it for all requests for the duration of the lambda? (The recommended practice is to use a single client, because initialization is a heavy weight operation.)

After initializing are you using the promise method provided, waitForInitialization? This isn't mandatory, but I am curious if it is causing your timeout behavior (because if it cannot initialize, then it will never become initialized). If you are, then I would recommend using a promise.race in combination with a timer promise as a general safety mechanism.

Filed internally as 190309

Thanks,
Ryan

[github-actions]

This issue is marked as stale because it has been open for 30 days without activity. Remove the stale label or comment, or this will be closed in 7 days.