Cached Datadog api key doesn't play well with snapstart in V84+

[kirkrd]

We are using .NET lambdas with snapstart and rotate our DD key every day, after upgrading to the latest version we noticed that the inflow of data and logs stop after a certain period of time (the keys are stuck and expire).

Suspected commit causing this:
#717

Comment found that might be causing the issue
#717 (comment)

Our solution now is to downgrade, but would love to have keep the same behaviour as pre V84.

[lym953]

Thanks for the report! Looking.

[lym953]

To help me better understand your issue: is your DD key stored as a Secret Manager secret and specified using an env var DD_API_KEY_SECRET_ARN?
If api key is in a secret and snapstart is on, I expect the function to time out from the first invocation. This is what this PR (#853) is trying to fix. Is this not the case for you?

[kirkrd]

To help me better understand your issue: is your DD key stored as a Secret Manager secret and specified using an env var DD_API_KEY_SECRET_ARN? If api key is in a secret and snapstart is on, I expect the function to time out from the first invocation. This is what this PR (#853) is trying to fix. Is this not the case for you?

It is stored in Secretmanager, then used as apiKeySecret property in the DatadogLambda construct from datadog-cdk-constructs-v2. Tried downgrading to extensionLayerVersion: 82 but the issue still seems to arise approximently 1-2 days after deployment through CDK

[kirkrd]

But perhaps its better to use the arn directly instead of reading the secret value directly, will try that and report back

[lym953]

@kirkrd Thanks for the context. Using apiKeySecret of the DatadogLambda construct results in DD_API_KEY_SECRET_ARN env var being set, which is affected by this bug. You don't need to try using the arn directly. The bug will be fixed in the next release.

[lym953]

@kirkrd Could you try v87?

[kirkrd]

Hello again, we tried v87 and using apiKeySecretArn instead of apiKeySecret, same issue @lym953

[lym953]

@kirkrd Thanks for the update. Let me reopen this issue and take a look.