Intermittent AWS InvalidSignatureException with v9.116.0

[ex-nerd]

Expected Behavior

No errors.

Actual Behavior

InvalidSignatureException error intermittently appearing in our logs, resulting in invalid token validation that caused auth issues for users.

Steps to Reproduce the Problem

Unfortunately, this is not something we've been able to consistently replicate. The error started happening when we upgraded from v9.115.0 to v9.116.0 and went away when we downgraded.

Specifications

• Datadog Lambda Layer version: v9.116.0
• Node version: AWS Lambda Node.js 20.x

Stacktrace

InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
    at constructor.CIi (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/protocol/json.js:80:27)
    at constructor.callListeners (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at constructor.emit (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at constructor.emitEvent (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/request.js:686:14)
    at constructor.t (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/request.js:22:10)
    at ONe.ONe.runTo (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at done (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/state_machine.js:26:10)
    at constructor.<anonymous> (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/request.js:38:9)
    at constructor.<anonymous> (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/request.js:688:12)
    at constructor.callListeners (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
    at constructor.emit (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at constructor.emitEvent (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/request.js:686:14)
    at constructor.t (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/request.js:22:10)
    at ONe.ONe.runTo (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at done (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/state_machine.js:26:10)
    at constructor.<anonymous> (/node_modules/.pnpm/aws-sdk@2.1502.0/node_modules/aws-sdk/lib/request.js:38:9)

Assuming your internal IDs are unique, an example can be found via:

span id: 7530370475899081053
trace id: 3907063422949516603

If necessary, I can share more information about this through your regular private support channel.

[astuyve]

Hi! I believe this is related to this issue, as this library doesn't actually trace or modify the aws-sdk.

Can you open an issue in dd-trace-js?

Thank you!

[ex-nerd]

We've been using dd-trace=5.24.0 but the only change we made to break/fix the issue here was upgrading the js version of the lambda layer to 116, along with the lambda extension from 65 to 66. Is it possible that one of those pulled in its own broken copy of dd-trace?

[astuyve]

This library, when used as a layer, packages dd-trace. If you install it from npm, then you pick your own version of ddtrace.

If you're using the lambda layer then I'd expect (due to layer dependency collisions), that your packaged version of the layer is not being used at all.

That said you can test it by using NPM to install this library along with ddtrace 5.26 which reverted this PR.

The Lambda Extension is not relevant here, however this library may need to publish a new Lambda Layer with the reverted version of dd-trace-js. That said it still may be helpful for that team to understand your use case to prevent it from happening again.

[ex-nerd]

If you're using the lambda layer

Yes, using the lambda layer per recommendation from the official docs. I can try to back out of this, but that's a larger change.

I'll test the updated ddtrace, though it sounds like you're suggesting there might be an incoming v9.117.0 that includes dd-trace >= 5.26 as well?

[astuyve]

Yup, v117 was just released and contains ddtrace 4.50, which contains the revert.

[ex-nerd]

Closing this because it's hopefully fixed, but we also don't plan to update to the latest version any time soon to verify.