[BUG]: Instrumented Lambda Authorizer throws "Error: Could not extract context"

[LukaHedtSV]

Tracer Version(s)

.> 5.51.0

Node.js Version(s)

22.x

Bug Report

Hi There,

We are using the dd-cdk-constructs-v2 library with some custom auto-instrumentation, and noticed that using Node Lambda Layer >= 126 (which ships dd-trace-js@5.56.0) crashes our authorizers on the start of an invocation.

(See cloudwatch extract in the error logs)

This behaviour doesn't occur in layer 125, so I assume the change went in between 5.51 and 5.56.

Based on my analysis, Lambda Authorizers don't include the context object as a second argument to the lambda handler, so when the tracer fails to find this argument, it throws an error instead of logging or just dealing with it.

This is undesirable, as it limits the default lambda layer version we can set in our internal library, so we miss out on all the newer features for most users.

Reproduction Code

// src/lambda/authorizer/handler/index.ts

import type { APIGatewayAuthorizerWithContextResult, APIGatewayRequestAuthorizerEvent } from 'aws-lambda';

export const handler = async (
  event: APIGatewayRequestAuthorizerEvent
): Promise<APIGatewayAuthorizerWithContextResult<any>> => {
  console.log('Hello World');
  // This is bad but illustrates the point
  return {
    principalId: '*',
    policyDocument: {
      Version: '2012-10-17',
      Statement: [
        {
          Action: 'execute-api:Invoke',
          Effect: 'Allow,
          Resource: ['*'],
        },
      ],
    },
  };
};

Error Logs

Tracer Config

N/A

Operating System

Amazon Linux 2

Bundling

ESBuild

[joeyzhao2018]

Thank you for raising the issue. We have created a PR to address the no-context cases. However, one thing puzzles me is that you mentioned the issue doesn't occur in layer 125 but the problematic logic was introduced in that version already. I will try set up a reproduction example to confirm.

[purple4reina]

Hi @LukaHedtSV, we've got a fix in place and are currently working to merge and release it, which should solve this issue for you.

In the meantime, we've been unable to reproduce on our end, despite using the exact same code that you shared with us here.

I'm wondering, how exactly are you bundling your code. Can you please share the exact commands that you run and any configuration files that it uses?

Thanks.

[LukaHedtSV]

Oh that's fascinating, okay - maybe it's something upstream.

This specific authorizer is being pulled in as a construct library from our internal NPM repo, so I can't share the specific precisely, but I'll see what I can do...

The authorizer construct is being bundled by JSII 5.9.0 for JS alone, and the lambda itself is bundled by the NodeJsLambda aws-cdk-lib construct with esbuild. The only notably set defaults are that it's an ARM lambda, minified, with these compilerOptions in tsconfig.json:

{
"compilerOptions": {
    "alwaysStrict": true,
    "declaration": true,
    "esModuleInterop": true,
    "experimentalDecorators": true,
    "inlineSourceMap": true,
    "inlineSources": true,
    "lib": [
      "es2020"
    ],
    "module": "CommonJS",
    "noEmitOnError": false,
    "noFallthroughCasesInSwitch": true,
    "noImplicitAny": true,
    "noImplicitReturns": true,
    "noImplicitThis": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "resolveJsonModule": true,
    "strict": true,
    "strictNullChecks": true,
    "strictPropertyInitialization": true,
    "stripInternal": true,
    "target": "ES2020"
  }
}

In the consuming repo, it's picked up by a CDK aspect and fed in by essentially this (the DatadogLambda is not created for every function the aspect finds, it's just created the once per stack, for clarity).

CDK Version: 2.237.1
datadog-cdk-constructs-v2: 3.1.0
NodeJS: 22.16.0

const datadog = new DatadogLambda(this, id, {
  addLayers: functionProps.addDatadogLayers,
  enableMergeXrayTraces: functionProps.enableMergeXrayTraces,
  pythonLayerVersion: functionProps.pythonLayerVersion,
  nodeLayerVersion: functionProps.nodeLayerVersion,
  extensionLayerVersion: functionProps.extensionLayerVersion,
  flushMetricsToLogs: false, // We don't support this method. Use Cloudwatch Custom Metrics
  site: this.props.site,
  apiKeySecretArn: this.apiKeySecret.secretArn,
  enableDatadogTracing: functionProps.enableDatadogTracing,
  enableDatadogLogs: this.props.loggingMethod == DatadogLogStreamMethod.EXTENSION && functionProps.enableDatadogLogs,
  injectLogContext: functionProps.injectLogContext,
  logLevel: functionProps.agentLogLevel,
  captureLambdaPayload: functionProps.captureLambdaPayload,
  env: this.env.name,
  service: functionProps.service,
  version: functionProps.version,
  tags: this.generateTagString(functionProps.tags!),
});

datadog.addLambdaFunctions(lambdas);

for (const lambda of lambdas) {
  this.apiKeySecret.grantRead(lambda);
  lambda.node.addMetadata(DD_ENROLLED_METADATA, true);
  lambda.addEnvironment('DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED', 'true');

  // See https://docs.datadoghq.com/serverless/aws_lambda/installation/nodejs/?tab=awscdk#minimize-cold-start-duration
  if (functionProps.nodeLayerVersion! >= 67) {
    lambda.addEnvironment('DD_TRACE_OTEL_ENABLED', 'false');
    lambda.addEnvironment('DD_PROFILING_ENABLED', 'false');
    lambda.addEnvironment('DD_SERVERLESS_APPSEC_ENABLED', 'false');
  }
}

The defaults for which are (currently):

{
  addDatadogLayers: true,
  pythonLayerVersion: 122,
  nodeLayerVersion: 125,
  extensionLayerVersion: 92,
  enableDatadogTracing: true,
  enableMergeXrayTraces: true,
  enableDatadogLogs: true,
  injectLogContext: true,
  agentLogLevel: 'error',
  captureLambdaPayload: false,
  tags: {},
  service: 'service' in cdk.Stack.of(this) ? (cdk.Stack.of(this) as any).service : undefined,
  version: 'version' in cdk.Stack.of(this) ? (cdk.Stack.of(this) as any).version : undefined,
}

Consuming repo's tsconfig.json:

{
  "compilerOptions": {
    "rootDir": ".",
    "outDir": "lib",
    "alwaysStrict": true,
    "declaration": true,
    "esModuleInterop": true,
    "experimentalDecorators": true,
    "inlineSourceMap": true,
    "inlineSources": true,
    "lib": [
      "es2022",
      "ES2021.String"
    ],
    "module": "NodeNext",
    "noEmitOnError": false,
    "noFallthroughCasesInSwitch": false,
    "noImplicitAny": true,
    "noImplicitReturns": true,
    "noImplicitThis": true,
    "noUnusedLocals": false,
    "noUnusedParameters": false,
    "resolveJsonModule": true,
    "strict": true,
    "strictNullChecks": true,
    "strictPropertyInitialization": false,
    "stripInternal": true,
    "target": "ES2022",
    "moduleResolution": "nodenext",
    "typeRoots": [
      "./node_modules/@types"
    ],
    "baseUrl": ".",
    "paths": {
      "@/*": [
        "src/app/*"
      ],
      "~/*": [
        "test/*"
      ]
    }
  },
}

Oh and it's possibly important - the authorizer lambda is being used as an Authorizer for a RestAPI, not a HTTP API.

I hope that brings some clarity? I have no idea why 125 works if the issue I thought I found turns out to be a red herring, sorry about that!

[purple4reina]

What's most strange is that we received another similar report, but pointing to different layer versions (#715). They're seeing the same error as you are.

I tried building the lambda handler using the config defaults you provided but was still unable to reproduce.

Either way, we know what is causing the error and have opened a PR to fix it (DataDog/dd-trace-js#7445). We should be able to get it merged and released within the next few days.

[LukaHedtSV]

Thanks @purple4reina - Once dd-trace-js releases and the new lambda layer version ships, I'll be sure to test it out and report back :)

[rithikanarayan]

Hi @LukaHedtSV, thanks for your patience! We have just released v134 of datadog-lambda-js, packaging dd-trace-js v5.83.0, which should solve your issue! Please let us know if you have any problems with it. Thanks for reporting this!

[LukaHedtSV]

Hiya @rithikanarayan - one of my colleagues just did a test, and unfortunately the issue is still around.

He gave me this stack trace:

"Error: Could not extract context",
"    at extractContext (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/lambda/handler.js:74:13)",
"    at Object.get [as handler] (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/lambda/handler.js:87:21)",
"    at /opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:102:32",
"    at Array.reduce (<anonymous>)",
"    at _resolveHandler (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:101:38)",
"    at /opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:338:35",
"    at step (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:43:23)",
"    at Object.next (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:24:53)",
"    at fulfilled (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:15:58)"

My guess is that there's something wrong with our authorizer, so I'll bark up that tree for a while before I ask for any more support.

Thanks!