"Error decrypting key" when upgrading to recent versions of extension

[janfranco27]

My team is installing the datadog lambda extension in a container image, and we have noticed an issue that is causing us some trouble when deploying it to AWS Lambda.
We have the DD_API_KEY_SECRET_ARN environment variable set to the arn of a secret manager were the datadog key is stored.
Previously we were using version v53 of the extension, and datadog-lambda == 5.88.0 and everything was working as expected.
However, recently we decided to bump the lambda extension version and the python datadog lambda library as well.

We have done so to v72 and 6.106.0 respectively, but now, after deploying, the following error is captured in Cloudwatch:

DD_EXTENSION \| ERROR \| Error decrypting key: {     "Message": "User: arn:aws:sts::<accountId>:assumed-role/core-service-develop-us-east-1-lambdaRole/core-service-develop-core is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-west-2:<accountId>:secret:uptogether/datadog/api_key-4QC7nT because no identity-based policy allows the secretsmanager:GetSecretValue action",     "__type": "AccessDeniedException" }
--
DD_EXTENSION \| ERROR \| Failed to resolve secrets, Datadog extension will be idle

The GetSecretValue permission is granted (otherwise it would not work in the past as well), but after the update, datadog is not able to capture logs.

Additional notes

• We have tested the same by using v69 and datadog-lambda == 6.105.0 but again, the same error is captured.
• I was able to upgrade to v66 and datadog-lambda == 6.102.0 and the logs were captured as expected, so maybe this is an issue related to the Next Generation of Datadog Lambda Extension?

Additional question

I would also like to take the opportunity to ask something about the documentation that is not fully clear (not sure if this is the right place, but if not, I would appreciate if someone can point me out to the right place)
The documentation in Installation steps for Container Images states:

Note that the minor version of the datadog-lambda package always matches the layer version. For example, datadog-lambda v0.5.0 matches the content of layer version 5.

However, it seems that the layer version is not longer matching the datadog-lambda-python package. (For instance, current version of the extension is 73 and the most recent python version is v6.106.0
Is there a guide or something similar to know which version should be used in each case?
As of now, I have been trying to use versions that were released in similar days, assuming they were meant to go together, but I can not ensure this is true. I would appreciate some guidance here.

[duncanista]

Hey @janfranco27,

I can answer your second question right away.

The v73 for this repository is for the Datadog Lambda Extension, as opposed to the version v6.106.0 which corresponds to the layer version for the Datadog Lambda Library, which is provided as a package/layer for multiple runtimes, in this case, you're using the one for Python.

Since you're using a container, it looks like you have installed datadog-lambda (or datadog-lambda-python) as a package, as opposed to a layer.

You can see in the releases of v6.106.0 that there are layers specified with version 106, for customers who use it as a layer – mostly those with managed runtime.

Let me know if you have any other questions regarding this.

I'll get back to you soon with the response for the issue.

[duncanista]

Me again 👋🏽

Normally, that issue would indicate that the AWS Lambda is missing permissions to read the secret ARN resource, but as you mentioned, this Lambda should have those permissions as you have stated.

I've seen in some cases that one just has to wait or force a cold start for the permissions to work as expected – an AWS mistery if you ask me. Could you report back if the issue persist after forcing out a cold start (by, let's say updating an env var or a new deploy)?

It's been almost 2 years of updates since v53 was released, and v67 introduced the breaking change of booting into our Next Generation Lambda Extension, yet we haven't had any incidents of the Lambda not being able to access secret resources.
Our internal Lambdas using secrets manager and our latest versions work and send data as we would expected.

I'd suggest using v73 (latest) and check if this issue persist.
This shouldn't have any issues with whichever datadog-lambda (datadog-lambda-python) package you use.

I'll jump back into this issue next week!
Have a great weekend!

[janfranco27]

Hi @duncanista
Thanks for the answer 🙏
just wanted to mention that I have tested again and updated the custom docker image we use so it includes the lambda extension v73 as suggested.
Every update triggers a new deployment. Also, just to be sure, I have waited some time to hit our lambda again so it is triggered via a cold start. In both cases, I get the same error message from the DD extension

DD_EXTENSION | ERROR | Error decrypting key: 
{
    "Message": "User: arn:aws:sts::<accountid>:assumed-role/core-service-develop-us-east-1-lambdaRole/core-service-develop-authorize is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-west-2:<accountid>:secret:uptogether/datadog/api_key-4QC7nT because no identity-based policy allows the secretsmanager:GetSecretValue action",
    "__type": "AccessDeniedException"
}

Let me know if there is something else needed or some other suggestion I could try to see if the issue is finally solved.

[duncanista]

Hey @janfranco27,

Thanks for testing again, I'll make sure to try to reproduce your scenario. Let me know if I'm missing anything from here:

1. Python Runtime in a container – is this custom image or provided Amazon Linux 2023? If it's a custom image, could you share non sensitive information about it?

custom docker image

2. Secret ARN
3. v73

I'll get an update tomorrow – thanks!

[janfranco27]

Hi @duncanista , thanks for your assistance here. Some extra details:

we use python 3.10, so the base OS that we get from AWS is Amazon Linux 2 (Amazon Linux 2023 is available starting python 3.12 and unfortunately we can not make that migration yet)

Here I left a simplified version of our dockerfile:

ARG DATADOG_EXTENSION_VERSION=73
ARG DATADOG_VERSION=6.106.0
ARG PYTHON_VERSION=3.10
ARG LAMBDA_IMAGE_TAG=${PYTHON_VERSION}
ARG POETRY_VERSION=2.1.1

FROM public.ecr.aws/datadog/lambda-extension:${DATADOG_EXTENSION_VERSION} AS dd_extension
FROM public.ecr.aws/lambda/python:$LAMBDA_IMAGE_TAG AS lambda_base

# Stage for building the project dependencies
FROM lambda_base AS builder
ARG POETRY_VERSION

## install OS packages
RUN yum groupinstall -y "Development Tools"
## Setup known hosts
RUN mkdir -p /root/.ssh && ssh-keyscan github.com >> /root/.ssh/known_hosts
RUN pip install poetry==${POETRY_VERSION}
WORKDIR /app
ENV POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    POETRY_CACHE_DIR=/tmp/poetry_cache

COPY pyproject.toml poetry.lock ./


FROM builder AS deps
RUN --mount=type=cache,target=${POETRY_CACHE_DIR} \
    --mount=type=ssh \
    poetry install --no-root

# Container for collecting all source files
FROM lambda_base AS src
COPY . /app
WORKDIR /app

# Base stage for runtime images
FROM lambda_base AS lambda_runtime_base
ARG PYTHON_VERSION
ARG DATADOG_VERSION

# copy datadog extension
COPY --from=dd_extension /opt/. /opt/
# install global packages
RUN pip install "ddtrace==2.21.3" datadog-lambda==${DATADOG_VERSION}

ENV PYTHONPATH=${LAMBDA_TASK_ROOT}/.venv/lib/python${PYTHON_VERSION}/site-packages
ENV AWS_LAMBDA_LOG_LEVEL="INFO"
ENV DD_SITE="datadoghq.com"
ENV DD_LAMBDA_HANDLER="lambda_handler.app"
ENV DD_LOG_LEVEL="INFO"
ENV DD_MERGE_XRAY_TRACES=true
ENV DD_SERVICE="service.core"
ENV DD_TRACE_OTEL_ENABLED=false
ENV DD_PROFILING_ENABLED=false
ENV DD_SERVERLESS_APPSEC_ENABLED=false
ENV DD_API_KEY_SECRET_ARN="arn of secret manager goes here"

# Use datadog wrapper
CMD [ "datadog_lambda.handler.handler" ]

# Main runtime image
FROM lambda_runtime_base AS main_lambda_runtime

COPY --from=deps /app/.venv ${LAMBDA_TASK_ROOT}/.venv
COPY --from=src /app ${LAMBDA_TASK_ROOT}

You will notice that DATADOG_EXTENSION_VERSION=73 and we also set the DD_API_KEY_SECRET_ARN

Happy to give you more details in case you need them.

[astuyve]

Hey @janfranco27! I'm taking over for Jordan on support this week.

I've attempted to reproduce this in us-west-2 and I'm unable to with the following setup:
Latest layers (extension v73 and python v106), with DD_API_KEY_SECRET_ARN as so

and the following permissions:

Everything executes okay and I see data in datadog:

Initially this leads me to suspect some kind of setup/configuration issue. Can you open a support ticket with us so we can troubleshoot this further? You can use the live chat feature of the app or just email support@datadoghq.com

Please include a link to this discussion and ask the support person to escalate the ticket to me so that we can move things along swiftly.

Thanks!
AJ

[janfranco27]

Hi @astuyve
I have opened a ticket with number #2068752 and requested to escalate it to you. Appreciate your help!

[astuyve]

Looks like we tracked this down to a cross-region secrets manager secret. I'll keep users posted here when we're able to patch this and get the fix out.

Thanks!

[astuyve]

@janfranco27 will be fixed in v74!

[astuyve]

Fixed in v74