Add ability to use AWS Cloud-based Authentication within the agent config

[wynbennett]

What does this PR do?

This adds a new authentication method for the agent. This give the agent the ability to exchange a AWS Cloud Auth Proof for an API key which is automatically managed and rotated on behalf of the customer. This is essentially extending the https://docs.datadoghq.com/account_management/cloud_provider_authentication into the agent.

This PR integrates the following PR into the config #46272

The flow is as so:

• Agent get AWS credentials from the environment the agent is running in
• Agent signs a request to AWS which will prove it's access to the credentials
• Agent passed the signed request to Datadog service
• Service validates the signed request against AWS and validates it against Datadog configuration
• Service response with a Managed and automatically rotated API key
• Agent propagates that API key throughout it's config
• If the delegated auth flow fails it will fallback to using the API key provided in the yaml file. The allows customers to onboard with limited risk to their current flow.

Configuration

Global provider config + per-key-prefix enablement:

# Global delegated auth provider settings (used for all instances)
delegated_auth:
  org_uuid: ${YOUR_ORG_UUID}
  # Optional: explicitly specify provider (auto-detects if omitted)
  provider: aws
  aws:
    region: us-east-1  # Optional: auto-detects from IMDS if omitted

# Per-prefix enablement (logs can use a different org)
logs_config:
  delegated_auth:
    org_uuid: ${LOGS_ORG_UUID}  # Can be different org

Motivation

This should enable customers to not need to manage the API used by the agent and instead use the AWS credentials in the AWS environment the agent is deploy to.

Describe how you validated your changes

Ran the agent locally to validate changes and deployed to multiple staging clusters charcadet and entei. I have deployed it to staging clusters with both the feature enabled and the feature disabled.

I have marked this a something for RC testing as it is a large shift in how we can authenticate so it seems like it is more RC related.

To test this see the above YAML settings to turn on this method of getting an API key. You will need to ensure the AWS account you are authenticating with is allowed by our configuration. Please reach out to allow us to enable any AWS ARNs to use cloud auth.

Additional Notes

[vickenty]

Looks like this is added everywhere. Would it make sense to add this to the core bundle?

[wynbennett]

It is already part of core.Bundle() see https://github.com/DataDog/datadog-agent/blob/wyn.bennett/delegated_auth_integration/comp/core/bundle.go#L70-L78. When core.Bundle(core.WithSecrets()) is used, both secretsfx.Module() and delegatedauthfx.Module() are injected together.

However, there are a few entry points that don't use core.Bundle() and instead call secretsfx.Module() directly:

• cmd/trace-agent/subcommands/run/command.go
• cmd/serverless-init/main.go
• cmd/dogstatsd/subcommands/start/command.go

For these cases, delegatedauthfx.Module() must be added explicitly alongside secretsfx.Module()

[pgimalac]

This is something we'll want to improve later, I thought all commands used core.Bundle tbh

[pgimalac]

Some nitpicks but LGTM !

[pgimalac]

This is something we'll want to improve later, I thought all commands used core.Bundle tbh

[pgimalac]

This command doesn't use secrets so I don't think it needs delegated auth

[wynbennett]

Fixed by making it use delegatedauthnoopfx.Module(), but because the delegatedauth.Component is non-optional we do need to actually include the noop module.

[pgimalac]

Could we just use a sync map here ?

[wynbennett]

sync.Map was considered but not used because the map is small, writes happen during initialization, and we need maps.Clone() for bulk reads which sync.Map doesn't support efficiently.

[L3n41c]

LGTM for the file owned by @DataDog/container-integrations.