Authentication to AWS using GCP credentials

[mik-laj]

ÀHello,

We can use OpenID Connect Federation in AWS Security Token Service to be able to use Google identity credentials to access AWS services.
If we obtain identity credentials using the Google metaserver, we will not have to use any long-lived authorization data. This will increase security. In case of leakage of temporary credentials, their use is limited.

This sample shows how to make authenticated calls to AWS APIs from Google Cloud

This is a sample flow:

Requirements:

1. The flow shows the usage of the S3 service, but it should be possible to use any AWS service.
2. The flow shows how to get a token using the metaserver, but we should also support other methods of getting ID tokens.
   a) service account in connection (path/dict)
   b) environment variable
   c) gcloud tool credentials (service account/stored authorized user credential)
   d) meta server
3. Tokens should be automatically refreshed in both libraries - boto and google-auth

I would also be happy if we had comprehensive documentation. Together with terraform scripts at best. This is a feature that is not described often because AWS doesn't want to describe GCP services and AWS doesn't want to describe Google services. The exact example certainly helped to understand this feature.

To implement token refresh for AWS, this article can help.
https://dev.to/li_chastina/auto-refresh-aws-tokens-using-iam-role-and-boto3-2cjf
In the case of GCP, the problem will be many methods of creating a token, but I will be happy to share the code if anyone wants to implement it.

More information:
https://github.com/shrikant0013/gcp-aws-webidentityfederation
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role_with_web_identity

CC: @ad-m

[ad-m]

I don't have full knowledge of Airflow environment in Kubernetes, but can we use Google Kubernetes Service Account Token instead of Google ID Token? Then there is no limitation to the use of the Google environment. From the AWS SDK perspective, integration is limited to setting two environment variables:

• AWS_WEB_IDENTITY_TOKEN_FILE - path to file with Id Token
• AWS_ROLE_ARN - target identification.

In AWS_WEB_IDENTITY_TOKEN_FILE, you can supply the Service Account Kubernetes token from the user's Kubernetes cluster exactly as the AWS webhook does.

[mik-laj]

@ad-m Not everyone uses Kubernetes on GCP. On the other hand, we can add more a well-known external OpenID Connect (OIDC)-compatible IdP in the nest step. Our GCP user community is very active and likes to create new solutions that can then be expanded with independent solutions. However, someone has to pay tax of the new feature.

If you are interested in how credentials for AWS are created then you can have a look here:
https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/base_aws.py#L87
In most cases, this means reading the configuration dictionary(connection_object.extra_dejson) and creating a credential object:

@olchas I know you work on Impersonation in Google integration. This is a similar concept, but we are not changing identity from Google to Google, but from Google to AWS. Is it interesting for you?

@potiuk @amithmathew What do you think about this ticket?

[olchas]

@mik-laj sounds interesting. I could take a look at that once I finish working on impersonation in GCP.