DynamoDBToS3Operator using native export functionality.

[Kuhlmann-Itagyba-bah]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.20.0

Apache Airflow version

2.9.1

Operating System

Debian GNU/Linux 12

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

DynamoDBToS3Operator does not offer the possibility to use DynamoDB Export native functionality; hence, it scans the entire table, requiring the airflow executor role to actually have access to data, which is potentially dangerous. Also, it does not allow incremental export, which can be used with boto3.

What you think should happen instead

It should add the possibility of using the AWS native export functionality, allowing the Airflow role to execute the data transfer without needing access to the data itself.

How to reproduce

Implement any DynamoDBToS3Operator task without "dynamodb:Scan" permission on the giving table.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jayceslesar]

Is this ask possible? The only method in the dynamo boto3 client listed that seems relevant is export_table_to_point_in_time which the existing codebase already supports

airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py

Line 156 in 6b92145

def _export_table_to_point_in_time(self):

.

Doesn't seem that a native export is possible without PITR enabled.

The only thing the existing Airflow implementation/passthrough seems to be missing are two (optional) arguments:

1. ExportType
2. IncrementalExportSpecification

[gyli]

I agree with @jayceslesar that ExportTableToPointInTime is the only API provided in boto to export DynamoDB data to S3, based on AWS's blog introducing this native export.

@Kuhlmann-Itagyba-bah You may try DynamoDBToS3Operator with argument export_time specified, so it will use _export_table_to_point_in_time method for export.

[Ghoul-SSZ]

Hello. I have also ran into this issue this week and have made a PR about it 👆 .
Please take a look if you have some time to spare. 🙏
Since its my first time making a PR here, there might be things or edge cases I haven't thought through.
In that case, sorry in advance 🙇

Any helps or suggestions are more than welcome. 😄