Skip to content

Conversation

@melugoyal
Copy link

@melugoyal melugoyal commented Jun 1, 2023

allow anonymous AWS access. this allows the consumer of any AWS operator to use the equivalent of aws s3 cp <s3 path> . --no-sign-request, to write a DAG that eg reads from a public S3 bucket without providing any credentials. to leverage this, an AWS connection can be created as such:

conn = Connection(
    conn_id="aws_demo",
    conn_type="aws",
    extra={
        "config_kwargs": {
            "signature_version": "unsigned",
        },
    },
)

this is necessary because botocore's UNSIGNED is not JSON-serializable, so using botocore.UNSIGNED when instantiating the connection will cause the connection processing to fail. instead we can use a string in the connection instantiation and convert it to botocore's expected value later. more history about this issue here: boto/botocore#2442


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jun 1, 2023
@melugoyal melugoyal force-pushed the allow-aws-anonymous-access branch from fd2bd39 to 6e6a397 Compare June 1, 2023 05:14
Copy link
Contributor

@vincbeck vincbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@jedcunningham jedcunningham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. Should we document this somehow as well?

@vincbeck
Copy link
Contributor

vincbeck commented Jun 2, 2023

Overall looks good. Should we document this somehow as well?

Good callout! I'd say yes :)

@melugoyal
Copy link
Author

@jedcunningham @vincbeck good call. added a sentence to the AWS connection docs. lmk how that looks

@o-nikolas
Copy link
Contributor

Other than the static check failures, lgtm 👍

@jedcunningham
Copy link
Member

@melugoyal I'd suggest you set up the pre-commit hooks, they help catch these issues early 👍.

@melugoyal melugoyal force-pushed the allow-aws-anonymous-access branch from 1c46e41 to 1c54aa5 Compare June 2, 2023 20:02
@jedcunningham jedcunningham merged commit 5dc6e5b into apache:main Jun 2, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 2, 2023

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@jedcunningham
Copy link
Member

Thanks @melugoyal! Congrats on your first commit 🎉

@melugoyal melugoyal deleted the allow-aws-anonymous-access branch June 2, 2023 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants