Skip to content

Conversation

@syedahsn
Copy link
Contributor

EMR launched a new feature which allows configuring runtime roles for EMR steps. This PR allows users to use this feature with the EmrAddStepsOperator. System test has been updated to use this feature.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@syedahsn syedahsn requested review from uranusjr and removed request for eladkal December 21, 2022 22:16
@uranusjr
Copy link
Member

Needs to resolve conflicts.

@syedahsn syedahsn force-pushed the syedahsn/emr-runtime-role branch from d80987f to c22326a Compare December 22, 2022 21:32
@syedahsn
Copy link
Contributor Author

Made a minor change with how the new parameter gets used in the hook. Because we are using @mock_emr decorator in some of the tests, it applies boto3 validation rules on the parameters passed, which include no None values, as well as a minimum length (otherwise I could have done a simpler ExecutionRoleArn=execution_role_arn or "").

@Taragolis
Copy link
Contributor

Made a minor change with how the new parameter gets used in the hook. Because we are using @mock_emr decorator in some of the tests, it applies boto3 validation rules on the parameters passed, which include no None values, as well as a minimum length (otherwise I could have done a simpler ExecutionRoleArn=execution_role_arn or "").

I think this originally comes from botocore service definition, example for input of AddJobFlowStepsInput.

I've also found that if set some of optional parameters set to None or empty string then botocore do not propagate it to API call but sometimes it raise an error ¯\_(ツ)_/¯

@syedahsn syedahsn force-pushed the syedahsn/emr-runtime-role branch from c22326a to 1dc0616 Compare December 23, 2022 17:06
@Taragolis Taragolis merged commit 297aa74 into apache:main Dec 23, 2022
@vandonr-amz vandonr-amz deleted the syedahsn/emr-runtime-role branch May 24, 2023 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants