Added retry to ECS Operator #14263

markhopson · 2021-02-16T19:49:40Z

This PR allows users to enable retry-behaviour for ECSOperator._start_task.

This feature was discussed briefly: here #13725

The design was lifted from hooks/base_google.py. There's 2 main differences with my AWS implementation ...

the user can configure the retry with a param quota_retry (Dict) that is passed in to the ECSOperator
the user can configure the retry without tenacity (e.g. passing in max=300 and multiplier=1 instead of tenacity.wait_exponential(multiplier=1, max=300)

boring-cyborg · 2021-02-16T19:49:45Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

airflow/www/yarn-error.log

markhopson · 2021-02-18T02:53:42Z

@turbaszek Thanks for the feedback. I also saw some checks fail (i.e. CI Build / Postgres9.6,Py3.6, CI Build / MySQL5.7, Py3.6, CI Build / Sqlite Py3.6) but I'm not sure what to do to fix them. Can you please advise me? Thanks!

markhopson · 2021-02-26T15:39:20Z

@turbaszek just a friendly bump to see what the next steps are here

ashb · 2021-02-26T15:57:30Z

airflow/providers/amazon/aws/hooks/base_aws.py

+from airflow.providers.amazon.aws.models.exceptions import ECSOperatorError
 from airflow.utils.log.logging_mixin import LoggingMixin

+ECS_QUOTA_ERROR_REASONS = [


Since these are ECS specific they shouldn't really be in the base_aws module.

Not sure what to do here. It feels weird to move this to ecs.py if I have code in base_aws.py that catches ECSOperatorError. For now, I've moved this variable into the is_permissible_error() scope to clarify intent. Any suggestions are welcomed.

airflow/providers/amazon/aws/operators/ecs.py

airflow/providers/amazon/aws/hooks/base_aws.py

ashb · 2021-02-26T16:01:16Z

airflow/providers/amazon/aws/hooks/base_aws.py

+        """
+
+        def decorator_f(self):
+            quota_retry = getattr(self, 'quota_retry', None)


Not sure about this parameter name -- it seems very specific to EcsOperator only.

Good point. Hm I've changed it to retry_args to be more generic, but I'm open to any suggestions.

airflow/providers/amazon/aws/models/exceptions.py

markhopson · 2021-03-06T02:15:20Z

@ashb Friendly bump here

ashb · 2021-03-06T10:12:07Z

Thanks, I'll take a look on Monday

ashb · 2021-03-08T09:50:08Z

airflow/providers/amazon/aws/hooks/base_aws.py

            return self.get_client_type("iam").get_role(RoleName=role)["Role"]["Arn"]

+    @staticmethod
+    def retry(fun: Callable):


If we change this to

def retry(should_retry: Callable[[Exception], bool], fun: Callable)

Then the retry_if_permissible_error can move out of BaseAWS in to ECSHook, used like this:

def should_retry(exception: Exception): """Check if exception is related to ECS resource quota (CPU, MEM).""" return isinstance(exception, ECSOperatorError) and any( quota_reason in failure['reason'] for quota_reason in ['RESOURCE:MEMORY', 'RESOURCE:CPU'] for failure in exception.failures ) ... @AwsBaseHook.retry(should_retry) def _start_task(self):

I wish I thought of this. I've made this change.

ashb · 2021-03-08T09:50:28Z

airflow/providers/amazon/aws/hooks/base_aws.py

+        limit.
+        """
+
+        def decorator_f(self):


Suggested change

def decorator_f(self):

@functools.wraps

def decorator_f(self, *args, **kwargs):

ashb · 2021-03-08T10:01:18Z

@markhopson See the review I just left -- I think that gives us a way to have the base retry functionality in the base hook, without having to put any service specific login in there. WDYT?

markhopson · 2021-03-10T02:48:48Z

@markhopson See the review I just left -- I think that gives us a way to have the base retry functionality in the base hook, without having to put any service specific login in there. WDYT?

@ashb Good tip. I like it. I've made the change.

github-actions · 2021-03-10T03:46:08Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2021-03-10T16:23:29Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

markhopson · 2021-03-15T03:25:19Z

@ashb Friendly bump.

ashb

Code looks good @markhopson

Could you fix up the static checks please?

github-actions · 2021-03-19T16:24:54Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.

ashb · 2021-03-19T22:39:56Z

TestECSOperator.test_execute_with_failures is now failing becuase the type of the exception it throws has changed -- I think that's fine, and the tests just need updating.

I haven't look in detail at the code/test though.

boring-cyborg · 2021-03-26T22:34:18Z

Awesome work, congrats on your first merged pull request!

Added retry to ECS Operator

0f23f9f

markhopson requested review from ashb and ryanahamilton as code owners February 16, 2021 19:49

boring-cyborg bot added area:webserver Webserver related Issues provider:amazon AWS/Amazon - related issues labels Feb 16, 2021

...

f2c62a0

turbaszek reviewed Feb 16, 2021

View reviewed changes

airflow/www/yarn-error.log Outdated Show resolved Hide resolved

turbaszek removed the request for review from ryanahamilton February 16, 2021 22:03

markhopson added 2 commits February 16, 2021 20:13

Remove airflow/www/yarn-error.log

a4774c8

Update decorator to not accept any params

41ad2c9

markhopson requested a review from turbaszek February 17, 2021 14:52

ashb requested changes Feb 26, 2021

View reviewed changes

markhopson added 3 commits February 27, 2021 22:45

...

0992473

...

ccfa3ed

...

2bb211e

markhopson requested a review from ashb February 28, 2021 04:01

lint

3085b21

ashb requested changes Mar 8, 2021

View reviewed changes

Add predicate argument in retry decorator

c333e7f

Add wraps and fixed test

16cb0d3

markhopson requested a review from ashb March 10, 2021 02:56

...

34a4e2b

Remove unnecessary retry_if_permissible_error and fix lint errors

dbbea3c

ashb approved these changes Mar 19, 2021

View reviewed changes

github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Mar 19, 2021

Static check fixes

eef6329

Fix TestECSOperator.test_execute_with_failures

5ea3df5

ashb approved these changes Mar 26, 2021

View reviewed changes

ashb merged commit 614be87 into apache:master Mar 26, 2021

zachliu mentioned this pull request Mar 4, 2022

retry on very specific eni provision failures #22002

Merged

	def decorator_f(self):
	@functools.wraps
	def decorator_f(self, args, *kwargs):

Added retry to ECS Operator #14263

Added retry to ECS Operator #14263

Uh oh!

Conversation

markhopson commented Feb 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boring-cyborg bot commented Feb 16, 2021

Uh oh!

Uh oh!

markhopson commented Feb 18, 2021

Uh oh!

markhopson commented Feb 26, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

markhopson commented Mar 6, 2021

Uh oh!

ashb commented Mar 6, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashb commented Mar 8, 2021

Uh oh!

markhopson commented Mar 10, 2021

Uh oh!

github-actions bot commented Mar 10, 2021

Uh oh!

github-actions bot commented Mar 10, 2021

Uh oh!

markhopson commented Mar 15, 2021

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 19, 2021

Uh oh!

ashb commented Mar 19, 2021

Uh oh!

boring-cyborg bot commented Mar 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

markhopson commented Feb 16, 2021 •

edited

Loading