Parameter invalid (cluster) in amazon-> operator -> EcsRunTaskOperator when using deferrable

[frantoja]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

No response

Apache Airflow version

2.8.1

Operating System

AWS MWAA

Deployment

Amazon (AWS) MWAA

Deployment details

No response

What happened

I have a custom class inheritance from EcsRunTaskOperator.
When I set deferrable = True, the Airflow task runs and changes to the deferred state successfully and also the ECS task runs successfully. However, when it returns (TaskDoneTrigger - task finished), the method execute_completes -> self._after_execution() -> self._check_success_task() -> response = self.client.describe_tasks(cluster=self.cluster, task=[self. arn]) shows an error:

botocore.exceptions.ParamValidationError: Parameter validation failed
Invalid type for parameter cluster, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

What you think should happen instead

I believe (and tested) that the variable self.cluster (approximately line 696 in the file airflow/providers/amazon/aws/operators/ecs.py) "looses the value"

How to reproduce

Set the parameter deferrable to true in EcsRunTaskOperator

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[vincbeck]

Indeed, the operator EcsRunTaskOperator use some variables that might not have been set when executing execute_complete. Ideally execute_complete should only use information from the event, the executor/worker running execute and execute_complete might (and most likely) be different

[ellisms]

I'll take it.