EmrContainerSensor On aws Using deferrable=True get exception #35882
Description
Apache Airflow version
2.7.3
What happened
when the trigger is running the execute_complete and the status is not success getting error
def execute_complete(self, context, event=None):
if event["status"] != "success":
raise AirflowException(f"Error while running job: {event}")
else:
self.log.info(event["message"])
ip-10-5-55-118.eu-central-1.compute.internal
*** Reading remote log from Cloudwatch log_group: airflow-tf-stg-euc1-airflow-analytics-engine-Task log_stream: dag_id=execute_model/run_id=manual__2023-11-26T12_53_43.787114+00_00/task_id=submit_emr_task/attempt=1.log.
[2023-11-26, 12:58:32 UTC] {{taskinstance.py:1937}} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 1606, in resume_execution
return execute_callable(context)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/operators/emr.py", line 618, in execute_complete
self.log.info("%s", event["message"])
~~~~~^^^^^^^^^^^
KeyError: 'message'
[2023-11-26, 12:58:32 UTC] {{standard_task_runner.py:104}} ERROR - Failed to execute job 2858894 for task submit_emr_task ('message'; 2194)
What you think should happen instead
not getting the rror
How to reproduce
run EmrContainerSensor with deferrable true on aws
Operating System
aws
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.4.0
apache-airflow-providers-apache-kafka==1.2.0
apache-airflow-providers-common-sql==1.6.1
apache-airflow-providers-ftp==3.4.2
apache-airflow-providers-http==4.5.0
apache-airflow-providers-imap==3.2.2
apache-airflow-providers-postgres==5.6.0
apache-airflow-providers-sqlite==3.4.3
Deployment
Amazon (AWS) MWAA
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct