Unable to view trigger log in real-time in Airflow UI when task is deferred #44683
Description
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.6.3
What happened?
when task is deferred, the webserver UI is unable to read trigger log:
*** Could not read served logs: Client error '404 NOT FOUND' for url 'http://<triggerer hostname>:8794/log/dag_id=xxx/run_id=xxx/task_id=xxx/attempt=18.log.trigger.3509.log'
after task is completed, the above error message will disappear, it says the trigger log is found on s3 (we enable remote logging):
*** Found logs in s3: *** * s3://<bucket>/path_to_log/attempt=18.log *** * s3://<bucket>/path_to_log/attempt=18.log.trigger.3509.log
and the content of the trigger log is displayed with the task log in the UI
[2024-12-05, 07:34:00 UTC] {triggerer_job_runner.py:616} INFO - Trigger xxx/-1/18 (ID 160) fired: TriggerEvent<{'state': 'SUCCESS', 'message': None, 'extra_data': {...}}>
When task is deferred I exec into the triggerer pod, I can't find the expected xxx/attempt=18.log.trigger.3509.log file in log folder; while after the task completes, I can find this log file in the folder.
What you think should happen instead?
while task is deferred, I expect to view the corresponding trigger log in real-time in the UI, like the video here
How to reproduce
run a DAG with deferrable task, and view the task log when it is deferred
Operating System
Ubuntu
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.1.0
apache-airflow-providers-celery==3.2.0
apache-airflow-providers-cncf-kubernetes==7.0.0
apache-airflow-providers-common-sql==1.5.1
apache-airflow-providers-docker==3.7.0
Deployment
Official Apache Airflow Helm Chart
Deployment details
triggerer pod has open 8794 port, and has a service listening on that port
triggerer ClusterIP None <none> 8794/TCP 22d
and webserver secret key has configured properly.
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct