Try number inconsistency between webserver and the actual log generated

[dheerajturaga]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.10.2

What happened?

Some tasks in airflow 2.10.2 are being launched with try number 0. However, the webserver is looking for logs starting with try number 1. Because of this we are having cases where tasks are running and passing however, the server throws a "cannot read served logs" error.

Executor=CeleryExecutor

What you think should happen instead?

Airflow should start attempt with try number 1 instead.

How to reproduce

This is happening intermittently. Not all tasks start with try number 0. I have checked to make sure both the webserver and the celery worker are on the same version 2.10.2

Operating System

SUSE Linux Enterprise Server 12 SP5

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.8.1
celery==5.4.0

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jscheffl]

I am not sure whether this is really a bug, might be rather a semantic gap or something that could be (visually) improved to just hide logs when no logs are there.

In Airflow 2.10 the internal semantic was changed to increment the try number of a task only at the time the task is really scheduled/started. As long as it is just there as instance (not scheduled) the try=0 is the sign for it has never tried. In this case the visual improvement might be to hide the log panel as no logs are existing.

Would you like to submit a small improvement?

[dheerajturaga]

@jscheffl, thats not really the case. Here the task has started and completed, and the log is captured under attempt-0.log. However, the webserver is trying to look for attempt-1.log
it appears that there are cases where the try number isn't incremented on the log side even when the task has started.

du -sh attempt-0.log 64K attempt-0.log

[2024-09-25T14:24:04.772+0000] {taskinstance.py:1402} DEBUG - previous_execution_date was called
[2024-09-25T14:24:04.822+0000] {taskinstance.py:1402} DEBUG - previous_execution_date was called
[2024-09-25T14:24:04.838+0000] {local_task_job_runner.py:123} INFO - ::group::Pre task execution logs
[2024-09-25T14:24:04.854+0000] {taskinstance.py:2612} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: [queued]>
[2024-09-25T14:24:04.867+0000] {taskinstance.py:2612} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: scheduled__2024-09-25T05:00:00+00:00 [queued]>
[2024-09-25T14:24:04.868+0000] {taskinstance.py:2865} INFO - Starting attempt 0 of 1