Make gantt view to show also retries

[alexInhert]

Description

Currently Gantt View shows only the last successful task retry.
This means that if we have t1 >> t2 >> t3
if we had:
t1 started on 08:00 finished 09:00
t2 1st retry started 09:01 failed on 13:00
t2 2nd retry started on 13:02 finished on 15:00
t3 started on 15:01 finished on 15:02

What the gantt view will show us is
t1 08:00-09:00
t2 13:02-15:00
t3 15:01-15:02

This makes it really hard to understand what really happen because it doesn't show what really happened.
This is very annoying for DAGs with so many tasks because you can't tell if the problem is due to tasks not being scheduled or this is something else. I understand the value of the current display but maybe it can add some toggle to consider also all the task instances?

Use case / motivation

The gantt somehow should show the actual information.

Are you willing to submit a PR?

Don't have the skills for that.

[bbovenzi]

I've been thinking about how to make the gantt view more useful (I think it has a lot of potential). Showing the full state history of a task instance would be a great part of that.

[ShakaibKhan]

I would like to try implementing this

[bbovenzi]

Sweet. Just assigned you. Feel free to reach out if you need anything!

[ShakaibKhan]

A little confused as to what is being asked for as it look like task retries are shown in the gantt view. So is the focus of this issue to have a toggle to choose which task history to display?

[alexInhert]

@ShakaibKhan @bbovenzi see this example

As you can see there is significant gap between the tasks. When you hover over the task you see it is the 2nd retry. If going into the log it shown that the first retry run for the duration of the purple rectangle.
My request is to add the purple rectangle. I want to see also the retries in the gantt without it - the Gantt just looks like the scheduler didn't schedule anything during this time which is incorrect view of what really happened.
I think the current setup is useful I'm asking for a way to enrich it also with the retries of the tasks to show the full picture.

[ShakaibKhan]

@alexInhert make sense to me and agree that retries should be shown; could you please share the DAG you used to make the above gantt chart

[ShakaibKhan]

Sorry I do not think I can take on this issue at this time. Unassigning to myself

[yxiao1996]

Reproduce the scenario described in this issue with a python operator which causes failed task at 50% chance

@task(task_id=f"sleep_and_randomly_fail", retries=1)
def my_sleeping_function():
    time.sleep(5)
    if random.uniform(0, 1) > 0.5:
        raise ValueError("Mock Exception.")

From the DAG run results it seems to me this issue is already addressed, as in the following snapshot we can see both failed 1st run and successful 2nd run in gantt chart. Under the hood, reruns of a tasks are stored as different task instances.

[bbovenzi]

Yes, we record Task Fails should be included in the Gantt chart. We also record Task Reschedules which would probably be helpful. But I think that's the most we could do with the existing tables in Airflow.