Display selected task outputs (xcom) in task UI

[seeholza]

Description

I often find myself checking the stats of a passed task, e.g. "inserted 3 new rows" or "discovered 4 new files" in the task logs. It would be very handy to see these on the UI directly, as part of the task details or elsewhere.

One idea would be to choose in the Task definition, which XCOM keys should be output in the task details, like so:

Use case/motivation

As a developer, I want to better monitor the results of my tasks in terms of key metrics, so I can see the data processed by them. While for production, this can be achieved by forwarding/outputting metrics to other systems, like notification hooks, or ingesting them into e.g. grafana, I would like to do this already in AirFlow to a certain extent. This would certainly cut down on my clicks while running beta DAGs.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[eladkal]

I'm not sure what is the request here?
There is Xcom tab.

[seeholza]

@eladkal indeed, this would be similar to the display of the xcom, but selecting some of the information to be exposed in the UI separately. My proposed solution above would duplicate the xcom tab information at a similar level of tab, so maybe that's not the route to go.

I was looking for a way to "expose a high level digest of task results in the UI" to make these available at a glance, instead of hiding them a few levels down in xcom.

[bbovenzi]

We're trying to bring more at-a-glance info into the grid view, like logs. This would be a good one too.

[pierrejeambrun]

Continuing our work to modernize the UI, I think adding a new xcom tab to task details to display xcom entries would be a good idea. I believe we already have an endpoint for retrieving XCOM entries.

[bbovenzi]

If anyone wants to pick it up:

• we can add an "Xcoms" tab like the details/logs tabs we have
• we can also use the list xcoms, and get xcom entry rest API endpoints

It should be relatively straightforward to follow how the rest of our API requests and react pages work.

[parth166]

Hi!

I am new to airflow and would like to contribute. I just completed the setup of my environment. Was wondering if I can pick this up?

Thanks.

If anyone wants to pick it up:

• we can add an "Xcoms" tab like the details/logs tabs we have
• we can also use the list xcoms, and get xcom entry rest API endpoints

It should be relatively straightforward to follow how the rest of our API requests and react pages work.

[potiuk]

sure.

[parth166]

Hey @potiuk I haven't been able to work on this since I was trying to work on a different issue and I am only a beginner in airflow. Can you assign this to someone else and perhaps later I can follow the resolution of this issue?

[bbovenzi]

@parth166 Unassigned you. But are you on the Airflow Slack? Reach out to me there and I'll help you get started!