Skip to content

Conversation

@guan404ming
Copy link
Member

@guan404ming guan404ming commented Nov 7, 2025

Related Issue

Closes #50873

Why

  • The current asset graph doesn’t display data dependencies.

How

  • add a toggle to switch between scheduling and data lineage gragh
demo.mov

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:translations area:UI Related to UI/UX. For Frontend Developers. translation:default labels Nov 7, 2025
@guan404ming guan404ming marked this pull request as ready for review November 8, 2025 03:25
@guan404ming guan404ming force-pushed the update-asset-graph branch 2 times, most recently from d20c28d to 5d2b047 Compare November 9, 2025 17:12
@bbovenzi
Copy link
Contributor

Wait, so is this to show the tasks inside the Dag instead?

@guan404ming
Copy link
Member Author

guan404ming commented Nov 21, 2025

Yes, that’s my current understanding. Based on the discussion in the issue, it seems the intention is to show the tasks inside the dag. But I’m not completely certain, so I’m happy to adjust the implementation if needed.

@pierrejeambrun pierrejeambrun added this to the Airflow 3.2.0 milestone Dec 2, 2025
Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are mixing two things and it will make the graph even more complicated for users.

  • On one side we have the 'dependencies' from logical execution flow / scheduling
  • On the other side we have data dependencies / lineage (which task is reading which asset event)

I'm not sure if baking those two pieces inside the same graph is a good idea. I believe it will make things hard to understand.

I assume the toggle is here for that.

Also the asset graph is really at the dag level I think. On the Dag graph on the other hand, we do have task level resolution. #51401, same for asset aliases.

Instead of having task level vs graph level, it sould be 'scheduling / execution flow' vs data dependencies

@guan404ming guan404ming changed the title Add task dependency graph to AssetGraph Add data lineage graph to AssetGraph Dec 9, 2025
@guan404ming
Copy link
Member Author

guan404ming commented Dec 9, 2025

I've updated the implementation to split them into two views and add a button to switch between two different graph

Screen.Recording.2025-12-09.at.8.06.22.PM.mov

Copy link
Member

@jason810496 jason810496 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.
It's really nice to see this cool feature!

Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. I'm not sure we want to call that 'lineage'. That could be 'task dependencies' or something else.

I don't want people to expect a full lineage solution, while this is for now limited to "producing/consuming task dependencies"

Also for asset alias, we might need to double check. If I run asset_alias_example_alias_producer, I see:

Screenshot 2026-01-14 at 16 27 26

But I don't see any consuming tasks, even after running asset_alias_example_alias_consumer. But i'm not sure if we do have the information at this point.

@guan404ming guan404ming marked this pull request as draft January 17, 2026 18:27
@guan404ming guan404ming changed the title Add data lineage graph to AssetGraph Add producing/consuming task dependencies to AssetGraph Jan 19, 2026
@guan404ming guan404ming force-pushed the update-asset-graph branch 2 times, most recently from 8fe8358 to 0dc4ba8 Compare January 19, 2026 04:25
@guan404ming guan404ming marked this pull request as ready for review January 19, 2026 08:27
@guan404ming
Copy link
Member Author

guan404ming commented Jan 19, 2026

But I don't see any consuming tasks, even after running asset_alias_example_alias_consumer. But i'm not sure if we do have the information at this point.

You're right. The data model doesn't currently store task-to-asset-alias relationships (only TaskInletAssetReference / TaskOutletAssetReference for actual assets). This would need new tables or querying serialized DAG data. Should we bring these in in this pr or in a follow up?

Nit. I'm not sure we want to call that 'lineage'. That could be 'task dependencies' or something else.

Make sense to me. I've updated to use the task dependencies wording.

image

@guan404ming guan404ming force-pushed the update-asset-graph branch 4 times, most recently from 671d7b1 to bf03b37 Compare January 26, 2026 07:11
Copy link
Contributor

@bbovenzi bbovenzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking much better.

In another PR, I think we should update the assets list and asset details pages to add scheduled as a column in the table or header card.

@guan404ming
Copy link
Member Author

In another PR, I think we should update the assets list and asset details pages to add scheduled as a column in the table or header card.

Make sense to me. I could do that in a follow up.

Copy link
Member

@pierrejeambrun pierrejeambrun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, we can improve with follow up PRs as suggested, I think it's a good base.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translations area:UI Related to UI/UX. For Frontend Developers. translation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Assets do not show up as consumed

4 participants