-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Add a task instance dependency for mapped dependencies #37498
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a task instance dependency for mapped dependencies #37498
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
4bd1303 to
c2e94c2
Compare
|
@stevenschaerer can you address the comments and rebase ? |
adbc13a to
325eace
Compare
|
@uranusjr Could you please review my latest changes and responses to your comments? Thanks! |
a6b93e7 to
dc90eb3
Compare
dc90eb3 to
ab8da53
Compare
42a694d to
2f69980
Compare
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
…om the base operator deps
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
86dbbe7 to
e4031b3
Compare
This closes #37091.
In particular, dynamically mapped tasks currently show as
failed_upstreaminstead ofskippedif some of the tasks that generate the mapping are skipped and the trigger rule isnone_failed,all_doneor similar.This is because when trying to expand the mapped task, a failed and skipped mapped dependency behave identically and both result in a NotFullyPopulated being raised.
This PR introduces a new task instance dependency that sets a dynamically mapped task to
skippedif any of the mapped dependencies are skipped (but not failed) and toupstream_failedif any of the mapped dependencies are failed. As such, an expansion of the mapped task is never attempted.Before (from #37091):

After:
