Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Oct 23, 2024

  • fix schedule_downstream_tasks bug

  • remove partial_subset

  • Update comment


Co-authored-by: 维湘 jiazhao.ljz@alibaba-inc.com
(cherry picked from commit 3fceaa6)


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

* fix schedule_downstream_tasks bug

* remove partial_subset

* Update comment

---------

Co-authored-by: 维湘 <jiazhao.ljz@alibaba-inc.com>
(cherry picked from commit 3fceaa6)
@potiuk potiuk requested review from XD-DENG, ashb and kaxil as code owners October 23, 2024 08:37
@potiuk
Copy link
Member Author

potiuk commented Oct 23, 2024

This is backport of #42582

Since we are closing to 2.10.3 I merged #42582 as I believe it solves often occuring "serialization" issues (with partial_subset) that happen in "mini-scheduler" and causes failures of celery workers - and tests had shown that partial_subset are also wrong when it comes to determining eligibility of dependent tasks in some cases.

@uranusjr and @ashb -> if you have any concerns about it this is the time to raise it before we merge - I think you were mostly those who knew most about using partial_subset and mini-scheduler so if you have any doubts whether we should release it in 2.10.3 - please comment. Details and discussion in #42582 and in the comments in this PR.

@potiuk
Copy link
Member Author

potiuk commented Oct 23, 2024

The failure here should be fixed by #43307

@potiuk
Copy link
Member Author

potiuk commented Oct 23, 2024

Failing tests fixed in tip of 2-10-test merging.

@potiuk potiuk merged commit 1e498d8 into apache:v2-10-test Oct 23, 2024
@potiuk potiuk deleted the remove-partial-cherry-pick branch October 23, 2024 14:44
@utkarsharma2 utkarsharma2 added the type:bug-fix Changelog: Bug Fixes label Oct 24, 2024
@utkarsharma2 utkarsharma2 added this to the Airflow 2.10.3 milestone Oct 24, 2024
utkarsharma2 pushed a commit that referenced this pull request Oct 24, 2024
* fix schedule_downstream_tasks bug

* remove partial_subset

* Update comment

---------

Co-authored-by: 维湘 <jiazhao.ljz@alibaba-inc.com>
(cherry picked from commit 3fceaa6)

Co-authored-by: luoyuliuyin <luoyuliuyin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants