LatestOnly operator should not compare to now, but to the DagRun dates
#49544
Conversation
Comparing to now means that if you clear a dagrun and it re-runs it will always show up as latest! Additionally remove the check/don't skip logic whe the data interval is zero-wdith. Even if a DAG doesn't have the concept of a data-interval (i.e. it is zero width) it still is logically consistent for it to have to concept of latest or not, so we now only compare against the end date of the interval. (And a few drive-by refactors too, `context["task"]` is `self`, `context["dag"]` is `self.dag`)
| return list(context["task"].get_direct_relative_ids(upstream=False)) | ||
| return list(self.get_direct_relative_ids(upstream=False)) | ||
|
|
||
| if not left_window < now <= right_window: |
There was a problem hiding this comment.
The change looks correct, but I have no idea why now was ever used
There was a problem hiding this comment.
Looks like it has been such since last 9 years.. So we should double-verify around the expectation. Maybe the test might help with that:;
There was a problem hiding this comment.
Oh. Dear. God.
Has this operator ever actually been useful? Is this change now actually a breaking change then?
There was a problem hiding this comment.
I'm not actually convinced the discussion in the original PR is right. I think Sid is confusing dag.start_date with dag_run.start_date or maybe task_instance.start_date.
Either way, I'll park this conversation.
|
I'm going to close this -- the switch away from |
2 participants
Comparing to now means that if you clear a dagrun and it re-runs it will
always show up as latest!
Additionally remove the check/don't skip logic whe the data interval is
zero-wdith. Even if a DAG doesn't have the concept of a data-interval (i.e. it
is zero width) it still is logically consistent for it to have to concept of
latest or not, so we now only compare against the end date of the interval.
(And a few drive-by refactors too,
context["task"]isself,context["dag"]isself.dag)Reported by @jscheffl here #48945 (comment)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.