Set logical_date and data_interval to None for asset-triggered dags and forbid them to be accessed in context/template #46460
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
the
area:Scheduler
Member
Author
Lee-W
force-pushed
the
none-as-logical-date-and-data-interval-for-asset-triggered-dag
branch
2 times, most recently
from
77e97e1 to
82081f9
Compare
Member
Author
|
Functionality-wise, it works ok for now. I'll take a look at other PRs to confirm whether the UI parts are blocked by other PR or something I should fixed in this one |
Lee-W
force-pushed
the
none-as-logical-date-and-data-interval-for-asset-triggered-dag
branch
from
dc45928 to
1a82214
Compare
Member
Author
Lee-W
changed the title
Member
Can you elaborate? I don’t have enough context to notice what’s wrong in the screenshot. |
Member
Author
Sure, the second dag |
Lee-W
force-pushed
the
none-as-logical-date-and-data-interval-for-asset-triggered-dag
branch
4 times, most recently
from
5a32895 to
49a250c
Compare
Lee-W
requested review from
bbovenzi,
jscheffl,
pierrejeambrun and
ryanahamilton
as code owners
Lee-W
force-pushed
the
none-as-logical-date-and-data-interval-for-asset-triggered-dag
branch
from
49a250c to
7e3cde7
Compare
Lee-W
force-pushed
the
none-as-logical-date-and-data-interval-for-asset-triggered-dag
branch
from
7e3cde7 to
08f8991
Compare
1 task
Lee-W
changed the title
…ata_interval_start_success, prev_data_interval_end_success for dag_run that has no data_interval
…ow nullable" This reverts commit a3ba5a1.
…ogical_date" This reverts commit a8be4a7ebb424b194c8dd3183243b2516e4edb66.
This reverts commit a1b4e6e8299b3e0e800db088bc95b30519fe6b1a.
…_date in grid_data" This reverts commit 24a79d6.
…ical_date in task_stat" This reverts commit 0f04880.
This reverts commit 5c90113.
… logic as we don't need to check existing logical_date for asset triggered dag runs
Lee-W
force-pushed
the
none-as-logical-date-and-data-interval-for-asset-triggered-dag
branch
from
0cc1160 to
b8660f8
Compare
Member
Author
|
@uranusjr The CI is green now 🙌 |
Member
Author
|
It's a bit late in our time zone now. @dstandish, it would be nice if you could take a look tomorrow if you have some bandwidth. Thanks! |
uranusjr
reviewed
Feb 13, 2025
uranusjr
reviewed
Feb 13, 2025
Member
|
Looks good to me aside from the one question above |
1 task
uranusjr
approved these changes
Feb 13, 2025
Lee-W
deleted the
none-as-logical-date-and-data-interval-for-asset-triggered-dag
branch
Member
Author
|
I thought merging it would close all the six issues 🤔 maybe the syntax is wrong, I'll close the rest 5 manually |
ambika-garg
pushed a commit
to ambika-garg/airflow
that referenced
this pull request
Feb 17, 2025
…nd forbid them to be accessed in context/template (apache#46460) * style(dag): improve type annotation * refactor(dag): rename by_dag as adrq_by_dag * feat(scheduler_job_runner): set logical_date, data_interval as none when creating dag runs for asset triggered dag * test(pytest_plugin): set run_after to now if data_interval is None * test(scheduler_job_runner): set logical_date and data_interval of asset triggered dag runs to none * feat(dagrun): order queued and running dag runs by run_after instead of logical_date * feat(dag): get task_instances based on run_after instead of logical_date * feat(taskinstance): change log_url base_date to use run_after instead of logical_date * test(test_common): rewrite create_dagrun as logical_date is now nullable * feat(dag): get the last_dagrun by run_after * feat(taskinstance): pass base_date to TaskInstance.log_uri only when logical_date exists * feat(www): fix last_dag_run through run_after * feat(www): fetch last_dag_runs using run_after instead of logical_date in task_stat * feat(www): fetch dag_run through run_after instead of logical_date in grid_data * feat(task_sdk): remove data_interval_start, data_interval_end, prev_data_interval_start_success, prev_data_interval_end_success for dag_run that has no data_interval * test(pytest_plugin): add DagRun.DATASET_TRIGERED for backward compat * Revert "test(test_common): rewrite create_dagrun as logical_date is now nullable" This reverts commit a3ba5a1. * Revert "feat(dag): get task_instances based on run_after instead of logical_date" This reverts commit a8be4a7ebb424b194c8dd3183243b2516e4edb66. * Revert "feat(dag): get the last_dagrun by run_after" This reverts commit a1b4e6e8299b3e0e800db088bc95b30519fe6b1a. * feat(timetable): remove AssetTriggeredTimetable.data_interval_for_events * feat(scheduler_job_runner): use start_date directly for asset triggered dag * Revert "feat(www): fetch dag_run through run_after instead of logical_date in grid_data" This reverts commit 24a79d6. * Revert "feat(www): fetch last_dag_runs using run_after instead of logical_date in task_stat" This reverts commit 0f04880. * Revert "feat(www): fix last_dag_run through run_after" This reverts commit 5c90113. * refactor(task_runner): merge the data_interval keys with logical_date check logic * feat(scheduler_job_runner): simplify _create_dag_runs_asset_triggered logic as we don't need to check existing logical_date for asset triggered dag runs
Closed
2 tasks
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why
closes: #46192, #46193, #46194, #46196, #46198, #46641
What
logical_dateanddata_intervalto None for asset-triggered dags.data_interval_start, data_interval_end, prev_data_interval_start_success, prev_data_interval_end_successif data_interval_start and data_interval_end is None^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.