Skip to content

Conversation

@Lee-W
Copy link
Member

@Lee-W Lee-W commented Jun 9, 2025

Why

TLDR: fix serialization for pre AIP-39 dag run history

The discussion in

# the older implementation has call to infer_automated_data_interval if data_interval is None, do we want to keep that or raise
# an exception?
if data_interval is None:
raise ValueError(f"Cannot calculate data interval for run {run}")
about LazyDeserializedDAG.get_run_data_interval raises the question of whether we should use infer_automated_data_interval when data_interval is None. After testing, I believe this is necessary to avoid forcing users to purge their pre-AIP-39 history.

Per

# Compatibility: runs created before AIP-39 implementation don't have an
# explicit data interval. Try to infer from the logical date.
return self.infer_automated_data_interval(run.logical_date)
, this is still managed in the dag model. However, instead of calling infer_automated_data_interval for custom intervals, comments suggest we shouldn't modify this function as stated #32074, but rather call timetable.infer_manual_data_interval(run_after=logical_date) which is extended and fixed in #32118.

What

Fallback to timetable.infer_manual_data_interval(run_after=logical_date) if data_interval is None when deserialize a dag


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@Lee-W Lee-W force-pushed the handle-none-data-interval branch from 5ac5497 to 95f88bf Compare June 11, 2025 02:50
@Lee-W Lee-W marked this pull request as ready for review June 11, 2025 02:51
@Lee-W Lee-W requested review from ashb and bolkedebruin as code owners June 11, 2025 02:51
@Lee-W Lee-W requested a review from amoghrajesh June 11, 2025 02:51
@Lee-W Lee-W added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Jun 11, 2025
@Lee-W Lee-W marked this pull request as draft June 11, 2025 03:01
@Lee-W Lee-W force-pushed the handle-none-data-interval branch from 95f88bf to bb301ae Compare June 11, 2025 03:14
@Lee-W Lee-W marked this pull request as ready for review June 11, 2025 04:07
@Lee-W Lee-W force-pushed the handle-none-data-interval branch from ab438b3 to 51c10a0 Compare June 14, 2025 04:22
@Lee-W Lee-W force-pushed the handle-none-data-interval branch 3 times, most recently from d7c591d to c0bf347 Compare June 15, 2025 12:20
@Lee-W Lee-W force-pushed the handle-none-data-interval branch from c0bf347 to 987f2c9 Compare June 15, 2025 14:09
@Lee-W Lee-W force-pushed the handle-none-data-interval branch from 987f2c9 to d79418c Compare June 15, 2025 14:13
@uranusjr uranusjr added this to the Airflow 3.0.3 milestone Jun 17, 2025
@Lee-W Lee-W merged commit 4f26e45 into apache:main Jun 17, 2025
52 checks passed
@Lee-W Lee-W deleted the handle-none-data-interval branch June 17, 2025 07:38
@github-actions
Copy link

Backport failed to create: v3-0-test. View the failure log Run details

Status Branch Result
v3-0-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 4f26e45 v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

RoyLee1224 pushed a commit to RoyLee1224/airflow that referenced this pull request Jun 17, 2025
…che#51530)

* fix(serialized_objects): try to infer data interval if it's none

it handles pre AIP-39 dag runs

* test(serialization_objects): add test case test_serialized_dag_get_run_data_interval

* fix(serialized_objects): do not infer data interval if the logical_date is None

* feat(serialized_objects): allow data_interval to be none
Lee-W added a commit to astronomer/airflow that referenced this pull request Jun 19, 2025
…'s none (apache#51530)

* fix(serialized_objects): try to infer data interval if it's none

it handles pre AIP-39 dag runs

* test(serialization_objects): add test case test_serialized_dag_get_run_data_interval

* fix(serialized_objects): do not infer data interval if the logical_date is None

* feat(serialized_objects): allow data_interval to be none
(cherry picked from commit 4f26e45)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk pushed a commit that referenced this pull request Jun 21, 2025
…'s none (#51530) (#51913)

* fix(serialized_objects): try to infer data interval if it's none

it handles pre AIP-39 dag runs

* test(serialization_objects): add test case test_serialized_dag_get_run_data_interval

* fix(serialized_objects): do not infer data interval if the logical_date is None

* feat(serialized_objects): allow data_interval to be none
(cherry picked from commit 4f26e45)
RoyLee1224 pushed a commit to RoyLee1224/airflow that referenced this pull request Jun 21, 2025
…che#51530)

* fix(serialized_objects): try to infer data interval if it's none

it handles pre AIP-39 dag runs

* test(serialization_objects): add test case test_serialized_dag_get_run_data_interval

* fix(serialized_objects): do not infer data interval if the logical_date is None

* feat(serialized_objects): allow data_interval to be none
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:serialization backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants