Skip to content

Conversation

@kaxil
Copy link
Member

@kaxil kaxil commented Jan 15, 2025

closes #45693

Adds support for the Magic Loop: https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html#optimizing-dag-parsing-delays-during-execution with Task SDK


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but we likely need a compat import shim for airflow/utils/dag_parsing_context.py -- do we have a place to capture those anywhere.

Also cc @Lee-W @uranusjr Another deprecated import to track in Ruff please.

@kaxil kaxil added the airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes label Jan 15, 2025
@kaxil
Copy link
Member Author

kaxil commented Jan 15, 2025

Looks good, but we likely need a compat import shim for airflow/utils/dag_parsing_context.py -- do we have a place to capture those anywhere.

Also cc @Lee-W @uranusjr Another deprecated import to track in Ruff please.

Good point, Yeah, #41641

Added newsfragment too

@kaxil kaxil merged commit 2841ca9 into apache:main Jan 15, 2025
60 of 61 checks passed
@kaxil kaxil deleted the magic-loop-sdk branch January 15, 2025 22:23
@Lee-W
Copy link
Member

Lee-W commented Jan 16, 2025

ruff PR created astral-sh/ruff#15525

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes area:CLI area:dev-tools area:providers area:Scheduler including HA (high availability) scheduler area:task-sdk kind:documentation provider:celery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support the "magic loop" with Task SDK

3 participants