Skip to content

Conversation

@kunaljubce
Copy link
Contributor

@kunaljubce kunaljubce commented Dec 30, 2025

Fixes #59003.

Issue 1: airflow dags reserialize fails to import related modules in CLI while working fine on UI: ModuleNotFoundError: No module named 'module.task_b'

Fix: So this was happening because when airflow tries to load the new module in:

loader.exec_module(new_module)

The airflow cli that runs airflow dags reserialize loads each DAG file with a synthetic, top‑level module name via SourceFileLoader and immediately executes it, but it never adds the DAG’s directory, i.e. /files/dags in this case, to sys.path. Once we add the helper package to the sys.path, it works as expected.

Issue 2: airflow tasks clear my_dag complains: AttributeError: 'DAG' object has no attribute 'clear'

image

Fix: This was happening because of a discrepancy in the type of the dag object returned by the task_clear() function in:

if args.dag_id and not args.bundle_name and not args.dag_regex and not args.task_regex:
dags = [get_dag_by_file_location(args.dag_id)]
else:
# todo clear command only accepts a single dag_id. no reason for get_dags with 's' except regex?
# Reading from_db because clear method still not implemented in Task SDK DAG
dags = get_dags(args.bundle_name, args.dag_id, use_regex=args.dag_regex, from_db=True)

Specifically, in the if block, the call to get_dag_by_file_location() was returning a DAG object which has no clear() attribute, whereas in the else block, the call to get_dags() returns a SerializedDAG object which the expected clear() attribute. Since this get_dag_by_file_location() does not seem to be called elsewhere, I simply altered the way we retrieve dags here from the DagModel to the SerializedDagModel.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@kunaljubce kunaljubce marked this pull request as ready for review December 30, 2025 15:52
@kunaljubce
Copy link
Contributor Author

Hello Maintainers - This is my first PR modifying a core component of dag processing, as far as I can tell. Please feel free to suggest if I am overlooking some other point of possible failure. Thanks!

Copy link
Contributor

@Nataneljpwd Nataneljpwd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few comments, maybe it is worth trying to change the loader

new_module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = new_module
loader.exec_module(new_module)
module_dir = os.fspath(Path(filepath).parent)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like a patch only of it is 1 directory above or below, what happens when a task is nested 2 modules deep?

# Insert module directory into sys.path to allow relative imports. Imp to note
# that this is just for the duration of `loader.exec_module(new_module)`, ensuring
# that helper modules located alongside the DAG are importable.
path_inserted = False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this intermediate variable? As it will be in position 0 only when we add it there

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Nataneljpwd You're right, this is an overkill. I will remove it.

@@ -439,7 +439,20 @@ def parse(mod_name, filepath):
spec = importlib.util.spec_from_loader(mod_name, loader)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should change to not use a source file loader and instead use spec_from_file_location and use the loader withing the spec?
I think it can solve this issue

@kunaljubce
Copy link
Contributor Author

Added a few comments, maybe it is worth trying to change the loader

Thanks, I will take a look and make the changes over the next few days :)

@kunaljubce kunaljubce marked this pull request as draft January 19, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow cli commands cant parse dags with imports from dags folder

2 participants