Handle default values from Airflow 2 gracefully #47371

sunank200 · 2025-03-05T06:25:59Z

Handle default values from Airflow 2.x gracefully, which breaks in Airflow 3.0. This change adds a method in the AirflowConfigParser class called handle_incompatible_airflow2_defaults for this. It runs by default while loading config.
Add two more option in airflow config lint
1. Automatically upgrade problematic default values:
  airflow config lint --upgrade-problematic-defaults
2. Skip checking default values:
  airflow config lint --skip-problematic-default-checks

Screenshot of testing done locally:

airflow config lint and importing conf in shell

Below is a summary table of the changes:

Parameter Key	Legacy (Airflow 2.x) Default	New (Airflow 3.0) Default
logging.log_filename_template	Either {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log or a variant like dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{%% if ti.map_index >= 0 %%}map_index={{ ti.map_index }}/{%% endif %%}attempt={{ try_number }}.log	Uses logical_date (with a Jinja filter on try_number), for example a template that looks like dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/ {%% if ti.map_index >= 0 %%}map_index={{ ti.map_index }}/{%% endif %%}attempt={{ try_number
core.dag_ignore_file_syntax	"regexp"	"glob"

closes: #46972

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

airflow/cli/commands/remote_commands/config_command.py

airflow/configuration.py

jedcunningham · 2025-03-13T06:42:06Z

airflow/cli/cli_config.py

    help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),


Suggested change

("--upgrade-defaults",),

("--upgrade-problematic-defaults",),

It's probably worth renaming these flags to include problematic - without it, it implies it'll upgrade all of the old defaults (which isn't a bad idea!).

Probably also means the Arg variable name should change too.

This only “upgrades” them in the current process, right? As far as I can tell the new values are not persisted. I wonder if there is a better verb to describe this.

@uranusjr any suggestion on prefix i could add to the name?

Not sure. Maybe --patch-problematic-defaults instead?

So, now that you bring it up, not sure this flag is even worth having. Upgrading it or not in the config lint command process doesn't seem that helpful?

But wait! In configuration.py you are already upgrading to the new default, meaning by the time config lint runs, well, it's already run and updated stuff. So having this run in config lint at all isn't doing anything really.

I can maybe have upgrade=False in configuration.py, which wouldn't upgrade the confs by default. Is this what we intend to do? Or do we always want to update in any case?

We do want upgrading by default, these are known bad values. I was just pointing out that doing thigns in config lint would never find/warn/upgrade anything - it's already too late.

airflow/cli/cli_config.py

airflow/cli/commands/remote_commands/config_command.py

airflow/configuration.py

jedcunningham · 2025-03-14T05:31:41Z

airflow/configuration.py

+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),


Suggested change

("core", "dag_ignore_file_syntax"): (

re.compile(r"^regexp$"),

"3.0",

"The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",

),

Looking at this, automatically changing folks away from the default may break things, not using an old default. e.g. if their airflowignore was regex, but we automatically swap to glob, things won't be happy.

We should leave this one out I believe.

jedcunningham · 2025-03-14T05:35:47Z

airflow/cli/cli_config.py

    help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),


So, now that you bring it up, not sure this flag is even worth having. Upgrading it or not in the config lint command process doesn't seem that helpful?

But wait! In configuration.py you are already upgrading to the new default, meaning by the time config lint runs, well, it's already run and updated stuff. So having this run in config lint at all isn't doing anything really.

jedcunningham · 2025-03-14T05:37:01Z

airflow/cli/commands/remote_commands/config_command.py

+        if not args.skip_problematic_default_checks:
+            conf.handle_incompatible_airflow2_defaults(upgrade=args.upgrade_problematic_defaults)
+


Which means, since this has already run and upgraded, you can get rid of this completely - and the args, etc. All of it.

It is unfortunate that this is what you get:

$ airflow config lint /Users/jedc/github/airflow/airflow/configuration.py:440 FutureWarning: [core] dag_ignore_file_syntax: The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0. Auto-upgraded from 'regexp' to 'glob'. No issues found in your airflow.cfg. It is ready for Airflow 3!

The warning is from configuration.py, where it got auto-upgraded.

This was a behaviour that I intentionally added where it automatically upgraded by default. If we want to change this to upgrade only via airflow config lint, I can pass upgrade=False in configuration.py method.

jedcunningham · 2025-03-14T05:38:50Z

tests/cli/commands/remote_commands/test_config_command.py

        assert expected_message in normalized_output
        assert config_change.suggestion in normalized_output
+
+    def test_lint_detects_default_value_change_upgrade(self):


Also means this test should be refactored and moved into test_configuration.py.

jedcunningham · 2025-03-14T05:44:54Z

airflow/configuration.py


+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(


fwiw, I took the default out of a fresh 2.10.5 config file, and this regex did not match against it. The regex may have been cut off at the end of the task_id line (that's naively what printed, at least).

Give that a test.

jedcunningham · 2025-03-14T06:41:14Z

Looking closer, this duplicates the functionality of deprecated_values. We can just update that.

I've opened #47761 to do that.

sunank200 · 2025-03-14T07:15:18Z

Closing this as discussed with @jedcunningham as per #47371 (comment)

boring-cyborg bot added the area:CLI label Mar 5, 2025

sunank200 mentioned this pull request Mar 5, 2025

Handle default values from Airflow 2 gracefully #46972

Closed

2 tasks

jedcunningham reviewed Mar 5, 2025

View reviewed changes

airflow/cli/commands/remote_commands/config_command.py Outdated Show resolved Hide resolved

airflow/cli/commands/remote_commands/config_command.py Outdated Show resolved Hide resolved

sunank200 force-pushed the handle-default-values branch from 10a4da2 to 7d539a4 Compare March 5, 2025 19:14

sunank200 requested a review from jedcunningham March 5, 2025 19:14

sunank200 force-pushed the handle-default-values branch 5 times, most recently from f8dd35a to 9ff5266 Compare March 7, 2025 06:57

phanikumv assigned vatsrahul1001 Mar 10, 2025

sunank200 force-pushed the handle-default-values branch 2 times, most recently from d00db25 to 69b7eb1 Compare March 12, 2025 07:58

sunank200 assigned sunank200 and unassigned vatsrahul1001 Mar 12, 2025

sunank200 force-pushed the handle-default-values branch from 69b7eb1 to 5aba5aa Compare March 12, 2025 08:19

sunank200 added the area:core label Mar 12, 2025

sunank200 force-pushed the handle-default-values branch 3 times, most recently from 4cb04be to 4c438c4 Compare March 12, 2025 11:26

sunank200 marked this pull request as ready for review March 12, 2025 12:38

sunank200 force-pushed the handle-default-values branch from 02372a6 to c03bac3 Compare March 13, 2025 06:27

Lee-W self-requested a review March 13, 2025 07:04

jedcunningham reviewed Mar 13, 2025

View reviewed changes

sunank200 force-pushed the handle-default-values branch 3 times, most recently from df0c37f to ae65d26 Compare March 13, 2025 10:28

sunank200 requested review from jedcunningham and uranusjr March 13, 2025 15:51

Handle default values from Airflow 2 gracefully

2e7ae29

sunank200 added 4 commits March 13, 2025 23:15

Add tests

1fdafc2

Add handle_incompatible_airflow2_defaults in configurations

a2e53dd

Add more default value changes between 2.10 to 3.0

421dff7

Fix PR comments

c39acba

sunank200 force-pushed the handle-default-values branch from 3cd1b18 to c39acba Compare March 13, 2025 17:30

jedcunningham reviewed Mar 14, 2025

View reviewed changes

sunank200 closed this Mar 14, 2025

sunank200 deleted the handle-default-values branch May 8, 2025 12:53

	("--upgrade-defaults",),
	("--upgrade-problematic-defaults",),

		if not args.skip_problematic_default_checks:
		conf.handle_incompatible_airflow2_defaults(upgrade=args.upgrade_problematic_defaults)

Handle default values from Airflow 2 gracefully #47371

Handle default values from Airflow 2 gracefully #47371

Uh oh!

Conversation

sunank200 commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uranusjr Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jedcunningham commented Mar 14, 2025

Uh oh!

sunank200 commented Mar 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sunank200 commented Mar 5, 2025 •

edited

Loading

uranusjr Mar 13, 2025 •

edited

Loading