Status of testing of Apache Airflow 2.7.2rc1

[ephraimbuddy]

Body

We are kindly requesting that contributors to Apache Airflow RC 2.7.2rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• Improve back-reference generation (#33149): @potiuk
• Better fix for add-back-references (#33207): @potiuk
• Refactor: Simplify code in breeze (#33273): @eumiro
• Limit connection resources used in mapped tasks (#33309): @potiuk
  Linked issues:
  • Flaky test_xcom_map_error_fails_task test (#33178)
• Refactor: Simplify conditions on len() (#33454): @eumiro
• Replace strftime with f-strings where nicer (#33455): @eumiro
• Refactor: Do not create lists we don't need (#33519): @eumiro
• Bump min-version for docker and docker-compose (#33572): @potiuk @uranusjr
  Linked issues:
  • Avoid WSL2 ones when finding a context for Breeze (#33547)
• Use str.splitlines() to split lines (#33592): @eumiro
• Hide Irrelevant Dag Processor from Cluster Activity Page (#33611): @erdos2n
  Linked issues:
  • DAG Processor should not be visible in the Cluster Activity Page if there is no stand alone processor (#33497)
• Refactor: Remove useless str() calls (#33629): @eumiro
• Refactor: Use random.choices (#33631): @eumiro
• Refactor: path.rglob("…") does not need "**" (#33669): @eumiro
• Refactor: Improve detection of duplicates and list sorting (#33675): @eumiro
• Refactor: Use f-strings (#33734): @eumiro
• Refactor: tmp_path in tests/dag_processing (#33740): @eumiro
• Refactor: Use inplace .sort() (#33743): @eumiro
• Refactor: Replace lambdas with comprehensions (#33745): @eumiro
• Refactor: remove unused state - SHUTDOWN (#33746): @Bisk1
• E731: replace lambda by a def method in Airflow core (#33758): @hussein-awala
• Replace type func by isinstance in fab_security manager (#33760): @hussein-awala
• Use literal dict instead of calling dict() in Airflow core (#33762): @hussein-awala
• remove unnecessary map and rewrite it using list in Airflow core (#33764): @hussein-awala
• Replace single quotes by double quotes in all Airflow modules (#33766): @hussein-awala
• Merge multiple isinstance calls for the same object in a single call (#33767): @hussein-awala
• Use a single statement with multiple contexts instead of nested statements in core (#33769): @hussein-awala
• Improve importing the modules in Airflow www package (#33810): @hussein-awala
• Refactor unneeded 'continue' jumps in utils (#33836): @eumiro
• Refactor unneeded 'continue' jumps in dag processing (#33839): @eumiro
• Refactor unneeded 'continue' jumps in api (#33842): @eumiro
• Refactor unneeded 'continue' jumps in configuration (#33844): @eumiro
• Refactor unneeded 'continue' jumps in cli (#33845): @eumiro
• Refactor unneeded 'continue' jumps in jobs (#33846): @eumiro
• Refactor unneeded 'continue' jumps in setup.py (#33848): @eumiro
• Refactor unneeded 'continue' jumps around the repo (#33849): @eumiro
• replace unnecessary dict comprehension by dict() in core (#33858): @hussein-awala
• Cleanup: remove remnants of 'shutdown' task instance state (#33893): @Bisk1
  Linked issues:
  • Refactor: remove unused state - SHUTDOWN (#33746)
• Replace sequence concatination by unpacking in Airflow core (#33934): @hussein-awala
• Replace single element slice by next() in Airflow tests (#33938): @hussein-awala
• Replace dict.items by dict.values when key is not used in Airflow core (#33940): @hussein-awala
• Avoid creating unnecessary list when parsing stats datadog tags (#33943): @hussein-awala
• Add cancel_trigger_ids to to_cancel dequeue in batch (#33944): @hussein-awala
• Remove useless string join from core (#33969): @hussein-awala
• Move the try outside the loop when this is possible in Airflow core (#33975): @hussein-awala
• Replace == by is for types comparison (#33983): @hussein-awala
• replace loop by any when looking for a positive value in core (#33985): @hussein-awala
• Combine similar if logics in core (#33988): @hussein-awala
• Restore rendering of provider configuration with version_addded (#34011): @potiuk @Taragolis
  Linked issues:
  • version_added field in configuration option doesn't work correctly in providers documentation (#34005)
• Remove [project] section from pyproject.toml (#34014): @risicle @Taragolis
  Linked issues:
  • pyproject.toml [project] section without name and version attributes is not pep 621 compliant (#33854)
• Remove infinite animation for pinwheel, spin for 1.5s (#34020): @pankajkoti @mj-dd
  Linked issues:
  • please disable pinwheel animation as it violates ada guidelines (#34019)
• Fix scheduler logic to plan new dag runs by ignoring manual runs (#34027): @Bisk1
• Consolidate importing of os.path.* (#34060): @eumiro
• Bring back SHUTDOWN-related constants (#34063): @potiuk @uranusjr
  Linked issues:
  • Add state utils to Public Airflow Interface (#34059)
• Fix for TaskGroup toggles for duplicated labels (#34072): @Usiel
  Linked issues:
  • Toggling TaskGroup toggles all TaskGroups with the same label on Graph/Grid (#34066)
• Add missing audit logs for default flask action: add, edit and delete (#34090): @Raul824 @hussein-awala
  Linked issues:
  • Delete record of dag run is not being logged in audit logs when clicking on delete button in list dag runs (#34069)
• Refactor: Consolidate import and usage of random (#34108): @eumiro
• Fix issues related to access_control={} (#34114): @SamWheating
• Refactor: Simplify string generation (#34118): @eumiro
• Fix unfound ab_user table in the CLI session (#34120): @mbarugelCA @hussein-awala
  Linked issues:
  • sqlalchemy error when running CLI command airflow tasks test (#34109)
• Fix the required permissions to clear a TI from the UI (#34123): @Raul824 @hussein-awala
  Linked issues:
  • Can Delete on Task Instances permission is required to clear task (#34092)
• Ensure details panel is shown when any tab is selected (#34136): @Usiel
• Fix FAB-related logging format interpolation (#34139): @Taragolis
• Fix query bug in next_run_datasets_summary endpoint (#34143): @ephraimbuddy
• correcting wrong time showing in grid view (#34179): @Abhishek-kumar-samsung
• Refactor integer division (#34180): @eumiro
• Refactor: Simplify comparisons (#34181): @eumiro
• Only tag :latest image when default python version is used (#34182): @potiuk
• Move default timezone to except block (#34245): @hussein-awala
• Make param validation consistent for DAG validation and triggering (#34248): @jens-scheffler-bosch @ldacey
  Linked issues:
  • Params Definition is Required to be JSON Schema Valid also if Manually Triggered (#31299)
  • 100+ DAGs fail to import after update to 2.7.1 due to "params without default values" (#34227)
  • Fix handling of default value and serialization of Param class (#33141)
• Replace assert by if...raise in www package (#34249): @hussein-awala
• Replace assert by if...raise in decorators package (#34250): @hussein-awala
• Change "not all" to "any" for ease of readability (#34259): @pankajkoti
• Set loglevel=DEBUG in 'Not syncing DAG-level permissions' (#34268): @csp33
• Fix www cluster_activity view not loading due to standaloneDagProcessor templating (#34274): @erdos2n @yermalov-here
  Linked issues:
  • Hide Irrelevant Dag Processor from Cluster Activity Page (#33611)
• Update min-sqlalchemy version to account for latest features used (#34293): @potiuk @Taragolis
  Linked issues:
  • Use NOT EXISTS subquery instead of tuple_not_in_condition (#33527)
• REST API: Fix task instance access issue in the batch endpoint (#34315): @ephraimbuddy
• Fixed rows count in the migration script (#34348): @alartum
• Refactor os.path.splitext to Path.* (#34352): @eumiro
• Fix dag warning endpoint permissions (#34355): @hussein-awala
• Deprecate numeric type python version in PythonVirtualEnvOperator (#34359): @Xiroo
• Fix datetime deserialization by replacing timezone short code by its name in serialized values (#34492): @hussein-awala
• Change Hide/Show Details button title in Grid View (#34519): @yiannis-had
• Add more exemptions to lengthy metric list (#34531): @sa1
  Linked issues:
  • Airflow DAG fails to run if dag_id + task_id is too long with OTEL integration enabled. (#34416)
• Fix: make dry run optional for patch task instance (#34568): @Calder-Ty
  Linked issues:
  • DryRun is not optional for patch task instance (#34563)
• Import Dataset from airflow.datasets to avoid circular dependency (#34586): @drobert-bfm @hussein-awala
  Linked issues:
  • Circular dependency issue when trying to locally test DAG (#34558)
• Fix is_parent_mapped value by checking if any of the parent tg is mapped (#34587): @fritz-astronomer @hussein-awala
  Linked issues:
  • Unable to retrieve logs for nested task group when parent is mapped (#34535)
• Fail dag test if defer without triggerer (#34619): @dstandish
• Use iterative loop to look for mapped parent (#34622): @uranusjr @hussein-awala
  Linked issues:
  • Fix is_parent_mapped value by checking if any of the parent tg is mapped (#34587)
• Don't run isort when creating new alembic migrations (#34636): @jedcunningham
• Standardise output of "airflow connections export" to those of "airflow variables export" and "airflow users export" (#34640): @gbonazzoli @Jayden-Chiu
  Linked issues:
  • Standardise output of "airflow connections export" to those of "airflow variables export" and "airflow users export" (#34232)
• Restrict astroid version < 3 (#34658): @Taragolis
• Add support for ZoneInfo and generic UTC (#34683): @bolkedebruin
• misc: Fix spelling for SesssionExemptMixin (#34696): @dkalamar
• Check if the lower of provided values are sensitives in config endpoint (#34712): @hussein-awala
• Order cherrypicking candidates by the closed date (#34739): @ephraimbuddy
• Fix autodetect_docker_context for list of dict case (#34779): @utkarsharma2
• zoneinfo.ZoneInfo is not available on < Python 3.9 (#34804): @bolkedebruin
• Add explanation about audit log users to security model (#34814): @potiuk

Thanks to all who contributed to the release (probably not a complete list!):
@dkalamar @jedcunningham @Raul824 @yiannis-had @Xiroo @erdos2n @drobert-bfm @dstandish @ephraimbuddy @Usiel @potiuk @hussein-awala @eumiro @mbarugelCA @jens-scheffler-bosch @utkarsharma2 @yermalov-here @Taragolis @Abhishek-kumar-samsung @csp33 @pankajkoti @mj-dd @gbonazzoli @fritz-astronomer @sa1 @ldacey @SamWheating @risicle @Calder-Ty @Bisk1 @bolkedebruin @uranusjr @alartum @Jayden-Chiu

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[jedcunningham]

#34011 and #34636 both should have been skipped - providers and dev env respectively.

[ephraimbuddy]

#34011 and #34636 both should have been skipped - providers and dev env respectively.

Mistake

[Calder-Ty]

#34568 Is working as expected

[utkarsharma2]

Tested #34779 <#34779>, working as expected. Thanks, Utkarsh Sharma

…

On Tue, Oct 10, 2023 at 7:54 PM Tyler Calder ***@***.***> wrote: #34568 <#34568> Is working as expected — Reply to this email directly, view it on GitHub <#34830 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDLAHOCBOWAFK7D2GOHDKTX6VLDLAVCNFSM6AAAAAA5YNKLICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJVGUZTIMRRGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[starone99]

#34359 is working as expected.

[Bisk1]

#33746, #33893 and #34027 work as expected

[Usiel]

#34072 and #34136 work as expected.

I also tested a few other PRs:

• Works as expected: Add missing audit logs for default flask action: add, edit and delete #34090
• Does not work for me: Fix unfound ab_user table in the CLI session #34120 (details below)

Details on #34120 issue (cc @hussein-awala)

Tested both

• locally with Breeze (Posgres 11, Python 3.8, git rev c8b25cb3ee aka 2.7.2rc1 tag) and
• on a test environment (MySQL 5.7, Python 3.9, PyPi package 2.7.2rc1).

In both cases, for different DAGs, I get the error below. #34656 seems to have broken this for me.

[2023-10-11T02:22:36.229+0000] {cli_action_loggers.py:85} DEBUG - Calling callbacks: []
Traceback (most recent call last):
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/__main__.py", line 60, in main
    args.func(args)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/utils/cli.py", line 113, in wrapper
    return f(*args, **kwargs)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 634, in task_test
    session.delete(ti.dag_run)
  File "/opt/.pyenv/versions/3.9.8/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/airflow/utils/session.py", line 37, in create_session
    session.commit()
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1454, in commit
    self._transaction.commit(_to_root=self.future)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 832, in commit
    self._prepare_impl()
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 811, in _prepare_impl
    self.session.flush()
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3449, in flush
    self._flush(objects)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3589, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3549, in _flush
    flush_context.execute()
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
    rec.execute(self)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 667, in execute
    util.preloaded.orm_persistence.delete_obj(
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 330, in delete_obj
    table_to_mapper = base_mapper._sorted_tables
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1184, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3386, in _sorted_tables
    sorted_ = sql_util.sort_tables(
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 1217, in sort_tables
    for (t, fkcs) in sort_tables_and_constraints(
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 1289, in sort_tables_and_constraints
    filtered = filter_fn(fkc)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 1207, in _skip_fn
    if skip_fn(fk):
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3369, in skip
    dep = table_to_mapper.get(fk.column.table)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 2532, in column
    return self._resolve_column()
  File "/opt/.pyenv/versions/3.9.8/envs/airflow/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 2543, in _resolve_column
    raise exc.NoReferencedTableError(
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'dag_run_note.user_id' could not find table 'ab_user' with which to generate a foreign key to target column 'id'
[2023-10-11T02:22:36.234+0000] {settings.py:353} DEBUG - Disposing DB connection pool (PID 5345)

[ephraimbuddy]

#34072 and #34136 work as expected.

I also tested a few other PRs:

• Works as expected: Add missing audit logs for default flask action: add, edit and delete #34090
• Does not work for me: Fix unfound ab_user table in the CLI session #34120 (details below)

I don't think the failing one should be considered a blocker since no new issue was introduced. cc @dstandish @hussein-awala

[hussein-awala]

#34072 and #34136 work as expected.
I also tested a few other PRs:

• Works as expected: Add missing audit logs for default flask action: add, edit and delete #34090
• Does not work for me: Fix unfound ab_user table in the CLI session #34120 (details below)

I don't think the failing one should be considered a blocker since no new issue was introduced. cc @dstandish @hussein-awala

Yes it should not considered as blocker, but if there is a different blocker for the RC (I hope not), we can cherry-pick #34656 in the next RC

[hussein-awala]

I tested my changes, they are present in the RC. I tested only the bug fixes, and all looks good, except #34120 as discussed above.

[ephraimbuddy]

#34072 and #34136 work as expected.
I also tested a few other PRs:

• Works as expected: Add missing audit logs for default flask action: add, edit and delete #34090
• Does not work for me: Fix unfound ab_user table in the CLI session #34120 (details below)

I don't think the failing one should be considered a blocker since no new issue was introduced. cc @dstandish @hussein-awala

Yes it should not considered as blocker, but if there is a different blocker for the RC (I hope not), we can cherry-pick #34656 in the next RC

#34656 is in the release, but not on the release note. You can see the milestone and changelog:skip label on it. I wanted to combine it with #34120 on the release note but forgot that.
We probably need another fix. Also, you can go to the v2-7-stable to confirm if the change is there

[jscheffl]

Checked for #34248 and can confirm that (1) code is correctly merged and (2) fix is working! Thanks for making the release!

[ephraimbuddy]

Airflow 2.7.2 has been released. Thank you all for testing the release candidate