Grid page visibility in the latest 2.3.1 release

[stepanof]

Apache Airflow version

2.3.1 (latest released)

What happened

Hello team!

I have faced the problem with Grid visibility in the latest 2.3.1 release.

My issue (example)

1. Here is the end of dag code:

...
<code>
...
telegram_alert = TelegramOperator(
    task_id='telegram_alert'
    ,telegram_conn_id='telegram_bot_id'
    ,chat_id='telegram_chat_id'
    ,text="blablabla"
    ,dag=dag
)
python_task_1 >> python_task_2 >> puller >> telegram_alert

Grid:

2. Delete task with telegram alert:
   Code in the end of dag code:

...
<code>
...
python_task_1 >> python_task_2 >> puller

Grid after deleting (Telegram alert task disappeared):

Grid after triggering dag again:

3. Return task with telegram alert in the dag as it was at the begining.
   Grid:
   

PS:

As I remember in 2.3.0 there wasn't such bug.
I tried to roll back airflow image (from 2.3.1 to 2.3.0) and restart containers but faced error during starting webserver:
alembic.util.exc.CommandError: Can't locate revision identified by '1de7bc13c950'

I see the same error when try to execute such command:
airflow db downgrade --to-version 2.3.0
Please help to me to roll back the airflow version or may be I can get some patch which fix this bug.

What you think should happen instead

The Grid must be visible at any time at any case

How to reproduce

Steps to reproduce:

1. Create dag with N tasks
2. Trigger dag
3. Delete 1 task from dag
4. Trigger dag
5. Return deleted task to dag
6. Try to see DagRun history on the Grid page.

Operating System

NAME="Ubuntu" VERSION="21.04 (Hirsute Hippo)" ID=ubuntu ID_LIKE=debian

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==3.3.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-docker==2.6.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-google==6.8.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.8.0
apache-airflow-providers-microsoft-mssql==2.1.3
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.3
apache-airflow-providers-telegram==2.0.4
apache-airflow-providers-vertica==2.1.3

Deployment

Docker-Compose

Deployment details

No response

Anything else

Many dags became unvisible on the Grid page and it is critical for us

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[stepanof]

WebServer log contains:

[2022-05-27 15:57:00,679] {manager.py:511} ERROR - Creation of Permission View Error: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "ab_permission_view_permission_id_view_menu_id_key"
DETAIL:  Key (permission_id, view_menu_id)=(13, 479) already exists.
[SQL: INSERT INTO ab_permission_view (id, permission_id, view_menu_id) VALUES (nextval('ab_permission_view_id_seq'), %(permission_id)s, %(view_menu_id)s) RETURNING ab_permission_view.id]
[parameters: {'permission_id': 13, 'view_menu_id': 479}]
(Background on this error at: http://sqlalche.me/e/14/gkpj)


[2022-05-27 15:57:00,795] {manager.py:570} ERROR - Add Permission to Role Error: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "ab_permission_view_role_permission_view_id_role_id_key"
DETAIL:  Key (permission_view_id, role_id)=(1648, 1) already exists.
[SQL: INSERT INTO ab_permission_view_role (id, permission_view_id, role_id) VALUES (nextval('ab_permission_view_role_id_seq'), %(permission_view_id)s, %(role_id)s) RETURNING ab_permission_view_role.id]
[parameters: {'permission_view_id': 1648, 'role_id': 1}]
(Background on this error at: http://sqlalche.me/e/14/gkpj)

[stepanof]

What DevTools found:

[stepanof]

Looks similar to problem in pull 23332
Similar things:
There is problem with visibility of grid actions as well I have.
There is problem with permissions as well my webserver log contain permissions error.

I can't go deeper. This is my limit of understanding.
I will be quite glad to your feedback and help.

[Smirn08]

@stepanof you can try to delete old dag dagruns at /dagrun/list/

[gbonazzoli]

@stepanof

Same problem here !

I have some DAGS that are not showing up in the grid layout.
In 2.3.0 there was no problem, it starts after upgrade Airflow to 2.3.1

I suspect that the cause is some problems on the permission due to problem with AUTH_LDAP config that I have.

[potiuk]

@gbonazzoli @stepanof almost for sure the problem is Duplicate of #23908 which is already solved in (very soon) 2.3.2 release where #23947 fixes it. It woudl be great if the both of you test the 2.3.2 RC1 that should appear likely within hours.

[stepanof]

@potiuk I will be happy to test 2.3.2 RC1. Please let me know how I can get it.

[gbonazzoli]

@potiuk @stepanof

I can snapshot my 2.3.1 Environment, install 2.3.2 RC1 and work on it in production.
And if there aren't important regressions keep it without regarding the final results on the grid page.

I only need that 2.3.2 RC1 is installable through pip install.

Is it possible ?