UI DAG counts including deleted DAGs

[joeknize-bc]

Apache Airflow version

2.7.0

What happened

On the DAGs page, the All, Active, and Paused counts include deleted DAGs. This is different from <= 2.6.1 (at least), where they were not included in the totals. Specifically this is for DAGs for which the DAG files have been removed, not DAGs that have been deleted via the UI.

What you think should happen instead

Including deleted DAGs in those counts is confusing, and this behavior should revert to the previous behavior.

How to reproduce

Create a DAG.
Wait for totals to increment.
Remove the DAG file.

The totals will not change.

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.3.2
apache-airflow-providers-cncf-kubernetes==7.4.2
apache-airflow-providers-common-sql==1.7.0
apache-airflow-providers-datadog==3.3.1
apache-airflow-providers-elasticsearch==5.0.0
apache-airflow-providers-ftp==3.5.0
apache-airflow-providers-google==10.6.0
apache-airflow-providers-http==4.5.0
apache-airflow-providers-imap==3.3.0
apache-airflow-providers-microsoft-azure==6.2.4
apache-airflow-providers-postgres==5.6.0
apache-airflow-providers-redis==3.3.1
apache-airflow-providers-salesforce==5.4.1
apache-airflow-providers-slack==7.3.2
apache-airflow-providers-snowflake==4.4.2
apache-airflow-providers-sqlite==3.4.3

Deployment

Astronomer

Deployment details

No response

Anything else

I suspect the issue is with DagModel.deactivate_deleted_dags, but I'm unable to verify.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[eladkal]

Create a DAG.
Wait for totals to increment.
Remove the DAG file.

Remove dag file does not remove the records from the DB.
the DAG entries are left. If you want to remove the history you need to remove the records

[joeknize-bc]

Remove dag file does not remove the records from the DB.
the DAG entries are left. If you want to remove the history you need to remove the records

This isn't about history, this is about how many DAGs are in my Airflow instance right now -- that's what I expect to see on the DAGs page.

Also, there is no way that I'm aware of to delete a DAG via the UI once the files are gone. This is new behavior that the DagModel.deactivate_deleted_dags function seems designed to deal with.

[boushphong]

I created a PR for this issue at: #33778
@potiuk @eladkal Could you take a look? 🙇

[potiuk]

Merged and marked for 2.7.1