Backfill job marked as completed immediately, though backfill dag runs still are executing

[assurah]

Apache Airflow version

3.1.6

If "Other Airflow 3 version" selected, which one?

3.1.6

What happened?

I have been running backfill commands multiple times for a specific DAG recently. Initially, it worked as expected; however, at some point, the behavior changed. Now, whenever I execute the backfill command, the process transitions to a "completed" status almost instantaneously.

Interestingly, while the backfill job itself marks itself as finished immediately, the actual DAG runs triggered by that backfill appear to be running and processing normally in the background.

What you think should happen instead?

No response

How to reproduce

The schedule information for this DAG is as follows:

schedule=CronTriggerTimetable('0 6 * * *', timezone='Asia/Seoul') # +09:00

The backfill job was executed using the following CLI command:

airflow backfill create \
    --dag-id etl_vitess_load_v2 \
    --from-date 2023-11-20 \
    --to-date 2024-06-10 \
    --max-active-runs 1 \
    --run-backwards

Operating System

Ubuntu 24.04.3 LTS

Versions of Apache Airflow Providers

From airflow info

apache-airflow-providers-amazon           | 9.19.0
apache-airflow-providers-celery           | 3.15.0
apache-airflow-providers-cncf-kubernetes  | 10.12.0
apache-airflow-providers-common-compat    | 1.11.0
apache-airflow-providers-common-io        | 1.7.0
apache-airflow-providers-common-messaging | 2.0.1
apache-airflow-providers-common-sql       | 1.30.2
apache-airflow-providers-edge3            | 2.0.1
apache-airflow-providers-fab              | 3.1.1
apache-airflow-providers-http             | 5.6.2
apache-airflow-providers-mysql            | 6.4.0
apache-airflow-providers-postgres         | 6.5.1
apache-airflow-providers-redis            | 4.4.1
apache-airflow-providers-smtp             | 2.4.1
apache-airflow-providers-standard         | 1.10.2

Deployment

Official Apache Airflow Helm Chart

Deployment details

normally deployed into k8s (1.23) using official helm chart.

meta-database postgresql 16.x also deployed in the same k8s cluster with ceph pv

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[SakshamSinghal20]

I would like to work on this could you please assign it to me?

[assurah]

In the current implementation, it appears that the Backfill record and its associated DagRuns may be committed in separate database transactions or phases. Specifically, the Backfill record is persisted first to ensure the request is registered.

However, the logic used by the scheduler to monitor and mark a backfill as completed seems to rely on the absence of "unfinished" dag runs associated with that backfill.

Since the query for unfinished DagRuns returns zero at that precise moment, the system might incorrectly mark the backfill as Completed before any actual work has been queued.

if some synchronization guards already in place, let me know.

[roykoand]

Reproducible on Airflow 3.1.7 as well

[shivaam]

Reproducible on Airflow 3.1.7 as well

I wasn't able to reproduce the issue in 3.0.6. Seems to be working properly

[shivaam]

@roykoand how are you reproducing?

[shivaam]

I was able to reproduce the issue. WIll submit a potential fix soon.

There is a race condition in creating the backfill entry, and creating the dag runs for the backfill. Scheduler assumes a backfill is complete before the dags are created.