Ensure that git-sync actually runs when dags.gitSync.enabled=true and dags.persistence.enabled=true #59123
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the official documentation, deploying Airflow with both git-sync (
dags.gitSync.enabled=true) and persistence enabled (dags.persistence.enabled=true) is a supported deployment pattern: https://airflow.apache.org/docs/helm-chart/1.8.0/manage-dags-files.html#mounting-dags-using-git-sync-sidecar-with-persistence-enabled. On Airflow 2.x this works just fine, the scheduler pod runs a git-sync sidecar container which downloads DAGs onto a shared volume on behalf of the entire Airflow installation.On Airflow 3.x, there is an if-statement that turns off the git-sync sidecar in the scheduler pod since the dag-processor pod is now present https://github.com/apache/airflow/blob/main/chart/templates/scheduler/scheduler-deployment.yaml#L254-L256. However, when we look at the dag-processor pod, it incorrectly disables the git-sync sidecar when DAG persistence is enabled https://github.com/apache/airflow/blob/main/chart/templates/dag-processor/dag-processor-deployment.yaml#L198. This sadly means that git-sync doesn't run anywhere, because the dag-processor was supposed to be the one doing the syncing.
This PR fixes the if statement so that the dag-processor always runs the git-sync sidecar irrespective of if persistence is enabled or not (it's the pod that's supposed to be syncing things for all of the others if persistence is enabled + it needs to sync for itself if persistence is disabled). So with this change, Airflow 3.x actually works with both git-sync and DAG persistence enabled again.
You can check this for yourself by doing
helm template airflow chart --set=dags.gitSync.enabled=true --set=dags.persistence.enabled=true | lesswith and without this PR. (With this PR's change, there is a git-sync container in the rendered manifest, and without this PR the git-sync container is totally missing.)^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.