[helm] default broker url doesn't match Redis service name

[jiajie-chen]

Official Helm Chart version

1.11.0 (latest released)

Apache Airflow version

v2.7.1

Kubernetes Version

v1.27.4 (minikube v1.31.2)

Helm Chart configuration

Enable the new standard naming option:

useStandardNaming: true

Using my-dag-test as the Helm release name.

Docker Image customizations

No docker customization, using default image from Helm.

What happened

When deploying the Helm chart, the Airflow worker crashes and restarts as it cannot find the Redis broker URL.

The Celery worker logs show the following:

... : ERROR/MainProcess] consumer: Cannot connect to redis://:**@my-dag-test-redis:6379/0: Error -2 connecting to my-dag-test-redis:6379. Name or service not known.
...

The Redis service name shows: my-dag-test-airflow-redis

The broker URL secret shows: redis://:***@my-dag-test-redis:6379/0

What you think should happen instead

The chart should template the correct Redis service name (and port) for the broker URL secret.

How to reproduce

Install a new Airflow release using Helm, with the following value overrides and release name:

helm upgrade \
  --install my-dag-test apache-airflow/airflow \
  --namespace airflow --create-namespace \
  --set 'useStandardNaming=true'

Compare the broker URL secret with the Redis service:

kubectl get secret --namespace airflow my-dag-test-broker-url -o jsonpath='{.data.connection}' | base64 --decode

kubectl get service --namespace airflow my-dag-test-airflow-redis -o jsonpath='{.metadata.name}'

Wait for the Airflow worker to start, and read the container logs:

kubectl logs --namespace airflow statefulset/my-dag-test-airflow-worker --container worker

Anything else

The Redis service name is defined here as {{ include "airflow.fullname" . }}-redis:

• airflow/chart/templates/redis/redis-service.yaml

  Line 27 in 142e67c

  name: {{ include "airflow.fullname" . }}-redis

The default broker URL is defined here as (printf "%s-redis:6379" .Release.Name ):

• airflow/chart/templates/secrets/redis-secrets.yaml

  Line 79 in 142e67c

  connection: {{ urlJoin (dict "scheme" "redis" "userinfo" (printf ":%s" ((default $random_redis_password .Values.redis.password) | urlquery)) "host" (printf "%s-redis:6379" .Release.Name ) "path" "/0") | b64enc | quote }}

If the Helm release name doesn't contain the chart name (airflow) then these won't match anymore.

Changing the printf to something like this should fix the issue:

(printf "%s-redis:%s" (include "airflow.fullname" .) .Values.ports.redisDB )

Note that the broker URL should also template the correct ports.redisDB value.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[sseelmann]

This should be fixed in #34825

[jiajie-chen]

Thanks @sseelmann , that PR looks like it resolved the issue I saw with Redis naming.

Is there a similar PR that addresses the issue when .Values.ports.redisDB is not set to 6379? I didn't see one open currently - I can push a PR with tests when I have time, if it's not currently being worked on.

[eladkal]

Fixed in #34825

I can push a PR with tests when I have time, if it's not currently being worked on.

PRs are always welcome (no need to create issues you can just open PRs directly)