Keytab file is not mounted in worker-kerberos container

[mikegit21]

Official Helm Chart version

1.5.0

Apache Airflow version

2.2.4

Kubernetes Version

1.20+

Helm Chart configuration

kerberos:
  enabled: true
  ccacheMountPath: /var/kerberos-ccache
  ccacheFileName: cache
  configPath: /etc/krb5.conf
  keytabBase64Content: "<base64contentofkeytabfile>"
  keytabPath: /etc/airflow.keytab
  principal: <principal>
  reinitFrequency: 3600
  config: "|
    [logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log

[libdefaults]
 default_realm = REALM1
 dns_lookup_realm = false
 dns_lookup_kdc = false
 renew_lifetime = 7d
 forwardable = true
 udp_preference_limit = 1
 kdc_timeout = 3000

[realms]
 REALM1 = {
 kdc = x
 kdc = x
 }
 REALM2 = {
 kdc = x
 kdc = x
 kdc = x
 kdc = x
 }

[domain_realm]
mapping = mapping"

Docker Image customisations

FROM apache/airflow:2.2.4

# Switch user because otherwise installing dependencies will not work
USER root
# Update package index and upgrade packages to prevent security issues
RUN apt-get update && apt-get upgrade -y
# Needed to install custom package
RUN apt-get install build-essential unixodbc-dev libkrb5-dev -y
USER airflow

RUN pip install pipenv

COPY Pipfile Pipfile.lock /
RUN pipenv install --system

What happened

When we are executing the Airflow pipeline which needs to connect to the kerberos database we are getting the following error:

File "/home/airflow/.local/lib/python3.7/site-packages/impala/dbapi.py", line 167, in connect
    retries=retries)
  File "/home/airflow/.local/lib/python3.7/site-packages/impala/hiveserver2.py", line 862, in connect
    transport.open()
  File "/home/airflow/.local/lib/python3.7/site-packages/thrift_sasl/__init__.py", line 82, in open
    ret, chosen_mech, initial_response = self.sasl.start(self.mechanism)
  File "/home/airflow/.local/lib/python3.7/site-packages/impala/sasl_compat.py", line 24, in start
    return True, self.mechanism, self.process()
  File "/home/airflow/.local/lib/python3.7/site-packages/puresasl/client.py", line 16, in wrapped
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/puresasl/client.py", line 148, in process
    return self._chosen_mech.process(challenge)
  File "/home/airflow/.local/lib/python3.7/site-packages/puresasl/mechanisms.py", line 505, in process
    kerberos.authGSSClientStep(self.context, '')
kerberos.GSSError: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))

This is intersting because we use the exact same krb5.conf, keytab file and principal for accessing the database in another tool. I jumped into the worker-kerberos container and verified the etc/krb5.conf and the principal in the opt/airflow/airflow.cfg - they do look good. But I stumbled about the fact that I cannot find the etc/airflow.keytab file, which from what I understand should be mounted into the container based on the secret that the airflow helm chart creates. I suspect that the missing keytab file actually causes the problem as it contains the login information needed to connect to the kerberos database.

I also verified that the airflow-kerberos-keytab secret exists. It looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: airflow-kerberos-keytab
 annotations:
   meta.helm.sh/release-name: airflow
   meta.helm.sh/release-namespace: namespace
type: Opaque
data:
  kerberos.keytab: <base64-content>

The base64 content looks good to me.

What you think should happen instead

The airflow.keytab file should be mounted into the worker-kerberos container, which in my opinion should solve the kerberos connection error.

How to reproduce

You can reproduce this issue by using a similar kerberos configuration like the one I provided and then trying to access the database in a DAG.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

You seem to know exactly what the problem is (and have a way of testing it) - would you maybe like to add a PR fixing it ? It does not seem like complex thing to do - testing and having a good environment to do so, seem like the most complex part.

Would you lile to pay back and contribute back to Airflow and become one of the > 2K contributors to Apache Airflow @mikegit21 ?

[mikegit21]

I can try to fix it in the HELM chart locally and if it solves the problem I could create a PR.

Otherwise I will come back to this thread

[potiuk]

I can try to fix it in the HELM chart locally and if it solves the problem I could create a PR.

Otherwise I will come back to this thread

Perfect! Thanks. Happy to help with your PR - just Ping me

[mikegit21]

I was accidentally looking in the worker container, I can see the keytab file in the worker-kerberos container and was able to check it using ktutil- looks good to me. The only thing I could think of would be a problem with the krb5.conf. I specified the log files like this:

[logging]
default = "FILE:{{ template "airflow_logs_no_quote" . }}/kerberos_libs.log"
kdc = "FILE:{{ template "airflow_logs_no_quote" . }}/kerberos_kdc.log"

But couldn't find anything in opt/airflow/ so I am a little bit lost what could be the problem. The config is correct as we are using it in another application (but as a seperate .conf file in this case). I also tried the same config with the airflow docker-compose setup which just accepts a krb5.conf file to be mounted and had no problems. It feels like there is some issue with parsing of .Values.kerberos.config

[potiuk]

Hard to say without deep looking, I know that kerberos was notoriously difficult to configure and even slightest typos made it stop working. I am pretty sure though that kerberos was working at some point in time, and there are tests covering that. I think you'd need to deep e bit deeper and try to see maybe some typo is there?

[potiuk]

You can see "test_kerberos_auth.py" for example - we actually start kerberos integration there and run testing with a test krb.conf - maybe trying to reproduce it there will help (you need to start breeze with --integration kerberos enabled.

[potiuk]

Also in tests/charts/test_kerberos.py you will find kerberos-related tests that do some generation of krb5.conf - maynbe those tests will give you some hints (they are actually running helm template under the hood and you can even grab generated template this way and analyse it.

[npsables]

Hi mikegit21, did you resolve this issue? How did you fix it?

[npsables]

I want to reuse kerberos.keytab file in workers but can't because the helm didn't mount it to worker container. The sidecar (airflow kerberos command) is also broken in my build. But I resolve all of this by reduce to 1 worker, mount /etc/kerberos to this, disable kerberos sidecar, and schedule an airflow task to refresh the kerberos ticket.