Integrate ldap

[manisha-tanwar]

Hi Team,

I tried using ldap authentication https://github.com/StackStorm/st2-auth-backend-ldap on my stackstorm environment on Linux server which works fine. But I'm having trouble in using it on K8.

I mounted /opt/stackstorm/st2 directory additionally on custom packs image(which includes pip install git+https://github.com/<github_account>/st2-auth-backend-ldap.git@master#egg=st2_auth_backend_ldap )

Also modified deployments.yaml and jobs.yaml to mount additional volume /opt/stackstorm/st2 and passed ldap configuration to /etc/st2/st2.user.conf through values.yaml.

I can see ldap backend is installed on st2actionrunner, st2client,st2sensorcontainer,st2api like this:

root@stackstorm-linux-gb5-st2actionrunner-7dfd5495cf-2zgrh:/opt/stackstorm# st2/bin/pip list | grep ldap
ldap2pg                        5.4
python-ldap                    3.3.1
st2-auth-backend-ldap          0.1.1

But didn't find any way to push this on st2auth pod and because of this it fails with

2020-07-13 12:50:50,018 ERROR [-] Invalid "st2auth.backends.backend" backend specified: ldap

What else do I need to modify in order to use ldap backend on st2auth. Please help!

[arm4b]

Here is the correct EWC LDAP repository that was open-sourced: https://github.com/stackstorm/st2-enterprise-auth-backend-ldap in https://stackstorm.com/2020/05/27/extreme-networks-donates-ewc-to-linux-foundation/.
The one you linked will deprecated in favor of previously enterprise one.

[manisha-tanwar]

Hi @armab
I was able to install enterprise ldap on Linux server but issue is still same while installing it on K8.

Do I need to mount /opt/stackstorm/st2 directory after pip installing ldap backend as by default only /opt/stackstorm/packs and /opt/stackstorm/virtualenvs are there in st2packs-runtime image.

$cat st2packs-runtime/Dockerfile
FROM alpine:3.8

ONBUILD VOLUME ["/opt/stackstorm/packs", "/opt/stackstorm/virtualenvs"]

# Copy packs and virtualenvs
ONBUILD COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs
ONBUILD COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs

[arm4b]

In order to install the library, package or extension on top of stackstorm core you need to fork https://github.com/stackstorm/st2-dockerfiles, install the lib in there and build, deploy and maintain infrastructure for your custom Docker images.

st2packs mechanisms which is for custom st2 packs won't help here.

[manisha-tanwar]

@armab Thanks for the clarification, I got confused.

Could you please confirm if modifying base/Dockerfile will work or do I need to modify all the Dockerfiles .

There are just 2 steps:

1.  RUN apt update &&  apt install -y     gcc     libldap2-dev     libsasl2-dev     python-dev      libssl-dev     ldap-utils
2. /opt/stackstorm/st2/bin/pip install git+https://github.com/StackStorm/st2-enterprise-auth-backend-ldap.git@master

[arm4b]

Modifying base/Dockerfile is the correct place, but you still need to re-build all Docker images based on that one.

[manisha-tanwar]

I added both the steps in base/Dockerfile and build/push st2 image.
Modified all Dockerfiles FROM to have latest image.
Modified Makefile to comment out st2 build & push.
Run below commands:
make build
make push
And able to see st2-enterprise-auth-backend-ldap 3.3.dev0 inside st2client pod. But other pods are failing due to error:

st2-linux-job-st2-apikey-load-rlht7                  0/1     Completed          0          10m
st2-linux-job-st2-key-load-6fw5v                     0/1     Completed          0          9m52s
st2-linux-job-st2-register-content-nfqnn             0/1     Completed          0          9m20s
st2-linux-mongodb-ha-0                               1/1     Running            0          10m
st2-linux-mongodb-ha-1                               1/1     Running            0          9m39s
st2-linux-mongodb-ha-2                               1/1     Running            0          9m5s
st2-linux-rabbitmq-ha-0                              1/1     Running            0          10m
st2-linux-rabbitmq-ha-1                              1/1     Running            0          9m54s
st2-linux-rabbitmq-ha-2                              1/1     Running            0          9m28s
st2-linux-st2actionrunner-6bd4c48894-5j2hr           0/1     CrashLoopBackOff   6          10m
st2-linux-st2actionrunner-6bd4c48894-br52v           0/1     CrashLoopBackOff   6          10m
st2-linux-st2actionrunner-6bd4c48894-c9p7x           0/1     CrashLoopBackOff   6          10m
st2-linux-st2actionrunner-6bd4c48894-g5x9j           0/1     CrashLoopBackOff   6          10m
st2-linux-st2actionrunner-6bd4c48894-xn4m2           0/1     CrashLoopBackOff   6          10m
st2-linux-st2api-74d5977c66-drvg8                    1/1     Running            6          10m
st2-linux-st2api-74d5977c66-zqwwd                    0/1     CrashLoopBackOff   6          10m
st2-linux-st2auth-dc66599fc-829np                    0/1     CrashLoopBackOff   6          10m
st2-linux-st2auth-dc66599fc-bd6sk                    0/1     CrashLoopBackOff   5          10m
st2-linux-st2client-5bc685d6cf-p6wvl                 1/1     Running            0          10m
st2-linux-st2garbagecollector-7cb9596d4b-n5jhf       0/1     CrashLoopBackOff   5          10m
st2-linux-st2notifier-6cb9795c7b-266vj               1/1     Running            6          10m
st2-linux-st2notifier-6cb9795c7b-5w7j5               0/1     CrashLoopBackOff   5          10m
st2-linux-st2rulesengine-66b8f457bc-5btr2            0/1     CrashLoopBackOff   6          10m
st2-linux-st2rulesengine-66b8f457bc-6hmmh            1/1     Running            6          10m
st2-linux-st2scheduler-85cd57db4c-52nx9              0/1     CrashLoopBackOff   6          10m
st2-linux-st2scheduler-85cd57db4c-xsrhs              0/1     CrashLoopBackOff   6          10m
st2-linux-st2sensorcontainer-67858b76c5-l62j6        0/1     CrashLoopBackOff   6          10m
st2-linux-st2stream-59b989f8c9-j7llw                 0/1     CrashLoopBackOff   5          10m
st2-linux-st2stream-59b989f8c9-trs4t                 0/1     CrashLoopBackOff   6          10m
st2-linux-st2timersengine-7cd85b68d-mcfmj            1/1     Running            6          10m
st2-linux-st2web-6d6f49ffcd-hvsvx                    1/1     Running            0          10m
st2-linux-st2web-6d6f49ffcd-wkkx5                    1/1     Running            0          10m
st2-linux-st2workflowengine-55997cc87-gtf4h          1/1     Running            6          10m
st2-linux-st2workflowengine-55997cc87-xn54v          0/1     CrashLoopBackOff   6          10m

~ $ kubectl logs st2-linux-st2actionrunner-6bd4c48894-xqtbs -n stackstorm
2020-07-20 22:39:09,023 DEBUG [-] Using Python: 3.6.9 (/opt/stackstorm/st2/bin/python)
2020-07-20 22:39:09,024 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2020-07-20 22:39:09,024 DEBUG [-] Using logging config: /etc/st2/logging.actionrunner.conf
2020-07-20 22:39:09,034 INFO [-] Connecting to database "st2" @ "st2-linux-mongodb-ha-0.st2-linux-mongodb-ha:27017,st2-linux-mongodb-ha-1.st2-linux-mongodb-ha:27017,st2-linux-mongodb-ha-2.st2-linux-mongodb-ha:27017 (replica set)" as user "admin".
2020-07-20 22:39:12,040 ERROR [-] Failed to connect to database "st2" @ "st2-linux-mongodb-ha-0.st2-linux-mongodb-ha:27017,st2-linux-mongodb-ha-1.st2-linux-mongodb-ha:27017,st2-linux-mongodb-ha-2.st2-linux-mongodb-ha:27017 (replica set)" as user "admin": No replica set members found yet
2020-07-20 22:39:12,041 ERROR [-] (PID=1) Worker quit due to exception.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/cmd/actionrunner.py", line 95, in main
    _setup()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/cmd/actionrunner.py", line 56, in _setup
    register_signal_handlers=True, service_registry=True, capabilities=capabilities)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/service_setup.py", line 160, in setup
    db_setup()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/database_setup.py", line 56, in db_setup
    connection = db_init.db_setup_with_retry(**db_cfg)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 75, in db_setup_with_retry
    ssl_match_hostname=ssl_match_hostname)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 58, in db_func_with_retry
    return retrying_obj.call(db_func, *args, **kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/six.py", line 696, in reraise
    raise value
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 169, in db_setup
    ssl_match_hostname=ssl_match_hostname)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 151, in _db_connect
    raise e
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 144, in _db_connect
    connection.admin.command('ismaster')
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/database.py", line 730, in command
    read_preference, session) as (sock_info, slave_ok):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1298, in _socket_for_reads
    server = self._select_server(read_preference, session)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1253, in _select_server
    server = topology.select_server(server_selector)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 235, in select_server
    address))
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 193, in select_servers
    selector, server_timeout, address)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 209, in _select_servers_loop
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: No replica set members found yet

I haven't pushed ldap configuration yet inside st2.user.conf or st2.docker.conf.

Am I missing something? Please help, seems like I'm very close but can't figure it out.

Edit: Issues looks like #139 but mine was working fine with default st2-dockerfiles.

I tried updating mongodb replicaset versions too as mentioned but no luck.

dependencies:
  - name: rabbitmq-ha
    version: 1.44.1
    repository: https://kubernetes-charts.storage.googleapis.com/
    condition: rabbitmq-ha.enabled
  - name: mongodb-replicaset
    version: 3.12.0
    repository: https://kubernetes-charts.storage.googleapis.com/
    alias: mongodb-ha
    condition: mongodb-ha.enabled
  - name: external-dns
    version: 1.6.1
    repository: https://kubernetes-charts.storage.googleapis.com/
    condition: external-dns.enabled
  - name: etcd-operator
    version: 0.10.3
    repository: https://kubernetes-charts.storage.googleapis.com/
    condition: etcd-operator.enabled```

[arm4b]

pymongo.errors.ServerSelectionTimeoutError: No replica set members found yet

That's the regression appeared recently in the current st2 v3.3dev, see StackStorm/st2#4997. The fix is planned t be merged this week.

[manisha-tanwar]

Thanks for the response @armab you saved my night, spent whole day trying to figure out what went wrong. Even fresh stackstorm-ha install isn't working. Might be worth waiting now and hopefully my ldap config works fine.
Cheers!

[manisha-tanwar]

So finally this error was resolved after your fix and I'm back to ldap now.
I'm able to see st2-enterprise-auth-backend-ldap, ldap confguration passed through st2.user.conf on st2client pod as below:

Configuration File:

root@stackstorm-linux-st2client-5bdfd7fd5b-t5jst:/opt/stackstorm# cat /etc/st2/st2.user.conf
[api]
allow_origin = '*'
#[actionrunner]
#logging = /etc/st2/logging.actionrunner.conf
#virtualenv_opts =
#python3_binary = /usr/bin/python3.6
[auth]
mode = standalone
backend = ldap
backend_kwargs = {"bind_dn": "***", "bind_password": "***", "base_ou": "**", "id_attr": "sAMAccountName", "group_dns": ["**","**"],"group_dns_check": "or", "host": "***", "port": 636, "use_ssl": true, "cacert": "/path/to/ca-bundle.crt"}
enable = True
use_ssl = True
logging = /etc/st2/logging.auth.gunicorn.conf
api_url =
debug = True

CA cert pushed through st2-dockerfiles:

root@stackstorm-linux-st2client-5bdfd7fd5b-t5jst:/opt/stackstorm# ll /path/to/ca-bundle.crt
-r--r--r-- 1 root root 267588 Jul 20 23:03 /path/to/ca-bundle.crt

Pip packages installed:

root@stackstorm-linux-st2client-5bdfd7fd5b-t5jst:/opt/stackstorm# st2/bin/pip list| grep backend
st2-auth-backend-flat-file       0.1.0
st2-auth-backend-pam             0.1.0
st2-enterprise-auth-backend-ldap 3.3.dev0

But st2auth pod has status "ErrImagePull" and error logs as:

Error from server (BadRequest): container "Init:ErrImagePull" in pod "stackstorm-linux-st2auth-6fcccb866c-lf589" is waiting to start: image can't be pulled

Do I need to comment this out:

  initContainers:

  - name: generate-htpasswd
    image: "{{ template "imageRepository" . }}/st2auth{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}"
    imagePullPolicy: {{ .Values.image.pullPolicy }}
    env:
    - name: ST2_AUTH_USERNAME
      valueFrom:
        secretKeyRef:
          name: {{ .Release.Name }}-st2-auth
          key: username
    - name: ST2_AUTH_PASSWORD
      valueFrom:
        secretKeyRef:
          name: {{ .Release.Name }}-st2-auth
          key: password
    volumeMounts:
    - name: htpasswd-vol
      mountPath: /tmp/st2
    command:
      - 'sh'
      - '-ec'
      - printf "${ST2_AUTH_USERNAME}:$(openssl passwd -apr1 "${ST2_AUTH_PASSWORD}")\n" > /tmp/st2/htpasswd

[manisha-tanwar]

After commenting out this initcontainer, I got new errors, but ldap is working :)

$ kubectl logs -f stackstorm-linux-st2auth-56c668cc58-hpsm4 --all-containers
2020-07-24 01:10:09,902 DEBUG [-] Using Python: 3.6.9 (/opt/stackstorm/st2/bin/python)
2020-07-24 01:10:09,993 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2020-07-24 01:10:09,996 DEBUG [-] Using logging config: /etc/st2/logging.auth.gunicorn.conf
2020-07-24 01:10:10,399 INFO [-] Connecting to database "st2" @ "stackstorm-linux-mongodb-ha-0.stackstorm-linux-mongodb-ha:27017,stackstorm-linux-mongodb-ha-1.stackstorm-linux-mongodb-ha:27017,stackstorm-linux-mongodb-ha-2.stackstorm-linux-mongodb-ha:27017 (replica set)" as user "admin".
2020-07-24 01:10:10,902 INFO [-] Successfully connected to database "st2" @ "stackstorm-linux-mongodb-ha-0.stackstorm-linux-mongodb-ha:27017,stackstorm-linux-mongodb-ha-1.stackstorm-linux-mongodb-ha:27017,stackstorm-linux-mongodb-ha-2.stackstorm-linux-mongodb-ha:27017 (replica set)" as user "admin".
2020-07-24 01:10:39,098 INFO [-] ST2 Auth API running in "standalone" auth mode
2020-07-24 01:10:39,099 INFO [-] (PID=1) ST2 Auth API is serving on http://0.0.0.0:9100.
2020-07-24 01:10:39,099 INFO [-] Creating st2auth: StackStorm v3.3dev as OpenAPI app.
2020-07-24 01:12:04,979 INFO [-] 684334ee-3a71-4b13-be14-ce7d894f3849 - POST /tokens with query={} (method='POST',path='/tokens',remote_addr='10.33.209.232',query={},request_id='684334ee-3a71-4b13-be14-ce7d894f3849')
2020-07-24 01:12:05,351 ERROR [-] Unable to identify user for "sAMAccountName=st2admin".
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth_enterprise_ldap_backend/ldap_backend.py", line 162, in authenticate
    user_dn = self._get_user_dn(connection=connection, username=username)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth_enterprise_ldap_backend/ldap_backend.py", line 307, in _get_user_dn
    user_dn, _ = self._get_user(connection=connection, username=username)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth_enterprise_ldap_backend/ldap_backend.py", line 329, in _get_user
    raise ValueError(msg)
ValueError: Unable to identify user for "sAMAccountName=st2admin".
2020-07-24 01:12:05,393 ERROR [-] Failed to call controller function "post" for operation "st2auth.controllers.v1.auth:token_controller.post": Invalid or missing credentials
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 515, in __call__
    resp = func(**kw)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/controllers/v1/auth.py", line 78, in post
    **kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/handlers.py", line 224, in handle_auth
    abort_request()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/handlers.py", line 38, in abort_request
    return abort(status_code, message)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 77, in abort
    raise exc.status_map[status_code](message)
webob.exc.HTTPUnauthorized: Invalid or missing credentials

[arm4b]

With Invalid or missing credentials you're probably missing correct st2 username:password pair for st2admin.
Make sure st2admin username and password placed in Helm values and in LDAP backend actually match.

[manisha-tanwar]

Does that mean I've to create st2admin on Ldap too? Or is there any way to remove this check at all. Like comment out st2-apikey-load job or something

[manisha-tanwar]

I commmented everything related to st2admin user in deployments.yaml & jobs.yaml, not getting the error anymore and my ldap is also working fine. Thanks for the help @armab
Issue is good to close!