Investigate authentication options

[lfrancke]

If possible - I have to admit that I don't know - we'd like to support authentication for Spark as well.

This ticket is meant to just investigate the options that we have and then create follow-up issues once we've made a decision on what to do.

[maltesander]

I read a little bit within the spark doc and wrote down my findings.

We already support the basic authentication via secret provided in the custom resource. Depending on the Kubernetes settings this is not very safe.

In general we have to differentiate between inner spark communication (e.g. executors, drivers etc.) and the web Ui.

Spark process authentication

The secret covers the inner spark process communication (spark automatically shares the one(!) defined secret to all participants ) [source]. Currently it is shared in plain text and not encrypted in the custom resource, which can be read by anyone having the rights to list custom resources or config maps in that namespace.

From spark 3.0.0 we can mount secrets into the pods [source]. It will still be up to us to ensure the safety of that process.

Within Kubernetes, Spark automatically generates and shares unique secrets for each application which are propagated to the pods using environment variables.

Web Ui authentication

For the web UI, servlet filters are used. That means (if written by us) we can support anything we need [source]. Or fallback on existing solutions. Spark does not offer any filters itself.

ACLs with "view" (who can see the Ui) and "modify" (who can e.g. kill jobs) permissions are supported on a group and user basis [source].