Streamline access permissions across catalog, query engine and superset

[martyngigg]

We don't have a harmonized system for access permissions across the various applications. Currently we

• deploy Keycloak as an identity provider for the Lakekeeper catalog. It connects to LDAP for authentication.
• connect Superset directly to LDAP for authentication and use its own roles mechanism for assigning permissions to datasets.
• connect Trino to Keycloak for authorisation and define access rules in Trino

As more people and groups get onboard this will become a challenge to manage. We should aim to centralize all of this information such that it is consistent across all of the catalog access points, namely BI tools (superset) & programmatic tools (Python/PyIceberg/).

Useful links:

• https://docs.lakekeeper.io/docs/nightly/authorization/: See OpenFGA, OPA & Trino.
• Superset user groups: User Groups for Simplified User and Role Management

[martyngigg]

Authorization backend is now enabled on the deployed Lakekeeper instance allowing permissions to be set for users in the warehouse.

For now we will pause the additional work to sync permissions as the main access root for the FASE data (this will need the most fine-grained permissions) can all be handled in Superset. The work involved to sync across all entry points could become involved and is not yet worth spending time on.