When `authorizationEnabled=true` in proxy.conf the proxy does not appear to perform Authorization check

[frankjkelly]

Describe the bug
When authorizationEnabled=true in proxy.conf the proxy does not appear to execute the authorization plugin.

To Reproduce
Steps to reproduce the behavior:

1. GIVEN we enable Authorization on proxy

root@pulsar-proxy-6f798754db-r9gbw:/pulsar/conf# grep -i authorization proxy.conf 
### ---Authorization --- ###
# Whether authorization is enforced by the Pulsar proxy
authorizationEnabled=true
# Authorization provider as a fully qualified class name
authorizationProvider=com.cogito.platform.signal.stream.pulsar.authn.broker.CogitoAuthorizationProvider
# Whether client authorization credentials are forwared to the broker for re-authorization.
forwardAuthorizationCredentials=false

2. AND GIVEN the Authorization plugin is loaded

[16:02:04] fkelly@Franks-Cogito-Work-Computer:[~/platform2-test]: (feature/sdlc-31257-minikube-integration) klf pulsar-proxy-6f798754db-r9gbw | grep -i authorization
[conf/proxy.conf] Applying config authorizationEnabled = true
[conf/proxy.conf] Applying config authorizationProvider = com.cogito.platform.signal.stream.pulsar.authn.broker.CogitoAuthorizationProvider
19:55:31.069 [main] INFO  com.cogito.platform.signal.stream.pulsar.authn.broker.CogitoAuthorizationProvider - ==> Initialize()
19:55:31.074 [main] INFO  org.apache.pulsar.broker.authorization.AuthorizationService - com.cogito.platform.signal.stream.pulsar.authn.broker.CogitoAuthorizationProvider has been loaded.

3. WHEN a Pulsar command is picked up by the Proxy
4. THEN the Authentication Plugin is executed but the Authorization Plugin is not (and the Role token is passed onto the Broker)

I was also able to replicate this without use of Custom AUth*n providers e.g. My proxy.conf

# grep -i auth proxy.conf | grep -v "#"
authorizationEnabled=true
authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
forwardAuthorizationCredentials=false
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=file:///pulsar/token-proxy/proxy.jwt
tokenAuthClaim=

and on startup

[11:21:23] fkelly@Franks-Cogito-Work-Computer:[~/platform2-test]: (feature/sdlc-31257-minikube-integration) klf pulsar-proxy-6f798754db-5nz4h | grep -i Auth| more
[conf/proxy.conf] Applying config authenticationEnabled = true
[conf/proxy.conf] Applying config authenticationProviders = org.apache.pulsar.broker.authentication.AuthenticationProviderToken
[conf/proxy.conf] Applying config authorizationEnabled = true
[conf/proxy.conf] Applying config brokerClientAuthenticationParameters = file:///pulsar/token-proxy/proxy.jwt
[conf/proxy.conf] Applying config brokerClientAuthenticationPlugin = org.apache.pulsar.client.impl.auth.AuthenticationToken
15:14:20.005 [main] INFO  org.apache.pulsar.broker.authentication.AuthenticationService - org.apache.pulsar.broker.authentication.AuthenticationProviderToken has been loaded.

and again the Authentication provider seems to be executed but not the Authorization Provider

Expected behavior
Unless I am misunderstanding the Auth*n deployment / configuration then I would expect Authorization to occur at the Proxy.

Screenshots
See above

Desktop (please complete the following information):

• Apache Pulsar: 2.5.2
• Kubernetes: minikube version: v1.9.0

Additional context
Add any other context about the problem here.

[sijie]

@frankjkelly Are you using zookeeper or brokerServiceURL for broker discovery? If you don't set zookeeper, proxy will not enable authorization. Because the default authorization plugin is implemented based on zookeeper.

There might be places that we can improve. E.g. we don't need zookeeper if users provide a customized implementation of Authorization plugin.

[frankjkelly]

Hmmm I'm not sure. In proxy.conf we have the following for brokerService

root@pulsar-proxy-764f4b6569-flg8m:/pulsar/conf# grep broker proxy.conf  | grep -v "#"
brokerServiceURL=http://pulsar-broker:6650
brokerServiceURLTLS=https://pulsar-broker:6651
brokerWebServiceURL=http://pulsar-broker:8080
brokerWebServiceURLTLS=https://pulsar-broker:8443
authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
brokerClientAuthenticationPlugin=
brokerClientAuthenticationParameters=
brokerClientTrustCertsFilePath=

and the following zookeeper related

root@pulsar-proxy-764f4b6569-flg8m:/pulsar/conf# grep zoo proxy.conf 
zookeeperServers=pulsar-zookeeper-0.pulsar-zookeeper
configurationStoreServers=pulsar-zookeeper-0.pulsar-zookeeper
# These settings are unnecessary if `zookeeperServers` is specified
zookeeperSessionTimeoutMs=30000

I see from this StreamNative documentation that by defining the brokerServiceURL we are effectively disabling Authorization in the proxy?

If that's right I guess we can defer the Authorization to the Broker but the problem I believe I have is that our JWT token contains custom claims that indicate whether a topic is accessible or not (it's not Role-based) so we need to pass the token along to the Authorization plugin for parsing. In standalone - when the Authentication and Authorization plugin are colocated this is fine but it seems that when the Proxy is doing Authentication and the Broker is doing Authorization the original token is no longer present at the Broker? I tried forwardAuthorizationCredentials=true but it did not seem to make a difference but perhaps I am missing something.

Thanks in advance

[sijie]

If you set forwardAuthorizationCredentials to true, proxy will forward the authentication credentials to the brokers and brokers will do both authentication and authorization. But in proxy, you need to do configure a super-user token, because proxy is required to read the partitioned topic metadata and lookup topic ownerships.

So you need to configure the following:

forwardAuthorizationCredentials=true
brokerClientAuthenticationPlugin=[plugin]
brokerClientAuthenticationParameters=[parameters]

[sijie]

You can also check this video: https://www.youtube.com/watch?v=sTISVpyq73o

[vzhikserg]

We see the same behavior using 2.6.0 on AKS deployment when connecting (jwt) via proxy with authentication and authorization enabled (using zookeeper):

• Client has no permissions an a topic the client is rejected, as expected

• Client has been granted for example only produce, then the client is able to consume as well (besides producing) I assume because the superUserRole is being used.

When we configure the proxyRoles and apply the same permissions as the client it works, however this implies that proxy clients get the proxyRoles or superUserRoles.

Meaning that in case we have 2 clients connecting to the same topic: one with consume and the other with produce permissions, then proxyRole would need produce and consume permissions, which leads to that both clients can produce and consume.

P.S. we watched the video twice ;)

[sijie]

@vzhikserg: the proxyRole is used for topic lookup (to figure who is the owner broker for a given topic) before establishing the proxied connections to the brokers. Hence the proxyRole is required to perform topic lookup over all topics. That means the proxyRole has to be a super-user role. Does that make sense?

[frankjkelly]

Could be related to #7830

[sijie]

If you are using 2.6.0, I think there are some issues with handling proxy forwarded credentials. The change in #7788 fixes it.

[frankjkelly]

Thanks @sijie yes I think you are right - is that changed scheduled to go out with 2.6.1?