Pulsar proxy fails to start with pulsar Docker image that uses non-root user

[lhotari]

Problem

There's a permission issue in pulsar-proxy when using a Docker image that uses a non-root user.
The 2.8.0-SNAPSHOT Pulsar docker images use a non-root user. This change was made in apache/pulsar#8796 .

Since the default configuration uses port 80 and now when the default user is "pulsar", it cannot bind to port 80.

This is the error log in pulsar-proxy-0 pod:

14:38:21.477 [main] ERROR org.apache.pulsar.proxy.server.ProxyServiceStarter - Failed to start pulsar proxy service. error msg Failed to start HTTP server on ports [80]
java.io.IOException: Failed to start HTTP server on ports [80]
at org.apache.pulsar.proxy.server.WebServer.start(WebServer.java:246) ~[org.apache.pulsar-pulsar-proxy-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.proxy.server.ProxyServiceStarter.<init>(ProxyServiceStarter.java:177) [org.apache.pulsar-pulsar-proxy-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.proxy.server.ProxyServiceStarter.main(ProxyServiceStarter.java:186) [org.apache.pulsar-pulsar-proxy-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Caused by: java.net.SocketException: Permission denied
at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_282]
at sun.nio.ch.Net.bind(Net.java:461) ~[?:1.8.0_282]
at sun.nio.ch.Net.bind(Net.java:453) ~[?:1.8.0_282]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) ~[?:1.8.0_282]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) ~[?:1.8.0_282]
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:345) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[org.eclipse.jetty-jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.server.Server.doStart(Server.java:401) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[org.eclipse.jetty-jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.apache.pulsar.proxy.server.WebServer.start(WebServer.java:224) ~[org.apache.pulsar-pulsar-proxy-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
... 2 more
2021-03-19 02:38:21,480 [sun.misc.Launcher$AppClassLoader@7ef20235] error Uncaught exception in thread main: java.io.IOException: Failed to start HTTP server on ports [80]

The problem goes away after changing the port to 8080 in values.yaml:

proxy:
  ports:
    http: 8080

[michaeljmarshall]

I proposed the following solution in apache/pulsar#8796 (comment). It's copied here:

I think the easiest way to prevent a breaking change in the helm chart is to default to using the root user in the helm chart's deployment for the pulsar proxy. We could then add a configuration that allows users to opt in to running that container as a non root user.

Essentially, we need to default a pulsar proxy pod's security context to the following:

  securityContext:
    runAsUser: 0
    runAsGroup: 0

I believe all other pulsar containers should still be able to run as the non root user.

(I am assuming that we want to prevent breaking changes here as the 2.8.0 release is only a minor version bump.)

I'm happy to discuss alternatives.

[michaeljmarshall]

@sijie and @lhotari - what are your thoughts here? It'd be good to solve this soon so that it doesn't affect end users when we release 2.8.0 soon.

I see a couple options.

1. We could add a value for the targetPort on the pulsar proxy service where the targetPort defaults to 8080. In this case, we could update the template for the proxy service with a block like this:

   spec:
     type: {{ .Values.proxy.service.type }}
     ports:
       {{- if or (not .Values.tls.enabled) (not .Values.tls.proxy.enabled) }}
       - name: http
         port: {{ .Values.proxy.ports.http }}
         targetPort: {{ .Values.proxy.ports.targetHttp }}
         protocol: TCP

   Then, we would open up the targetHttp port on the proxy container and pod definition. This would be helpful because it'd be a seamless transition for end users targeting the service. Technically, any users hitting the pods directly would be affected by this breaking change. Although, I believe that kind of direct interations is an anti-pattern within kubernetes, so I think we could safely ignore that case. Further, the proxy's PodMonitor config is set to hit the named port http, which would mean that it'd make the transition to an arbitrary targetPort seamlessly. (I technically still need to double check that this whole solution works. I'm pretty confident it will, but I'll need to test before pushing up the PR.)
   A major benefit to this solution is that it lets us support rootless containers by default for our end users.
2. As I suggested in my earlier comment, we could default the Pulsar Proxy container to run as the root user, and then let end users manage the upgrade themselves. This solution seems more complicated to me, and it doesn't result in a great experience for end users.

[michaeljmarshall]

In thinking more about this more, we have a problem for our users that upgrade to 2.8.0 from any previous version of pulsar. The persistent data written by prior bookie and zookeeper pods will be owned by the root user, not the new pulsar user, and it's possible that directories like the data directory in the bookie will not be writable for the pulsar user. Given that zookeepers and bookies delete files, they will need more permission in order to function properly after upgrade.

Solution 1 (mentioned above) is not sufficient to prevent breaking clusters during upgrade. We need to either run bookie and zookeeper containers as the root user by default and have users opt in to running them as non-root containers, or we need to solve the upgrade problem in the chart. Perhaps we can solve this by providing an optional init container that properly changes file system ownership for the bookie/zookeeper pods. Given that our users have already been running the containers as the root user, I don't see any conflict in using a privileged init container just this once.

Whatever path we choose, I would like to try to give new users a rootless deployment by default. I am not familiar with how helm manages upgrades versus new deployments, but perhaps there is a way to have conditional logic that branches on upgrade vs new deployment?

[cdbartholomew]

@michaeljmarshall Helm has a built-in variable that can tell you whether it is doing an install or an upgrade. For example, you can do something like this:

{{- if .Release.IsInstall }}
  do something
{{ end }}

Alternatively, you can use Helm Hooks.

[qifei888]

Setting the proxy port to 8080 can resolve this issue.
we can use --set proxy.ports.http=8080 during helm install to change the port to 8080
helm install
--values examples/values-minikube.yaml
--set initialize=true
--set proxy.ports.http=8080
--namespace pulsar
pulsar-mini apache/pulsar

[lhotari]

Fixed.