[Bug] pulsar-all 3.0.0 docker image contains Pulsar 2.11.0

[packageman]

Version

Ubuntu 20.04, Pulsar 3.0.0

Minimal reproduce step

What did you expect to see?

pulsar-related libs in pulsar-all:3.0.0 image should be 3.0.0 also.

What did you see instead?

pulsar-related libs in pulsar-all:3.0.0 is 2.11.0.

[tisonkun]

cc @cbornet @RobertIndie the Release Manager for this version.

[RobertIndie]

I think the root cause is that we use apachepulsar/pulsar:latest to build the pulsar-all image.

pulsar/docker/pulsar-all/Dockerfile

Lines 29 to 31 in ab810f4

	FROM apachepulsar/pulsar:latest
	COPY --from=pulsar-all /connectors /pulsar/connectors
	COPY --from=pulsar-all /offloaders /pulsar/offloaders

But at the time of building Pulsar 3.0.0, apachepulsar/pulsar:latest was pointing to version 2.11.0(I'm curious why it's not 2.11.1).

We can see that the connectors and offloaders are pointing to the 3.0.0:

docker run apachepulsar/pulsar-all:3.0.0 ls connectors
LICENSE
README
pulsar-io-aerospike-3.0.0.nar
pulsar-io-alluxio-3.0.0.nar
pulsar-io-batch-data-generator-3.0.0.nar
pulsar-io-canal-3.0.0.nar
pulsar-io-cassandra-3.0.0.nar
pulsar-io-data-generator-3.0.0.nar
pulsar-io-debezium-mongodb-3.0.0.nar
pulsar-io-debezium-mssql-3.0.0.nar
pulsar-io-debezium-mysql-3.0.0.nar
pulsar-io-debezium-oracle-3.0.0.nar
pulsar-io-debezium-postgres-3.0.0.nar
pulsar-io-dynamodb-3.0.0.nar
pulsar-io-elastic-search-3.0.0.nar
pulsar-io-file-3.0.0.nar
pulsar-io-flume-3.0.0.nar
pulsar-io-hbase-3.0.0.nar
pulsar-io-hdfs2-3.0.0.nar
pulsar-io-hdfs3-3.0.0.nar
pulsar-io-http-3.0.0.nar
pulsar-io-influxdb-3.0.0.nar
pulsar-io-jdbc-clickhouse-3.0.0.nar
pulsar-io-jdbc-mariadb-3.0.0.nar
pulsar-io-jdbc-openmldb-3.0.0.nar
pulsar-io-jdbc-postgres-3.0.0.nar
pulsar-io-jdbc-sqlite-3.0.0.nar
pulsar-io-kafka-3.0.0.nar
pulsar-io-kafka-connect-adaptor-3.0.0.nar
pulsar-io-kinesis-3.0.0.nar
pulsar-io-mongo-3.0.0.nar
pulsar-io-netty-3.0.0.nar
pulsar-io-nsq-3.0.0.nar
pulsar-io-rabbitmq-3.0.0.nar
pulsar-io-redis-3.0.0.nar
pulsar-io-solr-3.0.0.nar
pulsar-io-twitter-3.0.0.nar

docker run apachepulsar/pulsar-all:3.0.0 ls offloaders
tiered-storage-file-system-3.0.0.nar
tiered-storage-jcloud-3.0.0.nar

But the libraries under lib/ are not.

If my assumption is correct, then this is a serious problem. The pulsar-all:3.0.0 is actually a version 2.11.0 of Pulsar but with 3.0.0 connectors and offloaders.

I'm rebuilding the pulsar-all image. And then I will publish it to the docker hub.
We also need to fix this dockerfile

[tisonkun]

Thanks for your investigation @RobertIndie!

Then this can be of the same reason #19544 (comment).

cc @nodece @michaeljmarshall for your information.

[gaecfov]

I think the root cause is that we use apachepulsar/pulsar:latest to build the pulsar-all image.

pulsar/docker/pulsar-all/Dockerfile

Lines 29 to 31 in ab810f4

	FROM apachepulsar/pulsar:latest
	COPY --from=pulsar-all /connectors /pulsar/connectors
	COPY --from=pulsar-all /offloaders /pulsar/offloaders

But at the time of building Pulsar 3.0.0, apachepulsar/pulsar:latest was pointing to version 2.11.0(I'm curious why it's not 2.11.1).

We can see that the connectors and offloaders are pointing to the 3.0.0:

docker run apachepulsar/pulsar-all:3.0.0 ls connectors
LICENSE
README
pulsar-io-aerospike-3.0.0.nar
pulsar-io-alluxio-3.0.0.nar
pulsar-io-batch-data-generator-3.0.0.nar
pulsar-io-canal-3.0.0.nar
pulsar-io-cassandra-3.0.0.nar
pulsar-io-data-generator-3.0.0.nar
pulsar-io-debezium-mongodb-3.0.0.nar
pulsar-io-debezium-mssql-3.0.0.nar
pulsar-io-debezium-mysql-3.0.0.nar
pulsar-io-debezium-oracle-3.0.0.nar
pulsar-io-debezium-postgres-3.0.0.nar
pulsar-io-dynamodb-3.0.0.nar
pulsar-io-elastic-search-3.0.0.nar
pulsar-io-file-3.0.0.nar
pulsar-io-flume-3.0.0.nar
pulsar-io-hbase-3.0.0.nar
pulsar-io-hdfs2-3.0.0.nar
pulsar-io-hdfs3-3.0.0.nar
pulsar-io-http-3.0.0.nar
pulsar-io-influxdb-3.0.0.nar
pulsar-io-jdbc-clickhouse-3.0.0.nar
pulsar-io-jdbc-mariadb-3.0.0.nar
pulsar-io-jdbc-openmldb-3.0.0.nar
pulsar-io-jdbc-postgres-3.0.0.nar
pulsar-io-jdbc-sqlite-3.0.0.nar
pulsar-io-kafka-3.0.0.nar
pulsar-io-kafka-connect-adaptor-3.0.0.nar
pulsar-io-kinesis-3.0.0.nar
pulsar-io-mongo-3.0.0.nar
pulsar-io-netty-3.0.0.nar
pulsar-io-nsq-3.0.0.nar
pulsar-io-rabbitmq-3.0.0.nar
pulsar-io-redis-3.0.0.nar
pulsar-io-solr-3.0.0.nar
pulsar-io-twitter-3.0.0.nar

docker run apachepulsar/pulsar-all:3.0.0 ls offloaders
tiered-storage-file-system-3.0.0.nar
tiered-storage-jcloud-3.0.0.nar

But the libraries under lib/ are not.

If my assumption is correct, then this is a serious problem. The pulsar-all:3.0.0 is actually a version 2.11.0 of Pulsar but with 3.0.0 connectors and offloaders.

I'm rebuilding the pulsar-all image. And then I will publish it to the docker hub. We also need to fix this dockerfile

The configuration file also needs to be checked. I encountered a problem of missing 'enableBrokerSideSubscriptionPatternEvaluation' parameter in the broker.conf, but it is correct in pulsar:3.0.0.

[lhotari]

I think the root cause is that we use apachepulsar/pulsar:latest to build the pulsar-all image.

pulsar/docker/pulsar-all/Dockerfile

Lines 29 to 31 in ab810f4

	FROM apachepulsar/pulsar:latest
	COPY --from=pulsar-all /connectors /pulsar/connectors
	COPY --from=pulsar-all /offloaders /pulsar/offloaders

But at the time of building Pulsar 3.0.0, apachepulsar/pulsar:latest was pointing to version 2.11.0(I'm curious why it's not 2.11.1).

@cbornet @RobertIndie
I don't think that this is the root cause. In the current build, all docker images should be built locally without pulling apachepulsar/pulsar:latest from the remote repository. In that case, there's no chance that versions get mixed regardless of the remote versions.

How did the docker images get built and pushed?

We should be updating the release guide if there are problems (which was very common in the past).
I'd rather have the release guide in the GitHub wiki since that would lower the barrier to update the docs if there are issues that need to be addressed.
The current release process is maintained on the website, https://pulsar.apache.org/contribute/release-process/#stage-docker-images . That's fine as long as we continue to maintain it.

[RobertIndie]

@lhotari
When we build the image while preparing the release candidate, we use our own account name as docker hub org name instead of apachepulsar. You can check the release process here: https://pulsar.apache.org/contribute/release-process/#stage-docker-images (But actually, we need to modify this process because we have changed our build script)

Therefore the apachepulsar/pulsar is not pointed to the locally built image, which causes the issue.

[RobertIndie]

Hi, @gaecfov

The configuration file also needs to be checked. I encountered a problem of missing 'enableBrokerSideSubscriptionPatternEvaluation' parameter in the broker.conf, but it is correct in pulsar:3.0.0.

This is the same issue. 'enableBrokerSideSubscriptionPatternEvaluation' only be included in 3.0.0. But currently, pulsar-all:3.0.0 is actually an image for 2.11.0. This problem will be fixed after I republish the image.
Here is the verification:

docker run snzkyang/pulsar-all:3.0.0 cat conf/broker.conf | grep enableBrokerSideSubscriptionPatternEvaluation

enableBrokerSideSubscriptionPatternEvaluation=true

[tisonkun]

I'd rather have the release guide in the GitHub wiki since that would lower the barrier to update the docs if there are issues that need to be addressed.

No. GitHub wiki is a half completed GitHub repo that few people know how to trace its history or collaborate with comment. As a committer you can already update the release process on the site repo without any barrier - what's extra here?

[nodece]

I suggest we only use the apachepulsar/X.Y.Z image, not the latest image, which can avoid this issue.

[lhotari]

I'd rather have the release guide in the GitHub wiki since that would lower the barrier to update the docs if there are issues that need to be addressed.

No. GitHub wiki is a half completed GitHub repo that few people know how to trace its history or collaborate with comment. As a committer you can already update the release process on the site repo without any barrier - what's extra here?

If the instructions aren't maintained, there might be a barrier. It could be resolved by requesting everyone performing releases to ask for help and/or update the instructions if there are problems.

[lhotari]

I suggest we only use the apachepulsar/X.Y.Z image, not the latest image, which can avoid this issue.

I don't think that solves the problem. Both images should be built locally and nothing should ever be pulled.

[nodece]

I submitted #20433 to fix this, please help to review, thank you for your time!