Bump to Pulsar 2.8.2

[315157973]

Motivation

Bump to Pulsar 2.8.2

@codelipenghui Please help me push the docker images to the docker hub, and then review this PR. Thanks.

[lhotari]

PR #180 should be merged before this change.

[lhotari]

It seems that #188 should also be resolved before switching to Pulsar 2.8.x .

[lhotari]

Failures in CI which points to #188. I'll push a fix to this PR.

Unrecognized VM option 'PrintGCTimeStamps'                    
Error: Could not create the Java Virtual Machine.             
Error: A fatal exception has occurred. Program will exit.     
Unrecognized VM option 'PrintGCTimeStamps'                    
Error: Could not create the Java Virtual Machine.             
Error: A fatal exception has occurred. Program will exit.     

[lhotari]

Postpone merging until there's a decision whether we should go forward with switching to Pulsar 2.8.x in the helm chart because of a known issue with Zookeeper (https://issues.apache.org/jira/browse/ZOOKEEPER-3988 / apache/pulsar#11070 ).

Mailing list discussion: https://lists.apache.org/thread/619tpn6q5xbbhngwsmhtq3121vhjxpt4

[eolivelli]

Also, why should we ship 2.8 as default instead of 2.9.1?

[lhotari]

Also, why should we ship 2.8 as default instead of 2.9.1?

I guess it could be about doing one step at a time (2.7.x -> 2.8.x, then 2.8.x->2.9.x) and also about stability.

[lhotari]

I created a PR #195 which adds better logging to CI which would help investigating CI failures. I have observed the "ZK TLS Only" CI job failing, presumably with the known problem. However it would be nice to see the logs too once #195 is reviewed and merged.

[ckdarby]

Also, why should we ship 2.8 as default instead of 2.9.1?

I guess it could be about doing one step at a time (2.7.x -> 2.8.x, then 2.8.x->2.9.x) and also about stability.

It would also be community friendly to do 2.8.X as a helm release version and then 2.9.X as it allows the community to run 2.8 from a helm chart and not need to override tags.

[lhotari]

I have created #202 as a workaround for the Zookeeper issue (when TLS is enabled).

[lhotari]

It's possible that the Zookeeper issue is simply caused by the probe getting stuck.
The changes in #179 fix the issue for 1.20+ . I'll send a separate PR to address the issue for Kubernetes <1.20.

[lhotari]

Rebased after #214 changes. Let's see if the Zookeeper TLS tests pass now.

[lhotari]

All tests pass now. I'll inform about this on the dev mailing list thread.

[lhotari]

Closing and re-opening to run the tests one more time to see that the problem is fixed.

[lhotari]

Problems with ZK & TLS aren't resolved yet.

[lhotari]

I'm hoping to get more logs from the failure after #215 changes are in place in CI.

[lhotari]

I finally got some logs from CI. It looks like the metadata initialization job is stuck. I'm adding timeouts in #218 to see if that could help mitigate the issue.

I'm aware of a Zookeeper fix in the works for https://issues.apache.org/jira/browse/ZOOKEEPER-3988 . I haven't been able to find signs of ZOOKEEPER-3988 issue in the logs that were collected from the failing CI jobs.

I uploaded the relevant log files to a gist, https://gist.github.com/lhotari/d1bf977cfbd5f1fdbd942df9fef3c952

@eolivelli Could you check if the logs match ZOOKEEPER-3988?

To me it looks like another issue. I found https://issues.apache.org/jira/browse/ZOOKEEPER-3466 , https://issues.apache.org/jira/browse/ZOOKEEPER-3828 and https://issues.apache.org/jira/browse/ZOOKEEPER-3706 . I wonder if it is ZOOKEEPER-3706, fixed by apache/zookeeper#1235 . @eolivelli Would it be possible to get this fix to the next Zookeeper release that we could use for Pulsar 2.8.x and above?

[lhotari]

Promising changes from @frederic-kneier in apache/pulsar#14088 . It might resolve the issue that has been blocking us from upgrading to Pulsar 2.8.x in the Helm chart.

[lhotari]

Rebased on latest master branch changes.

[frankjkelly]

@lhotari are any of these changes (esp. the JVM setting changes) backwards incompatible with 2.7.x Pulsar? Just wondering if I can mix-and-match this 2.8.x helm chart and override the image tags with a 2.7.x pulsar image? Thanks!

[michaeljmarshall]

We've moved passed 2.8, so I am going to close this PR.