Upgrade Apache BookKeeper Client to 4.11.1 #8270

eolivelli · 2020-10-15T11:05:40Z

Motivation

Upgrade Apache BookKeeper client to version 4.11.1 and RocksDB to 6.10.2 (needed by Bookie)

Modifications

Upgrade Apache BookKeeper client, fix a few build errors, due to the introduction of BookieServiceInfo structure (see BP-38).

Verifying this change

This change is already covered by existing tests

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes)
The public API: ( no)
The schema: (no )
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)
If a feature is not applicable for documentation, explain why? it is only a dependency upgrade, no new BK features enabled with this change
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

eolivelli · 2020-10-15T12:03:27Z

/pulsarbot run-failure-checks

eolivelli · 2020-10-20T14:27:28Z

some CI jobs failed with this error, it looks like a temporary error

Error: Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:add-source (default) on project flink: Execution default of goal net.alchim31.maven:scala-maven-plugin:4.3.0:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:4.3.0 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.maven:maven-builder-support:jar:3.3.9, org.apache.maven:maven-artifact:jar:3.3.9: Could not transfer artifact org.apache.maven:maven-builder-support:jar:3.3.9 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/org/apache/maven/maven-builder-support/3.3.9/maven-builder-support-3.3.9.jar: Connection timed out (Read failed) -> [Help 1]

eolivelli · 2020-10-20T14:27:40Z

/pulsarbot run-failure-checks

eolivelli · 2020-10-20T15:59:44Z

/pulsarbot run-failure-checks

sijie · 2020-10-21T05:43:35Z

/pulsarbot run-failure-checks

eolivelli · 2020-10-21T06:11:33Z

I have to fix LICENSE stuff.

eolivelli · 2020-10-21T08:34:13Z

/pulsarbot run-failure-checks

jiazhai · 2020-10-28T07:31:12Z

@eolivelli Thanks for the fix. there a license check failed. and would you please also help update this pr with latest master code?

https://github.com/apache/pulsar/pull/8270/checks?check_run_id=1285693542

Run src/check-binary-license ./distribution/server/target/apache-pulsar-*-bin.tar.gz
src/check-binary-license: line 61: echo: write error: Broken pipe
jakarta.activation-jakarta.activation-api-1.2.1.jar unaccounted for in LICENSE
jakarta.xml.bind-jakarta.xml.bind-api-2.3.2.jar unaccounted for in LICENSE
jakarta.activation-jakarta.activation-api-1.2.2.jar mentioned in LICENSE, but not bundled
com.sun.activation-jakarta.activation-1.2.2.jar mentioned in LICENSE, but not bundled
jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar mentioned in LICENSE, but not bundled
jetty-alpn-java-client-9.4.27.v20200227.jar mentioned in lib/presto/LICENSE, but not bundled
jakarta.activation-1.2.2.jar mentioned in lib/presto/LICENSE, but not bundled
jakarta.activation-api-1.2.2.jar mentioned in lib/presto/LICENSE, but not bundled
jakarta.xml.bind-api-2.3.3.jar mentioned in lib/presto/LICENSE, but not bundled

It looks like there are issues with the LICENSE/NOTICE.
Error: Process completed with exit code 2.

eolivelli · 2020-10-28T07:34:34Z

thank you @jiazhai
my plan is to commit #8360 before this one.

Also in that PR I have a problem with license check,
locally the script is passing on my machine, but it fails on CI

this is very weird

eolivelli · 2020-10-28T07:42:54Z

@jiazhai I have fixed the conflict on Presto Distribution license

eolivelli · 2020-10-29T14:07:35Z

This error is very nasty, investigating now.
It is probably due to some dependency imported from the new BK client

Error:  Tests run: 11, Failures: 1, Errors: 0, Skipped: 10, Time elapsed: 4.263 s <<< FAILURE! - in org.apache.pulsar.client.TlsProducerConsumerTest
Error:  setup(org.apache.pulsar.client.TlsProducerConsumerTest)  Time elapsed: 4.166 s  <<< FAILURE!
java.lang.NoSuchFieldError: xmss_with_SHA256
	at java.security.AccessController.doPrivileged(Native Method)
	at java.lang.Class.newInstance(Class.java:442)
	at org.apache.pulsar.common.util.SecurityUtility.getBCProviderFromClassPath(SecurityUtility.java:125)
	at org.apache.pulsar.common.util.SecurityUtility.getProvider(SecurityUtility.java:102)
	at org.apache.pulsar.common.util.SecurityUtility.<clinit>(SecurityUtility.java:69)
	at org.apache.pulsar.broker.web.WebService.<init>(WebService.java:114)
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:479)
	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.startBroker(MockedPulsarServiceBaseTest.java:269)
	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.startBroker(MockedPulsarServiceBaseTest.java:251)

eolivelli · 2020-10-29T14:08:08Z

/pulsarbot run-failure-checks

eolivelli · 2020-10-29T16:01:57Z

The problem was about the fact that now BookKeeper imports a dependency from BouncyCastle and this pollutes the complex handling of BC in Pulsar

eolivelli · 2020-10-30T07:31:07Z

/pulsarbot run-failure-checks

Motivation Upgrade Apache BookKeeper client to version 4.11.1 and RocksDB to 6.10.2 (needed by Bookie) Modifications Upgrade Apache BookKeeper client, fix a few build errors, due to the introduction of BookieServiceInfo structure (see BP-38). Verifying this change This change is already covered by existing tests * Upgrade Apache BookKeeper to 4.11.1 * fix build * fix LICENSE files * Use BK 4.11.1 * Upgrade RocksDB to 6.10.2 * Fix license and upgrade maven assembly plugin * update license * restore bouncy castle license * fix licenses * Exclude BouncyCastle inherited from BK Co-authored-by: Enrico Olivelli <enrico.olivelli@diennea.com> Co-authored-by: Enrico Olivelli <eolivelli@apache.org>

changed the lience file for branch-2.6 Motivation Upgrade Apache BookKeeper client to version 4.11.1 and RocksDB to 6.10.2 (needed by Bookie) Modifications Upgrade Apache BookKeeper client, fix a few build errors, due to the introduction of BookieServiceInfo structure (see BP-38). Verifying this change This change is already covered by existing tests * Upgrade Apache BookKeeper to 4.11.1 * fix build * fix LICENSE files * Use BK 4.11.1 * Upgrade RocksDB to 6.10.2 * Fix license and upgrade maven assembly plugin * update license * restore bouncy castle license * fix licenses * Exclude BouncyCastle inherited from BK Co-authored-by: Enrico Olivelli <enrico.olivelli@diennea.com> Co-authored-by: Enrico Olivelli <eolivelli@apache.org>(cherry picked from commit 465964b)

Enrico Olivelli added 2 commits October 15, 2020 12:21

Upgrade Apache BookKeeper to 4.11.1

8e3f5ad

fix build

b78e39f

eolivelli changed the title ~~Fix/upgrade bk 411~~ Upgrade Apache BookKeeper Client to 4.11.1 Oct 15, 2020

Enrico Olivelli added 2 commits October 15, 2020 14:05

fix LICENSE files

1a49848

Use BK 4.11.1

c8e67f0

eolivelli marked this pull request as ready for review October 20, 2020 12:09

Upgrade RocksDB to 6.10.2

887db96

sijie approved these changes Oct 21, 2020

View reviewed changes

sijie assigned eolivelli Oct 21, 2020

sijie added this to the 2.7.0 milestone Oct 21, 2020

sijie added the component/bookkeeper label Oct 21, 2020

eolivelli mentioned this pull request Oct 21, 2020

Remove bouncy castle shaded module to avoid bring error of verifySingleJar #7453

Merged

Fix license and upgrade maven assembly plugin

18e8cb0

jiazhai approved these changes Oct 28, 2020

View reviewed changes

Merge remote-tracking branch 'origin/master' into fix/upgrade-bk-411

6a99311

rdhabalia approved these changes Oct 28, 2020

View reviewed changes

eolivelli added 4 commits October 29, 2020 10:06

Merge remote-tracking branch 'origin/master' into fix/upgrade-bk-411

97320c5

update license

0a25aff

restore bouncy castle license

4c652f0

fix licenses

ccf90ec

Exclude BouncyCastle inherited from BK

2de6127

jiazhai merged commit 465964b into apache:master Oct 30, 2020

codelipenghui mentioned this pull request Nov 4, 2020

Bump bk version to 4.11.0 #7809

Closed

jiazhai mentioned this pull request Nov 18, 2020

[branch-2.6] update to stable version of bk 4.11.1 #8604

Merged

Upgrade Apache BookKeeper Client to 4.11.1 #8270

Upgrade Apache BookKeeper Client to 4.11.1 #8270

Uh oh!

Conversation

eolivelli commented Oct 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

eolivelli commented Oct 15, 2020

Uh oh!

eolivelli commented Oct 20, 2020

Uh oh!

eolivelli commented Oct 20, 2020

Uh oh!

eolivelli commented Oct 20, 2020

Uh oh!

sijie commented Oct 21, 2020

Uh oh!

eolivelli commented Oct 21, 2020

Uh oh!

eolivelli commented Oct 21, 2020

Uh oh!

jiazhai commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 28, 2020

Uh oh!

eolivelli commented Oct 29, 2020

Uh oh!

eolivelli commented Oct 29, 2020

Uh oh!

eolivelli commented Oct 29, 2020

Uh oh!

eolivelli commented Oct 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eolivelli commented Oct 15, 2020 •

edited

Loading