Skip to content

[Issue 9728] Upgrade GRPC and bookkeeper #9729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merlimat merged 9 commits into from
Mar 5, 2021
Merged

[Issue 9728] Upgrade GRPC and bookkeeper #9729

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7cbc36b
Upgraded bookkeeper to 4.13.0; grpc to 1.33
dlg99 Feb 10, 2021
66f7393
updated LICENSE
dlg99 Feb 25, 2021
0fc6eac
fixed the rest of the license check
dlg99 Feb 26, 2021
5760089
no idea why license check results on CI are different from local :/
dlg99 Feb 26, 2021
1ba4bdd
one more license fix
dlg99 Feb 26, 2021
69e8d6f
Integration tests failed to start container/service there because of …
dlg99 Feb 26, 2021
a12ab19
another try to get the permark.jar in
dlg99 Feb 26, 2021
5fc4c94
another take on perfmark.jar and integration tests
dlg99 Mar 1, 2021
5178309
add perfmark.jar to LICENSE
dlg99 Mar 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions distribution/server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,12 @@
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>compile</scope>
</dependency>

<!-- bookkeeper http server -->
<dependency>
<groupId>org.apache.bookkeeper.http</groupId>
Expand Down
108 changes: 61 additions & 47 deletions distribution/server/src/assemble/LICENSE.bin.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -320,8 +320,11 @@ The Apache Software License, Version 2.0
- com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.11.1.jar
- com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.11.1.jar
- com.fasterxml.jackson.module-jackson-module-jsonSchema-2.11.1.jar
- org.codehaus.jackson-jackson-core-asl-1.9.11.jar
- org.codehaus.jackson-jackson-mapper-asl-1.9.11.jar
* Caffeine -- com.github.ben-manes.caffeine-caffeine-2.6.2.jar
* Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.12.0.jar
* Conscrypt -- org.conscrypt-conscrypt-openjdk-uber-2.5.1.jar
* Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.17.0.jar
* Joda -- org.joda-joda-convert-2.2.1.jar
* Bitbucket -- org.bitbucket.b_c-jose4j-0.7.2.jar
* Gson
Expand Down Expand Up @@ -395,32 +398,32 @@ The Apache Software License, Version 2.0
- org.apache.logging.log4j-log4j-1.2-api-2.14.0.jar
* Java Native Access JNA -- net.java.dev.jna-jna-4.2.0.jar
* BookKeeper
- org.apache.bookkeeper-bookkeeper-common-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-common-allocator-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-proto-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-server-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-tools-framework-4.12.1.jar
- org.apache.bookkeeper-circe-checksum-4.12.1.jar
- org.apache.bookkeeper-cpu-affinity-4.12.1.jar
- org.apache.bookkeeper-statelib-4.12.1.jar
- org.apache.bookkeeper-stream-storage-api-4.12.1.jar
- org.apache.bookkeeper-stream-storage-common-4.12.1.jar
- org.apache.bookkeeper-stream-storage-java-client-4.12.1.jar
- org.apache.bookkeeper-stream-storage-java-client-base-4.12.1.jar
- org.apache.bookkeeper-stream-storage-proto-4.12.1.jar
- org.apache.bookkeeper-stream-storage-server-4.12.1.jar
- org.apache.bookkeeper-stream-storage-service-api-4.12.1.jar
- org.apache.bookkeeper-stream-storage-service-impl-4.12.1.jar
- org.apache.bookkeeper.http-http-server-4.12.1.jar
- org.apache.bookkeeper.http-vertx-http-server-4.12.1.jar
- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.12.1.jar
- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.12.1.jar
- org.apache.bookkeeper.tests-stream-storage-tests-common-4.12.1.jar
- org.apache.distributedlog-distributedlog-common-4.12.1.jar
- org.apache.distributedlog-distributedlog-core-4.12.1-tests.jar
- org.apache.distributedlog-distributedlog-core-4.12.1.jar
- org.apache.distributedlog-distributedlog-protocol-4.12.1.jar
- org.apache.bookkeeper.stats-codahale-metrics-provider-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-common-4.13.0.jar
- org.apache.bookkeeper-bookkeeper-common-allocator-4.13.0.jar
- org.apache.bookkeeper-bookkeeper-proto-4.13.0.jar
- org.apache.bookkeeper-bookkeeper-server-4.13.0.jar
- org.apache.bookkeeper-bookkeeper-tools-framework-4.13.0.jar
- org.apache.bookkeeper-circe-checksum-4.13.0.jar
- org.apache.bookkeeper-cpu-affinity-4.13.0.jar
- org.apache.bookkeeper-statelib-4.13.0.jar
- org.apache.bookkeeper-stream-storage-api-4.13.0.jar
- org.apache.bookkeeper-stream-storage-common-4.13.0.jar
- org.apache.bookkeeper-stream-storage-java-client-4.13.0.jar
- org.apache.bookkeeper-stream-storage-java-client-base-4.13.0.jar
- org.apache.bookkeeper-stream-storage-proto-4.13.0.jar
- org.apache.bookkeeper-stream-storage-server-4.13.0.jar
- org.apache.bookkeeper-stream-storage-service-api-4.13.0.jar
- org.apache.bookkeeper-stream-storage-service-impl-4.13.0.jar
- org.apache.bookkeeper.http-http-server-4.13.0.jar
- org.apache.bookkeeper.http-vertx-http-server-4.13.0.jar
- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.13.0.jar
- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.13.0.jar
- org.apache.bookkeeper.tests-stream-storage-tests-common-4.13.0.jar
- org.apache.distributedlog-distributedlog-common-4.13.0.jar
- org.apache.distributedlog-distributedlog-core-4.13.0-tests.jar
- org.apache.distributedlog-distributedlog-core-4.13.0.jar
- org.apache.distributedlog-distributedlog-protocol-4.13.0.jar
- org.apache.bookkeeper.stats-codahale-metrics-provider-4.13.0.jar
* Apache HTTP Client
- org.apache.httpcomponents-httpclient-4.5.5.jar
- org.apache.httpcomponents-httpcore-4.4.9.jar
Expand Down Expand Up @@ -452,24 +455,32 @@ The Apache Software License, Version 2.0
* Google Error Prone Annotations - com.google.errorprone-error_prone_annotations-2.3.4.jar
* Apache Thrifth - org.apache.thrift-libthrift-0.12.0.jar
* OkHttp
- com.squareup.okhttp-okhttp-2.5.0.jar
- com.squareup.okhttp-okhttp-2.7.4.jar
* Okio - com.squareup.okio-okio-1.13.0.jar
* Javassist -- org.javassist-javassist-3.25.0-GA.jar
* gRPC
- io.grpc-grpc-all-1.18.0.jar
- io.grpc-grpc-auth-1.18.0.jar
- io.grpc-grpc-context-1.18.0.jar
- io.grpc-grpc-core-1.18.0.jar
- io.grpc-grpc-netty-1.18.0.jar
- io.grpc-grpc-okhttp-1.18.0.jar
- io.grpc-grpc-protobuf-1.18.0.jar
- io.grpc-grpc-protobuf-lite-1.18.0.jar
- io.grpc-grpc-stub-1.18.0.jar
- io.grpc-grpc-protobuf-nano-1.18.0.jar
- io.grpc-grpc-testing-1.18.0.jar
* gRPC
- io.grpc-grpc-all-1.33.0.jar
- io.grpc-grpc-auth-1.33.0.jar
- io.grpc-grpc-context-1.33.0.jar
- io.grpc-grpc-core-1.33.0.jar
- io.grpc-grpc-netty-1.33.0.jar
- io.grpc-grpc-okhttp-1.33.0.jar
- io.grpc-grpc-protobuf-1.33.0.jar
- io.grpc-grpc-protobuf-lite-1.33.0.jar
- io.grpc-grpc-stub-1.33.0.jar
- io.grpc-grpc-testing-1.33.0.jar
- io.grpc-grpc-alts-1.33.0.jar
- io.grpc-grpc-api-1.33.0.jar
- io.grpc-grpc-grpclb-1.33.0.jar
- io.grpc-grpc-netty-shaded-1.33.0.jar
- io.grpc-grpc-services-1.33.0.jar
- io.grpc-grpc-xds-1.33.0.jar
* Perfmark
- io.perfmark-perfmark-api-0.19.0.jar
* OpenCensus
- io.opencensus-opencensus-api-0.18.0.jar
- io.opencensus-opencensus-contrib-grpc-metrics-0.18.0.jar
- io.opencensus-opencensus-contrib-http-util-0.24.0.jar
- io.opencensus-opencensus-proto-0.2.0.jar
* Jodah
- net.jodah-typetools-0.5.0.jar
* Apache Avro
Expand Down Expand Up @@ -514,10 +525,16 @@ The Apache Software License, Version 2.0
- org.apache.zookeeper-zookeeper-jute-3.6.2.jar
* Snappy Java
- org.xerial.snappy-snappy-java-1.1.7.jar
* Google HTTP Client
- com.google.http-client-google-http-client-jackson2-1.34.0.jar
- com.google.http-client-google-http-client-1.34.0.jar
- com.google.auto.value-auto-value-annotations-1.7.jar
- com.google.re2j-re2j-1.2.jar

BSD 3-clause "New" or "Revised" License
* Google auth library
- com.google.auth-google-auth-library-credentials-0.9.0.jar -- licenses/LICENSE-google-auth-library.txt
- com.google.auth-google-auth-library-credentials-0.20.0.jar -- licenses/LICENSE-google-auth-library.txt
- com.google.auth-google-auth-library-oauth2-http-0.20.0.jar -- licenses/LICENSE-google-auth-library.txt
* LevelDB -- (included in org.rocksdb.*.jar) -- licenses/LICENSE-LevelDB.txt
* JSR305 -- com.google.code.findbugs-jsr305-3.0.2.jar -- licenses/LICENSE-JSR305.txt
* JavaHamcrest -- org.hamcrest-hamcrest-core-1.3.jar -- licenses/LICENSE-Hamcrest.txt
Expand All @@ -531,17 +548,13 @@ MIT License
- org.slf4j-jul-to-slf4j-1.7.25.jar
- org.slf4j-slf4j-api-1.7.25.jar
- org.slf4j-jcl-over-slf4j-1.7.25.jar
* Animal Sniffer Annotations
- org.codehaus.mojo-animal-sniffer-annotations-1.17.jar
* The Checker Framework
- org.checkerframework-checker-compat-qual-2.5.2.jar
- org.checkerframework-checker-qual-3.5.0.jar

Protocol Buffers License
* Protocol Buffers
- com.google.protobuf-protobuf-java-3.11.4.jar -- licenses/LICENSE-protobuf.txt
- com.google.protobuf-protobuf-java-util-3.11.4.jar -- licenses/LICENSE-protobuf.txt
- com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar -- licenses/LICENSE-protobuf.txt

CDDL-1.1 -- licenses/LICENSE-CDDL-1.1.txt
* Java Annotations API
Expand Down Expand Up @@ -571,7 +584,8 @@ CDDL-1.1 -- licenses/LICENSE-CDDL-1.1.txt
* Mimepull -- org.jvnet.mimepull-mimepull-1.9.13.jar

Eclipse Distribution License 1.0 -- licenses/LICENSE-EDL-1.0.txt
* Jakarta Activation -- jakarta.activation-jakarta.activation-api-1.2.1.jar
* Jakarta Activation
- jakarta.activation-jakarta.activation-api-1.2.1.jar
* Jakarta XML Binding -- jakarta.xml.bind-jakarta.xml.bind-api-2.3.2.jar

Eclipse Public License 1.0 -- licenses/LICENSE-AspectJ.txt
Expand Down
1 change: 0 additions & 1 deletion distribution/server/src/assemble/NOTICE.bin.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,6 @@ http://gcc.gnu.org/onlinedocs/libstdc++/manual/license.html
* Performance and memory usage improvement

------------------------------------------------------------------------------------
- org.checkerframework-checker-compat-qual-2.5.2.jar

A few parts of the Checker Framework have more permissive licenses.

Expand Down
5 changes: 5 additions & 0 deletions docker/pulsar-all/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@
<type>pom</type>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>pulsar-offloader-distribution</artifactId>
Expand Down
18 changes: 16 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ flexible messaging model and an intuitive client API.</description>
<!-- apache commons -->
<commons-compress.version>1.19</commons-compress.version>

<bookkeeper.version>4.12.1</bookkeeper.version>
<bookkeeper.version>4.13.0</bookkeeper.version>
<zookeeper.version>3.6.2</zookeeper.version>
<snappy.version>1.1.7</snappy.version> <!-- ZooKeeper server -->
<dropwizardmetrics.version>3.2.5</dropwizardmetrics.version> <!-- ZooKeeper server -->
Expand Down Expand Up @@ -124,7 +124,8 @@ flexible messaging model and an intuitive client API.</description>
<typetools.version>0.5.0</typetools.version>
<protobuf3.version>3.11.4</protobuf3.version>
<protoc3.version>${protobuf3.version}</protoc3.version>
<grpc.version>1.18.0</grpc.version>
<grpc.version>1.33.0</grpc.version>
Copy link
Copy Markdown
Contributor

@eolivelli eolivelli Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we jump to 1.36.0 ?

I known that in BK we stick to 1.33 but IIRC it is due to the etcd client. In Pulsar we do not have an etcd client.
As it will still take more time to test 2.8 before a release, we could upgrade to latest GRPC version now and have all of the time to see if it is working well with Pulsar

Copy link
Copy Markdown
Contributor Author

@dlg99 dlg99 Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eolivelli Etcd client broke because grpc 1.36 depends on netty 1.52.
From what I understood, the culprit was was this change netty/netty#10407 in 4.1.52 that got reverted netty/netty#10980
The change got reverted because TLS_FALSE_START slightly changes the "flow" during handshake which may cause suprises for the end-user
We can try mixing netty 1.51 (or 1.59, where the change is supposedly reverted, it was not out when I tested bookkeeper) with grpc 1.36 but I am not sure how much I can rely on existing tests (and many of them don't pass currently anyways, with or without my changes). I'd very much prefer not to experiment with this.

Copy link
Copy Markdown
Contributor

@eolivelli eolivelli Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine to me

Copy link
Copy Markdown
Member

@lhotari lhotari Mar 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be useful to switch to use gprc-bom also in Pulsar, in the similar way as it was done for Bookkeeper: https://github.com/apache/bookkeeper/blob/7327e3b1c636ed42bdbe231fc30bcb73e527fee8/pom.xml#L462-L468 .
gprc-bom became available in 1.25 and that's why it hasn't been used before.

<perfmark.version>0.19.0</perfmark.version>
<protoc-gen-grpc-java.version>${grpc.version}</protoc-gen-grpc-java.version>
<gson.version>2.8.6</gson.version>
<sketches.version>0.8.3</sketches.version>
Expand Down Expand Up @@ -838,6 +839,19 @@ flexible messaging model and an intuitive client API.</description>
<version>${grpc.version}</version>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<version>${perfmark.version}</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
<artifactId>error_prone_annotations</artifactId>
<groupId>com.google.errorprone</groupId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
Expand Down
1 change: 1 addition & 0 deletions pulsar-broker-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@
<include>net.java.dev.jna:*</include>
<include>com.carrotsearch:*</include>
<include>io.prometheus:*</include>
<include>io.perfmark:*</include>
<include>com.github.ben-manes.caffeine:*</include>
<include>org.glassfish.jersey.*:*</include>
<include>org.rocksdb:*</include>
Expand Down
1 change: 1 addition & 0 deletions pulsar-client-admin-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@
<include>org.glassfish.hk2*:*</include>
<include>com.fasterxml.jackson.*:*</include>
<include>io.grpc:*</include>
<include>io.perfmark:*</include>
<include>com.yahoo.datasketches:*</include>
<include>com.squareup.*:*</include>
<include>com.google.*:*</include>
Expand Down
1 change: 1 addition & 0 deletions pulsar-client-all/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@
<include>org.glassfish.hk2*:*</include>
<include>com.fasterxml.jackson.*:*</include>
<include>io.grpc:*</include>
<include>io.perfmark:*</include>
<include>com.yahoo.datasketches:*</include>
<include>io.netty:*</include>
<include>com.squareup.*:*</include>
Expand Down
1 change: 1 addition & 0 deletions pulsar-client-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@
<include>com.fasterxml.jackson.core:jackson-core</include>
<include>com.fasterxml.jackson.dataformat</include>
<include>io.netty:*</include>
<include>io.perfmark:*</include>
<include>org.eclipse.jetty:*</include>
<include>com.yahoo.datasketches:*</include>
<include>commons-*:*</include>
Expand Down
6 changes: 6 additions & 0 deletions pulsar-functions/instance/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,12 @@
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>org.apache.bookkeeper</groupId>
<artifactId>bookkeeper-common</artifactId>
Expand Down
4 changes: 4 additions & 0 deletions pulsar-functions/localrun-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,10 @@
<pattern>io.grpc</pattern>
<shadedPattern>org.apache.pulsar.functions.runtime.shaded.io.grpc</shadedPattern>
</relocation>
<relocation>
<pattern>io.perfmark</pattern>
<shadedPattern>org.apache.pulsar.functions.runtime.shaded.io.perfmark</shadedPattern>
</relocation>
<relocation>
<pattern>org.joda</pattern>
<shadedPattern>org.apache.pulsar.functions.runtime.shaded.org.joda</shadedPattern>
Expand Down
7 changes: 7 additions & 0 deletions pulsar-functions/localrun/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,13 @@
<groupId>io.grpc</groupId>
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>

</dependencies>

<build>
Expand Down
6 changes: 6 additions & 0 deletions pulsar-functions/proto/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@
</exclusions>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>

</dependencies>
<build>
<plugins>
Expand Down
24 changes: 12 additions & 12 deletions pulsar-sql/presto-distribution/LICENSE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -412,18 +412,18 @@ The Apache Software License, Version 2.0
- async-http-client-2.12.1.jar
- async-http-client-netty-utils-2.12.1.jar
* Apache Bookkeeper
- bookkeeper-common-4.12.1.jar
- bookkeeper-common-allocator-4.12.1.jar
- bookkeeper-proto-4.12.1.jar
- bookkeeper-server-4.12.1.jar
- bookkeeper-stats-api-4.12.1.jar
- bookkeeper-tools-framework-4.12.1.jar
- circe-checksum-4.12.1.jar
- codahale-metrics-provider-4.12.1jar
- cpu-affinity-4.12.1.jar
- http-server-4.12.1.jar
- prometheus-metrics-provider-4.12.1.jar
- codahale-metrics-provider-4.12.1.jar
- bookkeeper-common-4.13.0.jar
- bookkeeper-common-allocator-4.13.0.jar
- bookkeeper-proto-4.13.0.jar
- bookkeeper-server-4.13.0.jar
- bookkeeper-stats-api-4.13.0.jar
- bookkeeper-tools-framework-4.13.0.jar
- circe-checksum-4.13.0.jar
- codahale-metrics-provider-4.13.0.jar
- cpu-affinity-4.13.0.jar
- http-server-4.13.0.jar
- prometheus-metrics-provider-4.13.0.jar
- codahale-metrics-provider-4.13.0.jar
* Apache Commons
- commons-cli-1.2.jar
- commons-codec-1.10.jar
Expand Down
6 changes: 6 additions & 0 deletions pulsar-zookeeper-utils/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,12 @@
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>org.apache.bookkeeper.stats</groupId>
<artifactId>prometheus-metrics-provider</artifactId>
Expand Down