Skip to content

[Branch 2.7] Upgrade Netty to 4.1.77.Final, Netty Tcnative, grpc and protobuf#16896

Closed
Jason918 wants to merge 11 commits intoapache:branch-2.7from
Jason918:cp-15212+15646
Closed

[Branch 2.7] Upgrade Netty to 4.1.77.Final, Netty Tcnative, grpc and protobuf#16896
Jason918 wants to merge 11 commits intoapache:branch-2.7from
Jason918:cp-15212+15646

Conversation

@Jason918
Copy link
Copy Markdown
Contributor

@Jason918 Jason918 commented Aug 1, 2022

Motivation

Cherry-pick #15212 and #15646 to branch 2.7

There are conflict when cherry-pick them directly.

Modifications

Cherry-pick #15212 and #15646 and resolve License error.

Verifying this change

  • Make sure that the change passes the CI checks.

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-not-needed

@Jason918 Jason918 self-assigned this Aug 1, 2022
This was referenced Aug 1, 2022
Merged
Merged
@Jason918
Copy link
Copy Markdown
Contributor Author

Jason918 commented Aug 1, 2022

/pulsarbot run-failure-checks

lhotari and others added 4 commits August 1, 2022 20:00
@lhotari @Jason918
Upgrade Netty to 4.1.76.Final, Netty Tcnative, grpc and protobuf (apa…
78c6f94 
…che#15212)

* Upgrade Netty to 4.1.76.Final and Netty Tcnative to 2.0.51.Final

Fixes apache#14015
- release notes https://netty.io/news/2022/04/12/4-1-76-Final.html
  - contains fix for netty/netty#11695

* Upgrade grpc to 1.45.1 and protobuf to 3.19.2

- grpc < 1.45.1 is not compatible with Netty > 4.1.74.Final
  - grpc/grpc-java#9004

(cherry picked from commit 332a3c7)
@lhotari @Jason918
Upgrade Netty to 4.1.77.Final and netty-tcnative to 2.0.52.Final (apa…
6930e9d 
…che#15646)

- release notes https://netty.io/news/2022/05/06/2-1-77-Final.html
  - improves Alpine / musl compatibility
    - could help issues such as apache#14534 apache#11415 apache#11224 apache#10798
  - improves shading compatibility
  - fixes a bug related to the native epoll transport and epoll_pwait2

(cherry picked from commit a8045fc)
@Jason918
Fix License
2bd9674
@Jason918
fix testTxnCommitMarker
6572c36
@Jason918
Copy link
Copy Markdown
Contributor Author

Jason918 commented Aug 2, 2022

/pulsarbot run-failure-checks

@Jason918 Jason918 closed this Aug 4, 2022
Jason918 and others added 2 commits August 4, 2022 11:17
@Jason918
[Build] Use grpc-bom to align grpc library versions
8dc05e4
@hangc0276 @Jason918
Upgrade BookKeeper to 4.14.1 (apache#10686)
9769bae 
* upgrade BookKeeper to 4.14.0

* format code

* upgrade bookkeeper to 4.14.1 to fix prometheus metric bug

* update license

Co-authored-by: chenhang <chenhang@bigo.sg>
@Jason918 Jason918 reopened this Aug 4, 2022
Jason918 and others added 2 commits August 4, 2022 13:07
@Jason918
fix compile
80a1c30
@dlg99 @Jason918
[Issue 9728] Upgrade GRPC and bookkeeper (apache#9729)
3b3b5a1 
* Upgraded bookkeeper to 4.13.0; grpc to 1.33

* updated LICENSE

* fixed the rest of the license check

* no idea why license check results on CI are different from local :/

* one more license fix

* Integration tests failed to start container/service there because of java.lang.NoClassDefFoundError: io/perfmark/PerfMark

* another try to get the permark.jar in

* another take on perfmark.jar and integration tests

* add perfmark.jar to LICENSE
@Jason918
Copy link
Copy Markdown
Contributor Author

Jason918 commented Aug 4, 2022

Closing this PR for branch 2.7
The initial purpose of this PR is upgrade Netty to 4.1.76.Final, which requires GRPC 1.45.1.
But GRPC 1.45.1 is not compatible with Bookkeeper 4.12.1 with several following CI errors:

11:38:36.527 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Failed to start Component: storage-service
java.lang.AbstractMethodError: io.grpc.NameResolver$Factory.newNameResolver(Ljava/net/URI;Lio/grpc/NameResolver$Args;)Lio/grpc/NameResolver;
    at io.grpc.internal.ManagedChannelImpl.getNameResolver(ManagedChannelImpl.java:753) ~[io.grpc-grpc-core-1.45.1.jar:1.45.1]
    at io.grpc.internal.ManagedChannelImpl.getNameResolver(ManagedChannelImpl.java:785) ~[io.grpc-grpc-core-1.45.1.jar:1.45.1]
    at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:665) ~[io.grpc-grpc-core-1.45.1.jar:1.45.1]
    at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630) ~[io.grpc-grpc-core-1.45.1.jar:1.45.1]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297) ~[io.grpc-grpc-core-1.45.1.jar:1.45.1]
    at org.apache.bookkeeper.clients.impl.internal.LocationClientImpl.<init>(LocationClientImpl.java:66) ~[org.apache.bookkeeper-stream-storage-java-client-4.12.1.jar:4.12.1]
    at org.apache.bookkeeper.clients.impl.internal.StorageServerClientManagerImpl.<init>(StorageServerClientManagerImpl.java:76) ~[org.apache.bookkeeper-stream-storage-java-client-4.12.1.jar:4.12.1]
    at org.apache.bookkeeper.stream.server.StorageServer.lambda$buildStorageServer$6(StorageServer.java:311) ~[org.apache.bookkeeper-stream-storage-server-4.12.1.jar:4.12.1]
    at org.apache.bookkeeper.stream.storage.StorageContainerStoreBuilder.build(StorageContainerStoreBuilder.java:162) ~[org.apache.bookkeeper-stream-storage-service-impl-4.12.1.jar:4.12.1]
    at org.apache.bookkeeper.stream.server.service.StorageService.doStart(StorageService.java:46) ~[org.apache.bookkeeper-stream-storage-server-4.12.1.jar:4.12.1]
    at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.12.1.jar:4.12.1]
    at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$4(LifecycleComponentStack.java:144) ~[org.apache.bookkeeper-bookkeeper-common-4.12.1.jar:4.12.1]
    at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406) [com.google.guava-guava-30.1-jre.jar:?]
    at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:144) [org.apache.bookkeeper-bookkeeper-common-4.12.1.jar:4.12.1]
    at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doStart(StreamStorageLifecycleComponent.java:49) [org.apache.bookkeeper-stream-storage-server-4.12.1.jar:4.12.1]
    at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) [org.apache.bookkeeper-bookkeeper-common-4.12.1.jar:4.12.1]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runStreamStorage(LocalBookkeeperEnsemble.java:350) [org.apache.pulsar-pulsar-zookeeper-utils-2.7.5.jar:2.7.5]
    at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:432) [org.apache.pulsar-pulsar-zookeeper-utils-2.7.5.jar:2.7.5]
    at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:258) [org.apache.pulsar-pulsar-broker-2.7.5.jar:2.7.5]
    at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:115) [org.apache.pulsar-pulsar-broker-2.7.5.jar:2.7.5]

Bookkeeper 4.14.4 updated GRPC version to 1.42.1.
But we can't upgrade BK major version (4.12.x->4.14.x) in Pulsar minor versions (Point out by @hangc0276 ).
So this PR is dropped.

@Jason918 Jason918 closed this Aug 4, 2022
@Jason918 Jason918 mentioned this pull request Aug 4, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Jason918 Jason918

Labels

release/2.7.5

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Jason918 @lhotari @hangc0276 @dlg99