PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first

[sceee]

1. Specify the API: PubSub
2. OS type and version: Docker image gradle:5.2-jdk11-slim (also happens with jdk8 and other images)
3. Java version: jdk11 and jdk8
4. google-cloud-java version(s): 1.62.0 and above (tested with 1.65.0, too)

Steps to reproduce

1. Use google-cloud-pubsub library with version 1.62.0 or above as dependency
2. Create a subscription in own code (see code example below)

Code example

ProjectSubscriptionName subscriptionName = ProjectSubscriptionName.of(projectId, subscriptionId);
try (SubscriptionAdminClient subscriptionAdminClient = SubscriptionAdminClient.create()) {
    subscriptionAdminClient.getSubscription(subscriptionName); // Exception is thrown in this line
}
...

Stack trace

Mar 19, 2019 6:20:52 PM io.grpc.internal.ManagedChannelImpl$1 uncaughtException
SEVERE: [io.grpc.internal.ManagedChannelImpl-1] Uncaught exception in the SynchronizationContext. Panic!
java.lang.IllegalStateException: Could not find LoadBalancer pick_first. The build probably threw away META-INF/services/io.grpc.LoadBalancerProvider
        at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.<init>(AutoConfiguredLoadBalancerFactory.java:74)
        at io.grpc.internal.AutoConfiguredLoadBalancerFactory.newLoadBalancer(AutoConfiguredLoadBalancerFactory.java:45)
        at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:351)
        at io.grpc.internal.ManagedChannelImpl$ChannelTransportProvider$1ExitIdleModeForTransport.run(ManagedChannelImpl.java:447)
        at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:101)
        at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:130)
        at io.grpc.internal.ManagedChannelImpl$ChannelTransportProvider.get(ManagedChannelImpl.java:451)
        at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:241)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:392)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:689)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32)
        at com.google.api.gax.grpc.GrpcHeaderInterceptor$1.start(GrpcHeaderInterceptor.java:95)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:308)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:280)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58)
        at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64)
        at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:86)
        at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
        at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
        at com.google.api.gax.tracing.TracedUnaryCallable.futureCall(TracedUnaryCallable.java:75)
        at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
        at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
        at com.google.cloud.pubsub.v1.SubscriptionAdminClient.getSubscription(SubscriptionAdminClient.java:467)
        at com.google.cloud.pubsub.v1.SubscriptionAdminClient.getSubscription(SubscriptionAdminClient.java:420)
        at my.package.MyClass.start(MyClass.java:37)
        at my.package.App$1.run(App.java:31)
        at java.base/java.lang.Thread.run(Thread.java:834)

External references such as API reference guides used

• None

Any additional information below

The error only occurs since google-cloud-pubsub 1.62.0 .
When using google-cloud-pubsub 1.61.0 or below, the error does not occur with the same user code.

[kolea2]

Thank you for the report!

@sduskis, @andreamlin, Pub/Sub version 1.62 included an upgrade in gax - any idea if there were changes that could have caused this?

[sduskis]

@kolea2, can you tell if there was a grpc upgrade? That might be one possible reason.

[kolea2]

Yep - looks like grpc went from 1.17.1 to 1.18.0: #4496

[ajaaym]

@sceee can you please share your docker file with full repro? I tried the steps you mentioned and couldn't reproduce.

[olavloite]

@sceee I can reproduce this when I build a 'thick' jar using the maven shade plugin and add the Google Cloud client libraries as dependencies in the jar. Are you doing something like that?

If so, the files io.grpc.LoadBalancerProvider and io.grpc.NameResolverProvider from grpc-core/META-INF/services are probably not included in you final jar and this exception will occur.
A work-around is to add these files to your own jar by adding them either manually or by including them in your build process. The files should be in the META-INF/services/ folder of your jar.

If you add the files to your own project, you can include them in your final jar by using a maven-shade-plugin resource transformer like this:

<transformer implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
  <resource>META-INF/services</resource>
  <file>io.grpc.LoadBalancerProvider</file>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
  <resource>META-INF/services</resource>
  <file>io.grpc.NameResolverProvider</file>
</transformer>

[sceee]

Thank you for your response guys. @olavloite indeed I am shading using gradle with com.github.jengelman.gradle.plugins:shadow:5.0.0

Is this an issue that will be fixed in the future in grpc or the google-java library or do I permanently need this workaround from now on?

And with the workaround, do you mean including these files in my own project and including in them in the final jar?

• https://github.com/grpc/grpc-java/blob/master/core/src/bazel-internal/resources/META-INF/services/io.grpc.LoadBalancerProvider
• https://github.com/grpc/grpc-java/blob/master/core/src/bazel-internal/resources/META-INF/services/io.grpc.NameResolverProvider

[olavloite]

@sceee Yes, you should include those files in the META-INF/services directory of your jar, which means that you need to add them to your own project (at least for now as a workaround).
I'll have to discuss with someone else on what could be a possible long-term solution for this. @sduskis Could you point me in a direction of someone who might be able to help with that?