Skip to content

Conversation

@Technoboy-
Copy link
Contributor

Motivation

When create a topic with schema - AVRO, if a producer sends bytes data directly to the topic, a consumer with HTTP and TCP lookup has different prompt message. See below test:

# Consume
Consumer<User> consumer = pulsarClient
                .newConsumer(Schema.AVRO(User.class))
                .topic("persistent://public/default/test")
                .subscriptionName("sub-2")
                .subscriptionType(SubscriptionType.Shared)
                .subscribe();
while(true){
     Message<User> message = consumer.receive();
     System.out.println("received msg : " + message.getValue());
}

# Produce
bin/pulsar-client produce -n 5 -m "hello" test

If the consumer is using TCP lookup, it will get the below message :

Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException: org.apache.commons.lang3.SerializationException: Failed at fetching schema info for EMPTY
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
	at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.getSchemaReader(AbstractMultiVersionReader.java:83)

But with HTTP lookup, the error message change to :

com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.nio.BufferUnderflowException
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2055)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950)
	at org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.getSchemaReader(AbstractMultiVersionReader.java:82)

The root cause is that : if producer produce with bytes data, the schema version is empty. When in TCP lookup, the server will throw unchecked exception here(line-244) :

public SchemaVersion versionFromBytes(byte[] version) {
// The schema storage converts the schema from bytes to long
// so it handles both cases 1) version is 64 bytes long pre 2.4.0;
// 2) version is 8 bytes long post 2.4.0
//
// NOTE: if you are planning to change the logic here. you should consider
// both 64 bytes and 8 bytes cases.
ByteBuffer bb = ByteBuffer.wrap(version);
return new LongSchemaVersion(bb.getLong());
}

When in HTTP lookup, the error will throw by the client here(line-164):

if (version != null) {
path = String.format("admin/v2/schemas/%s/schema/%s",
schemaName,
ByteBuffer.wrap(version).getLong());
}

So in order to keep the consistent prompt message, we will check the empty schema at the client-side and throw the same exception.

Modifications

  • Keep checking the empty version at the client-side.
  • Using the same exception.
  • Add server-side check at ServerCnx#handleGetSchema to help non-java client to print the same error.

Documentation

  • no-need-doc

@Technoboy- Technoboy- self-assigned this Mar 9, 2022
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 9, 2022
Copy link
Member

@nodece nodece left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@congbobo184 congbobo184 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@gaoran10 gaoran10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codelipenghui codelipenghui modified the milestones: 2.10.0, 2.11.0 Mar 12, 2022
@codelipenghui codelipenghui merged commit 190e5db into apache:master Mar 12, 2022
codelipenghui pushed a commit that referenced this pull request Mar 18, 2022
@codelipenghui codelipenghui added cherry-picked/branch-2.9 Archived: 2.9 is end of life and removed release/2.8.4 labels Mar 18, 2022
codelipenghui pushed a commit that referenced this pull request Apr 19, 2022
Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022
Comment on lines +2030 to +2034
if (commandGetSchema.getSchemaVersion().length == 0) {
commandSender.sendGetSchemaErrorResponse(requestId, ServerError.IncompatibleSchema,
"Empty schema version");
return;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I debugged testAvroSchemaWithTcpLookup and testAvroSchemaWithHttpLookup, it never went here. Then I removed these lines, tests still passed.

@Technoboy- Technoboy- deleted the fix-get-schema branch August 10, 2022 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants