Skip to content

Conversation

@codelipenghui
Copy link
Contributor

@codelipenghui codelipenghui commented Sep 7, 2022

Motivation

We can see many logs like the following (not master since #16345 has changed to level to debug)

image

And it is easy to reproduce. You can just create a geo-replicated cluster
and produce the bytes messages.

It should be related to this change #17049 (haven't released) which fixed the replicated schema issue.
But we shouldn't get the schema from the broker for the BYTES schema, the client side will not cache the BYTES schema.
So, the replicator tries to get the schema again and again.

log.info("[{}] [{}] GetOrCreateSchema succeed", topic, producerName);
// In broker, if schema version is an empty byte array, it means the topic doesn't have schema. In this
// case, we should not cache the schema version so that the schema version of the message metadata will
// be null, instead of an empty array.
if (v.length != 0) {
schemaCache.putIfAbsent(msg.getSchemaHash(), v);
msg.getMessageBuilder().setSchemaVersion(v);
}

Modifications

The fix is the same as the normal messages with AUTO_PRODUCE producer do

if (msg.getSchemaInternal() == schema) {
schemaVersion.ifPresent(v -> msgMetadataBuilder.setSchemaVersion(v));
msg.setSchemaState(MessageImpl.SchemaState.Ready);
return true;
}

If the users will not change the message schema(send with bytes), the message schema is also AUTO_PRODUCE,
so the producer will not try to fetch schema from the broker.

Verifying this change

New test added.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@codelipenghui codelipenghui self-assigned this Sep 7, 2022
@codelipenghui codelipenghui added this to the 2.11.0 milestone Sep 7, 2022
@codelipenghui codelipenghui added type/bug The PR fixed a bug or issue reported a bug area/broker release/blocker Indicate the PR or issue that should block the release until it gets resolved release/2.10.2 release/2.9.4 labels Sep 7, 2022
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Sep 7, 2022
@codelipenghui codelipenghui merged commit 2ed5614 into apache:master Sep 8, 2022
@codelipenghui codelipenghui deleted the penghui/fix-replicate-byte-schema branch September 8, 2022 05:37
codelipenghui added a commit that referenced this pull request Sep 8, 2022
codelipenghui added a commit that referenced this pull request Sep 8, 2022
codelipenghui added a commit that referenced this pull request Sep 8, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Sep 16, 2022
…ytes schema (apache#17523)

(cherry picked from commit 2ed5614)
(cherry picked from commit d189846)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/broker cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 doc-not-needed Your PR changes do not impact docs release/blocker Indicate the PR or issue that should block the release until it gets resolved release/2.9.4 release/2.10.2 type/bug The PR fixed a bug or issue reported a bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants