Skip to content

[fix][broker] Fix calculate avg message per entry #17046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
codelipenghui merged 2 commits into from
Aug 17, 2022
Merged

[fix][broker] Fix calculate avg message per entry #17046

codelipenghui merged 2 commits into from
Aug 17, 2022

Conversation

@coderzc
Copy link
Member

@coderzc coderzc commented Aug 10, 2022

Motivation

Calculating avg message per entry using entries.size() is incorrect, because the entries may have null values.

Modifications

Calculate avg message per entry using the number of filter-out Non-null value total entries.

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Aug 10, 2022
Technoboy-
Technoboy- reviewed Aug 11, 2022
View reviewed changes
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
// Note
// Must ensure that the message is written to the pendingAcks before sent is first,
// because this consumer is possible to disconnect at this time.
if (pendingAcks != null) {
Copy link
Contributor

@Technoboy- Technoboy- Aug 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why move pendingAcks here ?

Copy link
Member Author

@coderzc coderzc Aug 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure the loop is executed.

@coderzc coderzc force-pushed the fix_avgMessagesPerEntry branch from 21e1859 to bfbfc3d Compare August 11, 2022 02:20
@coderzc coderzc requested a review from Technoboy- August 11, 2022 02:22
@Technoboy- Technoboy- added this to the 2.11.0 milestone Aug 11, 2022
gaoran10
gaoran10 approved these changes Aug 12, 2022
View reviewed changes
Copy link
Contributor

@gaoran10 gaoran10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I'm not sure why the entry could be null. Do you know the reason?

@coderzc
Copy link
Member Author

coderzc commented Aug 12, 2022

LGTM, but I'm not sure why the entry could be null. Do you know the reason?

Because some entries were set to null when the message was previously filtered.
Please see:

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java

Line 98 in b1a29b5

public int filterEntriesForConsumer(Optional<MessageMetadata[]> optMetadataArray, int startOffset,

codelipenghui
codelipenghui reviewed Aug 13, 2022
View reviewed changes
Copy link
Contributor

@codelipenghui codelipenghui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderzc The change looks good. Could you please help add a test? to avoid the regression in the future.

@coderzc coderzc force-pushed the fix_avgMessagesPerEntry branch from 41efd83 to bfbfc3d Compare August 13, 2022 09:53
@coderzc
Copy link
Member Author

coderzc commented Aug 13, 2022

@coderzc The change looks good. Could you please help add a test? to avoid the regression in the future.

ok, I have already added a test.

eolivelli
eolivelli approved these changes Aug 13, 2022
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

Good catch!

Please cherry pick to 2.10 branch

@eolivelli
Copy link
Contributor

eolivelli commented Aug 13, 2022

I don't think we need this in 2.9, filters don't exist there

@Technoboy-
Copy link
Contributor

Technoboy- commented Aug 15, 2022

I don't think we need this in 2.9, filters don't exist there

We need to cherry pick to 2.9 and 2.8, the same issue in sendMessages

@codelipenghui codelipenghui merged commit 2c2b75e into apache:master Aug 17, 2022
Technoboy- pushed a commit that referenced this pull request Aug 17, 2022
@coderzc @Technoboy-
[fix][broker] Fix calculate avg message per entry (#17046)
72810a0
@mattisonchao
Copy link
Member

mattisonchao commented Aug 25, 2022

Hi, @coderzc
Would you like to push a PR to branch-2.9 to fix this problem?

@coderzc
Copy link
Member Author

coderzc commented Aug 25, 2022

Hi, @coderzc Would you like to push a PR to branch-2.9 to fix this problem?

Yes, I will push a PR to branch-2.9.

@coderzc coderzc mentioned this pull request Aug 25, 2022
Merged
4 tasks
@mattisonchao mattisonchao added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Sep 1, 2022
Jason918 pushed a commit that referenced this pull request Sep 4, 2022
@coderzc @Jason918
[fix][broker] Fix calculate avg message per entry (#17046)
4fda75b 
(cherry picked from commit 2c2b75e)
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Sep 7, 2022
@coderzc @nicoloboschi
[fix][broker] Fix calculate avg message per entry (apache#17046)
ea426df 
(cherry picked from commit 2c2b75e)
@zymap
Copy link
Member

zymap commented Sep 15, 2022

Remove from release 2.8.5. Since the original code is introduced in 2.9.3.#14666

nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Sep 16, 2022
@coderzc @nicoloboschi
[fix][broker] Fix calculate avg message per entry (apache#17046)
241ada6 
(cherry picked from commit 2c2b75e)
(cherry picked from commit 4fda75b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Technoboy- Technoboy- Technoboy- approved these changes

@eolivelli eolivelli eolivelli approved these changes

@AnonHxy AnonHxy AnonHxy approved these changes

@codelipenghui codelipenghui codelipenghui approved these changes

@gaoran10 gaoran10 gaoran10 approved these changes

@mattisonchao mattisonchao mattisonchao approved these changes

@BewareMyPower BewareMyPower Awaiting requested review from BewareMyPower

Assignees

@coderzc coderzc

Labels

area/broker cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 cherry-picked/branch-2.11 doc-not-needed Your PR changes do not impact docs release/2.9.4 release/2.10.2 type/bug The PR fixed a bug or issue reported a bug

Projects

None yet

Milestone

2.11.0

Development

Successfully merging this pull request may close these issues.

9 participants

@coderzc @eolivelli @Technoboy- @mattisonchao @zymap @AnonHxy @codelipenghui @gaoran10 @Jason918