KAFKA-10048: Possible data gap for a consumer after a failover when u…#8730
KAFKA-10048: Possible data gap for a consumer after a failover when u…#8730mimaison merged 1 commit intoapache:trunkfrom
Conversation
ryannedolan
left a comment
ryannedolan
left a comment
There was a problem hiding this comment.
Thanks for discovering, verifying, and fixing this issue! Change makes sense to me.
Requested change: add deadline to consumeAllMessages() -- otherwise lgtm.
|
@ijuma would you please review this or suggest who is the right committer to help with this? |
|
@mimaison, would you be able to review this? |
|
Sure I'll take a look this week |
mimaison
left a comment
mimaison
left a comment
There was a problem hiding this comment.
Good catch, this is indeed a problem if some partitions don't have records before the failover.
I've left a few comments. Overall I find the test pretty hard to follow. I wonder if more comments or some refactoring would help.
|
@mimaison Thanks for the feedback. I've refactored the tests. Could you please give it another review. |
|
Thanks @asdaraujo, I'll try to take another look later on this week. Can you rebase on trunk? to pick up a change (241e144) that is now necessary to run the CI tests |
|
Thanks, @mimaison . I've rebased it. |
asdaraujo
force-pushed
the
trunk
branch
3 times, most recently
from
5a84cc9 to
2c6b47c
Compare
mimaison
left a comment
mimaison
left a comment
There was a problem hiding this comment.
Thanks for the update.
Considering the code change (a single line), the test is really complicated. Other existing tests already cover MM2 use cases. The new test could only focus on the difference of behaviour caused by the fix.
asdaraujo
force-pushed
the
trunk
branch
2 times, most recently
from
edb9ead to
0ee597e
Compare
mimaison
left a comment
mimaison
left a comment
There was a problem hiding this comment.
Thanks for the updates. I think we're almost there! I've left a couple of comments
…sing MM2 Ensure that the MM2 checkpoint mirror task replicates consumer offsets even when they are zero to avoid issues with consumers after failovers. Author: Andre Araujo <asdaraujo@gmail.com>
…sing MM2 (apache#8730) Ensure that the MM2 checkpoint mirror task replicates consumer offsets even when they are zero to avoid issues with consumers after failovers. Author: Andre Araujo <asdaraujo@gmail.com> Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ryanne Dolan <ryannedolan@gmail.com>, Edoardo Comar <ecomar@uk.ibm.com>, heritamas
8 participants
Changed the MM2 checkpoint mirror task to ensure it replicates consumer offsets even when they are equal to zero to avoid issues with consumers after failovers.
Modified the test case to cover the possible scenario of consumer gap, as described on KAFKA-10048.
Committer Checklist (excluded from commit message)