KAFKA-12571: Eliminate LeaderEpochFileCache constructor dependency on logEndOffset#10426
Merged
junrao merged 2 commits intoapache:trunkfrom Mar 31, 2021
Conversation
Contributor
Author
|
cc @junrao @dhruvilshah3 @satishd: this PR is ready for review. |
kowshik
force-pushed
the
refactor_eliminate_LogEndOffset_dependency_on_LeaderEpochFileCache
branch
from
61ccacc to
4208aad
Compare
dhruvilshah3
approved these changes
Mar 29, 2021
Contributor
dhruvilshah3
left a comment
dhruvilshah3
left a comment
There was a problem hiding this comment.
LGTM, thanks for the PR
kowshik
force-pushed
the
refactor_eliminate_LogEndOffset_dependency_on_LeaderEpochFileCache
branch
from
4208aad to
c699987
Compare
Contributor
Author
|
The rebase has started the Jenkins run now. We can wait for the tests to pass. |
kowshik
force-pushed
the
refactor_eliminate_LogEndOffset_dependency_on_LeaderEpochFileCache
branch
from
c699987 to
b5d41d2
Compare
Contributor
|
@kowshik : Do you know why the tests timed out? Thanks. |
kowshik
force-pushed
the
refactor_eliminate_LogEndOffset_dependency_on_LeaderEpochFileCache
branch
2 times, most recently
from
4bd5bc1 to
e01f613
Compare
kowshik
force-pushed
the
refactor_eliminate_LogEndOffset_dependency_on_LeaderEpochFileCache
branch
from
e01f613 to
53f8433
Compare
Contributor
Author
|
@junrao Jenkins doesn't seem to be starting up tests after I rebased this morning. I'm not sure what's going on. I'll try again today. |
Contributor
Author
|
@junrao The jenkins tests have finished. I checked the test failures, they seem to be unrelated to this PR: |
Terrdi
pushed a commit
to Terrdi/kafka
that referenced
this pull request
Apr 1, 2021
… logEndOffset (apache#10426) This PR is a precursor to the recovery logic refactor work (KAFKA-12553). Problems: For refactoring the recovery logic (KAFKA-12553), we would like to move the logic to initialize LeaderEpochFileCache out of the Log class and into a separate static function. In the future, once we successfully initialize LeaderEpochFileCache outside Log, we will be able pass it as a dependency into both the Log recovery module and Log class constructor. However, currently the LeaderEpochFileCache constructor takes a dependency on logEndOffset (via a callback), which poses the following problems: Blocks the instantiation of LeaderEpochFileCache outside Log class. Because, outside Log the logEndOffset is unavailable to be passed into LeaderEpochFileCache constructor. As a result, this situation blocks the recovery logic (KAFKA-12553) refactor work. It turns out the logEndOffset dependency is used only in 1 of the LeaderEpochFileCache methods: LeaderEpochFileCache.endOffsetFor, and just for 1 particular case. Therefore, it is overkill to pass it in the constructor as a dependency. Also a callback is generally not a neat way to access dependencies and it poses code readability problems too. Solution: This PR modifies the code such that we only pass the logEndOffset as a parameter into LeaderEpochFileCache.endOffsetFor whenever the method is called, thus eliminating the constructor dependency. This will also unblock the recovery logic refactor work (KAFKA-12553). Tests: I have modified the existing tests to suit the above refactor. Reviewers: Dhruvil Shah <dhruvil@confluent.io>, Jun Rao <junrao@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is a precursor to the recovery logic refactor work (KAFKA-12553).
Problems:
For refactoring the recovery logic (KAFKA-12553), we would like to move the logic to initialize
LeaderEpochFileCacheout of theLogclass and into a separate static function. In the future, once we successfully initializeLeaderEpochFileCacheoutsideLog, we will be able pass it as a dependency into both theLogrecovery module andLogclass constructor. However, currently theLeaderEpochFileCacheconstructor takes a dependency onlogEndOffset(via a callback), which poses the following problems:LeaderEpochFileCacheoutsideLogclass. Because, outsideLogthelogEndOffsetis unavailable to be passed intoLeaderEpochFileCacheconstructor. As a result, this situation blocks the recovery logic (KAFKA-12553) refactor work.logEndOffsetdependency is used only in 1 of theLeaderEpochFileCachemethods:LeaderEpochFileCache.endOffsetFor, and just for 1 particular case. Therefore, it is overkill to pass it in the constructor as a dependency. Also a callback is generally not a neat way to access dependencies and it poses code readability problems too.Solution:
This PR modifies the code such that we only pass the
logEndOffsetas a parameter intoLeaderEpochFileCache.endOffsetForwhenever the method is called, thus eliminating the constructor dependency. This will also unblock the recovery logic refactor work (KAFKA-12553).Tests:
I have modified the existing tests to suit the above refactor.