KAFKA-12154: Snapshot Loading API#10085
Merged
hachikuji merged 27 commits intoapache:trunkfrom May 1, 2021
Merged
Conversation
dengziming
reviewed
Feb 18, 2021
hachikuji
reviewed
Mar 17, 2021
hachikuji
reviewed
Mar 18, 2021
jsancio
commented
Mar 24, 2021
Member
Author
|
Update the PR's description to match the latest changes included here. |
3 tasks
jsancio
commented
Mar 31, 2021
mumrah
reviewed
Mar 31, 2021
Member
mumrah
left a comment
There was a problem hiding this comment.
Thanks for the patch! First pass of questions/comments inline
dengziming
reviewed
Apr 4, 2021
hachikuji
reviewed
Apr 5, 2021
jsancio
commented
Apr 6, 2021
10cd065 to
029fea5
Compare
Member
Author
|
@mumrah @hachikuji @dengziming thanks for the feedback. I should have addressed all of your suggestions. Let me know if I missed anything. |
hachikuji
reviewed
Apr 28, 2021
hachikuji
reviewed
Apr 30, 2021
hachikuji
reviewed
Apr 30, 2021
hachikuji
approved these changes
Apr 30, 2021
Contributor
hachikuji
left a comment
There was a problem hiding this comment.
LGTM. Thanks for the patch!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement Raft Snapshot loading API.
Adds a new method
handleSnapshottoraft.Listenerwhich is called whenever theRaftClientdetermines that theListenerneeds to load a new snapshot before reading the log. This happens when theListener's next offset is less than the log start offset also known as the earliest snapshot.Adds a new type
SnapshotReader<T>which provides aIterator<Batch<T>>interface and de-serializes records in theRawSnapshotReaderintoTsAdds a new type
RecordsIterator<T>that implements anIterator<Batch<T>>by scanning aRecordsobject and deserializes the batches and records intoBatch<T>. This type is used by bothSnapshotReader<T>andRecordsBatchReader<T>internally to implement theIteratorinterface that they expose.Changes the
MockLogimplementation to read one batch at a time. The previous implementation always read from the given offset to the high-watermark. This made it impossible to test interesting snapshot loading scenarios.Removed
throws IOExceptionfrom some methods. Some of types were inconsistently throwingIOExceptionin some cases and throwingRuntimeException(..., new IOException(...))in others. This PR improves the consistent by wrappingIOExceptioninRuntimeExceptionin a few more places and replacingCloseablewithAutoCloseable.Updated the Kafka Raft simulation test to take into account snapshot.
ReplicatedCounterwas updated to generate snapshot after 10 records get committed. This means that theConsistentCommittedDatavalidation was extended to take snapshots into account.More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)