Add packed format support for consumer offset decoding #388
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
July 13, 2022 18:25
- add : and , as literal characters
- hopefully fix endian swap for 16 bit integers
- Add "S" - count + string, used by Kafka for consumer offsets
- Add "U" - UUID
- Add "v" - varint (used by Kafka for some values)
- Add "C" - varint count + string
- Add "t" - 64 bit millisecond timestamp - note milliseconds are not displayed
this uses ctime(3) which isn't the best (especially since the newline
is removed through direct munging of the buffer) but should work well
enough for this use case
The problem with consumer offset is that it's versioned, so we need to have some special logic for it. This is a bit of a hack, reusing macros from the regular unpack and thus creating a bunch of fake variables so the macros work. But it does work properly.
Handle v3 group metadata.
Reformat help a little bit.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We needed to be able to decode the exact consumer offset data (and not Kafka's idea of it through its builtin deserializer - it glosses over version differences and we wanted to know the exact data stored).
This PR adds: