Persist legacy LatestPairs for now#13378
Merged
abhishekagarwal87 merged 1 commit intoapache:masterfrom Nov 17, 2022
Merged
Conversation
clintropolis
approved these changes
Nov 17, 2022
| } | ||
| } | ||
|
|
||
| private static final ObjectStrategy<SerializablePairLongString> LEGACY_STRATEGY = |
Member
There was a problem hiding this comment.
nit: maybe worth leaving a comment about why this is different from getObjectStrategy() here too (sad all big endian all the time, etc)
f5b3cde to
281dbb5
Compare
We added compression to the latest/first pair storage, but the code change was forcing new things to be persisted with the new format, meaning that any segment created with the new code cannot be read by the old code. Instead, we need to default to creating the old format and then remove that default in a future version.
281dbb5 to
f2eabcd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #13377
Description
We added compression to the latest/first pair storage, but the code change was forcing new things to be persisted with the new format, meaning that any segment created with the new code cannot be read by the old code. Instead, we need to default to creating the old format and then remove that default in a future version.
This introduces a java system property to toggle the compression on, but I do not believe we should publicly document it as it doesn't play well with the runtime.properties. Instead, it's there just for intrepid developers who want to use the functionality/validate it ahead of the next major release.
Along with creating the tests for this, it became clear to me that various implementations of ComplexColumn were breaking contract (they were returning "number of cells" for a getter that is documented as returning number of bytes). I also fixed the usages of the method that were expecting number of rows and adjusted the various implementations to return
-1when they don't know the size.This PR has: