Skip to content

Persist legacy LatestPairs for now#13378

Merged
abhishekagarwal87 merged 1 commit intoapache:masterfrom
imply-cheddar:latest-compat
Nov 17, 2022
Merged

Persist legacy LatestPairs for now#13378
abhishekagarwal87 merged 1 commit intoapache:masterfrom
imply-cheddar:latest-compat

Conversation

@imply-cheddar
Copy link
Copy Markdown
Contributor

Fixes #13377

Description

We added compression to the latest/first pair storage, but the code change was forcing new things to be persisted with the new format, meaning that any segment created with the new code cannot be read by the old code. Instead, we need to default to creating the old format and then remove that default in a future version.

This introduces a java system property to toggle the compression on, but I do not believe we should publicly document it as it doesn't play well with the runtime.properties. Instead, it's there just for intrepid developers who want to use the functionality/validate it ahead of the next major release.

Along with creating the tests for this, it became clear to me that various implementations of ComplexColumn were breaking contract (they were returning "number of cells" for a getter that is documented as returning number of bytes). I also fixed the usages of the method that were expecting number of rows and adjusted the various implementations to return -1 when they don't know the size.

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

}
}

private static final ObjectStrategy<SerializablePairLongString> LEGACY_STRATEGY =
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe worth leaving a comment about why this is different from getObjectStrategy() here too (sad all big endian all the time, etc)

We added compression to the latest/first pair storage, but
the code change was forcing new things to be persisted
with the new format, meaning that any segment created with
the new code cannot be read by the old code.  Instead, we
need to default to creating the old format and then remove that default in a future version.
@abhishekagarwal87 abhishekagarwal87 merged commit 6b9344c into apache:master Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backwards incompatible change to String 'first'/'last' complex columns

3 participants