Skip to content

KAFKA-14834: [10/N] Reserve repartition topic formats to include isLatest#13566

Merged
mjsax merged 1 commit intoapache:trunkfrom
vcrfxia:kip-914-repartition-topic-format
Apr 14, 2023
Merged

KAFKA-14834: [10/N] Reserve repartition topic formats to include isLatest#13566
mjsax merged 1 commit intoapache:trunkfrom
vcrfxia:kip-914-repartition-topic-format

Conversation

@vcrfxia
Copy link
Copy Markdown
Contributor

@vcrfxia vcrfxia commented Apr 14, 2023

#13564 introduced a new boolean isLatest into Change to indicate whether a change update represents the latest for the key. Even though Change is serialized into the table repartition topic, the new boolean does not need to be serialized in, because the table repartition map processor performs an optimization to drop records for which isLatest = false. If not for this optimization, the downstream table aggregate would have to drop such records instead, and isLatest would need to be serialized into the repartition topic.

In light of the possibility that isLatest may need to be serialized into the repartition topic in the future, e.g., if other downstream processors are added which need to distinguish between records for which isLatest = true vs isLatest = false, this PR reserves repartition topic formats which include isLatest. Reserving these formats now comes at no additional cost to users since a rolling bounce is already required for the upcoming release due to #10747. If we don't reserve them now and instead have to add them later, then another bounce would be required at that time. Reserving formats is cheap, so we choose to do it now.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@mjsax mjsax added streams kip Requires or implements a KIP labels Apr 14, 2023
@mjsax mjsax force-pushed the kip-914-repartition-topic-format branch from 76584d5 to c16fb68 Compare April 14, 2023 04:54
@mjsax mjsax merged commit f1eb260 into apache:trunk Apr 14, 2023
@vcrfxia vcrfxia deleted the kip-914-repartition-topic-format branch April 14, 2023 05:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kip Requires or implements a KIP streams

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants