MINOR: improve JavaDocs about auto-repartitioning in Streams DSL#6269
MINOR: improve JavaDocs about auto-repartitioning in Streams DSL#6269bbejeck merged 1 commit intoapache:trunkfrom
Conversation
|
Call for review @guozhangwang @bbejeck @vvcephei @ableegoldman The question about this comes up on a regular basis -- this should help to clarify and reduce number of questions. |
| * } | ||
| * }); | ||
| * }</pre> | ||
| * <p> |
There was a problem hiding this comment.
Side cleanup: not necessary after a <pre></pre> block
| * Transforming records might result in an internal data redistribution if a key based operator (like an aggregation | ||
| * or join) is applied to the result {@code KStream}. | ||
| * (cf. {@link #transformValues(ValueTransformerSupplier, String...)}) | ||
| * </p> |
There was a problem hiding this comment.
JavaDocs is not HTML. No closing </p> tag required.
There was a problem hiding this comment.
Heh, I actually just looked it up, because I was surprised all those other <p> elements were not closed... Apparently, even in HTML, you don't have to close <p> elements:
Paragraphs are block-level elements, and notably will automatically close if another block-level element is parsed before the closing </p> tag.
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/p
Sometimes, you just have to stop and marvel at HTML...
| * {@link StreamsConfig#APPLICATION_ID_CONFIG APPLICATION_ID_CONFIG}, "storeName" is an | ||
| * internally generated name, and "-changelog" is a fixed suffix. | ||
| * | ||
| * <p> |
There was a problem hiding this comment.
add some missing paragraph breaks
vvcephei
left a comment
vvcephei
left a comment
There was a problem hiding this comment.
LGTM. I asked a question, but it's independent of your proposed change.
| * Transforming records might result in an internal data redistribution if a key based operator (like an aggregation | ||
| * or join) is applied to the result {@code KStream}. | ||
| * (cf. {@link #transformValues(ValueTransformerSupplier, String...)}) | ||
| * </p> |
There was a problem hiding this comment.
Heh, I actually just looked it up, because I was surprised all those other <p> elements were not closed... Apparently, even in HTML, you don't have to close <p> elements:
Paragraphs are block-level elements, and notably will automatically close if another block-level element is parsed before the closing </p> tag.
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/p
Sometimes, you just have to stop and marvel at HTML...
| * } | ||
| * }</pre> | ||
| * Even if any upstream operation was key-changing, no auto-repartition is triggered. | ||
| * If repartitioning is required, a call to {@link #through(String)} should be performed before {@code transform()}. |
There was a problem hiding this comment.
It's a bummer, since it forces you to come up with a name for the repartition topic, which also makes its lifecycle your responsibility. And also, Streams won't know that it's "just" a repartition topic, so it can't be optimized.
Can I create a Jira to add a way to mark the transform as "repartition required"? Or maybe we can automatically do this when the transform has a state store?
There was a problem hiding this comment.
There is already a JIRA: https://issues.apache.org/jira/browse/KAFKA-7608 and also inactive KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
It's not fully backed out yet, but there is stuff we could do to improve the user experience.
|
cherry-picked to |
* ak/trunk: (45 commits) KAFKA-7487: DumpLogSegments misreports offset mismatches (apache#5756) MINOR: improve JavaDocs about auto-repartitioning in Streams DSL (apache#6269) KAFKA-7935: UNSUPPORTED_COMPRESSION_TYPE if ReplicaManager.getLogConfig returns None (apache#6274) KAFKA-7895: Fix stream-time reckoning for suppress (apache#6278) KAFKA-6569: Move OffsetIndex/TimeIndex logger to companion object (apache#4586) MINOR: add log indicating the suppression time (apache#6260) MINOR: Make info logs for KafkaConsumer a bit more verbose (apache#6279) KAFKA-7758: Reuse KGroupedStream/KGroupedTable with named repartition topics (apache#6265) KAFKA-7884; Docs for message.format.version should display valid values (apache#6209) MINOR: Save failed test output to build output directory MINOR: add test for StreamsSmokeTestDriver (apache#6231) MINOR: Fix bugs identified by compiler warnings (apache#6258) KAFKA-6474: Rewrite tests to use new public TopologyTestDriver [part 4] (apache#5433) MINOR: fix bypasses in ChangeLogging stores (apache#6266) MINOR: Make MockClient#poll() more thread-safe (apache#5942) MINOR: drop dbAccessor reference on close (apache#6254) KAFKA-7811: Avoid unnecessary lock acquire when KafkaConsumer commits offsets (apache#6119) KAFKA-7916: Unify store wrapping code for clarity (apache#6255) MINOR: Add missing Alter Operation to Topic supported operations list in AclCommand KAFKA-7921: log at error level for missing source topic (apache#6262) ...
3 participants
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)