KAFKA-18026: KIP-1112 migrate KTableSuppressProcessorSupplier#18150
Merged
ableegoldman merged 2 commits intoapache:trunkfrom Dec 18, 2024
Merged
KAFKA-18026: KIP-1112 migrate KTableSuppressProcessorSupplier#18150ableegoldman merged 2 commits intoapache:trunkfrom
ableegoldman merged 2 commits intoapache:trunkfrom
Conversation
| name, | ||
| new ProcessorParameters<>(suppressionSupplier, name), | ||
| StoreBuilderWrapper.wrapStoreBuilder(storeBuilder) | ||
| new String[]{storeName} |
Contributor
There was a problem hiding this comment.
Is this intentional? The callee seems not using the storeName below: https://github.com/apache/kafka/pull/18150/files#diff-c1133b80fb36ee7e4fbf7a26007f63e41a411e7f36e45aa96373afd9ebdc5559R22
Member
There was a problem hiding this comment.
it's definitely not being used in the PR, but actually we do still need it so it should/will be used (see https://github.com/apache/kafka/pull/18150/files#r1885482579)
| super(nodeName, processorParameters); | ||
|
|
||
| this.storeNames = storeNames; | ||
| this.storeNames = null; |
Member
There was a problem hiding this comment.
I don't think we can change this yet since not all StatefulProcessorNode users have been migrated yet. For example the FKJ. Pretty sure this is what's causing the test failures.
Member
There was a problem hiding this comment.
Mm yeah almost definitely the problem:
Caused by: org.apache.kafka.streams.errors.StreamsException: Processor KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000022 has no access to StateStore KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-0000000021 as the store is not connected to the processor. If you add stores manually via '.addStateStore()' make sure to connect the added store to the processor by providing the processor name to '.addStateStore()' or connect them via '.connectProcessorAndStateStores()'. DSL users need to provide the store name to '.process()', '.transform()', or '.transformValues()' to connect the store to the corresponding operator, or they can provide a StoreBuilder by implementing the stores() method on the Supplier itself. If you do not add stores manually, please file a bug report at https://issues.apache.org/jira/projects/KAFKA.
basically setting this to null skips the #connectProcessorAndStateStore call, and not every operator that uses this constructor (including children of StatefulProcessorNode) have been migrated to implementing stores (which will result in the store being connected during the ProcessorParameters#addProcessorTo call)
Member
|
failure is due to size limit on build so unrelated, merging to trunk |
ijuma
added a commit
to ijuma/kafka
that referenced
this pull request
Dec 19, 2024
…e-old-protocol-versions * apache-github/trunk: (25 commits) KAFKA-18270: FindCoordinator v0 incorrectly tagged as deprecated (apache#18262) KAFKA-18284: Add group coordinator records for Streams rebalance protocol (apache#18228) MINOR: Fix flaky state updater test (apache#18253) MINOR: improve StreamsResetter logging (apache#18237) KAFKA-18227: Ensure v2 partitions are not added to last transaction during upgrade (apache#18176) Add IT for share consumer with duration base offet auto reset (apache#18251) KAFKA-18283: Add StreamsGroupDescribe RPC definitions (apache#18230) KAFKA-18241: add docs check to CI (apache#18183) KAFKA-18223 Improve flaky test report (apache#18212) MINOR Remove triage label in nightly job (apache#18147) KAFKA-18294 Remove deprecated SourceTask#commitRecord (apache#18260) KAFKA-18264 Remove NotLeaderForPartitionException (apache#18211) KAFKA-13722: Refactor SerdeGetter (apache#18242) KAFKA-18094 Remove deprecated TopicListing(String, Boolean) (apache#18248) KAFKA-18282: Add StreamsGroupHeartbeat RPC definitions (apache#18227) KAFKA-18026: KIP-1112 migrate KTableSuppressProcessorSupplier (apache#18150) KAFKA-18026: transition KTable#filter impl to use processor wrapper (apache#18205) KAFKA-18293 Remove `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler` and `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler` (apache#18244) MINOR: add assertion about groupEpoch and targetAssignmentEpoch to testConsumerGroups (apache#18203) KAFKA-17960; PlaintextAdminIntegrationTest.testConsumerGroups fails with CONSUMER group protocol (apache#18234) ...
tedyu
pushed a commit
to tedyu/kafka
that referenced
this pull request
Jan 6, 2025
…#18150) Migrates KTableSuppressProcessorSupplier to use the the ProcessorSupplier#stores() method Reviewers: Guozhang Wang <guozhang.wang.us@gmail.com>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>
ableegoldman
pushed a commit
that referenced
this pull request
Jan 15, 2025
Migrates KTableSuppressProcessorSupplier to use the the ProcessorSupplier#stores() method Reviewers: Guozhang Wang <guozhang.wang.us@gmail.com>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>
Member
|
Forgot to cherrypick this earlier -- cherrypicked to 4.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates
KTableSuppressProcessorSupplierto use the theProcessorSupplier#stores()method