KAFKA-15045: (KIP-924 pt. 15) Implement #defaultStandbyTaskAssignment and finish rack-aware standby optimization#16129
Merged
ableegoldman merged 6 commits intoapache:trunkfrom May 30, 2024
Merged
Conversation
This fills in the implementation details of the standby task assignment utility functions within TaskAssignmentUtils.
apourchet
changed the title
Comment on lines
+308
to
+309
| final String rack1 = clientRacks.get(clientState1.processId().id()).get(); | ||
| final String rack2 = clientRacks.get(clientState2.processId().id()).get(); |
Member
There was a problem hiding this comment.
Future cleanup: we can actually get rid of this clientRacks data structure altogether, and just use the rack ids we already processed and nicely organized in the KafkaStreamsState which is addressable by UUID, ie
Suggested change
| final String rack1 = clientRacks.get(clientState1.processId().id()).get(); | |
| final String rack2 = clientRacks.get(clientState2.processId().id()).get(); | |
| final String rack1 = kafkaStreamsStates.get(clientState1.processId().id()).rackId().get(); | |
| final String rack2 = kafkaStreamsStates.get(clientState2.processId().id()).rackId().get(); |
Member
ableegoldman
left a comment
ableegoldman
left a comment
There was a problem hiding this comment.
A few more things that need to be addressed but can be done in a followup PR
| Function.identity(), | ||
| taskId -> { | ||
| final Set<String> stateStoreNames = topologyMetadata | ||
| .stateStoreNameToSourceTopicsForTopology(taskId.topologyName()) |
Member
There was a problem hiding this comment.
Ah somehow I missed this before -- this is actually returning all the state stores for this topology, it's not specific to the taskId. This was an existing issue so we don't need to fix it in this PR, it can be addressed in a followup. It might be a bit complicated so I'll take a look at how we can get this info
Would've caught this during testing since we definitely want tests with mixed stateless-and-stateful tasks, but still good to fix ASAP
Member
|
Test failures are unrelated. Merging to trunk |
ableegoldman
changed the title
chiacyu
pushed a commit
to chiacyu/kafka
that referenced
this pull request
Jun 1, 2024
… and finish rack-aware standby optimization (apache#16129) This fills in the implementation details of the standby task assignment utility functions within TaskAssignmentUtils. Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
wernerdv
pushed a commit
to wernerdv/kafka
that referenced
this pull request
Jun 3, 2024
… and finish rack-aware standby optimization (apache#16129) This fills in the implementation details of the standby task assignment utility functions within TaskAssignmentUtils. Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
TaiJuWu
pushed a commit
to TaiJuWu/kafka
that referenced
this pull request
Jun 8, 2024
… and finish rack-aware standby optimization (apache#16129) This fills in the implementation details of the standby task assignment utility functions within TaskAssignmentUtils. Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
gongxuanzhang
pushed a commit
to gongxuanzhang/kafka
that referenced
this pull request
Jun 12, 2024
… and finish rack-aware standby optimization (apache#16129) This fills in the implementation details of the standby task assignment utility functions within TaskAssignmentUtils. Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fills in the implementation details of the standby task assignment utility functions within TaskAssignmentUtils.