KAFKA-14057: Support dynamic reconfiguration in KRaft remote controllers#13332
Closed
cmccabe wants to merge 1 commit intoapache:trunkfrom
Closed
KAFKA-14057: Support dynamic reconfiguration in KRaft remote controllers#13332cmccabe wants to merge 1 commit intoapache:trunkfrom
cmccabe wants to merge 1 commit intoapache:trunkfrom
Conversation
This PR allows us to dynamically reconfigure the KRaft controller by setting either a cluster configuration (that is, a configuration that applies to all resource of type BROKER), or by configuring a combined broker/controller node. As described in DynamicBrokerConfig#addReconfigurables(ControllerServer), we currently support reconfiguring Yammer metrics, metrics reporters, the socket server, and the Authorizer. More dynamically reconfigurable Controller elements will be added later. This PR refactors MetadataPublisher's interface a bit. There is now a handleControllerChange callback. This is something that some publishers might want (a good example is ZkMigrationClient). There is now only a single publish() function rather than a separate function for publishing snapshots and log deltas. Most publishers didn't want to do anything different in those two cases. The ones that do want to do something different for snapshots can always check the manifest type. The close function now has a default empty implementation, since most publishers didn't need to do anything there. This PR fixes a bug where the KRaft broker's authorizer was not dynamically reconfigurable even if it had implemented Reconfigurable. The bug resulted from the fact that we called DynamicBrokerConfig.addReconfigurables prior to initializing BrokerServer.authorizer.
Contributor
Author
|
Let me split out the MetadataPublisher refactoring part. |
Contributor
Author
|
Superseded by #13116 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR allows us to dynamically reconfigure the KRaft controller by setting either a cluster configuration (that is, a configuration that applies to all resource of type BROKER), or by configuring a combined broker/controller node.
As described in DynamicBrokerConfig#addReconfigurables(ControllerServer), we currently support reconfiguring Yammer metrics, metrics reporters, the socket server, and the Authorizer. More dynamically reconfigurable Controller elements will be added later.
This PR refactors MetadataPublisher's interface a bit. There is now a handleControllerChange callback. This is something that some publishers might want (a good example is ZkMigrationClient). There is now only a single publish() function rather than a separate function for publishing snapshots and log deltas. Most publishers didn't want to do anything different in those two cases. The ones that do want to do something different for snapshots can always check the manifest type. The close function now has a default empty implementation, since most publishers didn't need to do anything there.
This PR fixes a bug where the KRaft broker's authorizer was not dynamically reconfigurable even if it had implemented Reconfigurable. The bug resulted from the fact that we called DynamicBrokerConfig.addReconfigurables prior to initializing BrokerServer.authorizer.