KAFKA-18431: Remove KafkaController by FrankYang0529 · Pull Request #18573 · apache/kafka

FrankYang0529 · 2025-01-16T10:15:28Z

Remove KafkaController and related unused references:

ControllerChannelContext
ControllerChannelManager
ControllerEventManager
ControllerState
PartitionStateMachine
ReplicaStateMachine
TopicDeletionManager
ZkBrokerEpochManager

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

ijuma · 2025-01-16T14:34:18Z

-    assertNotEquals(broker1.hashCode, broker3.hashCode)
-    assertNotEquals(broker1.hashCode, broker4.hashCode)
-
-    assertEquals(Some(1), Map(broker1 -> 1).get(broker1))


I guess this test is unrelated to the zk stuff, but not too useful if we convert BrookerEndPoint to a record. I submitted #18577 for the latter.

ijuma

Nice clean-up! LGTM.

ijuma · 2025-01-16T14:38:53Z

There were a few test failures, re-running the tests to see if these are flakes. If you get a chance, try to run the failures locally to see if they are related.

FrankYang0529 · 2025-01-16T14:44:09Z

If you get a chance, try to run the failures locally to see if they are related.

Yes, they're related, I get error with following cases:

DeleteSegmentsByRetentionSizeTest#executeTieredStorageTest
DeleteSegmentsByRetentionTimeTest#executeTieredStorageTest
RollAndOffloadActiveSegmentTest#executeTieredStorageTest
DisableRemoteLogOnTopicTest#executeTieredStorageTest
EnableRemoteLogOnTopicTest#executeTieredStorageTest
RemoteTopicCrudTest#testClusterWideDisablementOfTieredStorageWithEnabledTieredTopic
DynamicConfigChangeTest#testConfigChange
DynamicConfigChangeTest#testDynamicTopicConfigChange

FrankYang0529 · 2025-01-16T14:44:36Z

I will do some debug. Thanks for the review.

ijuma

Updating review to indicate we need to fix the test failures.

FrankYang0529 · 2025-01-16T14:48:43Z

I found root cause. I will push the comment again. Do we want to keep BrokerEndpointTest? If yes, I will remove zk related cases only.

Signed-off-by: PoAn Yang <payang@apache.org>

ijuma · 2025-01-16T15:09:23Z

Do we want to keep BrokerEndpointTest?

You can remove it. With the conversion to record via the other PR, the remaining test doesn't add value.

ijuma · 2025-01-16T15:12:03Z

Btw, when you fix a PR, it's much easier for the reviewer, if you add additional commits versus rewriting the original commit (which has 7k lines of changes and some small new change somewhere in this case). In any case, can you describe what the fix was in this case?

FrankYang0529 · 2025-01-16T15:33:02Z

can you describe what the fix was in this case?

Sorry for that. I deleted code in TopicConfigHandler#updateLogConfig. We should not remove this. We can just fix it by removing last argument in LogManager#updateTopicConfig.

Before (remove all following code):

    // kafkaController is only defined in Zookeeper's mode
    logManager.updateTopicConfig(topic, topicConfig, kafkaConfig.remoteLogManagerConfig.isRemoteStorageSystemEnabled(),
      wasRemoteLogEnabled, kafkaController.isDefined)

After (only remove last argument):

    logManager.updateTopicConfig(topic, topicConfig, kafkaConfig.remoteLogManagerConfig.isRemoteStorageSystemEnabled(),
      wasRemoteLogEnabled)

ijuma · 2025-01-16T18:28:03Z

Found 5 test failures:
FAILED ❌ SaslPlainPlaintextConsumerTest > testCoordinatorFailover(String, String).quorum=kraft.groupProtocol=consumer
FAILED ❌ TransactionsBounceTest > testWithGroupMetadata(String, String).quorum=kraft.groupProtocol=classic
FAILED ❌ ClientIdQuotaTest > testThrottledProducerConsumer(String, String).quorum=kraft.groupProtocol=classic
FAILED ❌ ClientIdQuotaTest > testThrottledProducerConsumer(String, String).quorum=kraft.groupProtocol=consumer
FAILED ❌ ClientIdQuotaTest > testQuotaOverrideDelete(String, String).quorum=kraft.groupProtocol=consumer
Found 4 flaky test failures:
FLAKY ⚠️ EosIntegrationTest > shouldCheckpointRestoredOffsetsWhenClosingCleanDuringRestoringStateUpdaterDisabled()
FLAKY ⚠️ AbstractCoordinatorTest > testWakeupAfterSyncGroupReceivedExternalCompletion()
FLAKY ⚠️ AbstractCoordinatorTest > testWakeupAfterSyncGroupSentExternalCompletion()
FLAKY ⚠️ KafkaAdminClientTest > testAdminClientApisAuthenticationFailure()

FrankYang0529 · 2025-01-17T01:06:38Z

@ijuma, Those cases can't be reproduced on my laptop. I merge trunk to trigger CI again. Thanks.

ijuma · 2025-01-17T02:55:17Z

Looks like the tests passed.

FrankYang0529 · 2025-01-17T03:15:27Z

Hi @ijuma, thanks for the review. Will you backport this to 4.0? I try to cherry-pick locally and there is no conflict. Thanks.

ijuma · 2025-01-17T04:35:23Z

Yes, I will. I do the backports in batches and update the JIRA once they're done.

Remove KafkaController and related unused references: * ControllerChannelContext * ControllerChannelManager * ControllerEventManager * ControllerState * PartitionStateMachine * ReplicaStateMachine * TopicDeletionManager * ZkBrokerEpochManager Reviewers: Ismael Juma <ismael@juma.me.uk>

Tests for this class is being removed via #18573 - making it a record avoids the need for the tests being removed. Reviewers: Andrew Schofield <aschofield@confluent.io>

chia7712

@FrankYang0529 I apologize for the delayed response. Could you please assist me in enhancing the content of zk2kraft.html?

chia7712 · 2025-01-20T13:26:19Z

-import scala.jdk.CollectionConverters._
-
-object ControllerChannelManager {
-  private val QueueSizeMetricName = "QueueSize"


Could you update zk2kraft.html to say all metrics under kafka.controller:type=ControllerChannelManager are removed?

chia7712 · 2025-01-20T13:31:26Z

-                             eventQueueTimeTimeoutMs: Long = 300000) {
-  import ControllerEventManager._
-
-  private val metricsGroup = new KafkaMetricsGroup(this.getClass)


ditto. all metrics under kafka.controller:type=ControllerEventManager are removed

chia7712 · 2025-01-20T13:32:21Z

-                      threadNamePrefix: Option[String] = None)
-  extends ControllerEventProcessor with Logging {
-
-  private val metricsGroup = new KafkaMetricsGroup(this.getClass)


ditto. all metrics under kafka.controller:type=KafkaController are removed

chia7712 · 2025-01-20T13:32:54Z

-}
-
-private[controller] class ControllerStats {
-  private val metricsGroup = new KafkaMetricsGroup(this.getClass)


ditto. kafka.controller:type=ControllerStats

FrankYang0529 · 2025-01-21T12:23:52Z

@chia7712 Thanks for the reminder. Create a PR for it.

#18654

Remove KafkaController and related unused references: * ControllerChannelContext * ControllerChannelManager * ControllerEventManager * ControllerState * PartitionStateMachine * ReplicaStateMachine * TopicDeletionManager * ZkBrokerEpochManager Reviewers: Ismael Juma <ismael@juma.me.uk>

Tests for this class is being removed via apache#18573 - making it a record avoids the need for the tests being removed. Reviewers: Andrew Schofield <aschofield@confluent.io>

Remove KafkaController and related unused references: * ControllerChannelContext * ControllerChannelManager * ControllerEventManager * ControllerState * PartitionStateMachine * ReplicaStateMachine * TopicDeletionManager * ZkBrokerEpochManager Reviewers: Ismael Juma <ismael@juma.me.uk>

Tests for this class is being removed via apache#18573 - making it a record avoids the need for the tests being removed. Reviewers: Andrew Schofield <aschofield@confluent.io>

Remove KafkaController and related unused references: * ControllerChannelContext * ControllerChannelManager * ControllerEventManager * ControllerState * PartitionStateMachine * ReplicaStateMachine * TopicDeletionManager * ZkBrokerEpochManager Reviewers: Ismael Juma <ismael@juma.me.uk>

Tests for this class is being removed via apache#18573 - making it a record avoids the need for the tests being removed. Reviewers: Andrew Schofield <aschofield@confluent.io>

github-actions Bot added the core Kafka Broker label Jan 16, 2025

FrankYang0529 force-pushed the KAFKA-18431 branch from cbd68cc to 2f155e9 Compare January 16, 2025 12:00

ijuma reviewed Jan 16, 2025

View reviewed changes

ijuma approved these changes Jan 16, 2025

View reviewed changes

ijuma mentioned this pull request Jan 16, 2025

MINOR: Convert BrokerEndPoint to record #18577

Merged

3 tasks

ijuma requested changes Jan 16, 2025

View reviewed changes

KAFKA-18431: Remove KafkaController

fce8a5e

Signed-off-by: PoAn Yang <payang@apache.org>

FrankYang0529 force-pushed the KAFKA-18431 branch from 2f155e9 to fce8a5e Compare January 16, 2025 14:57

Merge branch 'trunk' into KAFKA-18431

78804f5

ijuma merged commit 0e502e0 into apache:trunk Jan 17, 2025

FrankYang0529 deleted the KAFKA-18431 branch January 17, 2025 02:56

ijuma added a commit that referenced this pull request Jan 17, 2025

Convert BrokerEndPoint to record (#18577)

b90ef86

Tests for this class is being removed via #18573 - making it a record avoids the need for the tests being removed. Reviewers: Andrew Schofield <aschofield@confluent.io>

frankvicky mentioned this pull request Jan 19, 2025

MINOR: Remove unused CredentialProvider#updateCredentials #18622

Merged

3 tasks

chia7712 mentioned this pull request Jan 19, 2025

KAFKA-18591: Remove unused QuotaConfgHandler #18617

Merged

3 tasks

chia7712 reviewed Jan 20, 2025

View reviewed changes

FrankYang0529 mentioned this pull request Jan 21, 2025

MINOR: add removed controller metrics to zk2kraft #18654

Merged

3 tasks

Conversation

FrankYang0529 commented Jan 16, 2025

Committer Checklist (excluded from commit message)

Uh oh!

ijuma Jan 16, 2025

Choose a reason for hiding this comment

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

ijuma commented Jan 16, 2025

Uh oh!

FrankYang0529 commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrankYang0529 commented Jan 16, 2025

Uh oh!

ijuma left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankYang0529 commented Jan 16, 2025

Uh oh!

ijuma commented Jan 16, 2025

Uh oh!

ijuma commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrankYang0529 commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ijuma commented Jan 16, 2025

Uh oh!

FrankYang0529 commented Jan 17, 2025

Uh oh!

ijuma commented Jan 17, 2025

Uh oh!

FrankYang0529 commented Jan 17, 2025

Uh oh!

ijuma commented Jan 17, 2025

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

chia7712 Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

chia7712 Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

chia7712 Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

chia7712 Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

FrankYang0529 commented Jan 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

FrankYang0529 commented Jan 16, 2025 •

edited

Loading

ijuma left a comment •

edited

Loading

ijuma commented Jan 16, 2025 •

edited

Loading

FrankYang0529 commented Jan 16, 2025 •

edited

Loading