KAFKA-18592: Cleanup ReplicaManager by frankvicky · Pull Request #18621 · apache/kafka

frankvicky · 2025-01-19T07:16:26Z

JIRA: KAFKA-18592
There are a few methods in ReplicaManager unused now, we should remove them.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

chia7712 · 2025-01-19T07:21:11Z

-                                     isFromConsumer: Boolean,
-                                     fetchOnlyFromLeader: Boolean): Seq[Long] = {
-    val partition = getPartitionOrException(topicPartition)
-    partition.legacyFetchOffsetsForTimestamp(timestamp, maxNumOffsets, isFromConsumer, fetchOnlyFromLeader)


Could we remove it from partition also?

chia7712 · 2025-01-19T07:21:35Z

-  def hasDelayedElectionOperations: Boolean = delayedElectLeaderPurgatory.numDelayed != 0
-
-  def tryCompleteElection(key: DelayedOperationKey): Unit = {
-    val completed = delayedElectLeaderPurgatory.checkAndComplete(key)


Could we remove the purgatory too?

chia7712 · 2025-01-19T13:52:52Z

    new DelayedOperationPurgatory[DelayedDeleteRecords](
      "DeleteRecords", config.brokerId,
      config.deleteRecordsPurgatoryPurgeIntervalRequests))
-  val delayedElectLeaderPurgatory = delayedElectLeaderPurgatoryParam.getOrElse(


please remove delayedElectLeaderPurgatoryParam also

Also, this also remove the following metrics.

kafka.server:type=DelayedOperationPurgatory,delayedOperation=ElectLeader,name=PurgatorySize

kafka.server:type=DelayedOperationPurgatory,delayedOperation=ElectLeader,name=NumDelayedOperations

please add them into zk2kraft.html

Interesting, we still allow elect leader to be called. Why are these metrics no longer relevant?

The zk broker waits for the metadata update before returning a response to the ElectLeader request, whereas the Kraft broker does not. Consequently, the Kraft broker does not need delayedOperation metrics for ElectLeader requests. Instead, kraft broker has ForwardingManager metrics to monitor the forwarded requests.

Thanks for the explanation.

ijuma · 2025-01-22T02:33:41Z

Please resolve the conflicts.

ijuma

Thanks, a few comments.

ijuma · 2025-01-22T14:41:34Z

-                                     isFromConsumer: Boolean,
-                                     fetchOnlyFromLeader: Boolean): Seq[Long] = inReadLock(leaderIsrUpdateLock) {
-    val localLog = localLogWithEpochOrThrow(Optional.empty(), fetchOnlyFromLeader)
-    val allOffsets = localLog.legacyFetchOffsetsBefore(timestamp, maxNumOffsets)


Perhaps it can be removed from LocalLog too (in a follow-up).

You are right.
If this PR gets merged, this method would only be invoked in tests.
I will file a JIRA for this one.

https://issues.apache.org/jira/browse/KAFKA-18620

ijuma · 2025-01-22T14:44:09Z

        </li>
+        <li>
+            <p>
+                Remove the metrics which is monitoring the leader election.


This seems misleading - I think these metrics are related to the purgatory for the ElectLeader operation - we still have metrics for monitoring leader election.

Thanks for pointing out.
I have just updated the description. 🙇🏼

Do we need to explain the usage for each removed metric? Since there are numerous removed metrics, I propose adding a listed item to include all of them. This was also the suggestion I made in #18654.

@ijuma @frankvicky WDYT?

Make sense to me.

I think it may be helpful to indicate which metrics replace the ones being removed or if they're not applicable anymore. That would be useful, I believe.

@ijuma you are right. #18564 will address it

JIRA: KAFKA-18592 There are few methods in ReplicaManager unused now, we should remove them. # Conflicts: # docs/zk2kraft.html

clolov

Thanks for the contribution! A small comment from my side

clolov · 2025-01-23T11:59:07Z

-        return this;
-    }
-
-    public ReplicaManagerBuilder setDelayedDeleteRecordsPurgatory(DelayedOperationPurgatory<DelayedDeleteRecords> delayedDeleteRecordsPurgatory) {


Is there a reason why you can't remove the field(s) associated with this setter (and others) as well? As far as I can see it is only used as an argument for creating a ReplicaManager and you could just leave the instantiation of the purgatory in the constructor of ReplicaManager. I am happy for this to be done in a follow-up as long as it has already been accounted for and it isn't just a miss.

You are right. These fields are something we miss and also need to be clean.
I will file a jira for this one.

https://issues.apache.org/jira/browse/KAFKA-18630

clolov

With my last comment addressed, I am happy for this to go in assuming the tests pass

chia7712

LGTM

I'm going to merge this PR and #18654 will revise the zk2kraft.html

Reviewers: Ismael Juma <ismael@juma.me.uk>, Christo Lolov <lolovc@amazon.com>, Chia-Ping Tsai <chia7712@gmail.com>

github-actions Bot added triage PRs from the community core Kafka Broker small Small PRs labels Jan 19, 2025

chia7712 added the ci-approved label Jan 19, 2025

chia7712 reviewed Jan 19, 2025

View reviewed changes

frankvicky force-pushed the KAFKA-18592 branch from 34db24b to 5b59263 Compare January 19, 2025 07:27

chia7712 reviewed Jan 19, 2025

View reviewed changes

frankvicky force-pushed the KAFKA-18592 branch 2 times, most recently from 2d70136 to f142c30 Compare January 19, 2025 15:06

github-actions Bot removed the triage PRs from the community label Jan 20, 2025

frankvicky force-pushed the KAFKA-18592 branch from f142c30 to a2c5502 Compare January 21, 2025 02:30

frankvicky force-pushed the KAFKA-18592 branch from a2c5502 to 1f80a44 Compare January 22, 2025 02:37

ijuma reviewed Jan 22, 2025

View reviewed changes

frankvicky force-pushed the KAFKA-18592 branch from 1f80a44 to af3dff4 Compare January 22, 2025 14:57

KAFKA-18592: Cleanup ReplicaManager

76a7eee

JIRA: KAFKA-18592 There are few methods in ReplicaManager unused now, we should remove them. # Conflicts: # docs/zk2kraft.html

clolov reviewed Jan 23, 2025

View reviewed changes

frankvicky force-pushed the KAFKA-18592 branch from af3dff4 to 76a7eee Compare January 23, 2025 13:56

clolov approved these changes Jan 23, 2025

View reviewed changes

chia7712 approved these changes Jan 23, 2025

View reviewed changes

chia7712 merged commit 40890fa into apache:trunk Jan 23, 2025

clolov pushed a commit that referenced this pull request Jan 24, 2025

KAFKA-18592 Cleanup ReplicaManager (#18621)

f1b3634

Reviewers: Ismael Juma <ismael@juma.me.uk>, Christo Lolov <lolovc@amazon.com>, Chia-Ping Tsai <chia7712@gmail.com>

pranavt84 pushed a commit to pranavt84/kafka that referenced this pull request Jan 27, 2025

KAFKA-18592 Cleanup ReplicaManager (apache#18621)

228cd12

Reviewers: Ismael Juma <ismael@juma.me.uk>, Christo Lolov <lolovc@amazon.com>, Chia-Ping Tsai <chia7712@gmail.com>

airlock-confluentinc Bot pushed a commit to confluentinc/kafka that referenced this pull request Jan 27, 2025

KAFKA-18592 Cleanup ReplicaManager (apache#18621)

17200e4

Reviewers: Ismael Juma <ismael@juma.me.uk>, Christo Lolov <lolovc@amazon.com>, Chia-Ping Tsai <chia7712@gmail.com>

pdruley pushed a commit to pdruley/kafka that referenced this pull request Feb 12, 2025

KAFKA-18592 Cleanup ReplicaManager (apache#18621)

de27856

Reviewers: Ismael Juma <ismael@juma.me.uk>, Christo Lolov <lolovc@amazon.com>, Chia-Ping Tsai <chia7712@gmail.com>

manoj-mathivanan pushed a commit to manoj-mathivanan/kafka that referenced this pull request Feb 19, 2025

KAFKA-18592 Cleanup ReplicaManager (apache#18621)

967944b

Reviewers: Ismael Juma <ismael@juma.me.uk>, Christo Lolov <lolovc@amazon.com>, Chia-Ping Tsai <chia7712@gmail.com>

Conversation

frankvicky commented Jan 19, 2025

Committer Checklist (excluded from commit message)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ijuma Jan 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ijuma commented Jan 22, 2025

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clolov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clolov left a comment

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ijuma Jan 19, 2025 •

edited

Loading

clolov left a comment •

edited

Loading