MINOR: update the reassign-partition script results in docs by showuon · Pull Request #11845 · apache/kafka

showuon · 2022-03-04T09:16:11Z

Remove decommissioning brokers section
When reading decommissioning brokers section in document, it makes me confused:

The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet.

We should be able to auto generate a reassignment plan for decommissioning brokers via --generate argument

the reassignment needs to ensure that all the replicas are not moved from the decommissioned broker to only one other broker.

We already can move the replicas evenly across all the brokers

To make this process effortless, we plan to add tooling support for decommissioning brokers in the future.

This section is written in 2015, and I don't think we still have plan to support it (or we already supported)

Remove the section.

Update the --generate replicas plan with current version.

Also, the example in the --generate command, it assign all the partition preferred leader to the same one (broker 5), which will confuse users to think this tool is unreliable. After running a test, I confirmed we can generate a plan evenly assigning the replicas across the brokers. Update it.

Update the throttle updated command by adding --additional
After KIP-455 and this PR, we have to provide --additional flag to allow change the throttle config.
Update the reassign-partition script output with current version

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

showuon · 2022-03-04T09:19:44Z

-  "partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},
-                {"topic":"foo1","partition":0,"replicas":[3,4]},
-                {"topic":"foo2","partition":2,"replicas":[1,2]},
-                {"topic":"foo2","partition":0,"replicas":[3,4]},
-                {"topic":"foo1","partition":1,"replicas":[2,3]},
-                {"topic":"foo2","partition":1,"replicas":[2,3]}]
+  "partitions":[{"topic":"foo1","partition":0,"replicas":[2,1]},
+                {"topic":"foo1","partition":1,"replicas":[1,3]},
+                {"topic":"foo1","partition":2,"replicas":[3,4]},
+                {"topic":"foo2","partition":0,"replicas":[4,2]},
+                {"topic":"foo2","partition":1,"replicas":[2,1]},
+                {"topic":"foo2","partition":2,"replicas":[1,3]}]


Update the current partition replica assignment with the one in my local env, which has preferred partition leader evenly distributed across all brokers.

showuon · 2022-03-04T09:20:46Z

-  "partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},
-                {"topic":"foo1","partition":0,"replicas":[5,6]},
-                {"topic":"foo2","partition":2,"replicas":[5,6]},
-                {"topic":"foo2","partition":0,"replicas":[5,6]},
+  "partitions":[{"topic":"foo1","partition":0,"replicas":[6,5]},
                {"topic":"foo1","partition":1,"replicas":[5,6]},
-                {"topic":"foo2","partition":1,"replicas":[5,6]}]
+                {"topic":"foo1","partition":2,"replicas":[6,5]},
+                {"topic":"foo2","partition":0,"replicas":[5,6]},
+                {"topic":"foo2","partition":1,"replicas":[6,5]},
+                {"topic":"foo2","partition":2,"replicas":[5,6]}]


Updated the proposed reassignment with current script result, having the preferred partition leader evenly distributed.

showuon · 2022-03-04T09:21:22Z

-                {"topic":"foo1","partition":1,"replicas":[5,6]},
-                {"topic":"foo2","partition":1,"replicas":[5,6]}]
-  }</code></pre>
+  Successfully started partition reassignments for foo1-0,foo1-1,foo1-2,foo2-0,foo2-1,foo2-2


We only show Successfully started partition reassignments for... in current script. Update it.

showuon · 2022-03-04T09:21:50Z

-  Reassignment of partition [foo1,0] completed successfully
-  Reassignment of partition [foo1,1] is in progress
-  Reassignment of partition [foo1,2] is in progress
-  Reassignment of partition [foo2,0] completed successfully
-  Reassignment of partition [foo2,1] completed successfully
-  Reassignment of partition [foo2,2] completed successfully</code></pre>
+  Reassignment of partition [foo1,0] is completed
+  Reassignment of partition [foo1,1] is still in progress
+  Reassignment of partition [foo1,2] is still in progress
+  Reassignment of partition [foo2,0] is completed
+  Reassignment of partition [foo2,1] is completed
+  Reassignment of partition [foo2,2] is completed</code></pre>


Change the verify result with current script version

showuon · 2022-03-04T09:22:16Z

-  <h4 class="anchor-heading"><a id="basic_ops_decommissioning_brokers" class="anchor-link"></a><a href="#basic_ops_decommissioning_brokers">Decommissioning brokers</a></h4>
-  The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet. As such, the admin has to come up with a reassignment plan to move the replica for all partitions hosted on the broker to be decommissioned, to the rest of the brokers. This can be relatively tedious as the reassignment needs to ensure that all the replicas are not moved from the decommissioned broker to only one other broker. To make this process effortless, we plan to add tooling support for decommissioning brokers in the future.


Remove decommissioning brokers section

showuon · 2022-03-04T09:22:45Z

+  <pre class="language-bash">$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --additional --throttle 700000000
+  The inter-broker throttle limit was set to 700000000 B/s</code></pre>


We should add --additional argument in the script to change the throttle value.

showuon · 2022-03-04T09:24:03Z

@hachikuji @cmccabe , please take a look. Thanks.

showuon · 2022-04-21T08:01:08Z

@hachikuji @cmccabe , please take a look. Thanks.

showuon · 2022-04-30T02:55:04Z

@hachikuji @cmccabe , please take a look. Thanks.

showuon · 2022-05-19T02:37:26Z

@hachikuji @dajac , please take a look when available. Thanks.

showuon · 2022-05-31T02:40:32Z

@hachikuji @dajac , please take a look. Thank you.

showuon · 2022-06-02T13:33:10Z

Addressed in #12237

update the reassign-partition script results and argument

1851102

showuon commented Mar 4, 2022

View reviewed changes

Corlobin approved these changes Mar 6, 2022

View reviewed changes

showuon changed the title ~~MINOR: update the reassign-partition script results and argument~~ MINOR: update the reassign-partition script results in docs May 19, 2022

showuon closed this Jun 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MINOR: update the reassign-partition script results in docs#11845

MINOR: update the reassign-partition script results in docs#11845
showuon wants to merge 1 commit intoapache:trunkfrom
showuon:reassignPartitionUpdate

showuon commented Mar 4, 2022 •

edited

Loading

Uh oh!

showuon Mar 4, 2022

Uh oh!

showuon Mar 4, 2022

Uh oh!

showuon Mar 4, 2022

Uh oh!

showuon Mar 4, 2022

Uh oh!

showuon Mar 4, 2022

Uh oh!

showuon Mar 4, 2022

Uh oh!

showuon commented Mar 4, 2022

Uh oh!

showuon commented Apr 21, 2022

Uh oh!

showuon commented Apr 30, 2022

Uh oh!

showuon commented May 19, 2022

Uh oh!

showuon commented May 31, 2022

Uh oh!

showuon commented Jun 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		<h4 class="anchor-heading"><a id="basic_ops_decommissioning_brokers" class="anchor-link"></a><a href="#basic_ops_decommissioning_brokers">Decommissioning brokers</a></h4>
		The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet. As such, the admin has to come up with a reassignment plan to move the replica for all partitions hosted on the broker to be decommissioned, to the rest of the brokers. This can be relatively tedious as the reassignment needs to ensure that all the replicas are not moved from the decommissioned broker to only one other broker. To make this process effortless, we plan to add tooling support for decommissioning brokers in the future.

		<pre class="language-bash">$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --additional --throttle 700000000
		The inter-broker throttle limit was set to 700000000 B/s</code></pre>

Conversation

showuon commented Mar 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Committer Checklist (excluded from commit message)

Uh oh!

showuon Mar 4, 2022

Choose a reason for hiding this comment

Uh oh!

showuon Mar 4, 2022

Choose a reason for hiding this comment

Uh oh!

showuon Mar 4, 2022

Choose a reason for hiding this comment

Uh oh!

showuon Mar 4, 2022

Choose a reason for hiding this comment

Uh oh!

showuon Mar 4, 2022

Choose a reason for hiding this comment

Uh oh!

showuon Mar 4, 2022

Choose a reason for hiding this comment

Uh oh!

showuon commented Mar 4, 2022

Uh oh!

showuon commented Apr 21, 2022

Uh oh!

showuon commented Apr 30, 2022

Uh oh!

showuon commented May 19, 2022

Uh oh!

showuon commented May 31, 2022

Uh oh!

showuon commented Jun 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

showuon commented Mar 4, 2022 •

edited

Loading