KAFKA-12382: add a README for KIP-500 by cmccabe · Pull Request #10227 · apache/kafka

cmccabe · 2021-02-27T00:46:25Z

No description provided.

rondagostino

Looks good! I left a few minor comments.

Co-authored-by: Tom Bentley <tombentley@users.noreply.github.com>

hachikuji · 2021-03-09T22:12:16Z

+Self-managed mode in Kafka 2.8 is provided for testing only, *NOT* for production.  We do not yet support upgrading existing ZooKeeper-based Kafka clusters into this mode.  In fact, when Kafka 3.0 is released, it may not even be possible to upgrade your self-managed clusters from 2.8 to 3.0 without downtime.  There may be bugs, including serious ones.  You should *assume that your data could be lost at any time* if you try the early access release of KIP-500.
+
+## Generate a cluster ID
+The first step is to generate an ID for your new cluster, using the kafka-storage tool:


It would be useful to add a couple sentences of explanation for the clusterId generation and the storage tool since these are new. To keep the instructions concise here, maybe we could refer to a separate section which has more background.

Hmm, I'm not sure what else to add here. Can you give an example?

Why do we have this tool? Users will certainly ask that since these steps were not needed before. There are good reasons for it, so why don't we give a couple sentences of explanation?

hachikuji · 2021-03-09T22:18:33Z

+# Deployment
+Unlike in ZooKeeper-based mode, where any server can become the controller, in self-managed mode, only a small group of specially selected servers can act as controllers.  The specially selected controller servers will participate in the metadata quorum.  Each KIP-500 controller server is either active, or a hot standby for the current active controller server.
+
+Typically you will select either 3 or 5 servers for this role, depending on the size of your cluster.  Just like with ZooKeeper, you must keep a majority of the controllers alive in order to maintain availability.  So if you have 3 controllers, you can tolerate 1 failure; with 5 controllers, you can tolerate 2 failures.


I wonder if it's worth saying something about the relation between process.roles, controller.quorum.voters, and controller.listeners.

Hmm, I'm not sure what to add...

I think the main thing is emphasizing that they need to be set consistently. We don't say anything about quorum.voters in this README currently, but it's important to understand the basics of how to configure it to run a multi-node setup.

I added a section about quorum.voters

hachikuji

LGTM. Left two small comments.

hachikuji · 2021-03-18T17:11:04Z

+Nodes that act as both brokers and controllers are referred to as "combined" nodes.  The advantage of using combined nodes you will have uses fewer Java Virtual Machines (JVMs).  This will allow you to avoid some of the fixed memory overheads associated with JVMs.  The disdavantage is that the controller will be less isolated from the rest of the system.  For example, if activity on the broker causes an out of memory condition, the controller part of the server is not isolated from that OOM condition.
+
+## Quorum Voters
+All nodes in the system must set the `quorum.voters` configuration.  This identifies the quorum controller servers that should be used.  All the controllers must be enumerated.  This is similar to how, when using ZooKeeper, the `zookeeper.connect` configuration must contain all the ZooKeeper servers.  Unlike with the ZK config, however, `quorum.voters` also has IDs for each node.  The format is id1@host1:port1,id2@host2:port2, etc.


Should be controller.quorum.voters

hachikuji · 2021-03-18T17:13:22Z

+So if you have 10 brokers and 3 controllers named controller1, controller2, controller3, you might have the following `quorum.voters` configuration:
+```
+1@controller1.example.com:9093,2@controller2.example.com:9093,3@controller3.example.com:9093
+```


Can you include an example of the listener configuration. Just something which emphasizes that port 9093 is defined as the controller listener. For example:

node.id=1 listeners=CONTROLLER://controller1.example.com:9093 controller.listener.name=CONTROLLER

Reviewers: Tom Bentley <tbentley@redhat.com>, Ron Dagostino <rdagostino@confluent.io>, Jason Gustafson <jason@confluent.io>

Conflicts: * build.gradle: keep `dependencySubstitution` Confluent addition in `resolutionStrategy` and take upstream changes. Commits: * apache-github/trunk: KAFKA-12503: inform threads to resize their cache instead of doing so for them (apache#10356) KAFKA-10697: Remove ProduceResponse.responses (apache#10332) MINOR: Exclude KIP-500.md from rat check (apache#10354) MINOR: Move `configurations.all` to be a child of `allprojects` (apache#10349) MINOR: Remove use of `NoSuchElementException` in `KafkaMetadataLog` (apache#10344) MINOR: Start the broker-to-controller channel for request forwarding (apache#10340) KAFKA-12382: add a README for KIP-500 (apache#10227) MINOR: Fix BaseHashTable sizing (apache#10334) KAFKA-10357: Add setup method to internal topics (apache#10317) MINOR: remove redundant null check when testing specified type (apache#10314) KAFKA-12293: Remove JCenter from buildscript and delete buildscript.gradle KAFKA-12491: Make rocksdb an `api` dependency for `streams` (apache#10341) KAFKA-12454: Add ERROR logging on kafka-log-dirs when given brokerIds do not exist in current kafka cluster (apache#10304) KAFKA-12459; Use property testing library for raft event simulation tests (apache#10323) MINOR: fix failing ZooKeeper system tests (apache#10297) MINOR: fix client_compatibility_features_test.py (apache#10292)

cmccabe added the kraft label Feb 27, 2021

cmccabe changed the title ~~MINOR: add a README for KIP-500~~ KAFKA-12382: add a README for KIP-500 Feb 27, 2021

rondagostino reviewed Mar 1, 2021

View reviewed changes

Comment thread KIP-500.md Outdated

Comment thread KIP-500.md Outdated

Comment thread KIP-500.md Outdated

Comment thread KIP-500.md Outdated

Comment thread KIP-500.md Outdated

tombentley reviewed Mar 1, 2021

View reviewed changes

Comment thread KIP-500.md Outdated

Comment thread KIP-500.md Outdated

Comment thread KIP-500.md Outdated

tombentley reviewed Mar 2, 2021

View reviewed changes

Comment thread KIP-500.md Outdated

cmccabe added 3 commits March 9, 2021 13:41

MINOR: add a README for KIP-500

29a055c

Rename config/raft-* to config/nozk-*

8c21223

Revisions

c33b866

cmccabe force-pushed the kip-500-readme branch from abd94e2 to c33b866 Compare March 9, 2021 21:47

put spaces around em-dash

a182e46

Co-authored-by: Tom Bentley <tombentley@users.noreply.github.com>

hachikuji reviewed Mar 9, 2021

View reviewed changes

rondagostino reviewed Mar 16, 2021

View reviewed changes

Comment thread KIP-500.md Outdated

Some improvements

5ab87ca

cmccabe force-pushed the kip-500-readme branch from 90819e4 to 5ab87ca Compare March 17, 2021 20:52

hachikuji approved these changes Mar 18, 2021

View reviewed changes

Address review comments

97ce1d1

cmccabe merged commit dc6ea93 into apache:trunk Mar 18, 2021

cmccabe deleted the kip-500-readme branch March 18, 2021 17:27

cmccabe added a commit that referenced this pull request Mar 18, 2021

KAFKA-12382: add a README for KIP-500 (#10227)

67cb16b

Reviewers: Tom Bentley <tbentley@redhat.com>, Ron Dagostino <rdagostino@confluent.io>, Jason Gustafson <jason@confluent.io>

Conversation

cmccabe commented Feb 27, 2021

Uh oh!

rondagostino left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hachikuji left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants