KAFKA-14462; [4/N] Add GroupMetadataManager: ConsumerGroups Management, Members Management and Reconciliation Logic by dajac · Pull Request #13476 · apache/kafka

dajac · 2023-03-29T16:57:08Z

This PR adds the GroupMetadataManager to the group-coordinator module. This manager is responsible for handling the groups management, the members management and the entire reconciliation process. At this point, only the new consumer group type/protocol is implemented.

As you will see, the new manager is based on an architecture inspired from the quorum controller. A request can access/read the state but can't mutate it directly. Instead, a list of records is generated together with the response and those records will be applied to the state by the runtime framework. We use timeline data structures. Note that the runtime framework is not part of this PR. It will come in a following one.

For the reviewers, I suggest starting from the GroupMetadataManager.consumerGroupHeartbeat method. From there, you will how the consumer group heartbeat is handled and how all the classes fit together. Then, it is possible to review the classes independently, I suppose.

Note that I have diverged from the KIP in a few places. Firstly, I have adapted to assignor interface to be more convenient with the internals. This is something that we could still change. Secondly, I have adapted a few records. Especially, I have changed how the current state of the member is persisted. The way that we had in the KIP was a but naive and not practical at all for the implementation.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

jeffkbkim

Mainly looked at GroupMetadataManager.java and left some minor comments, will take a bit more to digest it

jeffkbkim · 2023-03-31T17:19:04Z

should this be The member's updated topic subscriptions?

jeffkbkim · 2023-03-31T17:25:48Z

to confirm, we will reach this line when a new member joins since the new ConsumerGroupMember's topic subscription is initialized as empty on group.getOrMaybeCreateMember()?

That's correct.

jeffkbkim · 2023-03-31T17:43:27Z

can we add a test case for when PartitionAssignor.assign() throws PartitionAssignorException or will this be handled elsewhere?

jeffkbkim · 2023-03-31T18:10:21Z

do we actually compute the delta here? it seems like we're comparing old vs. new and add the entire new assignment to the log if they're different.

The delta means that we only write a record for a member if its assignment is different from the previous one.

jeffkbkim · 2023-03-31T18:24:32Z

nit: only when the state has changed

jeffkbkim · 2023-03-31T20:45:16Z

"Assignment" and "Member" are swapped in ConsumerGroupCurrentMemberAssignmentKey and ConsumerGroupTargetAssignmentMemberKey which seems to be because we also have ConsumerGroupTargetAssignmentMetadataKey. Is it weird to use ConsumerGroupCurrentAssignmentMemberKey?

I see... The reason for putting Member and Metadata second in the target assignment case is because Member and Metadata are attributes of the target assignment. For the current assignment, we don't have this so I went with ConsumerGroupCurrentMemberAssignmentKey/Value which is a bit more readable in my opinion.

That being said, I am not attached to it at all. Should we just use ConsumerGroupCurrentAssignmentKey/Value without Member?

Hmm, I prefer to keep the Member keyword because it suggests that the key is unique per member.

i agree ConsumerGroupCurrentMemberAssignmentKey/Value is more readable, but it is also confusing that the ordering is different

jeffkbkim · 2023-03-31T21:08:14Z

nit: subset or not

jeffkbkim · 2023-03-31T21:13:49Z

To confirm, the consumer will send an empty list instead of null when heartbeating with no owned partitions?

That's correct. null means that there is not change since the last heartbeat.

jeffkbkim · 2023-03-31T21:15:05Z

nit: must be provided

Hangleton

Hi David, I started to read the PR and left very minor comments (mostly nits) while I am working on understanding the protocol. Thanks!

Hangleton · 2023-04-03T12:57:25Z

nit: stay -> state

Hangleton · 2023-04-03T13:30:16Z

nit: maybe Objects.hashCode() could be used?

I just relied on the auto-generated code for the hashcode. Let me try to clean this up.

Hangleton · 2023-04-03T13:45:01Z

nit: ordered -> sorted

Hangleton · 2023-04-03T15:30:46Z

nit: maybe a class could encapsulate all these parameters? Or maybe a DTO could be re-used?

Yeah, I considered whether passing RequestContext and ConsumerGroupHeartbeatRequestData would be better here. Would it be better?

Hangleton · 2023-04-03T15:35:00Z

nit: maybe this line and the one above could be merged (to avoid interleaving logs)?

I will take a look.

Hangleton · 2023-04-03T16:44:17Z

nit - this.records = Objects.requireNonNull(records);

Hangleton · 2023-04-04T11:23:28Z

Is this log provided when a new member joins the group? (or is it always re-joins?)

It is provided in both cases.

Hangleton · 2023-04-04T11:32:05Z

Should the consumer group creation be logged?

I don't think that it is necessary.

Hangleton · 2023-04-04T11:35:26Z

Should the new member creation be logged?

We already have the log printed when the member joins or re-joins. Based on this, we know that a member is implicitly created.

Hangleton · 2023-04-04T11:39:20Z

Just for my understanding, does the consumer group protocol allows the use case where the coordinator receives a LeaveGroup but the consumer group and/or member is not registered with the coordinator? Can it happen in case of heartbeat request loss, an invalid client request, or after a coordinator change?

If it is the case the group did not exist, do we need to compute a new assignment?

If the member or the group does not exist, the request is rejected with an unknown group id or an unknown member id error.

dajac · 2023-04-06T14:59:17Z

Extracted a few simple classes into #13520.

dajac · 2023-04-06T15:02:42Z

We will also do a separate PR for the changes in the assignor and the records.

jeffkbkim · 2023-04-10T17:14:26Z

Hmm, I prefer to keep the Member keyword because it suggests that the key is unique per member.

i agree ConsumerGroupCurrentMemberAssignmentKey/Value is more readable, but it is also confusing that the ordering is different

jeffkbkim · 2023-04-10T17:20:09Z

what's the soft state and how is it mutated?

There is actually no soft state at the moment. This will come with the implementation of the old protocol.

jeffkbkim · 2023-04-10T17:36:13Z

i don't see this being used right now. Should we update this once we accept a heartbeat request with SubscribedTopicNames?

i assume this will be used to listen to trigger new assignment computation when there are changes to a topic's metadata?

Removed it for now. You assumption is correct but this will come in another PR.

jeffkbkim · 2023-04-10T17:42:12Z

will this be updated when we read changes from the metadata log?

jeffkbkim · 2023-04-10T17:49:18Z

nit: has at least

jeffkbkim · 2023-04-10T17:50:10Z

nit: that the client/member

jeffkbkim · 2023-04-10T18:30:32Z

i realize it should be read as updated member's id but initially i read it as the updated memberId. can we just use memberId?

jeffkbkim · 2023-04-10T19:03:09Z

Curious as to why the name includes Partition and not Topics. my understanding is this record holds all subscribed topics per group id

Good question. I don't remember why we called it like this. Once we have merged all these PRs, I will do another one to rename records.

jeffkbkim · 2023-04-10T19:17:46Z

it's a bit unfortunate that we are iterating through the entire members map every time a single member changes its subscription. i wonder if we can keep a map of TopicMetadata to a set of member ids

let me see if i can do this.

…data

dajac · 2023-04-25T14:50:08Z

Closing this PR as I have opened smaller ones with the same code. @jeffkbkim I have addressed some of your last comments in #13639.

dajac added the KIP-848 The Next Generation of the Consumer Rebalance Protocol label Mar 29, 2023

dajac requested a review from hachikuji March 29, 2023 16:57

dajac force-pushed the KAFKA-14462-4 branch from 169de91 to 5fbe97b Compare March 30, 2023 07:57

jeffkbkim reviewed Mar 31, 2023

View reviewed changes

Hangleton reviewed Apr 4, 2023

View reviewed changes

dajac mentioned this pull request Apr 6, 2023

KAFKA-14462; [4/N] Add Group, Record and Result #13520

Merged

3 tasks

jeffkbkim reviewed Apr 10, 2023

View reviewed changes

dajac force-pushed the KAFKA-14462-4 branch from 1d9b7fb to 6fb0635 Compare April 11, 2023 07:47

dajac added 10 commits April 11, 2023 09:48

Update checkstyle

e341b2b

Update PartitionAssignor

b4cbd30

Update records

be43d6d

Add Group, Record and Result

b066649

Add ClientAssignor, MemberAssignment, TopicMetadata and VersionedMeta…

6302936

…data

Add ConsumerGroupMember

1180b41

Add RecordHelpers

6b6e503

Add TargetAssignmentBuilder

27dd2c0

Add CurrentAssignmentBuilder

4ff3251

Add GroupMetadataManager and ConsumerGroup

348b964

dajac marked this pull request as draft April 11, 2023 08:33

dajac force-pushed the KAFKA-14462-4 branch from 6fb0635 to 348b964 Compare April 11, 2023 08:33

dajac closed this Apr 25, 2023

jeffkbkim mentioned this pull request May 15, 2023

KAFKA-14462; [12/N] Add GroupMetadataManager and ConsumerGroup #13639

Merged

3 tasks

Conversation

dajac commented Mar 29, 2023

Committer Checklist (excluded from commit message)

Uh oh!

jeffkbkim left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dajac Apr 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Hangleton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dajac commented Apr 6, 2023

dajac Apr 6, 2023 •

edited

Loading