KAFKA-5505: Incremental cooperative rebalancing in Connect (KIP-415) by kkonstantine · Pull Request #6363 · apache/kafka

kkonstantine · 2019-03-04T05:50:40Z

Resolve the stop-the-world effect in Connect by allowing connectors and tasks to keep running if possible.
Maintain connector and task assignment in the presence of quick restarts or rolling upgrades of Connect workers
Upgrade Connect protocol to allow workers to report their active assignments and the leader to revoke assignments.
Select the Connect protocol during runtime in a fully backwards compatible way.
Introduce ConnectAssignor and make task assignment scheduling pluggable.
Implement incremental cooperative rebalancing of connectors and tasks
https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect

Tested via:

Unit tests for protocol compatibility
Unit tests on task assignment scheduling
Unit tests on incremental cooperative rebalancing
Integration tests on incremental cooperative rebalancing

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

kkonstantine

Marked a set of files that belong to different PRs and will be removed once those PRs are merged.
The PRs are:
#6340
#6342

kkonstantine · 2019-03-11T14:20:39Z

@ewencp @hachikuji @rhauch @mumrah the PR is ready for review. The following items are expected to be addressed along with your comments:

The code on task assignment is currently written to be readable and evaluate correctness. I'd like to add micro-benchmarks and I expect it'll be optimized.
Unit tests for DistributedHerder and WorkerCoordinatorIncrementalTest will be expanded to guard against changes in the new protocol more extensively.
More logging will be added appropriately and some integration with metrics will be considered.
A few more integration tests will be added.

Some files contain changes that are introduced by other outstanding PRs. If still present here, please skip or review the changes in their respective PRs.

Really looking forward to your comments! Thanks!

rhauch

Nice job, @kkonstantine. Took my first pass, and overall it looks good. With your review guidance above, I couldn't find any major issues, but I have quite a few comments/questions. Found and logged some nits when I happened to notice them, but I wasn't looking for them. :-D

BTW, #6342 is now merged.

rayokota

Looks great @kkonstantine ! I am very excited for this feature! Just a few comments and nits. Thanks!

kkonstantine

Thanks for your comments. @rayokota I think I've replied to all your comments.
@rhauch I didn't get to all your comments yet. Next I'll update the config and continue with the rest of the comments.

mumrah

Looks great @kkonstantine! My only real concern is the complexity and length of the methods in IncrementalCooperativeAssignor. I kind of wonder if a pattern other than procedural is warranted? I think we should at least consider changing the private methods to package-private and adding some unit tests.

rayokota · 2019-04-18T23:04:57Z

@kkonstantine , thanks for responding to my feedback! I just had one remaining comment. Looking great!

kkonstantine

@rhauch @mumrah I've addressed almost all your comments with changes or replies.
Would you mind returning to these discussions to see if we can resolve them?
There are a couple remaining items regarding javadocs and error handling during assignment, if I'm not mistaken, that I will definitely address before merging. Thanks!

kkonstantine

@ewencp thanks a lot for what I assume is your first round of comments!
I fixed/replied to the majority of the comments. Will return to the ones that need more work very soon in a second pass.

…s reached

…inComplete

…vious commit

kkonstantine · 2019-05-17T01:41:14Z

Thanks @rhauch @mumrah @rayokota @ryannedolan and @ewencp for all the insightful and useful comments! I believe I've addressed everything, except a few cleanup/refactoring suggestions that deemed high risk at this point and will be addressed in a follow up PR after this feature is merged.

Soak testing has been also performed and has confirmed correct execution for several days. More extensive testing and performance benchmarking will follow up in the next few days.

I'll be glad if we can get this in. Thanks!

rhauch

Fantastic work, @kkonstantine. I wish this weren't such a big PR, but I've been steadily tracking the progress of the latest commits as you've been running multiple tests. As you say, there are some minor things that could be cleaned up and improved, but given the size of the PR it'd be good to handle those separately in the coming days, since they shouldn't affect behavior or functionality but will be more about maintainability.

I'm approving pending a green build and successful Connect tests. Most of the recent PR builds have been great, but I know you changed just a few test-related things (e.g., Jenkinsfile to run the Connect tests many times) that you've now reverted, and they theoretically shouldn't affect the build.

…pache#6363) Added the incremental cooperative rebalancing in Connect to avoid global rebalances on all connectors and tasks with each new/changed/removed connector. This new protocol is backward compatible and will work with heterogeneous clusters that exist during a rolling upgrade, but once the clusters consist of new workers only some affected connectors and tasks will be rebalanced: connectors and tasks on existing nodes still in the cluster and not added/changed/removed will continue running while the affected connectors and tasks are rebalanced. This commit attempted to minimize the changes to the existing V0 protocol logic, though that was not entirely possible. This commit adds extensive unit and integration tests for both the old V0 protocol and the new v1 protocol. Soak testing has been performed multiple times to verify behavior while connectors and added, changed, and removed and while workers are added and removed from the cluster. Author: Konstantine Karantasis <konstantine@confluent.io> Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <me@ewencp.org>, Robert Yokota <rayokota@gmail.com>, David Arthur <mumrah@gmail.com>, Ryanne Dolan <ryannedolan@gmail.com>

findnanda · 2019-11-15T12:15:43Z

Hi Can you please suggest in which kafka version is this issue fixed as I am still seeing this problem every time i add a new connector, all the connector gets restarted?

rhauch · 2019-11-15T15:35:16Z

@findnanda, the Jira issue (https://issues.apache.org/jira/browse/KAFKA-5505) shows that this was merged and completed in AK 2.3.0.

If you're using AK 2.3.0 or later and still having problems, please create a new Jira issue and provide the Connect worker configs and a lot more detail about a procedure to replicate the problem. Thanks!

kkonstantine force-pushed the kafka-5505 branch 3 times, most recently from 0dd3063 to ef8d122 Compare March 6, 2019 21:50

kkonstantine commented Mar 6, 2019

View reviewed changes

kkonstantine commented Mar 7, 2019

View reviewed changes

Comment thread connect/runtime/src/test/resources/log4j.properties Outdated

kkonstantine force-pushed the kafka-5505 branch 2 times, most recently from 24cffd0 to abbf617 Compare March 11, 2019 04:23

rhauch reviewed Mar 26, 2019

View reviewed changes

kkonstantine force-pushed the kafka-5505 branch from f209531 to da659db Compare April 3, 2019 05:36

rayokota reviewed Apr 3, 2019

View reviewed changes

kkonstantine mentioned this pull request Apr 5, 2019

KAFKA-8013: Avoid underflow when reading a Struct from a partially correct buffer #6340

Merged

3 tasks

kkonstantine force-pushed the kafka-5505 branch from 7ce0d75 to e5b22b1 Compare April 16, 2019 00:32

kkonstantine commented Apr 16, 2019

View reviewed changes

kkonstantine force-pushed the kafka-5505 branch from e5b22b1 to 4f25d17 Compare April 16, 2019 00:39

mumrah reviewed Apr 16, 2019

View reviewed changes

kkonstantine force-pushed the kafka-5505 branch from 4f25d17 to 28402f1 Compare April 17, 2019 04:37

kkonstantine force-pushed the kafka-5505 branch 2 times, most recently from 9ce15ad to 662a259 Compare April 25, 2019 22:36

kkonstantine commented Apr 25, 2019

View reviewed changes

kkonstantine force-pushed the kafka-5505 branch from fc4cb39 to 4bebbdd Compare April 25, 2019 23:34

ewencp reviewed Apr 26, 2019

View reviewed changes

kkonstantine force-pushed the kafka-5505 branch 2 times, most recently from b6d69f2 to 79677ae Compare April 29, 2019 23:37

kkonstantine commented Apr 30, 2019

View reviewed changes

kkonstantine added 14 commits May 16, 2019 15:49

KIP-415: Skip consecutive revocations correctly

8d05ff7

KIP-415: Fix requesting a rebalance when schedule rebalance timeout i…

e6c2585

…s reached

KIP-415: Refactor WorkerCoordinatorIncrementalTest

8bb2dfa

KIP-415: Upgrade comment in protocol assignments to javadoc

d8234a9

KIP-415: Add validation for SCHEDULED_REBALANCE_MAX_DELAY_MS_CONFIG

8502032

KIP-415: More unit tests for the worker coordinator

8669030

KIP-415: Add herder unit tests for incremental cooperative rebalancing

44f2f59

KIP-415: Use a builder for WorkerLoad and ConnectorsAndTasks classes

c5d267d

KIP-415: Remove warning that is a unreachable

a6e307e

KIP-415: Improve and rearrange log messages in WorkerCoordinator#onJo…

3def33c

…inComplete

KIP-415: Fix handling of lost assignments and add specific unit tests

22a5f47

KIP-415: Factor out metadataRequest to a method in Connect protocols

fe66c21

KIP-415: Refactor ExtendedWorkerState as a separate public class

9545219

KIP-415: Rename ConnectAssignment to ExtendedAssignment

e2de086

kkonstantine force-pushed the kafka-5505 branch 2 times, most recently from 145a2fd to 736668b Compare May 16, 2019 23:22

kkonstantine added 5 commits May 16, 2019 18:29

KIP-415: Remove embed method from WorkerLoad which was missing in pre…

90ca7d0

…vious commit

KIP-415: Improve logging and javadocs in IncrementalCooperativeAssignor

b1c5424

KIP-415: Clarify javadocs

cffcbad

KIP-415: Rename enum method to fromProtocol to make meaning more clear

59785bb

KIP-415: Small fixes in connect integration tests to avoid flakiness

eb1853d

kkonstantine force-pushed the kafka-5505 branch from 736668b to eb1853d Compare May 17, 2019 01:31

rhauch approved these changes May 17, 2019

View reviewed changes

rhauch merged commit ce584a0 into apache:trunk May 17, 2019

kkonstantine added the connect label Oct 16, 2020

Conversation

kkonstantine commented Mar 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Committer Checklist (excluded from commit message)

Uh oh!

kkonstantine left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kkonstantine commented Mar 11, 2019

Uh oh!

rhauch left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rayokota left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kkonstantine left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mumrah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rayokota commented Apr 18, 2019

Uh oh!

kkonstantine left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kkonstantine commented Mar 4, 2019 •

edited

Loading

rhauch left a comment •

edited

Loading