KAFKA-8591: WorkerConfigTransformer NPE on connector configuration reloading by nachomdo · Pull Request #6991 · apache/kafka

nachomdo · 2019-06-24T12:10:22Z

A bug in WorkerConfigTransformer prevents the connector configuration reload when the ConfigData TTL expires.

The issue boils down to the fact that worker.herder().restartConnector is receiving a null callback.

[2019-06-17 14:34:12,320] INFO Scheduling a restart of connector workshop-incremental in 60000 ms (org.apache.kafka.connect.runtime.WorkerConfigTransformer:88)
[2019-06-17 14:34:12,321] ERROR Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:227)
java.lang.NullPointerException
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$19.onCompletion(DistributedHerder.java:1187)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$19.onCompletion(DistributedHerder.java:1183)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:273)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:219)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

This patch keeps the same behaviour than before in the WorkerConfigTransformer in terms of ignoring any error returned from the callback. Do we still want to behave in the same way or we would like to handle any potential error? 🤔 maybe we can rely on the scheduleReload retrying again whenever the TTL expires or the connector task fails due to a stale configuration.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

rayokota · 2019-07-02T15:14:02Z

Perhaps instead

CallBack<Void> cb = new Callback<Void>() { @Override public void onCompletion(Throwable error, Void result) { if (error != null) { log.error("Unexpected error during connector restart: ", error); } } };

rayokota

Thanks for the PR @nachomdo ! Left a small comment.

…PR feedback

nachomdo · 2019-07-03T14:33:38Z

Feedback applied ⚡️ Thanks for the 👀 @rayokota! 🏆

rayokota

Thanks @nachomdo ! LGTM

@rhauch, can you review and merge when you get a chance?

hachikuji

Thanks for the fix. Left a couple small comments.

hachikuji · 2019-07-03T18:28:48Z

    public void testReplaceVariableWithTTLFirstCancelThenScheduleRestart() {
        EasyMock.expect(worker.herder()).andReturn(herder);
-        EasyMock.expect(herder.restartConnector(1L, MY_CONNECTOR, null)).andReturn(requestId);
+        EasyMock.expect(herder.restartConnector(eq(1L), eq(MY_CONNECTOR),  anyObject())).andReturn(requestId);


nit: extra space before anyObject

hachikuji · 2019-07-03T18:35:00Z

        EasyMock.expect(worker.herder()).andReturn(herder);
-        EasyMock.expect(herder.restartConnector(1L, MY_CONNECTOR, null)).andReturn(requestId);
-
+        EasyMock.expect(herder.restartConnector(eq(1L), eq(MY_CONNECTOR), anyObject())).andReturn(requestId);


Hmm.. These tests all pass even if the callback is null. Perhaps we can use EasyMock.notNull()?

hachikuji · 2019-07-08T23:30:38Z

retest this please

hachikuji

Thanks for the patch!

…loading (#6991) A bug in `WorkerConfigTransformer` prevents the connector configuration reload when the ConfigData TTL expires. The issue boils down to the fact that `worker.herder().restartConnector` is receiving a null callback. ``` [2019-06-17 14:34:12,320] INFO Scheduling a restart of connector workshop-incremental in 60000 ms (org.apache.kafka.connect.runtime.WorkerConfigTransformer:88) [2019-06-17 14:34:12,321] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:227) java.lang.NullPointerException at org.apache.kafka.connect.runtime.distributed.DistributedHerder$19.onCompletion(DistributedHerder.java:1187) at org.apache.kafka.connect.runtime.distributed.DistributedHerder$19.onCompletion(DistributedHerder.java:1183) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:273) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:219) ``` This patch adds a callback which just logs the error. Reviewers: Robert Yokota <rayokota@gmail.com>, Jason Gustafson <jason@confluent.io>

…loading (apache#6991) A bug in `WorkerConfigTransformer` prevents the connector configuration reload when the ConfigData TTL expires. The issue boils down to the fact that `worker.herder().restartConnector` is receiving a null callback. ``` [2019-06-17 14:34:12,320] INFO Scheduling a restart of connector workshop-incremental in 60000 ms (org.apache.kafka.connect.runtime.WorkerConfigTransformer:88) [2019-06-17 14:34:12,321] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:227) java.lang.NullPointerException at org.apache.kafka.connect.runtime.distributed.DistributedHerder$19.onCompletion(DistributedHerder.java:1187) at org.apache.kafka.connect.runtime.distributed.DistributedHerder$19.onCompletion(DistributedHerder.java:1183) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:273) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:219) ``` This patch adds a callback which just logs the error. Reviewers: Robert Yokota <rayokota@gmail.com>, Jason Gustafson <jason@confluent.io>

* apache-github/2.3: MINOR: Update documentation for enabling optimizations (apache#7099) MINOR: Remove stale streams producer retry default docs. (apache#6844) KAFKA-8635; Skip client poll in Sender loop when no request is sent (apache#7085) KAFKA-8615: Change to track partition time breaks TimestampExtractor (apache#7054) KAFKA-8670; Fix exception for kafka-topics.sh --describe without --topic mentioned (apache#7094) KAFKA-8602: Separate PR for 2.3 branch (apache#7092) KAFKA-8530; Check for topic authorization errors in OffsetFetch response (apache#6928) KAFKA-8662; Fix producer metadata error handling and consumer manual assignment (apache#7086) KAFKA-8637: WriteBatch objects leak off-heap memory (apache#7050) KAFKA-8620: fix NPE due to race condition during shutdown while rebalancing (apache#7021) HOT FIX: close RocksDB objects in correct order (apache#7076) KAFKA-7157: Fix handling of nulls in TimestampConverter (apache#7070) KAFKA-6605: Fix NPE in Flatten when optional Struct is null (apache#5705) Fixes apache#8198 KStreams testing docs use non-existent method pipe (apache#6678) KAFKA-5998: fix checkpointableOffsets handling (apache#7030) KAFKA-8653; Default rebalance timeout to session timeout for JoinGroup v0 (apache#7072) KAFKA-8591; WorkerConfigTransformer NPE on connector configuration reloading (apache#6991) MINOR: add upgrade text (apache#7013) Bump version to 2.3.1-SNAPSHOT

… connector configuration reloading (apache#6991) TICKET = KAFKA-8591 LI_DESCRIPTION = EXIT_CRITERIA = HASH [4fdfe2b] ORIGINAL_DESCRIPTION = A bug in `WorkerConfigTransformer` prevents the connector configuration reload when the ConfigData TTL expires. The issue boils down to the fact that `worker.herder().restartConnector` is receiving a null callback. ``` [2019-06-17 14:34:12,320] INFO Scheduling a restart of connector workshop-incremental in 60000 ms (org.apache.kafka.connect.runtime.WorkerConfigTransformer:88) [2019-06-17 14:34:12,321] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:227) java.lang.NullPointerException at org.apache.kafka.connect.runtime.distributed.DistributedHerder$19.onCompletion(DistributedHerder.java:1187) at org.apache.kafka.connect.runtime.distributed.DistributedHerder$19.onCompletion(DistributedHerder.java:1183) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:273) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:219) ``` This patch adds a callback which just logs the error. Reviewers: Robert Yokota <rayokota@gmail.com>, Jason Gustafson <jason@confluent.io> (cherry picked from commit 4fdfe2b)

nachomdo changed the title ~~WorkerConfigTransformer NPE on connector configuration reloading~~ KAFKA-8591: WorkerConfigTransformer NPE on connector configuration reloading Jun 24, 2019

nachomdo force-pushed the config-provider-bug branch 2 times, most recently from 103f217 to 625a720 Compare July 1, 2019 16:18

rayokota reviewed Jul 2, 2019

View reviewed changes

nachomdo added 2 commits July 3, 2019 15:30

Fix NPE by passing a callback instead of null

8a5ed98

Replace FutureCallback and log error restarting the connector as per …

aee5672

…PR feedback

nachomdo force-pushed the config-provider-bug branch from 625a720 to aee5672 Compare July 3, 2019 14:30

rayokota approved these changes Jul 3, 2019

View reviewed changes

hachikuji reviewed Jul 3, 2019

View reviewed changes

Check for notNull in tests and fix extra space in arguments

19865b4

hachikuji approved these changes Jul 9, 2019

View reviewed changes

hachikuji merged commit 289ac09 into apache:trunk Jul 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-8591: WorkerConfigTransformer NPE on connector configuration reloading#6991

KAFKA-8591: WorkerConfigTransformer NPE on connector configuration reloading#6991
hachikuji merged 3 commits intoapache:trunkfrom
nachomdo:config-provider-bug

nachomdo commented Jun 24, 2019

Uh oh!

rayokota Jul 2, 2019 •

edited

Loading

Uh oh!

rayokota left a comment

Uh oh!

nachomdo commented Jul 3, 2019

Uh oh!

rayokota left a comment

Uh oh!

hachikuji left a comment

Uh oh!

hachikuji Jul 3, 2019

Uh oh!

hachikuji Jul 3, 2019

Uh oh!

hachikuji commented Jul 8, 2019

Uh oh!

hachikuji left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nachomdo commented Jun 24, 2019

Committer Checklist (excluded from commit message)

Uh oh!

rayokota Jul 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rayokota left a comment

Choose a reason for hiding this comment

Uh oh!

nachomdo commented Jul 3, 2019

Uh oh!

rayokota left a comment

Choose a reason for hiding this comment

Uh oh!

hachikuji left a comment

Choose a reason for hiding this comment

Uh oh!

hachikuji Jul 3, 2019

Choose a reason for hiding this comment

Uh oh!

hachikuji Jul 3, 2019

Choose a reason for hiding this comment

Uh oh!

hachikuji commented Jul 8, 2019

Uh oh!

hachikuji left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rayokota Jul 2, 2019 •

edited

Loading