Skip to content

Kafka indexing task can fail during rolling update #8837

@jihoonson

Description

@jihoonson

Affected Version

When updating 0.15 or earlier -> 0.16

Description

#7319 has removed an unused parameter, but it can cause task failures during rolling update with the following error.

2019-11-07T00:23:20,740 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.actions.RemoteTaskActionClient - Exception submitting action for task[index_kafka_clarity-cloud0_7c42ed11c91cf72_ghfhppbk]
org.apache.druid.java.util.common.IOE: Error with status[400 Bad Request] and message[{"error":"Instantiation of [simple type, class org.apache.druid.indexing.common.actions.CheckPointDataSourceMetadataAction] value failed: currentCheckPoint (through reference chain: org.apache.druid.indexing.common.actions.TaskActionHolder[\"action\"])"}]. Check overlord logs for details.
	at org.apache.druid.indexing.common.actions.RemoteTaskActionClient.submit(RemoteTaskActionClient.java:94) [druid-indexing-service-0.16.0-incubating-iap3.jar:0.16.0-incubating-iap3]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:736) [druid-indexing-service-0.16.0-incubating-iap3.jar:0.16.0-incubating-iap3]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:259) [druid-indexing-service-0.16.0-incubating-iap3.jar:0.16.0-incubating-iap3]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:177) [druid-indexing-service-0.16.0-incubating-iap3.jar:0.16.0-incubating-iap3]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.16.0-incubating-iap3.jar:0.16.0-incubating-iap3]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.16.0-incubating-iap3.jar:0.16.0-incubating-iap3]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

To fix this issue, CheckPointDataSourceMetadataAction can have a method as below to add the missing property in its serialized JSON.

  // For backwards compatibility
  @Deprecated
  @JsonProperty
  public SeekableStreamDataSourceMetadata getCurrentCheckPoint()
  {
    return checkpointMetadata;
  }

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions