Skip to content

KIS task keeps failing #7098

@pdeva

Description

@pdeva

Druid 0.13.0

i see this in the task status:

{"task":"index_kafka_pctile-minute_7b5f9c031810364_hbnefjko","status":{"id":"index_kafka_pctile-minute_7b5f9c031810364_hbnefjko","type":"index_kafka","createdTime":"2019-02-19T12:00:15.653Z","queueInsertionTime":"1970-01-01T00:00:00.000Z","statusCode":"FAILED","status":"FAILED","runnerStatusCode":"WAITING","duration":28811224,"location":{"host":null,"port":-1,"tlsPort":-1},"dataSource":"pctile-minute","errorMsg":"java.util.concurrent.ExecutionException: org.apache.druid.java.util.common.UOE: [org.apache.druid.qu..."}}

in the MM logs i see a lot of this:

2019-02-19 20:00:41,198 ERROR o.a.d.i.k.IncrementalPublishingKafkaIndexTaskRunner [task-runner-0-priority-0] Encountered exception while running task.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: No such file or directory
	at org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:232) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.runInternal(IncrementalPublishingKafkaIndexTaskRunner.java:626) ~[druid-kafka-indexing-service-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.run(IncrementalPublishingKafkaIndexTaskRunner.java:232) [druid-kafka-indexing-service-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:210) [druid-kafka-indexing-service-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:421) [druid-indexing-service-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:393) [druid-indexing-service-0.13.0-incubating.jar:0.13.0-incubating]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: No such file or directory
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_181]
	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_181]
	at org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:224) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
	... 9 more
Caused by: java.lang.RuntimeException: java.io.IOException: No such file or directory
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.persistHydrant(AppenderatorImpl.java:1287) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.access$100(AppenderatorImpl.java:109) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl$2.doCall(AppenderatorImpl.java:526) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.common.guava.ThreadRenamingCallable.call(ThreadRenamingCallable.java:44) ~[druid-common-0.13.0-incubating.jar:0.13.0-incubating]
	... 4 more
Caused by: java.io.IOException: No such file or directory
	at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:1.8.0_181]
	at java.io.File.createTempFile(File.java:2024) ~[?:1.8.0_181]
	at org.apache.druid.segment.writeout.TmpFileSegmentWriteOutMedium.makeWriteOutBytes(TmpFileSegmentWriteOutMedium.java:46) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.data.GenericIndexedWriter.open(GenericIndexedWriter.java:202) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.StringDimensionMergerV9.writeMergedValueDictionary(StringDimensionMergerV9.java:171) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.IndexMergerV9.writeDimValuesAndSetupDimConversion(IndexMergerV9.java:697) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:174) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:914) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.IndexMergerV9.persist(IndexMergerV9.java:779) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.IndexMergerV9.persist(IndexMergerV9.java:748) ~[druid-processing-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.persistHydrant(AppenderatorImpl.java:1265) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.access$100(AppenderatorImpl.java:109) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl$2.doCall(AppenderatorImpl.java:526) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
	at org.apache.druid.common.guava.ThreadRenamingCallable.call(ThreadRenamingCallable.java:44) ~[druid-common-0.13.0-incubating.jar:0.13.0-incubating]
	... 4 more

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions