Query fails because of potential bug in HTTPPostEmitter

[niketh]

Queries fail will the following stacktrace @leventov

2018-03-13T23:05:59,740 ERROR [groupBy_xxxx_[2018-03-08T00:00:00.000Z/2018-03-08T08:00:00.000Z]] com.metamx.emitter.core.HttpPostEmitter - Serious error during onSealExclusive(), set currentBatch to the failed Batch.batchNumber
java.lang.IllegalStateException: Attempted arrival of unregistered party for java.util.concurrent.Phaser@18dcd7c9[phase = 34658 parties = 1 arrived = 1]
        at java.util.concurrent.Phaser.doArrive(Phaser.java:391) ~[?:1.8.0_112]
        at java.util.concurrent.Phaser.arrive(Phaser.java:634) ~[?:1.8.0_112]
        at com.metamx.emitter.core.EmittedBatchCounter.batchEmitted(EmittedBatchCounter.java:40) ~[java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.HttpPostEmitter.batchFinalized(HttpPostEmitter.java:383) ~[java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.HttpPostEmitter.limitBuffersToEmitSize(HttpPostEmitter.java:368) ~[java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.HttpPostEmitter.addBatchToEmitQueue(HttpPostEmitter.java:357) ~[java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.HttpPostEmitter.doOnSealExclusive(HttpPostEmitter.java:328) ~[java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.HttpPostEmitter.onSealExclusive(HttpPostEmitter.java:304) [java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.Batch.tryReleaseShared(Batch.java:278) [java-util-1.3.4-SNAPSHOT.jar:?]
        at java.util.concurrent.locks.AbstractQueuedLongSynchronizer.releaseShared(AbstractQueuedLongSynchronizer.java:1119) [?:1.8.0_112]
        at com.metamx.emitter.core.Batch.unlockAndSeal(Batch.java:244) [java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.Batch.unlockAndSealIfNeeded(Batch.java:213) [java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.Batch.tryAddNonFirstEvent(Batch.java:181) [java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.Batch.tryAddEvent(Batch.java:132) [java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.HttpPostEmitter.emitAndReturnBatch(HttpPostEmitter.java:261) [java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.core.HttpPostEmitter.emit(HttpPostEmitter.java:226) [java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:72) [java-util-1.3.4-SNAPSHOT.jar:?]
        at com.metamx.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:77) [java-util-1.3.4-SNAPSHOT.jar:?]
        at io.druid.query.DefaultQueryMetrics.emit(DefaultQueryMetrics.java:290) [druid-processing-0.11.0.jar:0.11.0]
        at io.druid.query.MetricsEmittingQueryRunner$1.after(MetricsEmittingQueryRunner.java:120) [druid-processing-0.11.0.jar:0.11.0]
        at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:66) [java-util-0.11.0.jar:0.11.0]
        at io.druid.query.spec.SpecificSegmentQueryRunner$2.accumulate(SpecificSegmentQueryRunner.java:86) [druid-processing-0.11.0.jar:0.11.0]
        at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) [java-util-0.11.0.jar:0.11.0]
        at io.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:172) [druid-processing-0.11.0.jar:0.11.0]
        at io.druid.query.spec.SpecificSegmentQueryRunner.access$200(SpecificSegmentQueryRunner.java:45) [druid-processing-0.11.0.jar:0.11.0]
        at io.druid.query.spec.SpecificSegmentQueryRunner$3.wrap(SpecificSegmentQueryRunner.java:152) [druid-processing-0.11.0.jar:0.11.0]
        at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) [java-util-0.11.0.jar:0.11.0]
        at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) [java-util-0.11.0.jar:0.11.0]
        at io.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) [druid-processing-0.11.0.jar:0.11.0]
        at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) [java-util-0.11.0.jar:0.11.0]

[niketh]

Created a PR so that emitter failures don't lead to query failures #5484 . Root cause in HTTPEmitter still needs to be identified and fixed

[leventov]

Yes, this is a bug. I will create a PR.

[leventov]

#5488

[leventov]

Also @niketh the fact that you hit HttpPostEmitter.limitBuffersToEmitSize() means that your emitting service doesn't keep up with the rate of emitted events, and they are dropped. You may want to adjust HttpPostEmitter configurations, or look at it's metrics.