realtime nodes in a weird state

[pdeva]

Not sure if bug or my understanding is incorrect.

Screenshot:

3 historicals
2 realtime
replication factor 2

1. The 07:00 segment is already on both the historicals. Then why is it also on both the realtime?
2. Why is 06:00 segment still on realtime when its already on historical?

[nishantmonu51]

The realtime nodes periodically polls (1 minute interval by default) the coordinator for segment handoff and when it completes, it should drop the segment, there is also a possibility that the realtime node is running some long running query which may also be preventing the segment from being dropped.
was the realtime node eventually able to drop the segment eventually or the realtime node kept it loaded forever ? would you be able to share complete logs for coordinator and realtime node for this duration ?

[pdeva]

ok logs from when there is a similar state but at 16:00
Screenshot:
http://screencast.com/t/dmZjNvNN

Seems like its timing out connecting to coordinator.
Nothing bad in coordinator logs.

not sure if #2968 is related to this or not.

2016-05-28 17:40:07,237 ERROR i.d.s.r.p.CoordinatorBasedSegmentHandoffNotifier [coordinator_handoff_scheduled_0] Exception while checking handoff for dataSource[dripstat] Segment[SegmentDescriptor{interval=2016-05-28T16:00:00.000Z/2016-05-28T17:00:00.000Z, version='2016-05-28T16:00:00.000Z', partitionNumber=0}], Will try again after [60000]secs
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.jboss.netty.handler.timeout.ReadTimeoutException
    at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
    at io.druid.client.coordinator.CoordinatorClient.fetchServerView(CoordinatorClient.java:98) ~[druid-server-0.9.0.jar:0.9.0]
    at io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier.checkForSegmentHandoffs(CoordinatorBasedSegmentHandoffNotifier.java:101) [druid-server-0.9.0.jar:0.9.0]
    at io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier$1.run(CoordinatorBasedSegmentHandoffNotifier.java:86) [druid-server-0.9.0.jar:0.9.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_77]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_77]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_77]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.util.concurrent.ExecutionException: org.jboss.netty.handler.timeout.ReadTimeoutException
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
    at io.druid.client.coordinator.CoordinatorClient.fetchServerView(CoordinatorClient.java:82) ~[druid-server-0.9.0.jar:0.9.0]
    ... 9 more
Caused by: org.jboss.netty.handler.timeout.ReadTimeoutException
    at org.jboss.netty.handler.timeout.ReadTimeoutHandler.<clinit>(ReadTimeoutHandler.java:84) ~[netty-3.10.4.Final.jar:?]
    at com.metamx.http.client.NettyHttpClient.go(NettyHttpClient.java:176) ~[http-client-1.0.4.jar:?]
    at com.metamx.http.client.AbstractHttpClient.go(AbstractHttpClient.java:14) ~[http-client-1.0.4.jar:?]
    at io.druid.client.coordinator.CoordinatorClient.fetchServerView(CoordinatorClient.java:68) ~[druid-server-0.9.0.jar:0.9.0]
    ... 9 more

2016-05-28 17:25:07,143 ERROR i.d.s.r.p.CoordinatorBasedSegmentHandoffNotifier [coordinator_handoff_scheduled_0] Exception while checking handoff for dataSource[dripstat] Segment[SegmentDescriptor{interval=2016-05-28T16:00:00.000Z/2016-05-28T17:00:00.000Z, version='2016-05-28T16:00:00.000Z', partitionNumber=0}], Will try again after [60000]secs
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.jboss.netty.handler.timeout.ReadTimeoutException
    at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
    at io.druid.client.coordinator.CoordinatorClient.fetchServerView(CoordinatorClient.java:98) ~[druid-server-0.9.0.jar:0.9.0]
    at io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier.checkForSegmentHandoffs(CoordinatorBasedSegmentHandoffNotifier.java:101) [druid-server-0.9.0.jar:0.9.0]
    at io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier$1.run(CoordinatorBasedSegmentHandoffNotifier.java:86) [druid-server-0.9.0.jar:0.9.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_77]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_77]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_77]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.util.concurrent.ExecutionException: org.jboss.netty.handler.timeout.ReadTimeoutException
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
    at io.druid.client.coordinator.CoordinatorClient.fetchServerView(CoordinatorClient.java:82) ~[druid-server-0.9.0.jar:0.9.0]
    ... 9 more
Caused by: org.jboss.netty.handler.timeout.ReadTimeoutException
    at org.jboss.netty.handler.timeout.ReadTimeoutHandler.<clinit>(ReadTimeoutHandler.java:84) ~[netty-3.10.4.Final.jar:?]
    at com.metamx.http.client.NettyHttpClient.go(NettyHttpClient.java:176) ~[http-client-1.0.4.jar:?]
    at com.metamx.http.client.AbstractHttpClient.go(AbstractHttpClient.java:14) ~[http-client-1.0.4.jar:?]
    at io.druid.client.coordinator.CoordinatorClient.fetchServerView(CoordinatorClient.java:68) ~[druid-server-0.9.0.jar:0.9.0]

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.