Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoff information

[nishantmonu51]

• fixes Peons put high load on zookeeper on disconnects #1970
• extracted out segment handoff callbacks in SegmentHandoffNotifier
  which is responsible for tracking segment handoffs and doing callbacks
  when handoff is complete.
• Coordinator now maintains a timeline for the segments which makes it faster to provide serverView for an interval
• Added new end point to DatasourcesResource which exposes information about where nodes segments are loaded based on interval.
• realtime index task and realtime nodes now use HTTP end points exposed by the
  coordinator to get serverView

Below image shows the improvements by above changes on zookeeper read load with nearly 500 realtime index tasks running -

[xvrl]

allowIncomplete would be clearer I think. Maybe add some javadoc as well while we're at it.

[cheddar]

Can we instead expose the method

public Iterable<TimelineObjectHolder<VersionType, ObjectType>> lookupWithIncomplete(Interval interval);

Boolean flag style method often result in pretty confusing function signatures, because you are not always sure of what the boolean is supposed to mean. Especially if you don't actually have the code. If we can instead move those semantics into a completely different method name, it becomes easier for users to know what is going on.

Also, 👍 on adding javadoc.

[nishantmonu51]

fixed.

[xvrl]

👍 looks good to me.

If we change the way coordinator discovery is done, we should also change the way overlord discovery in RemoteTaskActionClient is done, since they are effectively the same. In order to avoid scope creep of this PR, I would be in favor of doing this change in a separate one. I'll file an issue for this.

[gianm]

Even if this is "druid/coordinator" by default, it should be set to "coordinator" in the example common.runtime.properties, since the example coordinator properties set the druid.service to "coordinator". Many people start off their clusters by copying the example configs.

I wonder if we should also just change the defaults in the code to match those…

[xvrl]

I think the configs were created before we had defaults. I'd rather just change the example configs to match the defaults, or leave it out of the example configs entirely, since it's not necessary.

[gianm]

The default overlord service name in tranquility is "overlord" to match the examples, so if we do change the examples we should change that too. All three of those things should match though (druid defaults, tranquility defaults, druid example configs)

[nishantmonu51]

#2046 to track

[nishantmonu51]

#2048

[nishantmonu51]

bouncing for travis.

[nishantmonu51]

Also added graph showing the improvement on zookeeper load with ~500 realtime index tasks running in PR description.

[gianm]

I think this patch makes the rolling update process not work, since it suggests doing coordinators after indexing nodes, but we'd need the coordinator endpoint up first.

[gianm]

added a "release notes" label

[gianm]

We also need to add to the release notes that you need to make sure your druid.selectors.coordinator.serviceName is set properly. Otherwise realtime indexing will stop working after updating because the handoff notifier won't be linked up.

[xvrl]

@gianm oddly this PR is not even mentioned at all in the release notes

[gianm]

Probably it was added after the first RC and missed. We should double check that all the stuff after that actually made it into the notes.

[gianm]

@xvrl will take a look now

[gianm]

@xvrl updated the release notes & added a few other missing things

[xvrl]

@gianm thx I was updating the release notes but you beat me to it.

[rasahner]

I just realized that the new endpoint isn't documented at http://druid.io/docs/latest/design/coordinator.html. Should it be? Or, are there some types of endpoints that are deliberately not documented at druid.io/docs and this is one of them?

[fjy]

@rasahner This should be documented. Can you submit a PR?

[rasahner]

see #2238

[fjy]

@nishantmonu51 @gianm @pjain1 @xvrl @cheddar

None of the examples were updated to reflect that serviceName now talks to coordinator and realtimes won't be able to do handoff.

[fjy]

Also, does anything that requires the coordinator to talk to the overlord still work? Kill tasks, merge tasks, etc?

Edit: yes they will, I thought we'd removed indexing.serviceName for a second there