Separate internal node discovery from external service discovery

[xvrl]

Currently we use CuratorServiceAnnouncer / ServerDiscoverySelector in two ways:

1. externally allow external services to Druid to resolve various node types.
2. internally for peons to find out how to talk to overlords in RemoteTaskActionClient, as well as in CoordinatorClient in Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoff information #2015

ServiceAnnouncer was initially meant to make Druid nodes discoverable by outside services. Using it internally requires to configure peons to know the external service name, and also makes it impossible to make ServiceAnnouncer pluggable.

At the same time CuratorServiceAnnouncer is not ideal for external announcement either. We currently share the same CuratorFramework for both CuratorServiceAnnouncer and the rest of Druid, which makes it impossible to use a dedicated ZooKeeper cluster for Druid, while announcing services externally on a different ZooKeeper cluster.

Here are some of the things we should do:

• separate the internal service announcement from the external one.
• standardize a way for Druid nodes to discover other Druid nodes. We currently have both the curator based service discovery as well as several home-grown ways for historical nodes or peons to announce themselves to other nodes. It might make sense to either use curator discovery for all internal announcements or share more code amongst the various announcement methods already implemented.
• separate the CuratorFramework used for ServiceAnnouncer from the one used for internal Druid coordination.

[guobingkun]

@xvrl I am more in favor of Announcer for internal announcements rather than CuratorServiceAnnouncer as its API gives more freedom on announcing stuff(Druid server metadata, leadership info, etc.) and it's already being used by Historical/Realtime to announce segments. This also means there should be an internal DruidServiceClients that discover Druid services announced internally.

That said, CuratorServiceAnnouncer should only be used for external discovery. Regarding announcing on a different Zookeeper cluster, we can annotate CuratorFramework differently for internal and external usages.

[xvrl]

@guobingkun curator is also pretty flexible in the payload that can be announced as part of their service discovery implementation, we don't have to be limited by what is currently in CuratorServiceAnnouncer.

We can still implement a separate DruidServerDiscovery for internal purposes that uses curator, removing the need to use our home-grown announcer code for discovery purposes, which still seems to be failing tests every now and then.

Historical nodes can keep using Announcer for segments, but since segments announcement is one of the first things we'll likely move off of ZK, that code will become less useful overtime.

[guobingkun]

@xvrl I agree with you that Announcer will become less useful overtime, and it's probably not necessary to use it for discovery purpose. If I understand correctly, are you suggesting carry payload in curator ServiceInstance? (currently we set payload to Void)

[xvrl]

@guobingkun yes, for internal discovery we can use the payload to pass information.
We will probably want to have two distinct announcers for internal and external purposes anyway for two reasons:

• we want to allow different ZK connection strings for internal and external use
• we want to allow someone to replace the external announcer with their own implementation that may carry other information in the payload

[cheddar]

Fwiw, I'm more in favor of using the Announcer. Curator service discovery leaves a lot of turds lying around in ZK and has a whole bunch of extra junk in the znodes. Also, the argument about replacing it. Using Curator discovery is likely to make it significantly more difficult to replace this, where using Announcer will keep the surface area of announcement contained and simpler to replace.

Fwiw, I think that replacing the "announcing" of service availability is the simplest of the things to swap off of ZK. After that, we can adjust the publishing logic (segment announcements). So, keeping in mind that we hope to follow this up with an HTTP implementation of the Announcer in the not too distant future, I'd be more in favor of sticking with an interface we have more control over instead of expanding the scope of the removal of ZK to also force us to reimplement the entire curator service discovery interface.

[xvrl]

@cheddar I don't think any implementation is going to be harder to replace than the other. The only interface we'd have to reimplement is whichever abstraction we are building on top of Announcer or Curator discovery to do internal node announcement.

That being said, I don't have a very strong preference over which announcement implementation we use, as long as we offer an easy abstraction to discover nodes within the cluster. Indeed, the reason curator was misused in the first place was due to its simplicity and because existing announcer based discovery protocols were tied to segment announcement or tied to indexing service specifics, making it harder to re-use those pieces for pure node discovery.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.