[Proposal] Implementing segment management using HTTP Endpoints instead of Zookeeper

[nishantmonu51]

This proposal is intended to do segment management using a REST interface instead of Zookeeper. Historical nodes can expose a REST Interface and possibly announce it to the coordinator using the listening-announcer (proposed in #2286) as a capability to load/unload segments. These HTTP endpoints will be used by the coordinator to manage the segments on a historical node.

Changes Include -

1. Adding a SegmentManagementResource on the historical node which can be used to -

   • Load a segment - queues a request to load a segment.
   • Unload a segment - queues a request to unload a segment.
   • Query the current used, max capacity on the historical node.
   • Query the list of segments served by the historical nodes - all segments, filtered by interval, datasource or both.
   • Query the status of a particular segment on whether its present on the historical or not.

2. Implementing a HTTPLoadQueuePeon on the coordinator which uses the REST Interface to manage segments on the historical nodes.

What this proposal is NOT Intended to do -
This is intended to be a small step in removing dependency from zookeeper and is not meant to change the way segments are Announced.

[drcrallen]

Note that extra metadata like current used and max capacity of a historical node is not in scope for #2286 so this proposal may need to clarify any changes in endpoints needed to accommodate the required metadata information.

[himanshug]

Fixed by #4966