Skip to content

[WIP] segment management using HTTP Endpoints#2368

Closed
nishantmonu51 wants to merge 4 commits intoapache:masterfrom
metamx:http-segment-assignment
Closed

[WIP] segment management using HTTP Endpoints#2368
nishantmonu51 wants to merge 4 commits intoapache:masterfrom
metamx:http-segment-assignment

Conversation

@nishantmonu51
Copy link
Copy Markdown
Member

Changes for Proposal - #2314
This PR abstracts out zookeeper interactions in the historical and coordinator for segment management.
Changes in Historical Nodes -

  1. Extracts segment management from ZKCoordinator into SegmentManager, ZKCoordinator now just listens to the SegmentChangeRequests from Zookeeper and uses SegmentManager to handle those requests
  2. Adds new HTTP End points in Historical Resource which uses SegmentManager to process SegmentChangeRequests

Changes in Coordinator -

  1. Abstract out LoadQueuePeon which handles load/unload queue and priority for load/unload.
  2. ZKLoadQueuePeon uses zookeeper to process SegmentChangeRequests
  3. HttpLoadQueuePeon uses the new http end-points for segment management
  4. Introduce new config "druid.coordinator.load.peon.type" which can be set to "zk" or "http" (default is http)

TODO:

  1. Add unit tests for new endpoints and HttpLoadQueuePeon
  2. Add docs

@nishantmonu51
Copy link
Copy Markdown
Member Author

@guobingkun @drcrallen: Since you are working on similar area, Can you guys do the initial review and provide feedback ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest being more descriptive like startStopLock

@drcrallen
Copy link
Copy Markdown
Contributor

A lot of places seem to be implementing ListenableFutures by hand. Is it reasonable to change the callback stuff to be ListenableFutrue callbacks?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parameters should be flipped i think

@himanshug
Copy link
Copy Markdown
Contributor

@nishantmonu51 are you still planning to work on this? if not, then I can take it up. thanks.

@himanshug
Copy link
Copy Markdown
Contributor

@nishantmonu51 ping!

@himanshug
Copy link
Copy Markdown
Contributor

@nishantmonu51 I'm closing this PR and will submit another to support optional segment assignment to http.

@himanshug himanshug closed this Sep 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants