Ability to blacklist historical node

[akashdw]

We had an issue where some historical nodes(with root partition filled up with the request logs) were not able to download segments and coordinator was keep assigning new segments to those historical nodes.
It would be nice if coordinator can blacklist historical nodes.

Blacklisting could be based on segment download status. Currently historical nodes don't pass segment load success/fail information to coordinator, in both cases historical deletes the loadQueue for the segment. Passing segment load status to coordinator will be useful to figure out if the problem is with the segment itself(say broken deep storage path) or with the historical node (disk issues), if it is with historical then blacklist historical node.

Haven't figured out the full details of the above approach and open for suggestions. what do you think @gianm @cheddar @himanshug @leventov ?

[niketh]

This might become easier after @himanshug work of replacing ZK based coordination with HTTP. After that PR goes through, the Coordinator can easily keep a count of failures from each historical and use that information for more "intelligent" processing

[himanshug]

i think @niketh is right in saying that "segment assignment migration to HTTP" ( partially done in #2368 ) will fix the problem that load success/fail status would get propagated to coordinator and coordinator would be able to act appropriately.

that said, once a historical is blacklisted, will user need to take some action to "un-blacklist" it or will it be something of a temporary blacklisting ?

[gianm]

that said, once a historical is blacklisted, will user need to take some action to "un-blacklist" it or will it be something of a temporary blacklisting ?

I think ideally, there should be some backoff. So let's say a node is blacklisted when it has some problems, and then after some time has passed, we'll let it load a few segments (applying a throttle) as a test. If it can do those, then take off the throttle and return it to full service. If it can't do those, then return it to a full blacklist for another while.

[himanshug]

yeah, i would prefer temporary blacklisting aka backoff instead of permanently marking a node blacklisted.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.