Add new configuration for kill task to skip segments that has been last modified within the config value time

[maytasm]

Description

Introduce a new configuration for the kill task with the key druid.coordinator.kill.fromLastModified. The value will be time period. The kill task will then skip any segments that has been last modified within the configured value.

Motivation

Currently we only have druid.coordinator.kill.durationToRetain which looks at the "end" time of the segments. However, this does not prevent accident / misconfigured segments to be kill. For example, a misconfigured drop rule accidentally drop segments from 6M to 1Year ago period. Most likely, these segments would be way pass druid.coordinator.kill.durationToRetain and be kill right away in the next kill cycle. Even worst would be if the kill cycle is very frequent, then it might be too late to catch the misconfigured rules and the segments would be kill (unable to recover). Obviously, the root cause is the human operator but that can and sometime do happens. This features aims to make mistake less costly if they do happen.

Want to see if anyone has any experience / recommendation and if something like this would be beneficial to others.

[yuanlihan]

Hi @maytasm, I like this druid.coordinator.kill.fromLastModified idea while I have a few confusion about it.

Does this druid.coordinator.kill.fromLastModified configuration works independently? For example, only one configuration will take effect, druid.coordinator.kill.fromLastModified or druid.coordinator.kill.durationToRetain.
Or these two configurations could be applied at the same time?

[maytasm]

@yuanlihan
I am thinking that the segment will have to satisfy both conditions to be eligible for the kill task. The new druid.coordinator.kill.fromLastModified offers additional safety to prevent accidental killing of segments. If the user does not want the druid.coordinator.kill.fromLastModified and want the existing behavior then they can simply set druid.coordinator.kill.fromLastModified to 0. Basically, this ignores druid.coordinator.kill.fromLastModified and only requires segment to satisfy druid.coordinator.kill.durationToRetain.
The benefit of druid.coordinator.kill.fromLastModified is for old segments that might have been dropped accidentally from incorrectly configured rules (usually rules are set for intervals of many months). For example, you might have a dropForever rule before a load rule (instead of the other way around). Those segments may be older than druid.coordinator.kill.durationToRetain but will not be kill due to druid.coordinator.kill.fromLastModified safety net.

[yuanlihan]

@maytasm thanks for the explanation.
It makes sense to me. This configuration will help a lot to reduce the impact of misoperation.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[maytasm]

Closing this as this is done in #12526 / #12599