Add feature to automatically remove rules based on retention period#11164
Add feature to automatically remove rules based on retention period#11164maytasm merged 3 commits intoapache:masterfrom
Conversation
|
LGTM |
| * @param timestamp timestamp in milliseconds | ||
| * @return number of rules removed | ||
| */ | ||
| int removeRulesOlderThan(long timestamp); |
There was a problem hiding this comment.
Maybe removeRulesForEmptyDatasourcesOlderThan is better
| handle -> { | ||
| Update sql = handle.createStatement( | ||
| StringUtils.format( | ||
| "DELETE FROM %1$s WHERE datasource NOT IN (SELECT DISTINCT datasource from %2$s) and datasource!=:default_rule and version < :date_time", |
There was a problem hiding this comment.
After this change, the version must be the timestamp of when the rule entry was added. It means this code must remain the same as it is unless we change this query together. Please add comments on both codes so that we won't forget it.
There was a problem hiding this comment.
Also, this query could be expensive when the segments table is large. I think maybe it's better to implement the join in this class. For example, this method can iterate all datasources in the rules table and check whether there is any entry in the segments table for each datasource. Then, you can use a much cheaper query for the existence check such as SELECT datasource FROM segments where datasource = ${rule_datasource} LIMIT 1. Maybe this approach needs to add a new index on datasource for the segments table.
There was a problem hiding this comment.
Added comment on version as suggested.
There was a problem hiding this comment.
Regarding the query "DELETE FROM %1$s WHERE datasource NOT IN (SELECT DISTINCT datasource from %2$s) and datasource!=:default_rule and version < :date_time", I think since it’s run very infrequent (by default once a day) and the inner query on segment table is a READ (no locking), it should be ok. Also, adding an index on datasource column in the segment table (for the alternative approach suggested) puts additional overhead on the database operation which is probably not worth it for a query that is run once a day. I think it is fine to keep it as is. I also added a comment in the code regarding this.
| handle -> { | ||
| Update sql = handle.createStatement( | ||
| StringUtils.format( | ||
| "DELETE FROM %1$s WHERE datasource NOT IN (SELECT DISTINCT datasource from %2$s) and datasource!=:default_rule and version < :date_time", |
| |`druid.coordinator.kill.audit.durationToRetain`| Duration of audit logs to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.audit.on` is set to True.| Yes if `druid.coordinator.kill.audit.on` is set to True| None| | ||
| |`druid.coordinator.kill.rule.on`| Boolean value for whether to enable automatic deletion of rules. If set to true, Coordinator will periodically remove rules of inactive datasource (datasource with no used and unused segments) from the rule table in metadata storage.| No | False| | ||
| |`druid.coordinator.kill.rule.period`| How often to do automatic deletion of rules in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.rule.on` is set to True.| No| `P1D`| | ||
| |`druid.coordinator.kill.rule.durationToRetain`| Duration of rules to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.rule.on` is set to True.| Yes if `druid.coordinator.kill.rule.on` is set to True| None| |
There was a problem hiding this comment.
We should update this to say that this will only work as expected if the version is a timestamp in utc
There was a problem hiding this comment.
Will add in a followup PR
suneet-s
left a comment
suneet-s
left a comment
There was a problem hiding this comment.
Overall LGTM
Is it possible for a user to create a segment with a custom version? What would happen in these cases? Do we need guardrails to prevent users from enabling this feature if there are segments that are not in the UTC format?
It is not possible for a user to specify a custom version. The only way possible would be for a user to connect to the metadata store directly and issue SQL to modify the version field. I don't think we need to consider this as that is not supported for regular operation. |
5 participants
Add feature to automatically remove rules based on retention period
Description
We currently already have tasklog auto cleanup (#3677) and audit logs auto cleanup (#11084). This PR adds a similar auto cleanup based on duration (time to retained) but for the rules table to auto clean up rules of inactive datasource (datasource with no used and unused segments)
This is useful when Druid user has a high churn of task / datasource in a short amount of time causing the metadata store size to grow uncontrollably.
This PR has: