Add feature to automatically remove supervisor based on retention period by maytasm · Pull Request #11200 · apache/druid

maytasm · 2021-05-05T01:40:11Z

Add feature to automatically remove supervisors based on retention period

Description

We currently already have tasklog auto cleanup (#3677) and audit logs auto cleanup (#11084). This PR adds a similar auto cleanup based on duration (time to retained) but for the supervisor table to auto clean up terminated supervisors

This is useful when Druid user has a high churn of task / datasource in a short amount of time causing the metadata store size to grow uncontrollably.

This PR has:

been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

loquisgon · 2021-05-05T23:57:30Z

 |`druid.coordinator.period.metadataStoreManagementPeriod`|How often to run metadata management tasks in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. |No | `PT1H`|
+|`druid.coordinator.kill.supervisor.on`| Boolean value for whether to enable automatic deletion of terminated supervisors. If set to true, Coordinator will periodically remove terminated supervisors from the supervisor table in metadata storage.| No | False| 
+|`druid.coordinator.kill.supervisor.period`| How often to do automatic deletion of terminated supervisor in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.| No| `P1D`|
+|`druid.coordinator.kill.supervisor.durationToRetain`| Duration of terminated supervisor to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.| Yes if `druid.coordinator.kill.supervisor.on` is set to True| None|


The code has 1S for durationToRetain default: PT-1s

The default in the code is used to fail cluster startup if the value is not set by the user. Basically code check for the value being positive (meaning that the user must set it and override the default value). For documentation simplicity sake, the default value in the code is the same effect as None and that the user must set the value.

abhishekagarwal87 · 2021-05-06T10:02:02Z

 |--------|-----------|---------|-------|
 |`druid.coordinator.period.metadataStoreManagementPeriod`|How often to run metadata management tasks in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. |No | `PT1H`|
+|`druid.coordinator.kill.supervisor.on`| Boolean value for whether to enable automatic deletion of terminated supervisors. If set to true, Coordinator will periodically remove terminated supervisors from the supervisor table in metadata storage.| No | False| 
+|`druid.coordinator.kill.supervisor.period`| How often to do automatic deletion of terminated supervisor in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.| No| `P1D`|


Suggested change

|`druid.coordinator.kill.supervisor.period`| How often to do automatic deletion of terminated supervisor in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.| No| `P1D`|

|`druid.coordinator.kill.supervisor.period`| How often to do automatic deletion of terminated supervisor in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be equal to or greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.supervisor.on` is set to "True".| No| `P1D`|

abhishekagarwal87 · 2021-05-06T10:03:31Z

 |`druid.coordinator.period.metadataStoreManagementPeriod`|How often to run metadata management tasks in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. |No | `PT1H`|
+|`druid.coordinator.kill.supervisor.on`| Boolean value for whether to enable automatic deletion of terminated supervisors. If set to true, Coordinator will periodically remove terminated supervisors from the supervisor table in metadata storage.| No | False| 
+|`druid.coordinator.kill.supervisor.period`| How often to do automatic deletion of terminated supervisor in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.| No| `P1D`|
+|`druid.coordinator.kill.supervisor.durationToRetain`| Duration of terminated supervisor to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.| Yes if `druid.coordinator.kill.supervisor.on` is set to True| None|


Suggested change

|`druid.coordinator.kill.supervisor.durationToRetain`| Duration of terminated supervisor to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.| Yes if `druid.coordinator.kill.supervisor.on` is set to True| None|

|`druid.coordinator.kill.supervisor.durationToRetain`| Duration of terminated supervisor to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.supervisor.on` is set to "True".| Yes if `druid.coordinator.kill.supervisor.on` is set to "True"| None|

abhishekagarwal87 · 2021-05-06T10:06:33Z

 |`interval/skipCompact/count`|Total number of intervals of this datasource that are skipped (not eligible for auto compaction) by the auto compaction.|datasource.|Varies.|
 |`coordinator/time`|Approximate Coordinator duty runtime in milliseconds. The duty dimension is the string alias of the Duty that is being run.|duty.|Varies.|
 |`coordinator/global/time`|Approximate runtime of a full coordination cycle in milliseconds. The `dutyGroup` dimension indicates what type of coordination this run was. i.e. Historical Management vs Indexing|`dutyGroup`|Varies.|
+|`metadata/kill/supervisor/count`|Total number of terminated supervisors automatically deleted from metadata store supervisor table per each Coordinator kill supervisor duty run. This metric can help adjust `druid.coordinator.kill.supervisor.durationToRetain` configuration based on if more or less terminated supervisors need to be deleted per cycle. Note that this metric is only emitted when `druid.coordinator.kill.supervisor.on` is set to true.| |Varies.|


Suggested change

|`metadata/kill/supervisor/count`|Total number of terminated supervisors automatically deleted from metadata store supervisor table per each Coordinator kill supervisor duty run. This metric can help adjust `druid.coordinator.kill.supervisor.durationToRetain` configuration based on if more or less terminated supervisors need to be deleted per cycle. Note that this metric is only emitted when `druid.coordinator.kill.supervisor.on` is set to true.| |Varies.|

|`metadata/kill/supervisor/count`|Total number of terminated supervisors that were automatically deleted from metadata store per each Coordinator kill supervisor duty run. This metric can help adjust `druid.coordinator.kill.supervisor.durationToRetain` configuration based on whether more or less terminated supervisors need to be deleted per cycle. Note that this metric is only emitted when `druid.coordinator.kill.supervisor.on` is set to true.| |Varies.|

abhishekagarwal87 · 2021-05-06T10:07:37Z

  Map<String, SupervisorSpec> getLatest();
+
+  /**
+   * Remove terminated supervisor created older than the given timestamp.


Suggested change

* Remove terminated supervisor created older than the given timestamp.

* Remove terminated supervisors created before the given timestamp.

abhishekagarwal87 · 2021-05-06T10:09:19Z

+    Map<String, SupervisorSpec> supervisors = getLatest();
+    for (Map.Entry<String, SupervisorSpec> supervisor : supervisors.entrySet()) {
+      final SupervisorSpec spec = supervisor.getValue();
+      if (spec instanceof NoopSupervisorSpec) {


does this mean that it is a terminated supervisor? can you add a comment clarifying this bit

Yep. Terminated supervisor will have it's latest supervisorSpec as NoopSupervisorSpec. (Basically NoopSupervisorSpec is used as a tombstone indicator)

abhishekagarwal87 · 2021-05-06T10:10:21Z

+            handle -> {
+              Update sql = handle.createStatement(
+                  StringUtils.format(
+                      "DELETE FROM %1$s WHERE spec_id = :spec_id AND created_date < :date_time",


does it make sense to delete in batch than doing one by one which will be slow?

abhishekagarwal87 · 2021-05-06T10:15:34Z

+import org.apache.druid.server.coordinator.DruidCoordinatorConfig;
+import org.apache.druid.server.coordinator.DruidCoordinatorRuntimeParams;
+
+public class KillSupervisors implements CoordinatorDuty


abhishekagarwal87

LGTM @maytasm. Does this PR need a "release notes" label?

add auto clean up

3007e27

clintropolis added the Area - Operations label May 5, 2021

maytasm added 4 commits May 4, 2021 23:43

add test

1e4f0b5

fix conflict

782efd1

add test

42e16ec

fix test

9582b9b

loquisgon reviewed May 5, 2021

View reviewed changes

abhishekagarwal87 reviewed May 6, 2021

View reviewed changes

maytasm added 2 commits May 6, 2021 17:51

Address comments

2710017

Address comments

e5249da

abhishekagarwal87 approved these changes May 7, 2021

View reviewed changes

maytasm added the Release Notes label May 7, 2021

maytasm merged commit d73f72e into apache:master May 7, 2021

maytasm deleted the IMPLY-2998 branch May 7, 2021 05:25

techdocsmith mentioned this pull request May 12, 2021

add docs for high-churn datasource cleanup #11245

Merged

1 task

maytasm mentioned this pull request May 14, 2021

Auto cleanup of metadata tables to enable high frequency creation/deletion of data sources #11173

Closed

clintropolis added this to the 0.22.0 milestone Aug 12, 2021

clintropolis mentioned this pull request Sep 3, 2021

[Draft] 0.22.0 Release Notes #11657

Closed

suneet-s mentioned this pull request Jan 21, 2022

Enable automatic metadata cleanup by default #12188

Merged

4 tasks

	\|`druid.coordinator.kill.supervisor.period`\| How often to do automatic deletion of terminated supervisor in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.\| No\| `P1D`\|
	\|`druid.coordinator.kill.supervisor.period`\| How often to do automatic deletion of terminated supervisor in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be equal to or greater than `druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if `druid.coordinator.kill.supervisor.on` is set to "True".\| No\| `P1D`\|

	\|`druid.coordinator.kill.supervisor.durationToRetain`\| Duration of terminated supervisor to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.supervisor.on` is set to True.\| Yes if `druid.coordinator.kill.supervisor.on` is set to True\| None\|
	\|`druid.coordinator.kill.supervisor.durationToRetain`\| Duration of terminated supervisor to be retained from created time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if `druid.coordinator.kill.supervisor.on` is set to "True".\| Yes if `druid.coordinator.kill.supervisor.on` is set to "True"\| None\|

	* Remove terminated supervisor created older than the given timestamp.
	* Remove terminated supervisors created before the given timestamp.

Conversation

maytasm commented May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

loquisgon May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhishekagarwal87 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

maytasm commented May 5, 2021 •

edited

Loading

loquisgon May 5, 2021 •

edited

Loading