From 5845ade432b33f5f0b8913369dfc24d9c6204d34 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Wed, 12 May 2021 14:37:31 -0700 Subject: [PATCH 1/8] add docs for high-churn datasource cleanup --- docs/operations/clean-metadata-store.md | 104 ++++++++++++++++++++++++ website/sidebars.json | 3 +- 2 files changed, 106 insertions(+), 1 deletion(-) create mode 100644 docs/operations/clean-metadata-store.md diff --git a/docs/operations/clean-metadata-store.md b/docs/operations/clean-metadata-store.md new file mode 100644 index 000000000000..003051d0f665 --- /dev/null +++ b/docs/operations/clean-metadata-store.md @@ -0,0 +1,104 @@ +--- +id: clean-metadata-store +title: "Automated cleanup for metadata records related to deleted datasources" +sidebar_label: Automated metadata store cleanup +description: "Defines a strategy to maintain Druid metadata store performance by automatically removing leftover records for deleted datasources. Most applicable to databases with 'high-churn' datasources." +--- + + +When you delete a datasource from Apache Druid, some records related to the datasource may remain in the metadata store including: + +- audit records +- supervisor records +- rule records +- compaction configuration records +- datasource records created by supervisors + +If you have a high datasource churn rate, meaning you frequently create and delete many short-lived datasources, the leftover records can start to fill your metadata store and cause performance issues. To maintain metadata store performance in this case, you can configure Apache Druid to automatically remove records associated with deleted datasources from the metadata store. + +## Automated cleanup strategies +There are several cases when you should consider automated cleanup of the metadata related to deleted datasources: +- Proactively, if you know you have many high-churn datasources. For example you have scripts that create and delete supervisors regularly. +- If you have issues with the hard disk for your metadata database filling up. +- If you run into performance issues with the metadata database. For example API calls are very slow or fail to execute. + +Do not use the metadata store automated cleanup features if you have requirements to retain metadata records. For example, you have compliance requirements to keep audit records. In these cases, you should come up with an alternate method to preserve the audit metadata while freeing up your active metadata store. + +## Configure automated metadata cleanup +Automated cleanup only removes records for deleted datasources. You can configure cleanup on a per-table basis as follows: + - `druid.coordinator.kill.*.on` enables cleanup for a particular metadata table. + - `druid.coordinator.kill.*.period` defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible records. + - `druid.coordinator.kill.*.durationToRetain` defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that the entity becomes eligible for deletion. For example to retain records for 30 days from their creation date: `P30D`. + + For full configuration details, see [Metadata management](../configuration/index.md#metadata-management). + +The following are eligible for automatic cleanup: +- `druid_audit`: Audit records for deleted datasources become eligible for deletion when the `durationToRetain` time has passed since their creation. +- `druid_supervisors`: Supervisor records become eligible for deletion when: + + - the supervisor is terminated. + - the `durationToRetain` time has passed since their creation. +- `druid_rules`: Rule records become eligible for deletion when: + + - no segments exist for the datasource. + - the `durationToRetain` time has passed since their creation. +- `compaction`: Compaction configuration records in the records in the `druid_config` table become eligible for deletion when no segments exist for the datasource. + >If you already have an extremely large compaction configuration, you may not be able to delete compaction configuration due to size limits with the audit log. In this case you can set `druid.audit.manager.maxPayloadSizeBytes` and `druid.audit.manager.skipNullField` to avoid the auditing issue. See [Audit logging](../configuration/index.md#audit-logging). +- `druid_datasources`: Datasource records become eligible for deletion when: + + - the supervisor is terminated or does not exist in the `druid_supervisors` table. + - the `durationToRetain` time has passed since their creation. + +## Automated metadata cleanup example configuration +Consider a scenario where you have scripts to create and delete supervisors for hundreds of datasources a day. You do not want to fill your metadata store with records for deleted datasources. Datasources tend to persist for only one or two days. Therefore, you want to run a cleanup job that identifies and removes leftover records that are at least four days old. The exception is for audit logs, which you need to retain for 30 days: + +``` +... +# Poll every day to delete audit records > 30 days old +druid.coordinator.kill.druid_audit.on=True +druid.coordinator.kill.druid_audit.period=P1D +druid.coordinator.kill.druid_audit.durationToRetain=P30D + +# Poll every day to delete supervisor records > 4 days old +druid.coordinator.kill.druid_supervisors.on=True +druid.coordinator.kill.druid_supervisors.period=P1D +druid.coordinator.kill.druid_supervisors.durationToRetain=P4D + +# Poll every day to delete rules records > 4 days old +druid.coordinator.kill.druid_rules.on=True +druid.coordinator.kill.druid_rules.period=P1D +druid.coordinator.kill.druid_rules.durationToRetain=P4D + +# Poll every day to delete compaction configuration records > 4 days old +druid.coordinator.kill.compaction.on=True +druid.coordinator.kill.compaction.period=P1D +druid.coordinator.kill.compaction.durationToRetain=P4D + +# Poll every day to delete datasource records > 4 days old +druid.coordinator.kill.druid_datasources.on=True +druid.coordinator.kill.druid_datasources.period=P1D +druid.coordinator.kill.druid_datasources.durationToRetain=P4D +... +``` + +## Learn more +See the following topics for more information: +- [Metadata management](../configuration/index.md#metadata-management) for metadata store configuration reference. +- [Metadata storage](../dependencies/metadata-storage.md) for an overview of the metadata storage database. diff --git a/website/sidebars.json b/website/sidebars.json index 8505bb2008d3..ba9d359f803f 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -133,7 +133,8 @@ "ids": [ "operations/basic-cluster-tuning", "operations/segment-optimization", - "operations/http-compression" + "operations/http-compression", + "operations/clean-metadata-store" ] }, "operations/api-reference", From 20db2dfe0cf44424e87a5ce10faeacdb355f08b7 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Wed, 12 May 2021 21:26:28 -0700 Subject: [PATCH 2/8] fix most comments except for task log --- docs/operations/clean-metadata-store.md | 96 ++++++++++++++++++------- 1 file changed, 70 insertions(+), 26 deletions(-) diff --git a/docs/operations/clean-metadata-store.md b/docs/operations/clean-metadata-store.md index 003051d0f665..66c686af5404 100644 --- a/docs/operations/clean-metadata-store.md +++ b/docs/operations/clean-metadata-store.md @@ -1,8 +1,8 @@ --- id: clean-metadata-store -title: "Automated cleanup for metadata records related to deleted datasources" -sidebar_label: Automated metadata store cleanup -description: "Defines a strategy to maintain Druid metadata store performance by automatically removing leftover records for deleted datasources. Most applicable to databases with 'high-churn' datasources." +title: "Automated cleanup for metadata records" +sidebar_label: Automated metadata cleanup +description: "Defines a strategy to maintain Druid metadata store performance by automatically removing leftover records for deleted entities: datasources, supervisors, rulles, compaction configuration, audit records, etc. Most applicable to databases with 'high-churn' datasources." --- -When you delete a datasource from Apache Druid, some records related to the datasource may remain in the metadata store including: +When you delete some entities from Apache Druid, records related to the entity may remain in the metadata store including: +- segments records - audit records - supervisor records - rule records - compaction configuration records - datasource records created by supervisors -If you have a high datasource churn rate, meaning you frequently create and delete many short-lived datasources, the leftover records can start to fill your metadata store and cause performance issues. To maintain metadata store performance in this case, you can configure Apache Druid to automatically remove records associated with deleted datasources from the metadata store. +If you have a high datasource churn rate, meaning you frequently create and delete many short-lived datasources or other releated entities like compaction configuration or rules, the leftover records can start to fill your metadata store and cause performance issues. + +To maintain metadata store performance in this case, you can configure Apache Druid to automatically remove records associated with deleted datasources from the metadata store. ## Automated cleanup strategies There are several cases when you should consider automated cleanup of the metadata related to deleted datasources: @@ -42,35 +45,77 @@ There are several cases when you should consider automated cleanup of the metada Do not use the metadata store automated cleanup features if you have requirements to retain metadata records. For example, you have compliance requirements to keep audit records. In these cases, you should come up with an alternate method to preserve the audit metadata while freeing up your active metadata store. ## Configure automated metadata cleanup -Automated cleanup only removes records for deleted datasources. You can configure cleanup on a per-table basis as follows: - - `druid.coordinator.kill.*.on` enables cleanup for a particular metadata table. - - `druid.coordinator.kill.*.period` defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible records. - - `druid.coordinator.kill.*.durationToRetain` defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that the entity becomes eligible for deletion. For example to retain records for 30 days from their creation date: `P30D`. - For full configuration details, see [Metadata management](../configuration/index.md#metadata-management). +You can configure cleanup on a per-entity basis with the following constraints: +- You have to configure a kill task for Segment records before you can configure automated cleanup for rules or compaction configuration. +- You have to configure the scheduler for the cleanup jobs to run at a the same frequency or more frequently than your most frequent cleanup job. For examlple, if your most frequent cleanup job is every hour, set it to one hour or less: `druid.coordinator.period.metadataStoreManagementPeriod=P1H`. + +For details on configuration properties, see [Metadata management](../configuration/index.md#metadata-management). + +### Segment records and segements in deep storage(kill task) +Segment records and segements in deep storage become eligible for deletion: + +- When they meet the elegibility requirement of kill task datasource configuration according to `killDataSourceWhitelist` and `killAllDataSources` set in the Coordinator dynamic configuration. See [Dynamic configuration](../configuration/index.md#ynamic-configuration). +- `durationToRetain` time has passed since their creation. + +Kill tasks use the following configuration: +- `druid.coordinator.kill.on`: When `True`, enables the Coordinator to submit kill task for unused Segments which deletes them completely from metadata store and from deep storage. Only applies dataSources according to allowed datasources or all datasources. +- `druid.coordinator.kill.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible Segments. Must be greater than `druid.coordinator.period.indexingPeriod`. +- `druid.coordinator.kill.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that Segments become eligible for deletion. +- `druid.coordinator.kill.maxSegments`: Defines the maximum number of Segments to delete per kill task. +>The kill task is the only configuration in this topic that affects actual data in deep storage and not simply metadata. + +### Audit records +All audit records become eligible for deletion when the `durationToRetain` time has passed since their creation. Audit cleanup uses the following configuration: + - `druid.coordinator.kill.druid_audit.on`: When `True`, enables cleanup for audit records. + - `druid.coordinator.kill.druid_audit.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. + - `druid.coordinator.kill.druid_audit.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that audit recores become eligible for deletion. -The following are eligible for automatic cleanup: -- `druid_audit`: Audit records for deleted datasources become eligible for deletion when the `durationToRetain` time has passed since their creation. -- `druid_supervisors`: Supervisor records become eligible for deletion when: +### Supervisor records +Supervisor records become eligible for deletion when the supervisor is terminated and the `durationToRetain` time has passed since their creation. Supervisor cleanup uses the following configuration: + - `druid.coordinator.kill.supervisor.on`: When `True`, enables cleanup for supervisor records. + - `druid.coordinator.kill.supervisor.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. + - `druid.coordinator.kill.supervisor.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that audit recores become eligible for deletion. - - the supervisor is terminated. - - the `durationToRetain` time has passed since their creation. -- `druid_rules`: Rule records become eligible for deletion when: +### Rules +Rule records become eligible for deletion when all Segments for the datasource have been killed by the kill task and the `durationToRetain` time has passed since their creation. Rule cleanup uses the following configuration: + - `druid.coordinator.kill.druid_rules.on`: When `True`, enables cleanup for a particular metadata table. + - `druid.coordinator.kill.druid_rules.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. + - `druid.coordinator.kill.druid_rules.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that rules records become eligible for deletion. - - no segments exist for the datasource. - - the `durationToRetain` time has passed since their creation. -- `compaction`: Compaction configuration records in the records in the `druid_config` table become eligible for deletion when no segments exist for the datasource. - >If you already have an extremely large compaction configuration, you may not be able to delete compaction configuration due to size limits with the audit log. In this case you can set `druid.audit.manager.maxPayloadSizeBytes` and `druid.audit.manager.skipNullField` to avoid the auditing issue. See [Audit logging](../configuration/index.md#audit-logging). -- `druid_datasources`: Datasource records become eligible for deletion when: + ### Compaction configuration +Compaction configuration records in the records in the `druid_config` table become eligible for deletion all Segments for the datasource have been killed by the kill task.Compaction cleanup uses the following configuration: + - `druid.coordinator.kill.compaction.on`: When `True`, enables cleanup for compaction records. + - `druid.coordinator.kill.compaction.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible compaction configuration records. - - the supervisor is terminated or does not exist in the `druid_supervisors` table. - - the `durationToRetain` time has passed since their creation. +>If you already have an extremely large compaction configuration, you may not be able to delete compaction configuration due to size limits with the audit log. In this case you can set `druid.audit.manager.maxPayloadSizeBytes` and `druid.audit.manager.skipNullField` to avoid the auditing issue. See [Audit logging](../configuration/index.md#audit-logging). + +### Datasources created by supervisors +Datasource records created by supervisors become eligible for deletion when the supervisor is terminated or does not exist in the `druid_supervisors` table and the `durationToRetain` time has passed since their creation. Datasource cleanup uses the following configuration: + - `druid.coordinator.kill.datasource.on`: When `True`, enables cleanup datasources created by supervisors. + - `druid.coordinator.kill.datasource.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible datasource records. + - `druid.coordinator.kill.druid_rules.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that datasource records become eligible for deletion. + +>The table names listed here assume the default value for of `druid` for table name prefix. This is configurable according to `druid.metadata.storage.tables.base`. See [Metadata storage](../configuration/index.md#metadata-storage). ## Automated metadata cleanup example configuration -Consider a scenario where you have scripts to create and delete supervisors for hundreds of datasources a day. You do not want to fill your metadata store with records for deleted datasources. Datasources tend to persist for only one or two days. Therefore, you want to run a cleanup job that identifies and removes leftover records that are at least four days old. The exception is for audit logs, which you need to retain for 30 days: +Consider a scenario where you have scripts to create and delete hundreds of datasources and related entities a day. You do not want to fill your metadata store with leftover records. The datasources and related entities tend to persist for only one or two days. Therefore, you want to run a cleanup job that identifies and removes leftover records that are at least four days old. The exception is for audit logs, which you need to retain for 30 days: ``` ... +# Schedule the metadata management task for every hour: +druid.coordinator.period.metadataStoreManagementPeriod=P1H + +# Set a kill task to poll every day to delete Segment records +# and Segments in deep storage > 4 days old. +# Required also for automated cleanup of rules +# and compaction configuration. + +druid.coordinator.kill.on=True +druid.coordinator.kill.period=P1D +druid.coordinator.kill.durationToRetain=P4D +druid.coordinator.kill.maxSegments=1000 + # Poll every day to delete audit records > 30 days old druid.coordinator.kill.druid_audit.on=True druid.coordinator.kill.druid_audit.period=P1D @@ -89,9 +134,8 @@ druid.coordinator.kill.druid_rules.durationToRetain=P4D # Poll every day to delete compaction configuration records > 4 days old druid.coordinator.kill.compaction.on=True druid.coordinator.kill.compaction.period=P1D -druid.coordinator.kill.compaction.durationToRetain=P4D -# Poll every day to delete datasource records > 4 days old +# Poll every day to delete datasource records created by supervisors > 4 days old druid.coordinator.kill.druid_datasources.on=True druid.coordinator.kill.druid_datasources.period=P1D druid.coordinator.kill.druid_datasources.durationToRetain=P4D From 8828d848a7526f5b36f201ffd9419918f377e504 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Thu, 13 May 2021 11:13:09 -0700 Subject: [PATCH 3/8] address comments --- docs/operations/clean-metadata-store.md | 118 ++++++++++++++---------- 1 file changed, 69 insertions(+), 49 deletions(-) diff --git a/docs/operations/clean-metadata-store.md b/docs/operations/clean-metadata-store.md index 66c686af5404..b6b61faef903 100644 --- a/docs/operations/clean-metadata-store.md +++ b/docs/operations/clean-metadata-store.md @@ -32,9 +32,9 @@ When you delete some entities from Apache Druid, records related to the entity m - compaction configuration records - datasource records created by supervisors -If you have a high datasource churn rate, meaning you frequently create and delete many short-lived datasources or other releated entities like compaction configuration or rules, the leftover records can start to fill your metadata store and cause performance issues. +If you have a high datasource churn rate, meaning you frequently create and delete many short-lived datasources or other related entities like compaction configuration or rules, the leftover records can start to fill your metadata store and cause performance issues. -To maintain metadata store performance in this case, you can configure Apache Druid to automatically remove records associated with deleted datasources from the metadata store. +To maintain metadata store performance in this case, you can configure Apache Druid to automatically remove records associated with deleted entities from the metadata store. ## Automated cleanup strategies There are several cases when you should consider automated cleanup of the metadata related to deleted datasources: @@ -47,67 +47,87 @@ Do not use the metadata store automated cleanup features if you have requirement ## Configure automated metadata cleanup You can configure cleanup on a per-entity basis with the following constraints: -- You have to configure a kill task for Segment records before you can configure automated cleanup for rules or compaction configuration. -- You have to configure the scheduler for the cleanup jobs to run at a the same frequency or more frequently than your most frequent cleanup job. For examlple, if your most frequent cleanup job is every hour, set it to one hour or less: `druid.coordinator.period.metadataStoreManagementPeriod=P1H`. +- You have to configure a [kill task for segment records](#kill-task) before you can configure automated cleanup for [rules](#rules-records) or [compaction configuration](#compaction-configuration-records). +- You have to configure the scheduler for the cleanup jobs to run at a the same frequency or more frequently than your most frequent cleanup job. For example, if your most frequent cleanup job is every hour, set the scheduler metadata store management period to one hour or less: `druid.coordinator.period.metadataStoreManagementPeriod=P1H`. For details on configuration properties, see [Metadata management](../configuration/index.md#metadata-management). -### Segment records and segements in deep storage(kill task) -Segment records and segements in deep storage become eligible for deletion: + +### Segment records and segments in deep storage (kill task) +Segment records and segments in deep storage become eligible for deletion: -- When they meet the elegibility requirement of kill task datasource configuration according to `killDataSourceWhitelist` and `killAllDataSources` set in the Coordinator dynamic configuration. See [Dynamic configuration](../configuration/index.md#ynamic-configuration). -- `durationToRetain` time has passed since their creation. +- When they meet the eligibility requirement of kill task datasource configuration according to `killDataSourceWhitelist` and `killAllDataSources` set in the Coordinator dynamic configuration. See [Dynamic configuration](../configuration/index.md#dynamic-configuration). +- The `durationToRetain` time has passed since their creation. Kill tasks use the following configuration: -- `druid.coordinator.kill.on`: When `True`, enables the Coordinator to submit kill task for unused Segments which deletes them completely from metadata store and from deep storage. Only applies dataSources according to allowed datasources or all datasources. -- `druid.coordinator.kill.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible Segments. Must be greater than `druid.coordinator.period.indexingPeriod`. -- `druid.coordinator.kill.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that Segments become eligible for deletion. -- `druid.coordinator.kill.maxSegments`: Defines the maximum number of Segments to delete per kill task. ->The kill task is the only configuration in this topic that affects actual data in deep storage and not simply metadata. +- `druid.coordinator.kill.on`: When `True`, enables the Coordinator to submit kill task for unused segments which deletes them completely from metadata store and from deep storage. Only applies dataSources according to allowed datasources or all datasources. +- `druid.coordinator.kill.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible segments. Must be greater than `druid.coordinator.period.indexingPeriod`. +- `druid.coordinator.kill.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that segments become eligible for deletion. +- `druid.coordinator.kill.maxSegments`: Defines the maximum number of segments to delete per kill task. +>The kill task is the only configuration in this topic that affects actual data in deep storage and not simply metadata or logs. ### Audit records -All audit records become eligible for deletion when the `durationToRetain` time has passed since their creation. Audit cleanup uses the following configuration: - - `druid.coordinator.kill.druid_audit.on`: When `True`, enables cleanup for audit records. - - `druid.coordinator.kill.druid_audit.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. - - `druid.coordinator.kill.druid_audit.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that audit recores become eligible for deletion. +All audit records become eligible for deletion when the `durationToRetain` time has passed since their creation. + +Audit cleanup uses the following configuration: + - `druid.coordinator.kill.audit.on`: When `True`, enables cleanup for audit records. + - `druid.coordinator.kill.audit.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. + - `druid.coordinator.kill.audit.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that audit records become eligible for deletion. ### Supervisor records -Supervisor records become eligible for deletion when the supervisor is terminated and the `durationToRetain` time has passed since their creation. Supervisor cleanup uses the following configuration: +Supervisor records become eligible for deletion when the supervisor is terminated and the `durationToRetain` time has passed since their creation. + +Supervisor cleanup uses the following configuration: - `druid.coordinator.kill.supervisor.on`: When `True`, enables cleanup for supervisor records. - - `druid.coordinator.kill.supervisor.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. - - `druid.coordinator.kill.supervisor.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that audit recores become eligible for deletion. - -### Rules -Rule records become eligible for deletion when all Segments for the datasource have been killed by the kill task and the `durationToRetain` time has passed since their creation. Rule cleanup uses the following configuration: - - `druid.coordinator.kill.druid_rules.on`: When `True`, enables cleanup for a particular metadata table. - - `druid.coordinator.kill.druid_rules.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. - - `druid.coordinator.kill.druid_rules.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that rules records become eligible for deletion. - - ### Compaction configuration -Compaction configuration records in the records in the `druid_config` table become eligible for deletion all Segments for the datasource have been killed by the kill task.Compaction cleanup uses the following configuration: - - `druid.coordinator.kill.compaction.on`: When `True`, enables cleanup for compaction records. + - `druid.coordinator.kill.supervisor.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible supervisor records. + - `druid.coordinator.kill.supervisor.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that supervisor records become eligible for deletion. + +### Rules records +Rule records become eligible for deletion when all segments for the datasource have been killed by the kill task and the `durationToRetain` time has passed since their creation. Automated cleanup for rules requires a [kill task](#kill-task). + +Rule cleanup uses the following configuration: + - `druid.coordinator.kill.rule.on`: When `True`, enables cleanup for rules records. + - `druid.coordinator.kill.rule.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible rules records. + - `druid.coordinator.kill.rule.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that rules records become eligible for deletion. + + ### Compaction configuration records +Compaction configuration records in the records in the `druid_config` table become eligible for deletion all segments for the datasource have been killed by the kill task. Automated cleanup for compaction configuration requires a [kill task](#kill-task). + +Compaction configuration cleanup uses the following configuration: + - `druid.coordinator.kill.compaction.on`: When `True`, enables cleanup for compaction configuration records. - `druid.coordinator.kill.compaction.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible compaction configuration records. >If you already have an extremely large compaction configuration, you may not be able to delete compaction configuration due to size limits with the audit log. In this case you can set `druid.audit.manager.maxPayloadSizeBytes` and `druid.audit.manager.skipNullField` to avoid the auditing issue. See [Audit logging](../configuration/index.md#audit-logging). -### Datasources created by supervisors -Datasource records created by supervisors become eligible for deletion when the supervisor is terminated or does not exist in the `druid_supervisors` table and the `durationToRetain` time has passed since their creation. Datasource cleanup uses the following configuration: +### Datasource records created by supervisors +Datasource records created by supervisors become eligible for deletion when the supervisor is terminated or does not exist in the `druid_supervisors` table and the `durationToRetain` time has passed since their creation. + +Datasource cleanup uses the following configuration: - `druid.coordinator.kill.datasource.on`: When `True`, enables cleanup datasources created by supervisors. - `druid.coordinator.kill.datasource.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible datasource records. - - `druid.coordinator.kill.druid_rules.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that datasource records become eligible for deletion. + - `druid.coordinator.kill.datasource.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that datasource records become eligible for deletion. + +### Indexer task logs +The Druid Overlord handles task log metadata management. + +Indexer task log cleanup on the Overlord uses the following configuration: +- `druid.indexer.logs.kill.enabled`: When `True`, enables cleanup of task logs. +- `druid.indexer.logs.kill.durationToRetain`: Defines the length of time in milliseconds to retain task logs. +- `druid.indexer.logs.kill.initialDelay`: Defines the length of time in milliseconds after the Overlord starts before it executes its first job to kill task logs. +- `druid.indexer.logs.kill.delay`: The length of time in milliseconds between jobs to kill task logs. ->The table names listed here assume the default value for of `druid` for table name prefix. This is configurable according to `druid.metadata.storage.tables.base`. See [Metadata storage](../configuration/index.md#metadata-storage). +For more detail, see [Task logging](../configuration/index.md#task-logging). ## Automated metadata cleanup example configuration Consider a scenario where you have scripts to create and delete hundreds of datasources and related entities a day. You do not want to fill your metadata store with leftover records. The datasources and related entities tend to persist for only one or two days. Therefore, you want to run a cleanup job that identifies and removes leftover records that are at least four days old. The exception is for audit logs, which you need to retain for 30 days: ``` ... -# Schedule the metadata management task for every hour: +# Schedule the metadata management store task for every hour: druid.coordinator.period.metadataStoreManagementPeriod=P1H # Set a kill task to poll every day to delete Segment records -# and Segments in deep storage > 4 days old. +# and segments in deep storage > 4 days old. # Required also for automated cleanup of rules # and compaction configuration. @@ -117,32 +137,32 @@ druid.coordinator.kill.durationToRetain=P4D druid.coordinator.kill.maxSegments=1000 # Poll every day to delete audit records > 30 days old -druid.coordinator.kill.druid_audit.on=True -druid.coordinator.kill.druid_audit.period=P1D -druid.coordinator.kill.druid_audit.durationToRetain=P30D +druid.coordinator.kill.audit.on=True +druid.coordinator.kill.audit.period=P1D +druid.coordinator.kill.audit.durationToRetain=P30D # Poll every day to delete supervisor records > 4 days old -druid.coordinator.kill.druid_supervisors.on=True -druid.coordinator.kill.druid_supervisors.period=P1D -druid.coordinator.kill.druid_supervisors.durationToRetain=P4D +druid.coordinator.kill.supervisor.on=True +druid.coordinator.kill.supervisor.period=P1D +druid.coordinator.kill.supervisor.durationToRetain=P4D # Poll every day to delete rules records > 4 days old -druid.coordinator.kill.druid_rules.on=True -druid.coordinator.kill.druid_rules.period=P1D -druid.coordinator.kill.druid_rules.durationToRetain=P4D +druid.coordinator.kill.rule.on=True +druid.coordinator.kill.rule.period=P1D +druid.coordinator.kill.rule.durationToRetain=P4D # Poll every day to delete compaction configuration records > 4 days old druid.coordinator.kill.compaction.on=True druid.coordinator.kill.compaction.period=P1D # Poll every day to delete datasource records created by supervisors > 4 days old -druid.coordinator.kill.druid_datasources.on=True -druid.coordinator.kill.druid_datasources.period=P1D -druid.coordinator.kill.druid_datasources.durationToRetain=P4D +druid.coordinator.kill.datasource.on=True +druid.coordinator.kill.datasource.period=P1D +druid.coordinator.kill.datasource.durationToRetain=P4D ... ``` ## Learn more See the following topics for more information: - [Metadata management](../configuration/index.md#metadata-management) for metadata store configuration reference. -- [Metadata storage](../dependencies/metadata-storage.md) for an overview of the metadata storage database. +- [Metadata storage](../dependencies/metadata-storage.md) for an overview of the metadata storage database. \ No newline at end of file From 91159b248428b756ca85285c260dd1b22a759dd2 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Thu, 13 May 2021 14:33:23 -0700 Subject: [PATCH 4/8] update strategy recommendation --- docs/operations/clean-metadata-store.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/clean-metadata-store.md b/docs/operations/clean-metadata-store.md index b6b61faef903..a8430471086d 100644 --- a/docs/operations/clean-metadata-store.md +++ b/docs/operations/clean-metadata-store.md @@ -42,7 +42,7 @@ There are several cases when you should consider automated cleanup of the metada - If you have issues with the hard disk for your metadata database filling up. - If you run into performance issues with the metadata database. For example API calls are very slow or fail to execute. -Do not use the metadata store automated cleanup features if you have requirements to retain metadata records. For example, you have compliance requirements to keep audit records. In these cases, you should come up with an alternate method to preserve the audit metadata while freeing up your active metadata store. +If you have compliance requirements to keep audit records, use alternative methods to preserve audit metadata if you enable automated cleanup for audit records. For example, periodically export audit metadata records to external storage. ## Configure automated metadata cleanup From 12b94fb1ce67f2883ff95d337ba3ad13b51d84c1 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Fri, 14 May 2021 10:46:25 -0700 Subject: [PATCH 5/8] address addtional comments --- docs/operations/clean-metadata-store.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/operations/clean-metadata-store.md b/docs/operations/clean-metadata-store.md index a8430471086d..cc4336e6dc54 100644 --- a/docs/operations/clean-metadata-store.md +++ b/docs/operations/clean-metadata-store.md @@ -61,7 +61,7 @@ Segment records and segments in deep storage become eligible for deletion: Kill tasks use the following configuration: - `druid.coordinator.kill.on`: When `True`, enables the Coordinator to submit kill task for unused segments which deletes them completely from metadata store and from deep storage. Only applies dataSources according to allowed datasources or all datasources. -- `druid.coordinator.kill.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible segments. Must be greater than `druid.coordinator.period.indexingPeriod`. +- `druid.coordinator.kill.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible segments. Defaults to `P1D`. Must be greater than `druid.coordinator.period.indexingPeriod`. - `druid.coordinator.kill.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that segments become eligible for deletion. - `druid.coordinator.kill.maxSegments`: Defines the maximum number of segments to delete per kill task. >The kill task is the only configuration in this topic that affects actual data in deep storage and not simply metadata or logs. @@ -71,7 +71,7 @@ All audit records become eligible for deletion when the `durationToRetain` time Audit cleanup uses the following configuration: - `druid.coordinator.kill.audit.on`: When `True`, enables cleanup for audit records. - - `druid.coordinator.kill.audit.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. + - `druid.coordinator.kill.audit.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. Defaults to `P1D`. - `druid.coordinator.kill.audit.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that audit records become eligible for deletion. ### Supervisor records @@ -79,7 +79,7 @@ Supervisor records become eligible for deletion when the supervisor is terminate Supervisor cleanup uses the following configuration: - `druid.coordinator.kill.supervisor.on`: When `True`, enables cleanup for supervisor records. - - `druid.coordinator.kill.supervisor.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible supervisor records. + - `druid.coordinator.kill.supervisor.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible supervisor records. Defaults to `P1D`. - `druid.coordinator.kill.supervisor.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that supervisor records become eligible for deletion. ### Rules records @@ -87,7 +87,7 @@ Rule records become eligible for deletion when all segments for the datasource h Rule cleanup uses the following configuration: - `druid.coordinator.kill.rule.on`: When `True`, enables cleanup for rules records. - - `druid.coordinator.kill.rule.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible rules records. + - `druid.coordinator.kill.rule.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible rules records. Defaults to `P1D`. - `druid.coordinator.kill.rule.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that rules records become eligible for deletion. ### Compaction configuration records @@ -95,7 +95,7 @@ Compaction configuration records in the records in the `druid_config` table beco Compaction configuration cleanup uses the following configuration: - `druid.coordinator.kill.compaction.on`: When `True`, enables cleanup for compaction configuration records. - - `druid.coordinator.kill.compaction.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible compaction configuration records. + - `druid.coordinator.kill.compaction.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible compaction configuration records. Defaults to `P1D`. >If you already have an extremely large compaction configuration, you may not be able to delete compaction configuration due to size limits with the audit log. In this case you can set `druid.audit.manager.maxPayloadSizeBytes` and `druid.audit.manager.skipNullField` to avoid the auditing issue. See [Audit logging](../configuration/index.md#audit-logging). @@ -104,7 +104,7 @@ Datasource records created by supervisors become eligible for deletion when the Datasource cleanup uses the following configuration: - `druid.coordinator.kill.datasource.on`: When `True`, enables cleanup datasources created by supervisors. - - `druid.coordinator.kill.datasource.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible datasource records. + - `druid.coordinator.kill.datasource.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible datasource records. Defaults to `P1D`. - `druid.coordinator.kill.datasource.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that datasource records become eligible for deletion. ### Indexer task logs From e7db81c321ab174d9e9e89dc120de98deb55c2b0 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Fri, 14 May 2021 13:01:59 -0700 Subject: [PATCH 6/8] fix --- docs/operations/clean-metadata-store.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/operations/clean-metadata-store.md b/docs/operations/clean-metadata-store.md index cc4336e6dc54..a1a18b88a47c 100644 --- a/docs/operations/clean-metadata-store.md +++ b/docs/operations/clean-metadata-store.md @@ -70,7 +70,7 @@ Kill tasks use the following configuration: All audit records become eligible for deletion when the `durationToRetain` time has passed since their creation. Audit cleanup uses the following configuration: - - `druid.coordinator.kill.audit.on`: When `True`, enables cleanup for audit records. + - `druid.coordinator.kill.audit.on`: When `true`, enables cleanup for audit records. - `druid.coordinator.kill.audit.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible audit records. Defaults to `P1D`. - `druid.coordinator.kill.audit.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that audit records become eligible for deletion. @@ -78,7 +78,7 @@ Audit cleanup uses the following configuration: Supervisor records become eligible for deletion when the supervisor is terminated and the `durationToRetain` time has passed since their creation. Supervisor cleanup uses the following configuration: - - `druid.coordinator.kill.supervisor.on`: When `True`, enables cleanup for supervisor records. + - `druid.coordinator.kill.supervisor.on`: When `true`, enables cleanup for supervisor records. - `druid.coordinator.kill.supervisor.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible supervisor records. Defaults to `P1D`. - `druid.coordinator.kill.supervisor.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that supervisor records become eligible for deletion. @@ -86,7 +86,7 @@ Supervisor cleanup uses the following configuration: Rule records become eligible for deletion when all segments for the datasource have been killed by the kill task and the `durationToRetain` time has passed since their creation. Automated cleanup for rules requires a [kill task](#kill-task). Rule cleanup uses the following configuration: - - `druid.coordinator.kill.rule.on`: When `True`, enables cleanup for rules records. + - `druid.coordinator.kill.rule.on`: When `true`, enables cleanup for rules records. - `druid.coordinator.kill.rule.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible rules records. Defaults to `P1D`. - `druid.coordinator.kill.rule.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that rules records become eligible for deletion. @@ -94,7 +94,7 @@ Rule cleanup uses the following configuration: Compaction configuration records in the records in the `druid_config` table become eligible for deletion all segments for the datasource have been killed by the kill task. Automated cleanup for compaction configuration requires a [kill task](#kill-task). Compaction configuration cleanup uses the following configuration: - - `druid.coordinator.kill.compaction.on`: When `True`, enables cleanup for compaction configuration records. + - `druid.coordinator.kill.compaction.on`: When `true`, enables cleanup for compaction configuration records. - `druid.coordinator.kill.compaction.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible compaction configuration records. Defaults to `P1D`. >If you already have an extremely large compaction configuration, you may not be able to delete compaction configuration due to size limits with the audit log. In this case you can set `druid.audit.manager.maxPayloadSizeBytes` and `druid.audit.manager.skipNullField` to avoid the auditing issue. See [Audit logging](../configuration/index.md#audit-logging). @@ -103,7 +103,7 @@ Compaction configuration cleanup uses the following configuration: Datasource records created by supervisors become eligible for deletion when the supervisor is terminated or does not exist in the `druid_supervisors` table and the `durationToRetain` time has passed since their creation. Datasource cleanup uses the following configuration: - - `druid.coordinator.kill.datasource.on`: When `True`, enables cleanup datasources created by supervisors. + - `druid.coordinator.kill.datasource.on`: When `true`, enables cleanup datasources created by supervisors. - `druid.coordinator.kill.datasource.period`: Defines the frequency in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) for the cleanup job to check for and delete eligible datasource records. Defaults to `P1D`. - `druid.coordinator.kill.datasource.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that datasource records become eligible for deletion. @@ -111,7 +111,7 @@ Datasource cleanup uses the following configuration: The Druid Overlord handles task log metadata management. Indexer task log cleanup on the Overlord uses the following configuration: -- `druid.indexer.logs.kill.enabled`: When `True`, enables cleanup of task logs. +- `druid.indexer.logs.kill.enabled`: When `true`, enables cleanup of task logs. - `druid.indexer.logs.kill.durationToRetain`: Defines the length of time in milliseconds to retain task logs. - `druid.indexer.logs.kill.initialDelay`: Defines the length of time in milliseconds after the Overlord starts before it executes its first job to kill task logs. - `druid.indexer.logs.kill.delay`: The length of time in milliseconds between jobs to kill task logs. @@ -131,32 +131,32 @@ druid.coordinator.period.metadataStoreManagementPeriod=P1H # Required also for automated cleanup of rules # and compaction configuration. -druid.coordinator.kill.on=True +druid.coordinator.kill.on=true druid.coordinator.kill.period=P1D druid.coordinator.kill.durationToRetain=P4D druid.coordinator.kill.maxSegments=1000 # Poll every day to delete audit records > 30 days old -druid.coordinator.kill.audit.on=True +druid.coordinator.kill.audit.on=true druid.coordinator.kill.audit.period=P1D druid.coordinator.kill.audit.durationToRetain=P30D # Poll every day to delete supervisor records > 4 days old -druid.coordinator.kill.supervisor.on=True +druid.coordinator.kill.supervisor.on=true druid.coordinator.kill.supervisor.period=P1D druid.coordinator.kill.supervisor.durationToRetain=P4D # Poll every day to delete rules records > 4 days old -druid.coordinator.kill.rule.on=True +druid.coordinator.kill.rule.on=true druid.coordinator.kill.rule.period=P1D druid.coordinator.kill.rule.durationToRetain=P4D # Poll every day to delete compaction configuration records > 4 days old -druid.coordinator.kill.compaction.on=True +druid.coordinator.kill.compaction.on=true druid.coordinator.kill.compaction.period=P1D # Poll every day to delete datasource records created by supervisors > 4 days old -druid.coordinator.kill.datasource.on=True +druid.coordinator.kill.datasource.on=true druid.coordinator.kill.datasource.period=P1D druid.coordinator.kill.datasource.durationToRetain=P4D ... From e2883f2b1840b6687d1564c4cfba8297849432cd Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Wed, 19 May 2021 13:36:59 -0500 Subject: [PATCH 7/8] address comments --- docs/operations/clean-metadata-store.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/operations/clean-metadata-store.md b/docs/operations/clean-metadata-store.md index a1a18b88a47c..78f39a08a31f 100644 --- a/docs/operations/clean-metadata-store.md +++ b/docs/operations/clean-metadata-store.md @@ -45,6 +45,7 @@ There are several cases when you should consider automated cleanup of the metada If you have compliance requirements to keep audit records, use alternative methods to preserve audit metadata if you enable automated cleanup for audit records. For example, periodically export audit metadata records to external storage. ## Configure automated metadata cleanup +By default, automatic cleanup for metadata is disabled. See [Metadata storage](../configuration/index.md#metadata-storage) for the default configuration settings after you enable the feature. You can configure cleanup on a per-entity basis with the following constraints: - You have to configure a [kill task for segment records](#kill-task) before you can configure automated cleanup for [rules](#rules-records) or [compaction configuration](#compaction-configuration-records). @@ -91,7 +92,7 @@ Rule cleanup uses the following configuration: - `druid.coordinator.kill.rule.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that rules records become eligible for deletion. ### Compaction configuration records -Compaction configuration records in the records in the `druid_config` table become eligible for deletion all segments for the datasource have been killed by the kill task. Automated cleanup for compaction configuration requires a [kill task](#kill-task). +Compaction configuration records in the `druid_config` table become eligible for deletion after all segments for the datasource have been killed by the kill task. Automated cleanup for compaction configuration requires a [kill task](#kill-task). Compaction configuration cleanup uses the following configuration: - `druid.coordinator.kill.compaction.on`: When `true`, enables cleanup for compaction configuration records. @@ -108,7 +109,7 @@ Datasource cleanup uses the following configuration: - `druid.coordinator.kill.datasource.durationToRetain`: Defines the retention period in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations) after creation that datasource records become eligible for deletion. ### Indexer task logs -The Druid Overlord handles task log metadata management. +You can configure the Overlord to delete indexer task log metadata and the indexer task logs from local disk or from cloud storage. Indexer task log cleanup on the Overlord uses the following configuration: - `druid.indexer.logs.kill.enabled`: When `true`, enables cleanup of task logs. @@ -126,10 +127,11 @@ Consider a scenario where you have scripts to create and delete hundreds of data # Schedule the metadata management store task for every hour: druid.coordinator.period.metadataStoreManagementPeriod=P1H -# Set a kill task to poll every day to delete Segment records -# and segments in deep storage > 4 days old. -# Required also for automated cleanup of rules -# and compaction configuration. +# Set a kill task to poll every day to delete Segment records and segments +# in deep storage > 4 days old. When druid.coordinator.kill.on is set to true, +# you must set either killAllDataSources or killDataSourceWhitelist in the dynamic +# configuration. For this example, assume killAllDataSources is set to true. +# Required also for automated cleanup of rules and compaction configuration. druid.coordinator.kill.on=true druid.coordinator.kill.period=P1D @@ -151,7 +153,7 @@ druid.coordinator.kill.rule.on=true druid.coordinator.kill.rule.period=P1D druid.coordinator.kill.rule.durationToRetain=P4D -# Poll every day to delete compaction configuration records > 4 days old +# Poll every day to delete compaction configuration records druid.coordinator.kill.compaction.on=true druid.coordinator.kill.compaction.period=P1D From 7f53c93e69cb3c3ab3e30dacac4d6b5e3400c7b6 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Wed, 19 May 2021 17:00:53 -0500 Subject: [PATCH 8/8] address comments from @sthetland --- docs/operations/clean-metadata-store.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/operations/clean-metadata-store.md b/docs/operations/clean-metadata-store.md index 78f39a08a31f..5eac3a7f8863 100644 --- a/docs/operations/clean-metadata-store.md +++ b/docs/operations/clean-metadata-store.md @@ -2,7 +2,7 @@ id: clean-metadata-store title: "Automated cleanup for metadata records" sidebar_label: Automated metadata cleanup -description: "Defines a strategy to maintain Druid metadata store performance by automatically removing leftover records for deleted entities: datasources, supervisors, rulles, compaction configuration, audit records, etc. Most applicable to databases with 'high-churn' datasources." +description: "Defines a strategy to maintain Druid metadata store performance by automatically removing leftover records for deleted entities: datasources, supervisors, rules, compaction configuration, audit records, etc. Most applicable to databases with 'high-churn' datasources." ---