diff --git a/docs/en/administrator-guide/dynamic-partition.md b/docs/en/administrator-guide/dynamic-partition.md index 9a947dd80fa630..402e37d7b2a681 100644 --- a/docs/en/administrator-guide/dynamic-partition.md +++ b/docs/en/administrator-guide/dynamic-partition.md @@ -151,6 +151,32 @@ The rules of dynamic partition are prefixed with `dynamic_partition.`: p20210523: ["2021-05-23", "2021-05-24") storage_medium=SSD storage_cooldown_time=2021-05-25 00:00:00 ``` +* `dynamic_partition.reserved_history_periods` + + The range of reserved history periods. It should be in the form of `[yyyy-MM-dd,yyyy-MM-dd],[...,...]` while the `dynamic_partition.time_unit` is "DAY, WEEK, and MONTH". And it should be in the form of `[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]` while the dynamic_partition.time_unit` is "HOUR". And no more spaces expected. The default value is `"NULL"`, which means it is not set. + + Let us give an example. Suppose today is 2021-09-06，partitioned by day, and the properties of dynamic partition are set to: + + ```time_unit="DAY/WEEK/MONTH", end=3, start=-3, reserved_history_periods="[2020-06-01,2020-06-20],[2020-10-31,2020-11-15]"```. + + The the system will automatically reserve following partitions in following period : + + ``` + ["2020-06-01","2020-06-20"], + ["2020-10-31","2020-11-15"] + ``` + or + + ```time_unit="HOUR", end=3, start=-3, reserved_history_periods="[2020-06-01 00:00:00,2020-06-01 03:00:00]"```. + + The the system will automatically reserve following partitions in following period : + + ``` + ["2020-06-01 00:00:00","2020-06-01 03:00:00"] + ``` + + Otherwise, every `[...,...]` in `reserved_history_periods` is a couple of properties, and they should be set at the same time. And the first date can't be larger than the second one. + #### Create History Partition Rules When `create_history_partition` is `true`, i.e. history partition creation is enabled, Doris determines the number of history partitions to be created based on `dynamic_partition.start` and `dynamic_partition.history_partition_num`. @@ -357,16 +383,16 @@ You can further view the scheduling of dynamic partitioned tables by using the f ``` mysql> SHOW DYNAMIC PARTITION TABLES; -+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+ -| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg | -+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+ -| d3 | true | WEEK | -3 | 3 | p | 1 | MONDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d5 | true | DAY | -7 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d4 | true | WEEK | -3 | 3 | p | 1 | WEDNESDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d6 | true | MONTH | -2147483648 | 2 | p | 8 | 3rd | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d2 | true | DAY | -3 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d7 | true | MONTH | -2147483648 | 5 | p | 8 | 24th | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+ ++-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+ +| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg | ReservedHistoryPeriods | ++-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+ +| d3 | true | WEEK | -3 | 3 | p | 1 | MONDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | [2021-12-01,2021-12-31] | +| d5 | true | DAY | -7 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | +| d4 | true | WEEK | -3 | 3 | p | 1 | WEDNESDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | +| d6 | true | MONTH | -2147483648 | 2 | p | 8 | 3rd | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | +| d2 | true | DAY | -3 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | +| d7 | true | MONTH | -2147483648 | 5 | p | 8 | 24th | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | ++-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+ 7 rows in set (0.02 sec) ``` diff --git a/docs/en/administrator-guide/operation/doris-error-code.md b/docs/en/administrator-guide/operation/doris-error-code.md index 1ee7bedcaabfc2..9e243222b0e463 100644 --- a/docs/en/administrator-guide/operation/doris-error-code.md +++ b/docs/en/administrator-guide/operation/doris-error-code.md @@ -172,4 +172,7 @@ under the License. | 5072 | The dynamic partition copy value is not a valid number | | 5073 | The original created table stmt is empty | | 5074 | Create historical dynamic partition parameters: create_history_partition is invalid, what is expected is: true or false | - +| 5076 | The specified dynamic partition reserved_history_periods is null or empty | +| 5077 | The specified dynamic partition reserved_history_periods is invalid | +| 5078 | The length of specified dynamic partition reserved_history_periods must have pairs of date value | +| 5079 | The specified dynamic partition reserved_history_periods' first date is larger than the second one | diff --git a/docs/en/sql-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md b/docs/en/sql-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md index d7dbeb305fa5ab..47bf39e05bc64d 100644 --- a/docs/en/sql-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md +++ b/docs/en/sql-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md @@ -287,6 +287,7 @@ distribution_info * `dynamic_partition.buckets`: Used to specify the number of partition buckets that are automatically created. * `dynamic_partition.create_history_partition`: Whether to create a history partition. * `dynamic_partition.history_partition_num`: Specify the number of historical partitions to be created. + * `dynamic_partition.reserved_history_periods`: Used to specify the range of reserved history periods. ### Example diff --git a/docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md b/docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md index 2664a002089faa..ddf25d81c410ed 100644 --- a/docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md +++ b/docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md @@ -311,6 +311,7 @@ Syntax: dynamic_partition.buckets: specifies the number of partition buckets that are automatically created dynamic_partition.create_history_partition: specifies whether create history partitions, default value is false dynamic_partition.history_partition_num: used to specify the number of history partitions when enable create_history_partition + dynamic_partition.reserved_history_periods: Used to specify the range of reserved history periods ``` 5) You can create multiple Rollups in bulk when building a table grammar: diff --git a/docs/zh-CN/administrator-guide/dynamic-partition.md b/docs/zh-CN/administrator-guide/dynamic-partition.md index efe84fa8ad8055..246559e1884a07 100644 --- a/docs/zh-CN/administrator-guide/dynamic-partition.md +++ b/docs/zh-CN/administrator-guide/dynamic-partition.md @@ -149,6 +149,33 @@ under the License. p20210523：["2021-05-23", "2021-05-24") storage_medium=SSD storage_cooldown_time=2021-05-25 00:00:00 ``` +* `dynamic_partition.reserved_history_periods` + + 需要保留的历史分区的时间范围。当`dynamic_partition.time_unit` 设置为 "DAY/WEEK/MONTH" 时，需要以 `[yyyy-MM-dd,yyyy-MM-dd],[...,...]` 格式进行设置。当`dynamic_partition.time_unit` 设置为 "HOUR" 时，需要以 `[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]` 的格式来进行设置。如果不设置，默认为 `"NULL"`。 + + 我们举例说明。假设今天是 2021-09-06，按天分类，动态分区的属性设置为： + + ```time_unit="DAY/WEEK/MONTH", end=3, start=-3, reserved_history_periods="[2020-06-01,2020-06-20],[2020-10-31,2020-11-15]"```。 + + 则系统会自动保留： + + ``` + ["2020-06-01","2020-06-20"], + ["2020-10-31","2020-11-15"] + ``` + + 或者 + + ```time_unit="HOUR", end=3, start=-3, reserved_history_periods="[2020-06-01 00:00:00,2020-06-01 03:00:00]"```. + + 则系统会自动保留： + + ``` + ["2020-06-01 00:00:00","2020-06-01 03:00:00"] + ``` + + 这两个时间段的分区。其中，`reserved_history_periods` 的每一个 `[...,...]` 是一对设置项，两者需要同时被设置，且第一个时间不能大于第二个时间``。 + #### 创建历史分区规则 当 `create_history_partition` 为 `true`，即开启创建历史分区功能时，Doris 会根据 `dynamic_partition.start` 和 `dynamic_partition.history_partition_num` 来决定创建历史分区的个数。 @@ -355,16 +382,16 @@ p20200521: ["2020-05-21", "2020-05-22") ``` mysql> SHOW DYNAMIC PARTITION TABLES; -+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+ -| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg | -+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+ -| d3 | true | WEEK | -3 | 3 | p | 1 | MONDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d5 | true | DAY | -7 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d4 | true | WEEK | -3 | 3 | p | 1 | WEDNESDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d6 | true | MONTH | -2147483648 | 2 | p | 8 | 3rd | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d2 | true | DAY | -3 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -| d7 | true | MONTH | -2147483648 | 5 | p | 8 | 24th | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | -+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+ ++-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+ +| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg | ReservedHistoryPeriods | ++-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+ +| d3 | true | WEEK | -3 | 3 | p | 1 | MONDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | [2021-12-01,2021-12-31] | +| d5 | true | DAY | -7 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | +| d4 | true | WEEK | -3 | 3 | p | 1 | WEDNESDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | +| d6 | true | MONTH | -2147483648 | 2 | p | 8 | 3rd | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | +| d2 | true | DAY | -3 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | +| d7 | true | MONTH | -2147483648 | 5 | p | 8 | 24th | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL | ++-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+ 7 rows in set (0.02 sec) ``` diff --git a/docs/zh-CN/administrator-guide/operation/doris-error-code.md b/docs/zh-CN/administrator-guide/operation/doris-error-code.md index 07123ed0936700..122c0d1b590da2 100644 --- a/docs/zh-CN/administrator-guide/operation/doris-error-code.md +++ b/docs/zh-CN/administrator-guide/operation/doris-error-code.md @@ -172,4 +172,8 @@ under the License. | 5072 | 动态分区副本值不是有效的数字 | | 5073 | 原始创建表stmt为空 | | 5074 | 创建历史动态分区参数：create_history_partition无效，期望的是：true或者false | +| 5076 | 指定的保留历史分区时间段为空 | +| 5077 | 指定的保留历史分区时间段无效 | +| 5078 | 指定的保留历史分区时间段必须是成对的时间 | +| 5079 | 指定的保留历史分区时间段对应位置的第一个时间比第二个时间大（起始时间大于结束时间） | diff --git a/docs/zh-CN/sql-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md b/docs/zh-CN/sql-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md index fdb00a47edb343..e61af64443fa58 100644 --- a/docs/zh-CN/sql-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md +++ b/docs/zh-CN/sql-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md @@ -287,6 +287,7 @@ distribution_info * `dynamic_partition.buckets`: 用于指定自动创建的分区分桶数量。 * `dynamic_partition.create_history_partition`: 是否创建历史分区。 * `dynamic_partition.history_partition_num`: 指定创建历史分区的数量。 + * `dynamic_partition.reserved_history_periods`: 用于指定保留的历史分区的时间段。 ### Example diff --git a/docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md b/docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md index 15abbbfbab036c..e287fd42c84cd8 100644 --- a/docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md +++ b/docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md @@ -334,6 +334,7 @@ under the License. dynamic_partition.buckets: 用于指定自动创建的分区分桶数量 dynamic_partition.create_history_partition: 用于创建历史分区功能是否开启。默认为 false。 dynamic_partition.history_partition_num: 当开启创建历史分区功能时，用于指定创建历史分区数量。 + dynamic_partition.reserved_history_periods: 用于指定保留的历史分区的时间段。 5) 建表时可以批量创建多个 Rollup 语法： diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDynamicPartitionStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDynamicPartitionStmt.java index ab8994ae5fae0c..a80e8e5de863b9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDynamicPartitionStmt.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDynamicPartitionStmt.java @@ -46,6 +46,7 @@ public class ShowDynamicPartitionStmt extends ShowStmt { .addColumn(new Column("State", ScalarType.createVarchar(20))) .addColumn(new Column("LastCreatePartitionMsg", ScalarType.createVarchar(20))) .addColumn(new Column("LastDropPartitionMsg", ScalarType.createVarchar(20))) + .addColumn(new Column("ReservedHistoryPeriods", ScalarType.createVarchar(20))) .build(); ShowDynamicPartitionStmt(String db) { @@ -95,4 +96,4 @@ public ShowResultSetMetaData getMetaData() { public RedirectStatus getRedirectStatus() { return RedirectStatus.FORWARD_NO_SYNC; } -} \ No newline at end of file +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/DynamicPartitionProperty.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/DynamicPartitionProperty.java index ecbe7f26419fbf..ca8bf9002ec55e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/DynamicPartitionProperty.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/DynamicPartitionProperty.java @@ -19,7 +19,9 @@ import org.apache.doris.analysis.TimestampArithmeticExpr.TimeUnit; import org.apache.doris.common.AnalysisException; +import org.apache.doris.common.DdlException; import org.apache.doris.common.FeConstants; +import org.apache.doris.common.util.DynamicPartitionUtil; import org.apache.doris.common.util.DynamicPartitionUtil.StartOfDate; import org.apache.doris.common.util.PropertyAnalyzer; import org.apache.doris.common.util.TimeUtils; @@ -43,11 +45,13 @@ public class DynamicPartitionProperty { public static final String CREATE_HISTORY_PARTITION = "dynamic_partition.create_history_partition"; public static final String HISTORY_PARTITION_NUM = "dynamic_partition.history_partition_num"; public static final String HOT_PARTITION_NUM = "dynamic_partition.hot_partition_num"; + public static final String RESERVED_HISTORY_PERIODS = "dynamic_partition.reserved_history_periods"; public static final int MIN_START_OFFSET = Integer.MIN_VALUE; public static final int MAX_END_OFFSET = Integer.MAX_VALUE; public static final int NOT_SET_REPLICATION_NUM = -1; public static final int NOT_SET_HISTORY_PARTITION_NUM = -1; + public static final String NOT_SET_RESERVED_HISTORY_PERIODS = "NULL"; private boolean exist; @@ -67,6 +71,7 @@ public class DynamicPartitionProperty { // This property are used to describe the number of partitions that need to be reserved on the high-speed storage. // If not set, default is 0 private int hotPartitionNum; + private String reservedHistoryPeriods; public DynamicPartitionProperty(Map properties) { if (properties != null && !properties.isEmpty()) { @@ -83,6 +88,7 @@ public DynamicPartitionProperty(Map properties) { this.createHistoryPartition = Boolean.parseBoolean(properties.get(CREATE_HISTORY_PARTITION)); this.historyPartitionNum = Integer.parseInt(properties.getOrDefault(HISTORY_PARTITION_NUM, String.valueOf(NOT_SET_HISTORY_PARTITION_NUM))); this.hotPartitionNum = Integer.parseInt(properties.getOrDefault(HOT_PARTITION_NUM, "0")); + this.reservedHistoryPeriods = properties.getOrDefault(RESERVED_HISTORY_PERIODS, NOT_SET_RESERVED_HISTORY_PERIODS); createStartOfs(properties); } else { this.exist = false; @@ -180,6 +186,14 @@ public ReplicaAllocation getReplicaAllocation() { return replicaAlloc; } + public String getReservedHistoryPeriods() { + return reservedHistoryPeriods; + } + + public String getSortedReservedHistoryPeriods(String reservedHistoryPeriods, String timeUnit) throws DdlException { + return DynamicPartitionUtil.sortedListedToString(reservedHistoryPeriods, timeUnit); + } + /** * use table replication_num as dynamic_partition.replication_num default value */ @@ -195,7 +209,8 @@ public String getProperties(ReplicaAllocation tableReplicaAlloc) { ",\n\"" + BUCKETS + "\" = \"" + buckets + "\"" + ",\n\"" + CREATE_HISTORY_PARTITION + "\" = \"" + createHistoryPartition + "\"" + ",\n\"" + HISTORY_PARTITION_NUM + "\" = \"" + historyPartitionNum + "\"" + - ",\n\"" + HOT_PARTITION_NUM + "\" = \"" + hotPartitionNum + "\""; + ",\n\"" + HOT_PARTITION_NUM + "\" = \"" + hotPartitionNum + "\"" + + ",\n\"" + RESERVED_HISTORY_PERIODS + "\" = \"" + reservedHistoryPeriods + "\""; if (getTimeUnit().equalsIgnoreCase(TimeUnit.WEEK.toString())) { res += ",\n\"" + START_DAY_OF_WEEK + "\" = \"" + startOfWeek.dayOfWeek + "\""; } else if (getTimeUnit().equalsIgnoreCase(TimeUnit.MONTH.toString())) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java index 62905a75aa498e..4a6fc1d2d2e6bc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java +++ b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java @@ -245,11 +245,29 @@ private void setStorageMediumProperty(HashMap partitionPropertie partitionProperties.put(PropertyAnalyzer.PROPERTIES_STORAGE_COLDOWN_TIME, cooldownTime); } + private Range getClosedRange(Database db, OlapTable olapTable, Column partitionColumn, String partitionFormat, + String lowerBorderOfReservedHistory, String upperBorderOfReservedHistory) { + Range reservedHistoryPartitionKeyRange = null; + PartitionValue lowerBorderPartitionValue = new PartitionValue(lowerBorderOfReservedHistory); + PartitionValue upperBorderPartitionValue = new PartitionValue(upperBorderOfReservedHistory); + try { + PartitionKey lowerBorderBound = PartitionKey.createPartitionKey(Collections.singletonList(lowerBorderPartitionValue), Collections.singletonList(partitionColumn)); + PartitionKey upperBorderBound = PartitionKey.createPartitionKey(Collections.singletonList(upperBorderPartitionValue), Collections.singletonList(partitionColumn)); + reservedHistoryPartitionKeyRange = Range.closed(lowerBorderBound, upperBorderBound); + } catch (AnalysisException e) { + // AnalysisException: keys.size is always equal to column.size, cannot reach this exception + // IllegalArgumentException: lb is greater than ub + LOG.warn("Error in gen reservePartitionKeyRange. Error={}, db: {}, table: {}", e.getMessage(), + db.getFullName(), olapTable.getName()); + } + return reservedHistoryPartitionKeyRange; + } + /** * 1. get the range of [start, 0) as a reserved range. * 2. get DropPartitionClause of partitions which range are before this reserved range. */ - private ArrayList getDropPartitionClause(Database db, OlapTable olapTable, Column partitionColumn, String partitionFormat) { + private ArrayList getDropPartitionClause(Database db, OlapTable olapTable, Column partitionColumn, String partitionFormat) throws DdlException { ArrayList dropPartitionClauses = new ArrayList<>(); DynamicPartitionProperty dynamicPartitionProperty = olapTable.getTableProperty().getDynamicPartitionProperty(); if (dynamicPartitionProperty.getStart() == DynamicPartitionProperty.MIN_START_OFFSET) { @@ -261,14 +279,16 @@ private ArrayList getDropPartitionClause(Database db, OlapT String lowerBorder = DynamicPartitionUtil.getPartitionRangeString(dynamicPartitionProperty, now, dynamicPartitionProperty.getStart(), partitionFormat); String upperBorder = DynamicPartitionUtil.getPartitionRangeString(dynamicPartitionProperty, - now, 0, partitionFormat); + now, dynamicPartitionProperty.getEnd() + 1, partitionFormat); PartitionValue lowerPartitionValue = new PartitionValue(lowerBorder); PartitionValue upperPartitionValue = new PartitionValue(upperBorder); + List> reservedHistoryPartitionKeyRangeList = new ArrayList>(); Range reservePartitionKeyRange; try { PartitionKey lowerBound = PartitionKey.createPartitionKey(Collections.singletonList(lowerPartitionValue), Collections.singletonList(partitionColumn)); PartitionKey upperBound = PartitionKey.createPartitionKey(Collections.singletonList(upperPartitionValue), Collections.singletonList(partitionColumn)); reservePartitionKeyRange = Range.closedOpen(lowerBound, upperBound); + reservedHistoryPartitionKeyRangeList.add(reservePartitionKeyRange); } catch (AnalysisException | IllegalArgumentException e) { // AnalysisException: keys.size is always equal to column.size, cannot reach this exception // IllegalArgumentException: lb is greater than ub @@ -276,23 +296,44 @@ private ArrayList getDropPartitionClause(Database db, OlapT db.getFullName(), olapTable.getName()); return dropPartitionClauses; } + + String reservedHistoryPeriods = dynamicPartitionProperty.getReservedHistoryPeriods(); + List ranges = DynamicPartitionUtil.convertStringToPeriodsList(reservedHistoryPeriods, dynamicPartitionProperty.getTimeUnit()); + + if (ranges.size() != 0) { + for (Range range : ranges) { + try { + String lowerBorderOfReservedHistory = DynamicPartitionUtil.getHistoryPartitionRangeString(dynamicPartitionProperty, range.lowerEndpoint().toString(), partitionFormat); + String upperBorderOfReservedHistory = DynamicPartitionUtil.getHistoryPartitionRangeString(dynamicPartitionProperty, range.upperEndpoint().toString(), partitionFormat); + Range reservedHistoryPartitionKeyRange = getClosedRange(db, olapTable, partitionColumn, partitionFormat, lowerBorderOfReservedHistory, upperBorderOfReservedHistory); + reservedHistoryPartitionKeyRangeList.add(reservedHistoryPartitionKeyRange); + } catch (IllegalArgumentException e) { + return dropPartitionClauses; + } + } + } RangePartitionInfo info = (RangePartitionInfo) (olapTable.getPartitionInfo()); List> idToItems = new ArrayList<>(info.getIdToItem(false).entrySet()); idToItems.sort(Comparator.comparing(o -> ((RangePartitionItem) o.getValue()).getItems().upperEndpoint())); + Map isContaineds = new HashMap<>(); for (Map.Entry idToItem : idToItems) { - try { - Long checkDropPartitionId = idToItem.getKey(); - Range checkDropPartitionKey = idToItem.getValue().getItems(); - RangeUtils.checkRangeIntersect(reservePartitionKeyRange, checkDropPartitionKey); - if (checkDropPartitionKey.upperEndpoint().compareTo(reservePartitionKeyRange.lowerEndpoint()) <= 0) { - String dropPartitionName = olapTable.getPartition(checkDropPartitionId).getName(); - // Do not drop the partition "by force", or the partition will be dropped directly instread of being in - // catalog recycle bin. This is for safe reason. - dropPartitionClauses.add(new DropPartitionClause(false, dropPartitionName, false, false)); + isContaineds.put(idToItem.getKey(), false); + Long checkDropPartitionId = idToItem.getKey(); + Range checkDropPartitionKey = idToItem.getValue().getItems(); + for (Range reserveHistoryPartitionKeyRange : reservedHistoryPartitionKeyRangeList) { + if (RangeUtils.checkIsTwoRangesIntersect(reserveHistoryPartitionKeyRange, checkDropPartitionKey)) { + isContaineds.put(checkDropPartitionId, true); } - } catch (DdlException e) { - break; + } + } + + for (Long dropPartitionId : isContaineds.keySet()) { + // Do not drop the partition "by force", or the partition will be dropped directly instread of being in + // catalog recycle bin. This is for safe reason. + if(!isContaineds.get(dropPartitionId)) { + String dropPartitionName = olapTable.getPartition(dropPartitionId).getName(); + dropPartitionClauses.add(new DropPartitionClause(false, dropPartitionName, false, false)); } } return dropPartitionClauses; @@ -311,8 +352,8 @@ private void executeDynamicPartition(Collection> dynamicPartiti } ArrayList addPartitionClauses = new ArrayList<>(); - ArrayList dropPartitionClauses; - String tableName; + ArrayList dropPartitionClauses = null; + String tableName = null; boolean skipAddPartition = false; OlapTable olapTable; olapTable = (OlapTable) db.getTableNullable(tableId); @@ -358,6 +399,8 @@ private void executeDynamicPartition(Collection> dynamicPartiti } dropPartitionClauses = getDropPartitionClause(db, olapTable, partitionColumn, partitionFormat); tableName = olapTable.getName(); + } catch (DdlException e) { + e.printStackTrace(); } finally { olapTable.readUnlock(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java index d6497dd3b85630..69b9baed27c5df 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java @@ -244,6 +244,14 @@ public enum ErrorCode { "Invalid dynamic partition create_history_partition: %s. Expected true or false"), ERROR_DYNAMIC_PARTITION_HISTORY_PARTITION_NUM_ZERO(5075, new byte[] {'4', '2', '0', '0', '0'}, "Dynamic history partition num must greater than 0"), + ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_PERIODS_EMPTY(5076, new byte[] {'4', '2', '0', '0', '0'}, + "Dynamic reserved history periods is empty."), + ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_PERIODS_INVALID(5077, new byte[] {'4', '2', '0', '0', '0'}, + "Invalid \" %s \" value %s. It must be like \"[yyyy-MM-dd,yyyy-MM-dd],[...,...]\" while time_unit is DAY/WEEK/MONTH or \"[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]\" while time_unit is HOUR."), + ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_PERIODS_START_ENDS_LENGTH_NOT_EQUAL(5078, new byte[] {'4', '2', '0', '0', '0'}, + "RESERVED_HISTORY_PERIODS must have pairs of date value. The input %s is not valid."), + ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_PERIODS_START_LARGER_THAN_ENDS(5079, new byte[] {'4', '2', '0', '0', '0'}, + "The first date is larger than the second date, [%s,%s] is invalid."), ERROR_LDAP_CONFIGURATION_ERR(5080, new byte[] {'4', '2', '0', '0', '0'}, "LDAP configuration is incorrect or LDAP admin password is not set."), ERROR_LDAP_USER_NOT_UNIQUE_ERR(5081, new byte[] {'4', '2', '0', '0', '0'}, diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java index 174673bb887eae..1725ff49055d7b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java @@ -41,20 +41,30 @@ import com.google.common.base.Preconditions; import com.google.common.base.Strings; +import com.google.common.collect.Range; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.sql.Timestamp; import java.text.ParseException; import java.text.SimpleDateFormat; import java.time.DayOfWeek; import java.time.Month; +import java.time.ZoneId; import java.time.ZonedDateTime; import java.time.format.DateTimeFormatter; -import java.util.Calendar; +import java.util.ArrayList; +import java.util.Comparator; import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.TimeZone; +import java.util.Date; +import java.util.Calendar; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.Collectors; public class DynamicPartitionUtil { private static final Logger LOG = LogManager.getLogger(DynamicPartitionUtil.class); @@ -230,6 +240,101 @@ private static void checkHotPartitionNum(String val) throws DdlException { } } + public static List convertStringToPeriodsList(String reservedHistoryPeriods, String timeUnit) throws DdlException { + List reservedHistoryPeriodsToRangeList = new ArrayList(); + if (DynamicPartitionProperty.NOT_SET_RESERVED_HISTORY_PERIODS.equals(reservedHistoryPeriods)) { + return reservedHistoryPeriodsToRangeList; + } + + Pattern pattern = getPattern(timeUnit); + Matcher matcher = pattern.matcher(reservedHistoryPeriods); + while (matcher.find()) { + String lowerBorderOfReservedHistory = matcher.group(1); + String upperBorderOfReservedHistory = matcher.group(2); + if (lowerBorderOfReservedHistory.compareTo(upperBorderOfReservedHistory) > 0) { + ErrorReport.reportDdlException(ErrorCode.ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_PERIODS_START_LARGER_THAN_ENDS, lowerBorderOfReservedHistory, upperBorderOfReservedHistory); + } else { + reservedHistoryPeriodsToRangeList.add(Range.closed(lowerBorderOfReservedHistory, upperBorderOfReservedHistory)); + } + } + return reservedHistoryPeriodsToRangeList; + } + + private static Pattern getPattern(String timeUnit) { + if (timeUnit.equalsIgnoreCase(TimeUnit.HOUR.toString())) { + return Pattern.compile("\\[([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}),([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2})\\]"); + } else { + return Pattern.compile("\\[([0-9]{4}-[0-9]{2}-[0-9]{2}),([0-9]{4}-[0-9]{2}-[0-9]{2})\\]"); + } + } + + public static String sortedListedToString(String reservedHistoryPeriods, String timeUnit) throws DdlException { + if (DynamicPartitionProperty.NOT_SET_RESERVED_HISTORY_PERIODS.equals(reservedHistoryPeriods)) { + return reservedHistoryPeriods; + } + List reservedHistoryPeriodsToRangeList = convertStringToPeriodsList(reservedHistoryPeriods, timeUnit); + reservedHistoryPeriodsToRangeList.sort(new Comparator() { + @Override + public int compare(Range o1, Range o2) { + return o1.lowerEndpoint().compareTo(o2.lowerEndpoint()); + } + }); + List sortedReservedHistoryPeriods = reservedHistoryPeriodsToRangeList.stream(). + map(e -> "[" + e.lowerEndpoint() + "," + e.upperEndpoint() + "]").collect(Collectors.toList()); + + return String.join(",", sortedReservedHistoryPeriods); + } + + private static void checkReservedHistoryPeriodValidate(String reservedHistoryPeriods, String timeUnit) throws DdlException { + if (Strings.isNullOrEmpty(reservedHistoryPeriods)) { + ErrorReport.reportDdlException(ErrorCode.ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_PERIODS_EMPTY); + } + if (DynamicPartitionProperty.NOT_SET_RESERVED_HISTORY_PERIODS.equals(reservedHistoryPeriods)) { + return; + } + // it has 5 kinds of situation + // 1. "dynamic_partition.reserved_history_periods" = "[2021-07-01,]," invalid one + // 2. "dynamic_partition.reserved_history_periods" = "2021-07-01", invalid. It must be surrounded by [] + // 1. "dynamic_partition.reserved_history_periods" = "[2021-07-01,]" invalid one, needs pairs of values + // 2. "dynamic_partition.reserved_history_periods" = "[,2021-08-01]" invalid one, needs pairs of values + // 3. "dynamic_partition.reserved_history_periods" = "[2021-07-01,2020-08-01,]" invalid format + if (!reservedHistoryPeriods.startsWith("[") || !reservedHistoryPeriods.endsWith("]")) { + ErrorReport.reportDdlException(ErrorCode.ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_PERIODS_INVALID, DynamicPartitionProperty.RESERVED_HISTORY_PERIODS, reservedHistoryPeriods); + } + + List reservedHistoryPeriodsToRangeList = convertStringToPeriodsList(reservedHistoryPeriods, timeUnit); + Integer sizeOfPeriods = reservedHistoryPeriods.split("],\\[").length; + SimpleDateFormat sdf = getSimpleDateFormat(timeUnit); + + if (reservedHistoryPeriodsToRangeList.size() != sizeOfPeriods) { + ErrorReport.reportDdlException(ErrorCode.ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_PERIODS_INVALID, DynamicPartitionProperty.RESERVED_HISTORY_PERIODS, reservedHistoryPeriods); + } else { + try { + for (Range range : reservedHistoryPeriodsToRangeList) { + String formattedLowerBound = sdf.format(sdf.parse(range.lowerEndpoint().toString())); + String formattedUpperBound = sdf.format(sdf.parse(range.upperEndpoint().toString())); + if (!range.lowerEndpoint().toString().equals(formattedLowerBound) || !range.upperEndpoint().toString().equals(formattedUpperBound)) { + throw new DdlException("Invalid " + DynamicPartitionProperty.RESERVED_HISTORY_PERIODS + + " value. It must be correct DATE value \"[yyyy-MM-dd,yyyy-MM-dd],[...,...]\" while time_unit is DAY/WEEK/MONTH " + + "or \"[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]\" while time_unit is HOUR."); + } + } + } catch (ParseException e) { + throw new DdlException("Invalid " + DynamicPartitionProperty.RESERVED_HISTORY_PERIODS + + " value. It must be like \"[yyyy-MM-dd,yyyy-MM-dd],[...,...]\" while time_unit is DAY/WEEK/MONTH " + + "or \"[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]\" while time_unit is HOUR."); + } + } + } + + private static SimpleDateFormat getSimpleDateFormat(String timeUnit) { + if (timeUnit.equalsIgnoreCase(TimeUnit.HOUR.toString())) { + return new SimpleDateFormat(DATETIME_FORMAT); + } else { + return new SimpleDateFormat(DATE_FORMAT); + } + } + public static boolean checkDynamicPartitionPropertiesExist(Map properties) { if (properties == null) { return false; @@ -261,6 +366,8 @@ public static boolean checkInputDynamicPartitionProperties(Map p String enable = properties.get(DynamicPartitionProperty.ENABLE); String createHistoryPartition = properties.get(DynamicPartitionProperty.CREATE_HISTORY_PARTITION); String historyPartitionNum = properties.get(DynamicPartitionProperty.HISTORY_PARTITION_NUM); + String reservedHistoryPeriods = properties.get(DynamicPartitionProperty.RESERVED_HISTORY_PERIODS); + if (!(Strings.isNullOrEmpty(enable) && Strings.isNullOrEmpty(timeUnit) && Strings.isNullOrEmpty(timeZone) && @@ -269,7 +376,8 @@ public static boolean checkInputDynamicPartitionProperties(Map p Strings.isNullOrEmpty(end) && Strings.isNullOrEmpty(buckets) && Strings.isNullOrEmpty(createHistoryPartition) && - Strings.isNullOrEmpty(historyPartitionNum))) { + Strings.isNullOrEmpty(historyPartitionNum) && + Strings.isNullOrEmpty(reservedHistoryPeriods))) { if (Strings.isNullOrEmpty(enable)) { properties.put(DynamicPartitionProperty.ENABLE, "true"); } @@ -298,6 +406,10 @@ public static boolean checkInputDynamicPartitionProperties(Map p properties.put(DynamicPartitionProperty.HISTORY_PARTITION_NUM, String.valueOf(DynamicPartitionProperty.NOT_SET_HISTORY_PARTITION_NUM)); } + if (Strings.isNullOrEmpty(reservedHistoryPeriods)) { + properties.put(DynamicPartitionProperty.RESERVED_HISTORY_PERIODS, + String.valueOf(DynamicPartitionProperty.NOT_SET_RESERVED_HISTORY_PERIODS)); + } } return true; } @@ -450,7 +562,12 @@ public static Map analyzeDynamicPartition(Map pr properties.remove(DynamicPartitionProperty.HOT_PARTITION_NUM); analyzedProperties.put(DynamicPartitionProperty.HOT_PARTITION_NUM, val); } - + if (properties.containsKey(DynamicPartitionProperty.RESERVED_HISTORY_PERIODS)) { + String reservedHistoryPeriods = properties.get(DynamicPartitionProperty.RESERVED_HISTORY_PERIODS); + checkReservedHistoryPeriodValidate(reservedHistoryPeriods, analyzedProperties.get(DynamicPartitionProperty.TIME_UNIT)); + properties.remove(DynamicPartitionProperty.RESERVED_HISTORY_PERIODS); + analyzedProperties.put(DynamicPartitionProperty.RESERVED_HISTORY_PERIODS, reservedHistoryPeriods); + } return analyzedProperties; } @@ -553,6 +670,23 @@ public static String getPartitionRangeString(DynamicPartitionProperty property, } } + public static String getHistoryPartitionRangeString(DynamicPartitionProperty dynamicPartitionProperty, String time, String format) { + ZoneId zoneId = dynamicPartitionProperty.getTimeZone().toZoneId(); + Date date = null; + Timestamp timestamp = null; + String timeUnit = dynamicPartitionProperty.getTimeUnit(); + DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.s").withZone(zoneId); + SimpleDateFormat simpleDateFormat = getSimpleDateFormat(timeUnit); + try { + date = simpleDateFormat.parse(time); + } catch (ParseException e) { + LOG.warn("Parse dynamic partition periods error. Error={}", e.getMessage()); + return getFormattedTimeWithoutMinuteSecond(ZonedDateTime.parse(timestamp.toString(), dateTimeFormatter), format); + } + timestamp = new Timestamp(date.getTime()); + return getFormattedTimeWithoutMinuteSecond(ZonedDateTime.parse(timestamp.toString(), dateTimeFormatter), format); + } + /** * return formatted string of partition range in HOUR granularity. * offset: The offset from the current hour. 0 means current hour, 1 means next hour, -1 last hour. diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/RangeUtils.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/RangeUtils.java index eeef22eac4e919..13e0f5ba0fc73f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/util/RangeUtils.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/RangeUtils.java @@ -51,6 +51,12 @@ public static void checkRangeIntersect(Range range1, Range range1, Range range2) { + if (range2.isConnected(range1) && !range2.intersection(range1).isEmpty()) { + return true; + } + return false; + } /* * Pass only if the 2 range lists are exactly same * What is "exactly same"? diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java index 8778104433c616..d40b8478dfdeed 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java @@ -1817,6 +1817,7 @@ private void handleShowDynamicPartition() { if (replicaAlloc.isNotSet()) { replicaAlloc = olapTable.getDefaultReplicaAllocation(); } + String unsortedReservedHistoryPeriods = dynamicPartitionProperty.getReservedHistoryPeriods(); rows.add(Lists.newArrayList( tableName, String.valueOf(dynamicPartitionProperty.getEnable()), @@ -1832,7 +1833,10 @@ private void handleShowDynamicPartition() { dynamicPartitionScheduler.getRuntimeInfo(olapTable.getId(), DynamicPartitionScheduler.LAST_SCHEDULER_TIME), dynamicPartitionScheduler.getRuntimeInfo(olapTable.getId(), DynamicPartitionScheduler.DYNAMIC_PARTITION_STATE), dynamicPartitionScheduler.getRuntimeInfo(olapTable.getId(), DynamicPartitionScheduler.CREATE_PARTITION_MSG), - dynamicPartitionScheduler.getRuntimeInfo(olapTable.getId(), DynamicPartitionScheduler.DROP_PARTITION_MSG))); + dynamicPartitionScheduler.getRuntimeInfo(olapTable.getId(), DynamicPartitionScheduler.DROP_PARTITION_MSG), + dynamicPartitionProperty.getSortedReservedHistoryPeriods(unsortedReservedHistoryPeriods, dynamicPartitionProperty.getTimeUnit().toUpperCase()))); + } catch (DdlException e) { + e.printStackTrace(); } finally { olapTable.readUnlock(); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java index efc2685d704e78..6c36ba8cf53656 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java @@ -1186,4 +1186,340 @@ public void testRuntimeInfo() throws Exception { scheduler.removeRuntimeInfo(tableId); Assert.assertTrue(scheduler.getRuntimeInfo(tableId, key1) == FeConstants.null_string); } + + @Test + public void testMissReservedHistoryPeriods() throws Exception { + String createOlapTblStmt = "CREATE TABLE test.`dynamic_partition_miss_reserved_history_periods` (\n" + + " `k1` date NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-06-01\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-12-01\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"day\",\n" + + "\"dynamic_partition.prefix\" = \"p\"\n" + + ");"; + createTable(createOlapTblStmt); + OlapTable table = (OlapTable) Catalog.getCurrentCatalog().getDbOrAnalysisException("default_cluster:test").getTableOrAnalysisException("dynamic_partition_miss_reserved_history_periods"); + Assert.assertEquals("NULL", table.getTableProperty().getDynamicPartitionProperty().getReservedHistoryPeriods()); + } + + @Test + public void testNormalReservedHisrotyPeriods() throws Exception { + String createOlapTblStmt = "CREATE TABLE test.`dynamic_partition_normal_reserved_history_periods` (\n" + + " `k1` date NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-06-01\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-12-01\"),\n" + + "PARTITION p4 VALUES LESS THAN (\"2020-06-01\"),\n" + + "PARTITION p5 VALUES LESS THAN (\"2020-06-20\"),\n" + + "PARTITION p6 VALUES LESS THAN (\"2020-10-25\"),\n" + + "PARTITION p7 VALUES LESS THAN (\"2020-11-01\"),\n" + + "PARTITION p8 VALUES LESS THAN (\"2020-11-11\"),\n" + + "PARTITION p9 VALUES LESS THAN (\"2020-11-21\"),\n" + + "PARTITION p10 VALUES LESS THAN (\"2021-04-20\"),\n" + + "PARTITION p11 VALUES LESS THAN (\"2021-05-20\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"day\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[2020-06-01,2020-06-20],[2020-10-25,2020-11-15],[2021-06-01,2021-06-20]\"\n" + + ");"; + createTable(createOlapTblStmt); + OlapTable table = (OlapTable) Catalog.getCurrentCatalog().getDbOrAnalysisException("default_cluster:test").getTableOrAnalysisException("dynamic_partition_normal_reserved_history_periods"); + Assert.assertEquals("[2020-06-01,2020-06-20],[2020-10-25,2020-11-15],[2021-06-01,2021-06-20]", table.getTableProperty().getDynamicPartitionProperty().getReservedHistoryPeriods()); + Assert.assertEquals(table.getAllPartitions().size(), 9); + + String createOlapTblStmt2 = "CREATE TABLE test.`dynamic_partition_normal_reserved_history_periods2` (\n" + + " `k1` datetime NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01 00:00:00\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-01-01 03:00:00\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-01-01 04:00:00\"),\n" + + "PARTITION p4 VALUES LESS THAN (\"2020-01-01 08:00:00\"),\n" + + "PARTITION p5 VALUES LESS THAN (\"2020-06-20 00:00:00\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"hour\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[2014-01-01 00:00:00,2014-01-01 03:00:00]\"\n" + + ");"; + createTable(createOlapTblStmt2); + OlapTable table2 = (OlapTable) Catalog.getCurrentCatalog().getDbOrAnalysisException("default_cluster:test").getTableOrAnalysisException("dynamic_partition_normal_reserved_history_periods2"); + Assert.assertEquals("[2014-01-01 00:00:00,2014-01-01 03:00:00]", table2.getTableProperty().getDynamicPartitionProperty().getReservedHistoryPeriods()); + Assert.assertEquals(table2.getAllPartitions().size(), 6); + + String createOlapTblStmt3 = "CREATE TABLE test.`dynamic_partition_normal_reserved_history_periods3` (\n" + + " `k1` int NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p202127 VALUES [(\"20200527\"), (\"20200628\"))\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k2`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"1\",\n" + + "\"dynamic_partition.time_unit\" = \"day\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[2020-06-01,2020-06-30]\"\n" + + ");"; + createTable(createOlapTblStmt3); + OlapTable table3 = (OlapTable) Catalog.getCurrentCatalog().getDbOrAnalysisException("default_cluster:test").getTableOrAnalysisException("dynamic_partition_normal_reserved_history_periods3"); + Assert.assertEquals("[2020-06-01,2020-06-30]", table3.getTableProperty().getDynamicPartitionProperty().getReservedHistoryPeriods()); + Assert.assertEquals(table3.getAllPartitions().size(), 5); + } + + @Test + public void testInvalidReservedHistoryPeriods() throws Exception { + String createOlapTblStmt1 = "CREATE TABLE test.`dynamic_partition_invalid_reserved_history_periods1` (\n" + + " `k1` date NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-06-01\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-12-01\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"day\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[20210101,2021-10-10]\"\n" + + ");"; + ExceptionChecker.expectThrowsWithMsg(DdlException.class, + "errCode = 2, detailMessage = Invalid \" dynamic_partition.reserved_history_periods \" value [20210101,2021-10-10]. " + + "It must be like \"[yyyy-MM-dd,yyyy-MM-dd],[...,...]\" while time_unit is DAY/WEEK/MONTH or " + + "\"[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]\" while time_unit is HOUR.", + () -> createTable(createOlapTblStmt1)); + + String createOlapTblStmt2 = "CREATE TABLE test.`dynamic_partition_invalid_reserved_history_periods2` (\n" + + " `k1` date NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-06-01\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-12-01\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"day\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[0000-00-00,2021-10-10]\"\n" + + ");"; + ExceptionChecker.expectThrowsWithMsg(DdlException.class, + "errCode = 2, detailMessage = Invalid dynamic_partition.reserved_history_periods value. " + + "It must be correct DATE value " + + "\"[yyyy-MM-dd,yyyy-MM-dd],[...,...]\" while time_unit is DAY/WEEK/MONTH or " + + "\"[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]\" while time_unit is HOUR.", + () -> createTable(createOlapTblStmt2)); + } + + @Test + public void testReservedHistoryPeriodsValidate() throws Exception { + String createOlapTblStmt1 = "CREATE TABLE test.`dynamic_partition_reserved_history_periods_validate1` (\n" + + " `k1` date NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-06-01\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-12-01\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"day\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[2021-01-01,]\"\n" + + ");"; + ExceptionChecker.expectThrowsWithMsg(DdlException.class, + "errCode = 2, detailMessage = Invalid \" dynamic_partition.reserved_history_periods \" value [2021-01-01,]. " + + "It must be like " + + "\"[yyyy-MM-dd,yyyy-MM-dd],[...,...]\" while time_unit is DAY/WEEK/MONTH " + + "or \"[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]\" while time_unit is HOUR.", + () -> createTable(createOlapTblStmt1)); + + String createOlapTblStmt2 = "CREATE TABLE test.`dynamic_partition_reserved_history_periods_validate2` (\n" + + " `k1` date NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-06-01\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-12-01\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"day\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[,2021-01-01]\"\n" + + ");"; + ExceptionChecker.expectThrowsWithMsg(DdlException.class, + "errCode = 2, detailMessage = Invalid \" dynamic_partition.reserved_history_periods \" value [,2021-01-01]. " + + "It must be like " + + "\"[yyyy-MM-dd,yyyy-MM-dd],[...,...]\" while time_unit is DAY/WEEK/MONTH or " + + "\"[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]\" while time_unit is HOUR.", + () -> createTable(createOlapTblStmt2)); + + String createOlapTblStmt3 = "CREATE TABLE test.`dynamic_partition_reserved_history_periods_validate3` (\n" + + " `k1` date NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-06-01\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-12-01\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"day\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[2020-01-01,2020-03-01],[2021-10-01,2021-09-01]\"\n" + + ");"; + ExceptionChecker.expectThrowsWithMsg(DdlException.class, + "errCode = 2, detailMessage = The first date is larger than the second date, [2021-10-01,2021-09-01] is invalid.", + () -> createTable(createOlapTblStmt3)); + + String createOlapTblStmt4 = "CREATE TABLE test.`dynamic_partition_reserved_history_periods_validate4` (\n" + + " `k1` datetime NULL COMMENT \"\",\n" + + " `k2` int NULL COMMENT \"\",\n" + + " `k3` smallint NULL COMMENT \"\",\n" + + " `v1` varchar(2048) NULL COMMENT \"\",\n" + + " `v2` datetime NULL COMMENT \"\"\n" + + ") ENGINE=OLAP\n" + + "DUPLICATE KEY(`k1`, `k2`, `k3`)\n" + + "COMMENT \"OLAP\"\n" + + "PARTITION BY RANGE (k1)\n" + + "(\n" + + "PARTITION p1 VALUES LESS THAN (\"2014-01-01 00:00:00\"),\n" + + "PARTITION p2 VALUES LESS THAN (\"2014-06-01 00:00:00\"),\n" + + "PARTITION p3 VALUES LESS THAN (\"2014-12-01 00:00:00\")\n" + + ")\n" + + "DISTRIBUTED BY HASH(`k1`) BUCKETS 32\n" + + "PROPERTIES (\n" + + "\"replication_num\" = \"1\",\n" + + "\"dynamic_partition.enable\" = \"true\",\n" + + "\"dynamic_partition.start\" = \"-3\",\n" + + "\"dynamic_partition.end\" = \"3\",\n" + + "\"dynamic_partition.buckets\" = \"3\",\n" + + "\"dynamic_partition.time_unit\" = \"hour\",\n" + + "\"dynamic_partition.prefix\" = \"p\",\n" + + "\"dynamic_partition.reserved_history_periods\" = \"[2020-01-01,2020-03-01]\"\n" + + ");"; + ExceptionChecker.expectThrowsWithMsg(DdlException.class, + "errCode = 2, detailMessage = Invalid \" dynamic_partition.reserved_history_periods \" value [2020-01-01,2020-03-01]. " + + "It must be like " + + "\"[yyyy-MM-dd,yyyy-MM-dd],[...,...]\" while time_unit is DAY/WEEK/MONTH " + + "or \"[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]\" while time_unit is HOUR.", + () -> createTable(createOlapTblStmt4)); + } }