Skip to content

[Dynamic Partition] reserve specific history periods by dynamic partition. #6554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
morningman merged 16 commits into from
Sep 28, 2021
Merged

[Dynamic Partition] reserve specific history periods by dynamic partition. #6554

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a3d6a79
modify dynamic partition.
Henry2SS Sep 2, 2021
bc7b70f
add some ut and add documents
Henry2SS Sep 3, 2021
78a7214
fix
Henry2SS Sep 7, 2021
fed84fb
change reserved history period range closedOpen to closed.
Henry2SS Sep 9, 2021
5a0d881
change it to list
Henry2SS Sep 13, 2021
5331b4e
fix
Henry2SS Sep 14, 2021
2f5e360
fix
Henry2SS Sep 15, 2021
b605066
change Pattern
Henry2SS Sep 16, 2021
91c0e3b
refactor
Henry2SS Sep 16, 2021
6f5639a
fix
Henry2SS Sep 17, 2021
7fe9cd9
fix2
Henry2SS Sep 18, 2021
346cc33
change default value of RESERVED_HISTORY_PERIODS
Henry2SS Sep 22, 2021
f7e4266
fix
Henry2SS Sep 23, 2021
89fc952
modify HOUR
Henry2SS Sep 24, 2021
1f61d20
modify HOUR fix
Henry2SS Sep 24, 2021
8a23015
change contains to RangeUtils.isTwoRangesIntesect && add more ut
Henry2SS Sep 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 36 additions & 10 deletions docs/en/administrator-guide/dynamic-partition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,32 @@ The rules of dynamic partition are prefixed with `dynamic_partition.`:
p20210523: ["2021-05-23", "2021-05-24") storage_medium=SSD storage_cooldown_time=2021-05-25 00:00:00
```

* `dynamic_partition.reserved_history_periods`

The range of reserved history periods. It should be in the form of `[yyyy-MM-dd,yyyy-MM-dd],[...,...]` while the `dynamic_partition.time_unit` is "DAY, WEEK, and MONTH". And it should be in the form of `[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]` while the dynamic_partition.time_unit` is "HOUR". And no more spaces expected. The default value is `"NULL"`, which means it is not set.

Let us give an example. Suppose today is 2021-09-06，partitioned by day, and the properties of dynamic partition are set to:

```time_unit="DAY/WEEK/MONTH", end=3, start=-3, reserved_history_periods="[2020-06-01,2020-06-20],[2020-10-31,2020-11-15]"```.

The the system will automatically reserve following partitions in following period :

```
["2020-06-01","2020-06-20"],
["2020-10-31","2020-11-15"]
```
or

```time_unit="HOUR", end=3, start=-3, reserved_history_periods="[2020-06-01 00:00:00,2020-06-01 03:00:00]"```.

The the system will automatically reserve following partitions in following period :

```
["2020-06-01 00:00:00","2020-06-01 03:00:00"]
```

Otherwise, every `[...,...]` in `reserved_history_periods` is a couple of properties, and they should be set at the same time. And the first date can't be larger than the second one.

#### Create History Partition Rules

When `create_history_partition` is `true`, i.e. history partition creation is enabled, Doris determines the number of history partitions to be created based on `dynamic_partition.start` and `dynamic_partition.history_partition_num`.
Expand Down Expand Up @@ -357,16 +383,16 @@ You can further view the scheduling of dynamic partitioned tables by using the f

```
mysql> SHOW DYNAMIC PARTITION TABLES;
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+
| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg |
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+
| d3 | true | WEEK | -3 | 3 | p | 1 | MONDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d5 | true | DAY | -7 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d4 | true | WEEK | -3 | 3 | p | 1 | WEDNESDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d6 | true | MONTH | -2147483648 | 2 | p | 8 | 3rd | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d2 | true | DAY | -3 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d7 | true | MONTH | -2147483648 | 5 | p | 8 | 24th | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+
| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg | ReservedHistoryPeriods |
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+
| d3 | true | WEEK | -3 | 3 | p | 1 | MONDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | [2021-12-01,2021-12-31] |
| d5 | true | DAY | -7 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
| d4 | true | WEEK | -3 | 3 | p | 1 | WEDNESDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
| d6 | true | MONTH | -2147483648 | 2 | p | 8 | 3rd | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
| d2 | true | DAY | -3 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
| d7 | true | MONTH | -2147483648 | 5 | p | 8 | 24th | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+
7 rows in set (0.02 sec)
```

Expand Down
5 changes: 4 additions & 1 deletion docs/en/administrator-guide/operation/doris-error-code.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,4 +172,7 @@ under the License.
| 5072 | The dynamic partition copy value is not a valid number |
| 5073 | The original created table stmt is empty |
| 5074 | Create historical dynamic partition parameters: create_history_partition is invalid, what is expected is: true or false |

| 5076 | The specified dynamic partition reserved_history_periods is null or empty |
| 5077 | The specified dynamic partition reserved_history_periods is invalid |
| 5078 | The length of specified dynamic partition reserved_history_periods must have pairs of date value |
| 5079 | The specified dynamic partition reserved_history_periods' first date is larger than the second one |
1 change: 1 addition & 0 deletions ...l-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ distribution_info
* `dynamic_partition.buckets`: Used to specify the number of partition buckets that are automatically created.
* `dynamic_partition.create_history_partition`: Whether to create a history partition.
* `dynamic_partition.history_partition_num`: Specify the number of historical partitions to be created.
* `dynamic_partition.reserved_history_periods`: Used to specify the range of reserved history periods.

### Example

Expand Down
1 change: 1 addition & 0 deletions docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,7 @@ Syntax:
dynamic_partition.buckets: specifies the number of partition buckets that are automatically created
dynamic_partition.create_history_partition: specifies whether create history partitions, default value is false
dynamic_partition.history_partition_num: used to specify the number of history partitions when enable create_history_partition
dynamic_partition.reserved_history_periods: Used to specify the range of reserved history periods
```
5) You can create multiple Rollups in bulk when building a table
grammar:
Expand Down
47 changes: 37 additions & 10 deletions docs/zh-CN/administrator-guide/dynamic-partition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,33 @@ under the License.
p20210523：["2021-05-23", "2021-05-24") storage_medium=SSD storage_cooldown_time=2021-05-25 00:00:00
```

* `dynamic_partition.reserved_history_periods`

需要保留的历史分区的时间范围。当`dynamic_partition.time_unit` 设置为 "DAY/WEEK/MONTH" 时，需要以 `[yyyy-MM-dd,yyyy-MM-dd],[...,...]` 格式进行设置。当`dynamic_partition.time_unit` 设置为 "HOUR" 时，需要以 `[yyyy-MM-dd HH:mm:ss,yyyy-MM-dd HH:mm:ss],[...,...]` 的格式来进行设置。如果不设置，默认为 `"NULL"`。

我们举例说明。假设今天是 2021-09-06，按天分类，动态分区的属性设置为：

```time_unit="DAY/WEEK/MONTH", end=3, start=-3, reserved_history_periods="[2020-06-01,2020-06-20],[2020-10-31,2020-11-15]"```。

则系统会自动保留：

```
["2020-06-01","2020-06-20"],
["2020-10-31","2020-11-15"]
```

或者

```time_unit="HOUR", end=3, start=-3, reserved_history_periods="[2020-06-01 00:00:00,2020-06-01 03:00:00]"```.

则系统会自动保留：

```
["2020-06-01 00:00:00","2020-06-01 03:00:00"]
```

这两个时间段的分区。其中，`reserved_history_periods` 的每一个 `[...,...]` 是一对设置项，两者需要同时被设置，且第一个时间不能大于第二个时间``。

#### 创建历史分区规则

当 `create_history_partition` 为 `true`，即开启创建历史分区功能时，Doris 会根据 `dynamic_partition.start` 和 `dynamic_partition.history_partition_num` 来决定创建历史分区的个数。
Expand Down Expand Up @@ -355,16 +382,16 @@ p20200521: ["2020-05-21", "2020-05-22")

```
mysql> SHOW DYNAMIC PARTITION TABLES;
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+
| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg |
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+
| d3 | true | WEEK | -3 | 3 | p | 1 | MONDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d5 | true | DAY | -7 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d4 | true | WEEK | -3 | 3 | p | 1 | WEDNESDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d6 | true | MONTH | -2147483648 | 2 | p | 8 | 3rd | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d2 | true | DAY | -3 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
| d7 | true | MONTH | -2147483648 | 5 | p | 8 | 24th | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A |
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+
| TableName | Enable | TimeUnit | Start | End | Prefix | Buckets | StartOf | LastUpdateTime | LastSchedulerTime | State | LastCreatePartitionMsg | LastDropPartitionMsg | ReservedHistoryPeriods |
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+
| d3 | true | WEEK | -3 | 3 | p | 1 | MONDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | [2021-12-01,2021-12-31] |
| d5 | true | DAY | -7 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
| d4 | true | WEEK | -3 | 3 | p | 1 | WEDNESDAY | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
| d6 | true | MONTH | -2147483648 | 2 | p | 8 | 3rd | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
| d2 | true | DAY | -3 | 3 | p | 32 | N/A | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
| d7 | true | MONTH | -2147483648 | 5 | p | 8 | 24th | N/A | 2020-05-25 14:29:24 | NORMAL | N/A | N/A | NULL |
+-----------+--------+----------+-------------+------+--------+---------+-----------+----------------+---------------------+--------+------------------------+----------------------+-------------------------+
7 rows in set (0.02 sec)
```

Expand Down
4 changes: 4 additions & 0 deletions docs/zh-CN/administrator-guide/operation/doris-error-code.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,4 +172,8 @@ under the License.
| 5072 | 动态分区副本值不是有效的数字 |
| 5073 | 原始创建表stmt为空 |
| 5074 | 创建历史动态分区参数：create_history_partition无效，期望的是：true或者false |
| 5076 | 指定的保留历史分区时间段为空 |
| 5077 | 指定的保留历史分区时间段无效 |
| 5078 | 指定的保留历史分区时间段必须是成对的时间 |
| 5079 | 指定的保留历史分区时间段对应位置的第一个时间比第二个时间大（起始时间大于结束时间） |

1 change: 1 addition & 0 deletions ...l-reference-v2/sql-statements/Data-Definition-Statements/Create/CREATE-TABLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ distribution_info
* `dynamic_partition.buckets`: 用于指定自动创建的分区分桶数量。
* `dynamic_partition.create_history_partition`: 是否创建历史分区。
* `dynamic_partition.history_partition_num`: 指定创建历史分区的数量。
* `dynamic_partition.reserved_history_periods`: 用于指定保留的历史分区的时间段。

### Example

Expand Down
1 change: 1 addition & 0 deletions docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,7 @@ under the License.
dynamic_partition.buckets: 用于指定自动创建的分区分桶数量
dynamic_partition.create_history_partition: 用于创建历史分区功能是否开启。默认为 false。
dynamic_partition.history_partition_num: 当开启创建历史分区功能时，用于指定创建历史分区数量。
dynamic_partition.reserved_history_periods: 用于指定保留的历史分区的时间段。

5) 建表时可以批量创建多个 Rollup
语法：
Expand Down
3 changes: 2 additions & 1 deletion fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDynamicPartitionStmt.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ public class ShowDynamicPartitionStmt extends ShowStmt {
.addColumn(new Column("State", ScalarType.createVarchar(20)))
.addColumn(new Column("LastCreatePartitionMsg", ScalarType.createVarchar(20)))
.addColumn(new Column("LastDropPartitionMsg", ScalarType.createVarchar(20)))
.addColumn(new Column("ReservedHistoryPeriods", ScalarType.createVarchar(20)))
.build();

ShowDynamicPartitionStmt(String db) {
Expand Down Expand Up @@ -95,4 +96,4 @@ public ShowResultSetMetaData getMetaData() {
public RedirectStatus getRedirectStatus() {
return RedirectStatus.FORWARD_NO_SYNC;
}
}
}
17 changes: 16 additions & 1 deletion fe/fe-core/src/main/java/org/apache/doris/catalog/DynamicPartitionProperty.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@

import org.apache.doris.analysis.TimestampArithmeticExpr.TimeUnit;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.util.DynamicPartitionUtil;
import org.apache.doris.common.util.DynamicPartitionUtil.StartOfDate;
import org.apache.doris.common.util.PropertyAnalyzer;
import org.apache.doris.common.util.TimeUtils;
Expand All @@ -43,11 +45,13 @@ public class DynamicPartitionProperty {
public static final String CREATE_HISTORY_PARTITION = "dynamic_partition.create_history_partition";
public static final String HISTORY_PARTITION_NUM = "dynamic_partition.history_partition_num";
public static final String HOT_PARTITION_NUM = "dynamic_partition.hot_partition_num";
public static final String RESERVED_HISTORY_PERIODS = "dynamic_partition.reserved_history_periods";

public static final int MIN_START_OFFSET = Integer.MIN_VALUE;
public static final int MAX_END_OFFSET = Integer.MAX_VALUE;
public static final int NOT_SET_REPLICATION_NUM = -1;
public static final int NOT_SET_HISTORY_PARTITION_NUM = -1;
public static final String NOT_SET_RESERVED_HISTORY_PERIODS = "NULL";

private boolean exist;

Expand All @@ -67,6 +71,7 @@ public class DynamicPartitionProperty {
// This property are used to describe the number of partitions that need to be reserved on the high-speed storage.
// If not set, default is 0
private int hotPartitionNum;
private String reservedHistoryPeriods;

public DynamicPartitionProperty(Map<String, String> properties) {
if (properties != null && !properties.isEmpty()) {
Expand All @@ -83,6 +88,7 @@ public DynamicPartitionProperty(Map<String, String> properties) {
this.createHistoryPartition = Boolean.parseBoolean(properties.get(CREATE_HISTORY_PARTITION));
this.historyPartitionNum = Integer.parseInt(properties.getOrDefault(HISTORY_PARTITION_NUM, String.valueOf(NOT_SET_HISTORY_PARTITION_NUM)));
this.hotPartitionNum = Integer.parseInt(properties.getOrDefault(HOT_PARTITION_NUM, "0"));
this.reservedHistoryPeriods = properties.getOrDefault(RESERVED_HISTORY_PERIODS, NOT_SET_RESERVED_HISTORY_PERIODS);
createStartOfs(properties);
} else {
this.exist = false;
Expand Down Expand Up @@ -180,6 +186,14 @@ public ReplicaAllocation getReplicaAllocation() {
return replicaAlloc;
}

public String getReservedHistoryPeriods() {
return reservedHistoryPeriods;
}

public String getSortedReservedHistoryPeriods(String reservedHistoryPeriods, String timeUnit) throws DdlException {
return DynamicPartitionUtil.sortedListedToString(reservedHistoryPeriods, timeUnit);
}

/**
* use table replication_num as dynamic_partition.replication_num default value
*/
Expand All @@ -195,7 +209,8 @@ public String getProperties(ReplicaAllocation tableReplicaAlloc) {
",\n\"" + BUCKETS + "\" = \"" + buckets + "\"" +
",\n\"" + CREATE_HISTORY_PARTITION + "\" = \"" + createHistoryPartition + "\"" +
",\n\"" + HISTORY_PARTITION_NUM + "\" = \"" + historyPartitionNum + "\"" +
",\n\"" + HOT_PARTITION_NUM + "\" = \"" + hotPartitionNum + "\"";
",\n\"" + HOT_PARTITION_NUM + "\" = \"" + hotPartitionNum + "\"" +
",\n\"" + RESERVED_HISTORY_PERIODS + "\" = \"" + reservedHistoryPeriods + "\"";
if (getTimeUnit().equalsIgnoreCase(TimeUnit.WEEK.toString())) {
res += ",\n\"" + START_DAY_OF_WEEK + "\" = \"" + startOfWeek.dayOfWeek + "\"";
} else if (getTimeUnit().equalsIgnoreCase(TimeUnit.MONTH.toString())) {
Expand Down
Loading