[Dynamic Partition] reserve specific history periods by dynamic partition. by Henry2SS · Pull Request #6554 · apache/doris

Henry2SS · 2021-09-02T10:41:09Z

Proposed changes

Add RESERVED_HISTORY_STARTS and RESERVED_HISTORY_ENDS.
Fixes #6514

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
[x ] New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation Update (if none of the other choices apply)
Code refactor (Modify the code structure, format the code, etc...)
Optimization. Including functional usability improvements and performance improvements.
Dependency. Such as changes related to third-party components.
Other.

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[ x] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If these changes need document changes, I have updated the document
Any dependent changes have been merged

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Henry2SS · 2021-09-02T10:43:27Z

I will add some unit-tests later.
And I have already compiled and validated this function in my own clusters.

morningman · 2021-09-06T05:00:33Z

hi @Henry2SS please update the document

Henry2SS · 2021-09-06T10:48:43Z

hi @Henry2SS please update the document

Done.

morningman · 2021-09-08T13:35:31Z

docs/zh-CN/administrator-guide/dynamic-partition.md

+
+    ```
+    ["2020-06-01","2020-06-20"),
+    ["2020-10-31","2020-11-15")


Why using closed open range? Not closed range?

morningman · 2021-09-08T14:23:43Z

fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java

+    private static void checkReservedHistoryStarts(String reservedHistoryStarts) throws DdlException{
+        String[] starts = reservedHistoryStarts.split(",");
+        if (starts.length == 0) {
+            //ErrorReport.reportDdlException(ErrorCode.ERROR_DYNAMIC_PARTITION_RESERVED_HISTORY_STARTS_EMPTY);


Why not using ErrorReport.reportDdlException?

Henry2SS · 2021-09-09T06:33:28Z

Hi, @morningman changed reserved history periods from closedOpen to closed. Thanks for your advice.

caiconghui · 2021-09-12T08:45:28Z

how about comibine two configs into one config, use reserved_history_list instead of reserved_history_starts and reserved_history_ends, and we can set the property like "[2021-01-01, 2021-01-31] , [2021-02-03, 2021-02-15]", and use pattern to check the property and extract property

Henry2SS · 2021-09-13T11:03:55Z

how about comibine two configs into one config, use reserved_history_list instead of reserved_history_starts and reserved_history_ends, and we can set the property like "[2021-01-01, 2021-01-31] , [2021-02-03, 2021-02-15]", and use pattern to check the property and extract property

Great idea. And i've finished changing it to reserved_history_periods. And ut will be modified later.

caiconghui · 2021-09-14T03:46:09Z

please rebase master to solve the conflicts problem

Henry2SS · 2021-09-14T06:26:22Z

please rebase master to solve the conflicts problem

Done.

Henry2SS · 2021-09-24T11:03:30Z

@morningman Hi, it could reserve history periods when the TIME_UNIT is set to HOUR now. And modified docs and add two more unit-test to do validation.

morningman

LGTM

github-actions · 2021-09-28T03:39:03Z

PR approved by at least one committer and no changes requested.

github-actions · 2021-09-28T03:39:04Z

PR approved by anyone and no changes requested.

Henry2SS force-pushed the whz_dynamic_partition branch 4 times, most recently from f779054 to bda1918 Compare September 3, 2021 12:15

morningman added area/dynamic-partition kind/feature Categorizes issue or PR as related to a new feature. labels Sep 6, 2021

Henry2SS force-pushed the whz_dynamic_partition branch from 88f8036 to d10cf87 Compare September 7, 2021 09:37

Henry2SS changed the title ~~modify dynamic partition.~~ [Dynamic Partition] reserve specific history periods by dynamic partition. Sep 8, 2021

morningman reviewed Sep 8, 2021

View reviewed changes

Henry2SS force-pushed the whz_dynamic_partition branch from a1605da to d12bccf Compare September 14, 2021 03:42

Henry2SS force-pushed the whz_dynamic_partition branch from a0f623b to f5726d1 Compare September 14, 2021 04:08

Henry2SS and others added 4 commits September 14, 2021 13:04

modify dynamic partition.

a3d6a79

add some ut and add documents

bc7b70f

fix

78a7214

change reserved history period range closedOpen to closed.

fed84fb

Henry2SS force-pushed the whz_dynamic_partition branch 3 times, most recently from d976679 to 8436d16 Compare September 14, 2021 05:15

change it to list

5a0d881

Henry2SS force-pushed the whz_dynamic_partition branch from 8436d16 to 5a0d881 Compare September 14, 2021 05:25

fix

5331b4e

Henry2SS force-pushed the whz_dynamic_partition branch from 864096b to 5331b4e Compare September 14, 2021 05:48

Henry2SS added 2 commits September 15, 2021 11:22

fix

2f5e360

change Pattern

b605066

Henry2SS force-pushed the whz_dynamic_partition branch from 8bc295f to b605066 Compare September 16, 2021 09:49

refactor

91c0e3b

Henry2SS force-pushed the whz_dynamic_partition branch from 8226393 to 91c0e3b Compare September 16, 2021 11:20

Henry2SS added 3 commits September 17, 2021 14:52

fix

6f5639a

fix2

7fe9cd9

change default value of RESERVED_HISTORY_PERIODS

346cc33

Henry2SS force-pushed the whz_dynamic_partition branch from c58a073 to 346cc33 Compare September 22, 2021 09:37

Henry2SS mentioned this pull request Sep 22, 2021

[Release] Prepare for release 0.15 #6703

Closed

3 tasks

fix

f7e4266

Henry2SS requested a review from morningman September 23, 2021 12:38

Henry2SS added 2 commits September 24, 2021 14:38

modify HOUR

89fc952

modify HOUR fix

1f61d20

Henry2SS force-pushed the whz_dynamic_partition branch 2 times, most recently from 3f6d2db to 1233b3b Compare September 26, 2021 07:58

change contains to RangeUtils.isTwoRangesIntesect && add more ut

8a23015

Henry2SS force-pushed the whz_dynamic_partition branch from 1233b3b to 8a23015 Compare September 26, 2021 07:59

morningman approved these changes Sep 28, 2021

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 28, 2021

github-actions bot added the reviewed label Sep 28, 2021

morningman merged commit cdf9f9e into apache:master Sep 28, 2021

morningman mentioned this pull request Oct 10, 2021

Release note 0.15.0 #6806

Closed

Conversation

Henry2SS commented Sep 2, 2021

Proposed changes

Types of changes

Checklist

Further comments

Uh oh!

Henry2SS commented Sep 2, 2021

Uh oh!

morningman commented Sep 6, 2021

Uh oh!

Henry2SS commented Sep 6, 2021

Uh oh!

morningman Sep 8, 2021

Choose a reason for hiding this comment

Uh oh!

morningman Sep 8, 2021

Choose a reason for hiding this comment

Uh oh!

Henry2SS commented Sep 9, 2021

Uh oh!

caiconghui commented Sep 12, 2021

Uh oh!

Henry2SS commented Sep 13, 2021

Uh oh!

caiconghui commented Sep 14, 2021

Uh oh!

Henry2SS commented Sep 14, 2021

Uh oh!

Henry2SS commented Sep 24, 2021

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 28, 2021

Uh oh!

github-actions bot commented Sep 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants