[Bug] Create dynamic partition table failed with enable create_history_partition and not specify the start value

[harveyyue]

fix the issue #5995
Add the property "dynamic_partition.history_partition_num" to specify the history partition number when enable create_history_partition to fix the invalid date format value
and add these two properties to docs

Proposed changes

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

• [ x] Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)
• Code refactor (Modify the code structure, format the code, etc...)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• [ x] I have created an issue on (Fix [bug] Create dynamic partition table failed with enable create_history_partition and not specify the start value #5995) and described the bug/feature there in detail
• Compiling and unit tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• If these changes need document changes, I have updated the document
• Any dependent changes have been merged

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[morningman]

Hi @harveyyue Thank for your PR.
But I think if neither start nor history_partition_num is set, we should not create any history partition.
the there should be no default value for history_partition_num.

[harveyyue]

Hi @harveyyue Thank for your PR.
But I think if neither start nor history_partition_num is set, we should not create any history partition.
the there should be no default value for history_partition_num.

when enable the property "dynamic_partition.create_history_partition" and not set property "dynamic_partition.start"
will getting following date format error, create table failed. because the default value of start is Integer.MIN_VALUE(-2147483648)

one more thing, i want to keep 100 partitions with set dynamic_partition.start=-100, and want to create 20 old partitions rather than getting 100 old partitions.

2021-06-08 12:19:21,255 WARN (doris-mysql-nio-pool-0|147) [DynamicPartitionScheduler.getAddPartitionClause():169] Error in gen addPartitionKeyRange. Error=errCode = 2, detailMessage = date literal [+178951742-12-01] is invalid, db: default_cluster:poc, table: dwd_tf_order2

[qidaye]

Hi @harveyyue , thanks for your PR.

one more thing, i want to keep 100 partitions with set dynamic_partition.start=-100, and want to create 20 old partitions rather than getting 100 old partitions.

Depending on your needs, it is necessary to add the dynamic_partition.history_partition_num parameter. But there is no need to set default value for history_partition_num.

Control the behavior of automatic history partition creation based on the three parameters, start,create_history_partition,history_partition_num.

Here's the suggested logic pseudocode:

expect_create_partition_num;

if (!create_history_partition) {
  start = 0;
  expect_create_partition_num = end - start;
} else {
  if (history_partition_num != 0) {
    expect_create_partition_num = end - MAX(start, -history_partition_num);
  } else {
    if (start == Integer.MIN_VALUE) {
      throw exception
    }
    expect_create_partition_num = end - start;
}

if (expect_create_partition_num > max_dynamic_partition_num) {
  throw exception
}

You can add this logic in function: https://github.com/apache/incubator-doris/blob/ba84eacb8c2d0d70bff9fc43f7d13b7382384aed/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java#L295

Here is detail code:
https://github.com/apache/incubator-doris/blob/ba84eacb8c2d0d70bff9fc43f7d13b7382384aed/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java#L351-L359

By the way, you also need to update dynamic partition doc https://github.com/apache/incubator-doris/blob/master/docs/en/administrator-guide/dynamic-partition.md, for both Chinese and English.

If you have any questions , feel free to reach out.

[harveyyue]

hi @qidaye
I have completed these code changes according your suggestion, I use "git rebase" to merge these two commits to one, and resubmit the following pr. Please help to review it.

#6129

[qidaye]

hi @qidaye
I have completed these code changes according your suggestion, I use "git rebase" to merge these two commits to one, and resubmit the following pr. Please help to review it.

#6129

Thanks for such a quick improvement, I'll look into it.