-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Description
Is your feature request related to a problem? Please describe.
In some situations, we need to keep partitions of some periods of time. To do so, we need to set dynamic_partition.start very big, and it will take lots of disk space.
For example, we need to keep partitions between p20191031 to p20191120, now we should set dynamic_partition.start = -730, and in some days, they will be dropped day by day. In fact, we just need to keep 20+ partitions and never drop them.
Describe the solution you'd like
Add two properties of DynamicPartition. dynamic_partition.reserved_history_starts and dynamic_partition.reserved_history_ends.
To support different periods of time, dynamic_partition.reserved_history_starts should be like "2021-01-01, 2021-03-01, 2021-05-01, ..." and correspondingly, dynamic_partition.reserved_history_ends should be like "2021-01-31, 2021-03-11, 2021-05-31, ...". And the periods of time that is reserved should be "2021-01-01" - "2021-01-31", "2021-03-01" - "2021-03-31", "2021-05-01" - "2021-05-31" and ....
And before dropping partition operation, we just need to exclude those partitions.