Skip to content

[DynamicPartition] support retention for specified period of history time. #6514

@Henry2SS

Description

@Henry2SS

Is your feature request related to a problem? Please describe.
In some situations, we need to keep partitions of some periods of time. To do so, we need to set dynamic_partition.start very big, and it will take lots of disk space.
For example, we need to keep partitions between p20191031 to p20191120, now we should set dynamic_partition.start = -730, and in some days, they will be dropped day by day. In fact, we just need to keep 20+ partitions and never drop them.

Describe the solution you'd like
Add two properties of DynamicPartition. dynamic_partition.reserved_history_starts and dynamic_partition.reserved_history_ends.
To support different periods of time, dynamic_partition.reserved_history_starts should be like "2021-01-01, 2021-03-01, 2021-05-01, ..." and correspondingly, dynamic_partition.reserved_history_ends should be like "2021-01-31, 2021-03-11, 2021-05-31, ...". And the periods of time that is reserved should be "2021-01-01" - "2021-01-31", "2021-03-01" - "2021-03-31", "2021-05-01" - "2021-05-31" and ....
And before dropping partition operation, we just need to exclude those partitions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions