[feature](backup) add property to remove snapshot before creating repo #25847

w41ter · 2023-10-24T11:07:46Z

Doris is not responsible for managing snapshots, but it needs to clear all snapshots before doing backup/restore regression testing, so a property is added to indicate that existing snapshots need to be cleared when creating a repo. This property is only used during testing and should not be exposed to users.

In addition, a regression test case for backup/restore has been added.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

w41ter · 2023-10-24T11:07:58Z

run buildall

doris-robot · 2023-10-24T11:51:37Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.93 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162114542 Bytes

github-actions · 2023-10-24T12:27:23Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-24T12:27:26Z

PR approved by anyone and no changes requested.

morningman

please also update the doc

fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java

fe/fe-core/src/main/java/org/apache/doris/common/util/S3URI.java

w41ter · 2023-10-25T08:07:55Z

run buildall

morningman

LGTM

github-actions · 2023-10-25T08:19:13Z

PR approved by at least one committer and no changes requested.

w41ter · 2023-10-25T10:45:32Z

run buildall

doris-robot · 2023-10-25T11:58:49Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.54 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162039232 Bytes

w41ter · 2023-10-26T01:27:11Z

run buildall

doris-robot · 2023-10-26T02:10:01Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.82 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162689492 Bytes

w41ter · 2023-10-26T02:29:30Z

run buildall

w41ter · 2023-10-27T08:01:38Z

run buildall

JackDrogon

LGTM

Doris is not responsible for managing snapshots, but it needs to clear all snapshots before doing backup/restore regression testing, so a property is added to indicate that existing snapshots need to be cleared when creating a repo. In addition, a regression test case for backup/restore has been added.

w41ter · 2023-10-27T10:17:44Z

run buildall

doris-robot · 2023-10-27T10:52:06Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.65 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162296434 Bytes

dataroaring

LGTM

github-actions · 2023-10-27T11:56:38Z

PR approved by at least one committer and no changes requested.

apache#25847) Doris is not responsible for managing snapshots, but it needs to clear all snapshots before doing backup/restore regression testing, so a property is added to indicate that existing snapshots need to be cleared when creating a repo. In addition, a regression test case for backup/restore has been added.

* [feature](backup) add property to remove snapshot before creating repo (#25847) Doris is not responsible for managing snapshots, but it needs to clear all snapshots before doing backup/restore regression testing, so a property is added to indicate that existing snapshots need to be cleared when creating a repo. In addition, a regression test case for backup/restore has been added. * [cases](regression-test) Add backup & restore with multi tables test (#26040) * [cases](regression-test) Add backup restore operation test (#26169) 1. restore overwrites an exists table 2. backup & restore with exclude table 3. restore to a new table 4. restore mix exists and new tables 5. restore with alias * [fix](regression) Add auto-expired prefix for backup/restore cases (#26272) * [test](regression) Add backup/restore DB case (#26312)

* [feature](backup) add property to remove snapshot before creating repo (apache#25847) Doris is not responsible for managing snapshots, but it needs to clear all snapshots before doing backup/restore regression testing, so a property is added to indicate that existing snapshots need to be cleared when creating a repo. In addition, a regression test case for backup/restore has been added. * [cases](regression-test) Add backup & restore with multi tables test (apache#26040) * [cases](regression-test) Add backup restore operation test (apache#26169) 1. restore overwrites an exists table 2. backup & restore with exclude table 3. restore to a new table 4. restore mix exists and new tables 5. restore with alias * [fix](regression) Add auto-expired prefix for backup/restore cases (apache#26272) * [test](regression) Add backup/restore DB case (apache#26312)

apache#25847) Doris is not responsible for managing snapshots, but it needs to clear all snapshots before doing backup/restore regression testing, so a property is added to indicate that existing snapshots need to be cleared when creating a repo. In addition, a regression test case for backup/restore has been added.

…itory delete_if_exists is a temporary solution introduced in apache#25847, to avoid concurrent testing conflicts.

…itory delete_if_exists is a temporary solution introduced in apache#25847, to avoid concurrent testing conflicts. Cherry-pick apache#38190

…itory (#38190) delete_if_exists is a temporary solution introduced in #25847, to avoid concurrent testing conflicts.

…itory (#38194) delete_if_exists is a temporary solution introduced in #25847, to avoid concurrent testing conflicts. Cherry-pick #38190

…itory (#38192) delete_if_exists is a temporary solution introduced in #25847, to avoid concurrent testing conflicts. Cherry-pick #38190

…itory (#38190) delete_if_exists is a temporary solution introduced in #25847, to avoid concurrent testing conflicts.

…itory (apache#38194) delete_if_exists is a temporary solution introduced in apache#25847, to avoid concurrent testing conflicts. Cherry-pick apache#38190

dataroaring previously approved these changes Oct 24, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 24, 2023

github-actions bot added the reviewed label Oct 24, 2023

ByteYue approved these changes Oct 24, 2023

View reviewed changes

morningman reviewed Oct 24, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/common/util/S3URI.java Outdated Show resolved Hide resolved

dataroaring added the dev/2.0.3 label Oct 25, 2023

w41ter dismissed dataroaring’s stale review via beb6394 October 25, 2023 08:01

w41ter force-pushed the feature/drop_snapshot_before_create_repo branch from 01647d5 to beb6394 Compare October 25, 2023 08:01

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 25, 2023

w41ter force-pushed the feature/drop_snapshot_before_create_repo branch from beb6394 to 50487b1 Compare October 25, 2023 08:05

w41ter requested review from ByteYue, dataroaring and morningman October 25, 2023 08:07

morningman previously approved these changes Oct 25, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023

w41ter dismissed morningman’s stale review via 88e05d8 October 25, 2023 09:30

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 25, 2023

w41ter force-pushed the feature/drop_snapshot_before_create_repo branch from 88e05d8 to 12120d9 Compare October 26, 2023 01:25

w41ter force-pushed the feature/drop_snapshot_before_create_repo branch from 12120d9 to 91f9159 Compare October 26, 2023 02:29

w41ter force-pushed the feature/drop_snapshot_before_create_repo branch from 25d4327 to f75555f Compare October 27, 2023 07:58

JackDrogon approved these changes Oct 27, 2023

View reviewed changes

w41ter force-pushed the feature/drop_snapshot_before_create_repo branch from f75555f to 43a6d94 Compare October 27, 2023 10:17

dataroaring approved these changes Oct 27, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 27, 2023

dataroaring merged commit 365fdd2 into apache:master Oct 27, 2023

w41ter deleted the feature/drop_snapshot_before_create_repo branch October 27, 2023 13:25

w41ter mentioned this pull request Nov 3, 2023

Cherry-pick backup/restore releated PRs to branch 2.0 #26350

Merged

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 3, 2023

w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 22, 2024

[chore](backup) Remove delete_if_exists properties for creating repos…

ab0e20f

…itory delete_if_exists is a temporary solution introduced in apache#25847, to avoid concurrent testing conflicts.

w41ter mentioned this pull request Jul 22, 2024

[chore](backup) Remove delete_if_exists properties for creating repository #38190

Merged

w41ter mentioned this pull request Jul 22, 2024

[chore](backup) Remove delete_if_exists properties for creating repository #38192

Merged

w41ter mentioned this pull request Jul 22, 2024

[chore](backup) Remove delete_if_exists properties for creating repository #38194

Merged

w41ter added a commit that referenced this pull request Jul 22, 2024

[chore](backup) Remove delete_if_exists properties for creating repos…

10aa083

…itory (#38190) delete_if_exists is a temporary solution introduced in #25847, to avoid concurrent testing conflicts.

dataroaring pushed a commit that referenced this pull request Jul 24, 2024

[chore](backup) Remove delete_if_exists properties for creating repos…

02d8b58

…itory (#38190) delete_if_exists is a temporary solution introduced in #25847, to avoid concurrent testing conflicts.

[feature](backup) add property to remove snapshot before creating repo #25847

[feature](backup) add property to remove snapshot before creating repo #25847

Uh oh!

Conversation

w41ter commented Oct 24, 2023

Proposed changes

Further comments

Uh oh!

w41ter commented Oct 24, 2023

Uh oh!

doris-robot commented Oct 24, 2023

Uh oh!

github-actions bot commented Oct 24, 2023

Uh oh!

github-actions bot commented Oct 24, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

w41ter commented Oct 25, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 25, 2023

Uh oh!

w41ter commented Oct 25, 2023

Uh oh!

doris-robot commented Oct 25, 2023

Uh oh!

w41ter commented Oct 26, 2023

Uh oh!

doris-robot commented Oct 26, 2023

Uh oh!

w41ter commented Oct 26, 2023

Uh oh!

w41ter commented Oct 27, 2023

Uh oh!

JackDrogon left a comment

Choose a reason for hiding this comment

Uh oh!

w41ter commented Oct 27, 2023

Uh oh!

doris-robot commented Oct 27, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants