[Enhancement](merge-on-write) use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist #24011

bobhan1 · 2023-09-07T03:01:02Z

Proposed changes

For Unique table with merge-on-write enabled, we can directly use delete bitmap to mark the rows with delete sign as deleted when sequence column doesn't exist to eliminate reading delete sign columns in later reads.
This PR can also improve the performance of #21773 if we disable insert ignore statement on mow Unique table with sequence column.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-09-07T03:09:23Z

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 · 2023-09-07T03:24:10Z

run buildall

github-actions · 2023-09-07T03:29:36Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-07T03:38:35Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-07T03:41:45Z

TeamCity be ut coverage result:
Function Coverage: 36.99% (7891/21333)
Line Coverage: 28.94% (63239/218514)
Region Coverage: 27.89% (32894/117933)
Branch Coverage: 24.46% (16891/69064)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d50bf46f3e29338fa37629af1c8406c97a4a7bac_d50bf46f3e29338fa37629af1c8406c97a4a7bac/report/index.html

doris-robot · 2023-09-07T04:35:06Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.92 seconds
stream load tsv: 550 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17161921611 Bytes

github-actions · 2023-09-07T06:22:48Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-07T07:34:12Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-07T08:15:03Z

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 · 2023-09-07T08:28:55Z

run buildall

github-actions · 2023-09-07T08:33:05Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-07T08:34:47Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-07T08:58:29Z

TeamCity be ut coverage result:
Function Coverage: 37.00% (7893/21332)
Line Coverage: 28.95% (63262/218500)
Region Coverage: 27.91% (32913/117938)
Branch Coverage: 24.47% (16899/69070)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0d4230adb762ae8113b7c02a1dda9a86ecaca646_0d4230adb762ae8113b7c02a1dda9a86ecaca646/report/index.html

be/src/olap/rowset/segment_v2/segment_writer.cpp

doris-robot · 2023-09-07T09:26:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.24 seconds
stream load tsv: 539 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17161959788 Bytes

github-actions · 2023-09-07T09:47:47Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-07T09:50:22Z

clang-tidy review says "All clean, LGTM! 👍"

be/src/olap/tablet_meta.h

bobhan1 · 2023-09-07T10:38:17Z

run buildall

github-actions · 2023-09-07T10:41:10Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-07T10:46:29Z

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 · 2023-09-07T10:57:41Z

run buildall

github-actions · 2023-09-08T08:38:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-08T08:38:49Z

PR approved by anyone and no changes requested.

doris-robot · 2023-09-08T08:57:31Z

TeamCity be ut coverage result:
Function Coverage: 36.94% (7897/21378)
Line Coverage: 28.95% (63390/218988)
Region Coverage: 27.89% (32918/118030)
Branch Coverage: 24.47% (16908/69086)
Coverage Report: http://coverage.selectdb-in.cc/coverage/296e546c5904e6df62fb7968d40d73175de25f59_296e546c5904e6df62fb7968d40d73175de25f59/report/index.html

doris-robot · 2023-09-08T09:22:15Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.93 seconds
stream load tsv: 532 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162113559 Bytes

be/src/olap/rowset/segment_v2/segment_writer.cpp

bobhan1 · 2023-09-08T14:43:38Z

run buildall

github-actions · 2023-09-08T14:50:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-08T15:11:41Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7897/21385)
Line Coverage: 28.94% (63389/219051)
Region Coverage: 27.88% (32915/118041)
Branch Coverage: 24.47% (16906/69100)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7ccc496f8bb594d1e2aae533b52452e89ffd5109_7ccc496f8bb594d1e2aae533b52452e89ffd5109/report/index.html

doris-robot · 2023-09-08T15:28:36Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.1 seconds
stream load tsv: 545 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17161909864 Bytes

liaoxin01

LGTM

zhannngchen

LGTM

github-actions · 2023-09-11T12:23:03Z

PR approved by at least one committer and no changes requested.

…for rows with delete sign when sequence column doesn't exist apache#24011"

…n merge-on-write unique table (#25071) * pick "[Fix](Partial update) Fix wrong position using in segment writer #22782" * pick "[Enhancement](merge-on-write) use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist #24011" * pick "[Fix](merge-on-write) Correct the alignment process when the existing rows with same key has marked delete sign #24877"

bobhan1 force-pushed the enhance-delete-sign branch from d50bf46 to f95b797 Compare September 7, 2023 03:30

bobhan1 force-pushed the enhance-delete-sign branch from 7f3e711 to 0d4230a Compare September 7, 2023 08:28

zhannngchen reviewed Sep 7, 2023

View reviewed changes

be/src/olap/rowset/segment_v2/segment_writer.cpp Outdated Show resolved Hide resolved

zhannngchen reviewed Sep 7, 2023

View reviewed changes

be/src/olap/rowset/segment_v2/segment_writer.cpp Outdated Show resolved Hide resolved

bobhan1 force-pushed the enhance-delete-sign branch 2 times, most recently from e958f53 to c5c76d3 Compare September 7, 2023 09:43

zhannngchen reviewed Sep 7, 2023

View reviewed changes

be/src/olap/tablet_meta.h Show resolved Hide resolved

bobhan1 force-pushed the enhance-delete-sign branch 2 times, most recently from 78edca8 to 69159b0 Compare September 7, 2023 10:37

bobhan1 force-pushed the enhance-delete-sign branch from 69159b0 to c31c094 Compare September 7, 2023 10:57

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 8, 2023

github-actions bot added the reviewed label Sep 8, 2023

liaoxin01 reviewed Sep 8, 2023

View reviewed changes

be/src/olap/rowset/segment_v2/segment_writer.cpp Outdated Show resolved Hide resolved

update

7ccc496

bobhan1 dismissed zhannngchen’s stale review via 7ccc496 September 8, 2023 14:43

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 8, 2023

liaoxin01 approved these changes Sep 10, 2023

View reviewed changes

bobhan1 requested a review from zhannngchen September 11, 2023 01:40

zhannngchen approved these changes Sep 11, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 11, 2023

zhannngchen merged commit 6913d68 into apache:master Sep 12, 2023

bobhan1 mentioned this pull request Sep 25, 2023

[Fix](merge-on-write) Correct the alignment process when the existing rows with same key has marked delete sign #24877

Merged

bobhan1 added a commit to bobhan1/doris that referenced this pull request Oct 7, 2023

pick "[Enhancement](merge-on-write) use delete bitmap to mark delete …

0d8ce0b

…for rows with delete sign when sequence column doesn't exist apache#24011"

bobhan1 mentioned this pull request Oct 7, 2023

pick "[Enhancement](merge-on-write) use delete bitmap to mark delete … #25065

Closed

bobhan1 added a commit to bobhan1/doris that referenced this pull request Oct 7, 2023

pick "[Enhancement](merge-on-write) use delete bitmap to mark delete …

1130ded

…for rows with delete sign when sequence column doesn't exist apache#24011"

bobhan1 added a commit to bobhan1/doris that referenced this pull request Oct 7, 2023

pick "[Enhancement](merge-on-write) use delete bitmap to mark delete …

d594cb9

…for rows with delete sign when sequence column doesn't exist apache#24011"

bobhan1 mentioned this pull request Oct 7, 2023

[branch-2.0-pick](merge-on-write) picks PRs about delete processing on merge-on-write unique table #25071

Merged

xiaokang added the dev/2.0.3-merged label Oct 8, 2023

zhannngchen mentioned this pull request Oct 20, 2023

[fix](partial update) fix some bugs about delete sign #25712

Merged

bobhan1 mentioned this pull request Nov 9, 2023

[Revert](merge-on-write) Don't use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist #26721

Merged

[Enhancement](merge-on-write) use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist #24011

[Enhancement](merge-on-write) use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist #24011

Uh oh!

Conversation

bobhan1 commented Sep 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

bobhan1 commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

doris-robot commented Sep 7, 2023

Uh oh!

doris-robot commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

bobhan1 commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

doris-robot commented Sep 7, 2023

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

Uh oh!

bobhan1 commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 7, 2023

Uh oh!

bobhan1 commented Sep 7, 2023

Uh oh!

github-actions bot commented Sep 8, 2023

Uh oh!

github-actions bot commented Sep 8, 2023

Uh oh!

doris-robot commented Sep 8, 2023

Uh oh!

doris-robot commented Sep 8, 2023

Uh oh!

Uh oh!

bobhan1 commented Sep 8, 2023

Uh oh!

github-actions bot commented Sep 8, 2023

Uh oh!

doris-robot commented Sep 8, 2023

Uh oh!

doris-robot commented Sep 8, 2023

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

Reviewers

Assignees

Labels

bobhan1 commented Sep 7, 2023 •

edited

Loading