Skip to content

[cherry-pick](branch-2.1) Pick "[Fix](full compaction) Full compaction should not do ordered data compaction (#44359)" #44529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 2 commits into from
Nov 26, 2024
Merged

[cherry-pick](branch-2.1) Pick "[Fix](full compaction) Full compaction should not do ordered data compaction (#44359)" #44529

yiguolei merged 2 commits into from
Nov 26, 2024

Conversation

@Yukang-Lian
Copy link
Collaborator

@Yukang-Lian Yukang-Lian commented Nov 25, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Pick #44359

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Yukang-Lian
[Fix](full compaction) Full compaction should not do ordered data com…
6846e71 
…paction (apache#44359)


Problem: For a duplicate table with the following distribution, if it
has already completed cumulative compaction and then undergoes full
compaction, it will cause a BE core issue. Check failed: new_point ==
Tablet::K_INVALID_CUMULATIVE_POINT || new_point >= _cumulative_point
Unexpected cumulative point: 1087, origin: 2801.
"rowsets": [
"[0-386] 0 DATA NONOVERLAPPING
02000000000198aabe4290f2b0f5f35610c08a233a061892 0",
"[387-387] 0 DELETE OVERLAP_UNKNOWN
0200000000541310ac4d76e7580a708a2823a4d7a4f06090 0",
"[388-388] 0 DELETE OVERLAP_UNKNOWN
0200000000541d76ac4d76e7580a708a2823a4d7a4f06090 0",
"[389-389] 0 DELETE OVERLAP_UNKNOWN
0200000000543b4dac4d76e7580a708a2823a4d7a4f06090 0",
"[390-390] 0 DELETE OVERLAP_UNKNOWN
02000000005453aeac4d76e7580a708a2823a4d7a4f06090 0",
"[391-391] 0 DELETE OVERLAP_UNKNOWN
0200000000546a44ac4d76e7580a708a2823a4d7a4f06090 0",
"[392-392] 0 DELETE OVERLAP_UNKNOWN
02000000005480dbac4d76e7580a708a2823a4d7a4f06090 0",
"[393-393] 0 DELETE OVERLAP_UNKNOWN
0200000000548cb3ac4d76e7580a708a2823a4d7a4f06090 0",
"[394-394] 0 DELETE OVERLAP_UNKNOWN
0200000000549a25ac4d76e7580a708a2823a4d7a4f06090 0",
"[395-395] 0 DELETE OVERLAP_UNKNOWN
020000000054b359ac4d76e7580a708a2823a4d7a4f06090 0",
"[396-396] 0 DELETE OVERLAP_UNKNOWN
020000000054c19dac4d76e7580a708a2823a4d7a4f06090 0",
"[397-397] 0 DELETE OVERLAP_UNKNOWN
020000000054d757ac4d76e7580a708a2823a4d7a4f06090 0",
        ...
"[1085-1085] 0 DELETE OVERLAP_UNKNOWN
02000000002a0b20bd4798638f237008ff42fbca276b52a2 0",
"[1087-1506] 1 DATA NONOVERLAPPING
020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB",
"[1087-1506] 1 DATA NONOVERLAPPING
020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB",
        ...
"[2800-2800] 0 DELETE OVERLAP_UNKNOWN
02000000002f12d6bd4798638f237008ff42fbca276b52a2 0"

Reason: The duplicate table will go through ordered data compaction. Due
to the special distribution of the table, the input rowset will be cut
by the ordered data compaction, resulting in the full compaction only
being performed on a part of the rowsets.

Solution: For full compaction, prohibit ordered data compaction.
@Yukang-Lian
2
f3fc7e3
@doris-robot
Copy link

doris-robot commented Nov 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian
Copy link
Collaborator Author

Yukang-Lian commented Nov 25, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 25, 2024

TeamCity be ut coverage result:
Function Coverage: 36.74% (9606/26145)
Line Coverage: 28.15% (79035/280776)
Region Coverage: 26.83% (40617/151397)
Branch Coverage: 23.65% (20669/87380)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f3fc7e3e2ab099d065031d7c4aa647b1fdf9f947_f3fc7e3e2ab099d065031d7c4aa647b1fdf9f947/report/index.html

@github-actions
Copy link
Contributor

github-actions bot commented Nov 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@yiguolei yiguolei merged commit 550aa9c into apache:branch-2.1 Nov 26, 2024
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Yukang-Lian @doris-robot @yiguolei