-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[enhance](compaction) optimize mow base compaction parameters #52321
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[enhance](compaction) optimize mow base compaction parameters #52321
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
run beut |
|
run buildall |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
|
run buildall |
|
run buildall |
TPC-H: Total hot run time: 33801 ms |
TPC-DS: Total hot run time: 184434 ms |
ClickBench: Total hot run time: 30 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
dataroaring
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
PR approved by at least one committer and no changes requested. |
## 1 Increase base compaction frequency In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes. ## 2. Increase the limit on compaction score per run The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.
## 1 Increase base compaction frequency In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes. ## 2. Increase the limit on compaction score per run The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.
…ters apache#52321 (apache#52605) Cherry-picked from apache#52321 Co-authored-by: Luwei <luwei@selectdb.com>
…#52321) ## 1 Increase base compaction frequency In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes. ## 2. Increase the limit on compaction score per run The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.
…#52321) ## 1 Increase base compaction frequency In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes. ## 2. Increase the limit on compaction score per run The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.
…#52321) ## 1 Increase base compaction frequency In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes. ## 2. Increase the limit on compaction score per run The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.
1 Increase base compaction frequency
In high-frequency mow load scenarios, the default two-hour compaction interval is too long, causing the compaction score to rise. Therefore, adjust base_compaction_freeze_interval_s from two hours to 30 minutes.
2. Increase the limit on compaction score per run
The base_compaction_max_compaction_score parameter limits the number of rowsets in a single compaction to control memory usage. However, in high-frequency mow load scenarios, this can slow down compaction and prevent it from keeping up with data growth. Thus, this parameter is increased to 200.