Skip to content

Conversation

@yujun777
Copy link
Contributor

@yujun777 yujun777 commented Nov 13, 2023

Be load score = replica num / avg replica num * replica num coef + used percent / avg used percent * capacity coef

And replica num coef = 1.0 - capacity coef

If capacity coef = 1.0, then replica num coef = 0, then the balance will ignore tablet num, only considering tablet size.
So the be balance will not move a empty tablet from high load backend to low load backend. Because moving this tablet, these two backends' load score will not change. And if there are lots of empty tablets, the balance may always pick them, but move failed.

So we let capacity coef <= 0.99, then the balance will always considering the tablet num. Then will make balance moving a tablet become possible.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@yujun777
Copy link
Contributor Author

run buildall

1 similar comment
@yujun777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.15 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17161989669 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 9ed275562de2510ed2f2745d2b437062fe137b10, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5723	5289	5109	5109
q2	384	191	200	191
q3	2155	2105	2074	2074
q4	1503	1463	1419	1419
q5	4181	4132	4131	4131
q6	257	135	132	132
q7	1540	947	948	947
q8	2856	2835	2814	2814
q9	10028	16665	9412	9412
q10	3486	3588	3578	3578
q11	418	265	271	265
q12	471	293	299	293
q13	4507	4152	4135	4135
q14	321	297	291	291
q15	626	557	531	531
q16	688	594	588	588
q17	1159	1086	1099	1086
q18	8064	7475	7606	7475
q19	1710	1688	1695	1688
q20	585	385	350	350
q21	4993	4617	4628	4617
q22	529	429	420	420
Total cold run time: 56184 ms
Total hot run time: 51546 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4986	4962	4979	4962
q2	332	220	252	220
q3	4107	4061	4043	4043
q4	2800	2788	2757	2757
q5	9562	9591	9645	9591
q6	244	127	125	125
q7	3019	2478	2478	2478
q8	4760	4764	4784	4764
q9	12973	12900	12813	12813
q10	4116	4201	4197	4197
q11	775	667	643	643
q12	982	806	849	806
q13	4315	3850	3922	3850
q14	382	342	347	342
q15	646	548	541	541
q16	804	707	669	669
q17	3958	3920	3908	3908
q18	9627	9429	9471	9429
q19	1839	1773	1755	1755
q20	2390	2070	2059	2059
q21	9023	8786	8816	8786
q22	914	873	844	844
Total cold run time: 82554 ms
Total hot run time: 79582 ms

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit f4d5e6d into apache:master Nov 13, 2023
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 14, 2023
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 14, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
yujun777 added a commit to yujun777/doris that referenced this pull request Feb 20, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants