Skip to content

[opt](routine-load) optimize allocate task to be algorithm for load balance #32021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Mar 20, 2024
Merged

[opt](routine-load) optimize allocate task to be algorithm for load balance #32021

liaoxin01 merged 1 commit into from
Mar 20, 2024

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Mar 9, 2024 •
edited
Loading

Proposed changes

The task always running on one BE may cause the BE to run too many tasks simultaneously while the other BE is not running few task. Here, a simple limitation is added to optimize this situation.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@sollhui
Copy link
Contributor Author

sollhui commented Mar 9, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Mar 9, 2024

PR approved by anyone and no changes requested.

@sollhui
Copy link
Contributor Author

sollhui commented Mar 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 15, 2024

TPC-H: Total hot run time: 38261 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72a0e9d98388da6dff99073a7bd49e63918bafaf, data reload: false

------ Round 1 ----------------------------------
q1	17624	4935	4136	4136
q2	2026	151	141	141
q3	10605	1089	916	916
q4	6741	757	703	703
q5	7467	2613	2619	2613
q6	183	123	126	123
q7	1253	821	800	800
q8	9321	1979	2012	1979
q9	7129	6420	6422	6420
q10	8510	3490	3616	3490
q11	423	226	222	222
q12	709	295	305	295
q13	17788	2865	2867	2865
q14	277	243	254	243
q15	499	454	451	451
q16	517	399	397	397
q17	948	563	554	554
q18	7190	6518	6346	6346
q19	4921	1417	1385	1385
q20	532	292	284	284
q21	6497	3613	3581	3581
q22	361	342	317	317
Total cold run time: 111521 ms
Total hot run time: 38261 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4171	4114	4114	4114
q2	321	231	222	222
q3	2996	2829	2845	2829
q4	1874	1550	1602	1550
q5	5250	5263	5240	5240
q6	195	117	121	117
q7	2221	1832	1817	1817
q8	3152	3312	3308	3308
q9	8573	8526	8551	8526
q10	3710	3679	3656	3656
q11	534	458	448	448
q12	715	547	555	547
q13	16922	2881	2857	2857
q14	276	268	249	249
q15	481	460	445	445
q16	444	401	431	401
q17	1751	1505	1461	1461
q18	7670	7176	7018	7018
q19	1637	1514	1568	1514
q20	1938	1714	1721	1714
q21	4843	4567	4794	4567
q22	530	462	460	460
Total cold run time: 70204 ms
Total hot run time: 53060 ms

liaoxin01
liaoxin01 approved these changes Mar 16, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 16, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 16, 2024

PR approved by at least one committer and no changes requested.

@liaoxin01
Copy link
Contributor

liaoxin01 commented Mar 18, 2024

run p0

@liaoxin01
Copy link
Contributor

liaoxin01 commented Mar 18, 2024

run performance

@liaoxin01 liaoxin01 merged commit e7d2b38 into apache:master Mar 20, 2024
yiguolei pushed a commit that referenced this pull request Mar 21, 2024
@sollhui @Doris-Extras
[opt](routine-load) optimize allocate task to be algorithm for load b…
acde3ae 
…alance (#32021)
yiguolei pushed a commit that referenced this pull request Mar 21, 2024
@sollhui @Doris-Extras
[opt](routine-load) optimize allocate task to be algorithm for load b…
22782fe 
…alance (#32021)
@yiguolei yiguolei mentioned this pull request Mar 24, 2024
Closed
@sollhui sollhui mentioned this pull request Jul 8, 2024
Merged
liaoxin01 pushed a commit that referenced this pull request Jul 8, 2024
@sollhui
[opt](routine-load) optimize allocate task to be algorithm for load b…
598c46d 
…alance (#32021) (#37434)
@xiaokang xiaokang mentioned this pull request Jul 14, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@sollhui @Baymine
[opt](routine-load) optimize allocate task to be algorithm for load b…
078d78e 
…alance (apache#32021) (apache#37434)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

+2 more reviewers

@kaijchen kaijchen kaijchen approved these changes

@feifeifeimoon feifeifeimoon feifeifeimoon approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.13-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sollhui @doris-robot @liaoxin01 @kaijchen @feifeifeimoon