Skip to content

branch-3.0: [opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm #51733 #52813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 8, 2025
Merged

branch-3.0: [opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm #51733 #52813

dataroaring merged 1 commit into from
Jul 8, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jul 4, 2025

Cherry-picked from #51733

@github-actions github-actions bot requested a review from dataroaring as a code owner July 4, 2025 14:19
@Thearas
Copy link
Contributor

Thearas commented Jul 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jul 4, 2025
@dataroaring dataroaring reopened this Jul 4, 2025
@Thearas
Copy link
Contributor

Thearas commented Jul 4, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 4, 2025

TPC-H: Total hot run time: 39576 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cdee51f5747aec2eb89b11ea7eecef927e01c98c, data reload: false

------ Round 1 ----------------------------------
q1	17579	6778	6615	6615
q2	2069	179	171	171
q3	10525	1100	1090	1090
q4	10232	737	734	734
q5	7718	2887	2782	2782
q6	213	131	129	129
q7	995	610	598	598
q8	9358	1913	1985	1913
q9	6644	6386	6362	6362
q10	7038	2215	2292	2215
q11	465	275	257	257
q12	393	206	208	206
q13	17806	2971	2987	2971
q14	234	210	211	210
q15	509	461	479	461
q16	513	385	371	371
q17	968	511	524	511
q18	7274	6684	6658	6658
q19	1394	1135	1088	1088
q20	454	196	206	196
q21	3862	3102	3071	3071
q22	1105	984	967	967
Total cold run time: 107348 ms
Total hot run time: 39576 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6574	6535	6504	6504
q2	336	232	229	229
q3	2880	2857	2918	2857
q4	2031	1743	1782	1743
q5	5734	5696	5706	5696
q6	205	121	128	121
q7	2186	1832	1841	1832
q8	3350	3545	3507	3507
q9	8670	8859	8807	8807
q10	3572	3500	3483	3483
q11	598	494	488	488
q12	816	622	587	587
q13	9265	3166	3168	3166
q14	303	277	264	264
q15	519	467	460	460
q16	485	457	423	423
q17	1853	1576	1588	1576
q18	8244	7603	7752	7603
q19	1675	1517	1577	1517
q20	2051	1863	1928	1863
q21	5387	5073	5011	5011
q22	1155	1072	1033	1033
Total cold run time: 67889 ms
Total hot run time: 58770 ms

@doris-robot
Copy link

doris-robot commented Jul 4, 2025

TPC-DS: Total hot run time: 195603 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cdee51f5747aec2eb89b11ea7eecef927e01c98c, data reload: false

query1	1310	915	907	907
query2	6291	1889	1865	1865
query3	10840	4366	4144	4144
query4	32943	23342	23431	23342
query5	3591	443	441	441
query6	254	178	174	174
query7	3991	304	320	304
query8	278	236	224	224
query9	9494	2550	2548	2548
query10	469	256	253	253
query11	17781	15186	15181	15181
query12	155	102	102	102
query13	1544	428	437	428
query14	9084	6590	6487	6487
query15	215	171	182	171
query16	7951	468	490	468
query17	1583	613	588	588
query18	2069	334	329	329
query19	223	157	164	157
query20	144	119	119	119
query21	207	115	115	115
query22	4765	4767	4467	4467
query23	35199	34508	34061	34061
query24	11498	2834	2949	2834
query25	659	444	429	429
query26	1401	167	165	165
query27	2927	353	348	348
query28	7744	2141	2095	2095
query29	940	477	445	445
query30	258	158	162	158
query31	1060	832	842	832
query32	96	57	54	54
query33	784	336	319	319
query34	947	491	525	491
query35	918	719	724	719
query36	1098	946	934	934
query37	131	66	65	65
query38	4098	3912	3920	3912
query39	1531	1471	1489	1471
query40	256	101	109	101
query41	49	48	47	47
query42	116	101	106	101
query43	517	468	465	465
query44	1257	831	813	813
query45	189	170	171	170
query46	1134	709	737	709
query47	2026	1859	1913	1859
query48	469	386	390	386
query49	1006	410	425	410
query50	828	417	406	406
query51	7393	7306	7391	7306
query52	98	92	89	89
query53	259	181	183	181
query54	1328	451	455	451
query55	78	77	78	77
query56	259	248	250	248
query57	1332	1231	1221	1221
query58	237	207	214	207
query59	3094	3037	2992	2992
query60	285	259	258	258
query61	112	105	114	105
query62	876	698	680	680
query63	220	196	197	196
query64	5161	678	649	649
query65	3323	3329	3242	3242
query66	1139	297	312	297
query67	16337	15602	15594	15594
query68	4701	568	565	565
query69	432	259	257	257
query70	1174	1100	1132	1100
query71	361	258	267	258
query72	6216	3953	4077	3953
query73	760	353	349	349
query74	10729	8923	9179	8923
query75	3385	2650	2612	2612
query76	2659	1054	1144	1054
query77	419	264	281	264
query78	10610	9607	9639	9607
query79	2266	618	593	593
query80	1138	433	422	422
query81	549	222	221	221
query82	889	86	85	85
query83	235	141	137	137
query84	235	78	74	74
query85	1346	309	293	293
query86	438	299	288	288
query87	4429	4191	4250	4191
query88	4096	2342	2337	2337
query89	413	295	286	286
query90	1894	180	181	180
query91	184	149	147	147
query92	66	47	47	47
query93	2409	544	543	543
query94	795	298	294	294
query95	364	256	254	254
query96	636	275	277	275
query97	3288	3119	3136	3119
query98	212	206	194	194
query99	1489	1292	1275	1275
Total cold run time: 304666 ms
Total hot run time: 195603 ms

@doris-robot
Copy link

doris-robot commented Jul 4, 2025

ClickBench: Total hot run time: 29.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cdee51f5747aec2eb89b11ea7eecef927e01c98c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.06
query4	1.62	0.11	0.10
query5	0.52	0.50	0.51
query6	1.14	0.73	0.72
query7	0.03	0.02	0.03
query8	0.04	0.03	0.03
query9	0.56	0.49	0.50
query10	0.56	0.55	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.12
query13	0.62	0.60	0.59
query14	0.77	0.80	0.79
query15	0.85	0.82	0.83
query16	0.40	0.39	0.40
query17	1.06	1.05	1.06
query18	0.23	0.21	0.21
query19	1.97	1.77	1.79
query20	0.01	0.00	0.01
query21	15.40	0.62	0.60
query22	2.41	1.94	1.43
query23	16.92	0.88	0.74
query24	3.02	1.57	0.70
query25	0.19	0.06	0.08
query26	0.55	0.13	0.13
query27	0.05	0.05	0.04
query28	10.25	0.47	0.49
query29	12.60	3.25	3.20
query30	0.24	0.07	0.06
query31	2.85	0.38	0.38
query32	3.23	0.46	0.46
query33	2.95	3.02	3.05
query34	16.91	4.56	4.44
query35	4.51	4.45	4.53
query36	0.68	0.46	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.26 s
Total hot run time: 29.3 s

dataroaring
dataroaring approved these changes Jul 7, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@deardeng @dataroaring
[opt](cloud) Optimize balance speed by reducing the complexity of the…
bd975a1 
… rebalance algorithm (#51733)

Arrary List remove，time complexity is too high
<img width="489" alt="image"
src="https://github.com/user-attachments/assets/ffa486f8-c9fb-4ee9-9f9c-408e189525d5"
/>
@dataroaring dataroaring force-pushed the auto-pick-51733-branch-3.0 branch from cdee51f to bd975a1 Compare July 7, 2025 06:47
@dataroaring
Copy link
Contributor

dataroaring commented Jul 7, 2025

run buildall

@github-actions
Copy link
Contributor Author

github-actions bot commented Jul 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 7, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Jul 7, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit a265588 into branch-3.0 Jul 8, 2025
24 of 28 checks passed
@github-actions github-actions bot deleted the auto-pick-51733-branch-3.0 branch July 8, 2025 01:36
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @deardeng