Skip to content

[fix](auto bucket)Set the estimated partition size to 5G in non cloud #51258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 2 commits into from
Jun 12, 2025
Merged

[fix](auto bucket)Set the estimated partition size to 5G in non cloud #51258

gavinchou merged 2 commits into from
Jun 12, 2025

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented May 26, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng
Copy link
Contributor Author

deardeng commented May 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 26, 2025

TPC-H: Total hot run time: 33715 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37c26997154e2f1fea457b77add68ab8edd3affe, data reload: false

------ Round 1 ----------------------------------
q1	25802	5016	5010	5010
q2	2063	278	181	181
q3	10402	1242	693	693
q4	10204	1101	521	521
q5	7533	2341	2308	2308
q6	180	162	133	133
q7	908	746	604	604
q8	9331	1209	1049	1049
q9	7052	5115	5265	5115
q10	6881	2317	1892	1892
q11	473	287	270	270
q12	344	353	208	208
q13	17819	3706	3157	3157
q14	233	228	213	213
q15	529	490	493	490
q16	418	420	369	369
q17	597	836	365	365
q18	7470	7069	7144	7069
q19	1856	956	538	538
q20	335	338	214	214
q21	3666	3148	2348	2348
q22	1059	1029	968	968
Total cold run time: 115155 ms
Total hot run time: 33715 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5120	5028	5025	5025
q2	238	317	225	225
q3	2180	2626	2312	2312
q4	1386	1775	1358	1358
q5	4540	4388	4365	4365
q6	213	173	131	131
q7	2029	1918	1777	1777
q8	2603	2703	2540	2540
q9	7203	7156	6894	6894
q10	3051	3237	2811	2811
q11	578	510	491	491
q12	686	767	594	594
q13	3509	3931	3291	3291
q14	302	294	278	278
q15	539	477	477	477
q16	439	497	469	469
q17	1119	1496	1402	1402
q18	7805	7538	7422	7422
q19	795	836	830	830
q20	1972	2066	1864	1864
q21	4588	4463	4480	4463
q22	1089	1047	1043	1043
Total cold run time: 51984 ms
Total hot run time: 50062 ms

@doris-robot
Copy link

doris-robot commented May 26, 2025

TPC-DS: Total hot run time: 193149 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37c26997154e2f1fea457b77add68ab8edd3affe, data reload: false

query1	1441	1083	1059	1059
query2	6362	1783	1781	1781
query3	11015	4486	4399	4399
query4	53981	25546	23589	23589
query5	5117	478	476	476
query6	361	216	200	200
query7	4945	487	297	297
query8	319	253	250	250
query9	5938	2611	2606	2606
query10	447	324	291	291
query11	15520	15014	14870	14870
query12	157	109	108	108
query13	1072	534	411	411
query14	10174	6286	6280	6280
query15	207	196	189	189
query16	7064	677	499	499
query17	1070	719	622	622
query18	1568	397	306	306
query19	195	205	189	189
query20	132	122	124	122
query21	209	129	117	117
query22	4363	4321	4383	4321
query23	34193	33433	33646	33433
query24	6612	2464	2460	2460
query25	472	476	409	409
query26	685	273	160	160
query27	2224	508	335	335
query28	2983	2134	2121	2121
query29	587	584	450	450
query30	273	224	186	186
query31	881	871	776	776
query32	70	67	62	62
query33	451	369	327	327
query34	776	857	538	538
query35	830	833	772	772
query36	914	977	923	923
query37	117	102	76	76
query38	4243	4247	4311	4247
query39	1523	1461	1497	1461
query40	217	122	109	109
query41	56	54	53	53
query42	129	112	114	112
query43	500	519	482	482
query44	1358	836	836	836
query45	178	170	164	164
query46	859	1034	668	668
query47	1865	1874	1771	1771
query48	399	441	371	371
query49	673	532	473	473
query50	676	707	421	421
query51	4226	4408	4217	4217
query52	120	120	100	100
query53	240	262	188	188
query54	589	577	524	524
query55	90	86	90	86
query56	304	312	331	312
query57	1261	1235	1129	1129
query58	278	254	271	254
query59	2705	2740	2809	2740
query60	346	336	323	323
query61	161	135	124	124
query62	722	712	650	650
query63	217	184	194	184
query64	1471	1028	674	674
query65	4267	4194	4222	4194
query66	725	404	306	306
query67	15844	15717	15346	15346
query68	5424	889	529	529
query69	529	311	263	263
query70	1231	1167	1118	1118
query71	438	323	293	293
query72	5876	5031	5132	5031
query73	1420	702	364	364
query74	8951	9283	8649	8649
query75	3490	3228	2733	2733
query76	3668	1203	751	751
query77	556	376	288	288
query78	10161	10268	9354	9354
query79	2577	797	583	583
query80	796	508	438	438
query81	542	255	220	220
query82	641	125	97	97
query83	263	251	251	251
query84	291	108	81	81
query85	799	358	318	318
query86	450	293	284	284
query87	4404	4470	4340	4340
query88	3399	2291	2286	2286
query89	413	318	287	287
query90	1917	218	201	201
query91	143	150	127	127
query92	69	65	56	56
query93	2598	908	580	580
query94	680	392	312	312
query95	386	294	295	294
query96	486	574	276	276
query97	2702	2758	2700	2700
query98	241	206	217	206
query99	1353	1404	1310	1310
Total cold run time: 296861 ms
Total hot run time: 193149 ms

@doris-robot
Copy link

doris-robot commented May 26, 2025

ClickBench: Total hot run time: 29.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37c26997154e2f1fea457b77add68ab8edd3affe, data reload: false

query1	0.04	0.03	0.04
query2	0.12	0.10	0.12
query3	0.24	0.19	0.19
query4	1.60	0.20	0.21
query5	0.48	0.44	0.44
query6	1.16	0.68	0.67
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.53	0.53
query10	0.58	0.58	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	0.79	0.81	0.80
query15	0.88	0.86	0.87
query16	0.40	0.38	0.38
query17	1.05	1.04	1.03
query18	0.22	0.21	0.22
query19	1.94	1.84	1.87
query20	0.01	0.01	0.02
query21	15.40	0.88	0.53
query22	0.77	1.32	0.69
query23	14.80	1.36	0.60
query24	7.34	1.29	0.85
query25	0.48	0.20	0.09
query26	0.63	0.17	0.14
query27	0.06	0.05	0.05
query28	9.22	0.90	0.45
query29	12.53	3.91	3.26
query30	0.25	0.09	0.06
query31	2.83	0.61	0.39
query32	3.23	0.54	0.48
query33	3.14	3.12	3.11
query34	15.77	5.12	4.48
query35	4.53	4.52	4.52
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.13	0.14
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.2 s
Total hot run time: 29.21 s

dataroaring
dataroaring previously approved these changes May 26, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 26, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 26, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 26, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 27, 2025
@deardeng
Copy link
Contributor Author

deardeng commented May 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 27, 2025

TPC-H: Total hot run time: 33644 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c1bcb30fb839f92b4b09031dfacfa54ec5433d11, data reload: false

------ Round 1 ----------------------------------
q1	26004	5037	5024	5024
q2	2067	281	182	182
q3	10398	1262	719	719
q4	10222	1000	524	524
q5	7554	2389	2317	2317
q6	178	164	132	132
q7	900	726	614	614
q8	9307	1214	1042	1042
q9	6825	5042	5073	5042
q10	6858	2313	1889	1889
q11	484	298	279	279
q12	344	347	210	210
q13	17788	3632	3094	3094
q14	225	226	209	209
q15	545	490	507	490
q16	422	432	376	376
q17	579	850	363	363
q18	7507	7059	7240	7059
q19	2983	954	542	542
q20	326	315	219	219
q21	3750	3134	2346	2346
q22	1043	1011	972	972
Total cold run time: 116309 ms
Total hot run time: 33644 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5278	5086	5063	5063
q2	245	325	233	233
q3	2185	2639	2321	2321
q4	1376	1792	1421	1421
q5	4495	4405	4417	4405
q6	219	173	128	128
q7	1993	1946	1739	1739
q8	2582	2602	2616	2602
q9	7201	7192	6996	6996
q10	3012	3216	2735	2735
q11	570	542	482	482
q12	661	777	616	616
q13	3541	3905	3282	3282
q14	286	286	281	281
q15	545	488	478	478
q16	440	478	431	431
q17	1129	1479	1386	1386
q18	7908	7546	7387	7387
q19	797	865	1056	865
q20	2084	2006	1833	1833
q21	4689	4462	4377	4377
q22	1117	1085	992	992
Total cold run time: 52353 ms
Total hot run time: 50053 ms

@doris-robot
Copy link

doris-robot commented May 27, 2025

TPC-DS: Total hot run time: 193716 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c1bcb30fb839f92b4b09031dfacfa54ec5433d11, data reload: false

query1	1436	1100	1076	1076
query2	6229	1901	1906	1901
query3	11055	4479	4603	4479
query4	55432	25363	23647	23647
query5	4850	550	482	482
query6	341	239	200	200
query7	4874	513	301	301
query8	308	257	235	235
query9	5292	2631	2611	2611
query10	444	331	274	274
query11	15099	15064	14928	14928
query12	159	107	105	105
query13	1027	529	429	429
query14	10229	6473	6442	6442
query15	206	197	184	184
query16	7145	647	493	493
query17	1118	769	619	619
query18	1560	426	337	337
query19	199	196	172	172
query20	135	135	128	128
query21	211	131	110	110
query22	4399	4335	4336	4335
query23	34196	33875	33568	33568
query24	6891	2437	2439	2437
query25	467	470	428	428
query26	721	280	151	151
query27	2439	516	339	339
query28	2877	2133	2134	2133
query29	553	557	440	440
query30	278	224	191	191
query31	863	859	788	788
query32	70	63	64	63
query33	439	383	343	343
query34	779	963	562	562
query35	793	861	759	759
query36	938	1013	871	871
query37	116	100	78	78
query38	4218	4290	4155	4155
query39	1495	1465	1455	1455
query40	218	119	113	113
query41	56	53	77	53
query42	129	112	106	106
query43	509	514	502	502
query44	1306	821	825	821
query45	178	175	164	164
query46	856	1032	681	681
query47	1836	1869	1801	1801
query48	390	419	313	313
query49	693	543	449	449
query50	660	711	426	426
query51	4276	4258	4255	4255
query52	119	111	99	99
query53	231	264	186	186
query54	575	562	506	506
query55	84	84	85	84
query56	309	313	287	287
query57	1145	1208	1133	1133
query58	264	260	264	260
query59	2776	2895	2846	2846
query60	329	332	312	312
query61	130	124	128	124
query62	755	731	684	684
query63	230	191	194	191
query64	1981	1031	703	703
query65	4362	4227	4260	4227
query66	731	405	306	306
query67	15696	15614	15300	15300
query68	7724	882	513	513
query69	535	295	261	261
query70	1188	1147	1087	1087
query71	517	321	299	299
query72	5679	4832	4815	4815
query73	1289	662	356	356
query74	9053	9239	8640	8640
query75	4108	3212	2721	2721
query76	4301	1223	778	778
query77	696	376	285	285
query78	10027	10188	9617	9617
query79	2408	814	582	582
query80	699	531	441	441
query81	482	254	221	221
query82	414	123	95	95
query83	246	254	234	234
query84	306	98	90	90
query85	770	350	307	307
query86	344	300	281	281
query87	4499	4507	4365	4365
query88	3255	2279	2258	2258
query89	407	319	288	288
query90	1805	206	212	206
query91	145	148	116	116
query92	73	60	58	58
query93	2143	954	576	576
query94	673	405	312	312
query95	364	297	297	297
query96	492	572	282	282
query97	2763	2797	2625	2625
query98	236	209	204	204
query99	1517	1406	1325	1325
Total cold run time: 299863 ms
Total hot run time: 193716 ms

@doris-robot
Copy link

doris-robot commented May 27, 2025

ClickBench: Total hot run time: 29.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c1bcb30fb839f92b4b09031dfacfa54ec5433d11, data reload: false

query1	0.04	0.05	0.03
query2	0.12	0.10	0.12
query3	0.26	0.20	0.20
query4	1.60	0.20	0.19
query5	0.46	0.44	0.46
query6	1.17	0.66	0.65
query7	0.02	0.01	0.01
query8	0.05	0.04	0.03
query9	0.58	0.52	0.53
query10	0.56	0.57	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.59
query14	0.78	0.81	0.80
query15	0.88	0.86	0.86
query16	0.40	0.38	0.38
query17	1.07	1.02	1.01
query18	0.23	0.21	0.21
query19	1.86	1.82	1.87
query20	0.01	0.01	0.01
query21	15.39	0.88	0.54
query22	0.75	1.18	0.64
query23	14.98	1.39	0.63
query24	7.05	1.25	1.00
query25	0.52	0.06	0.23
query26	0.71	0.16	0.14
query27	0.05	0.05	0.05
query28	9.93	0.90	0.42
query29	12.58	3.95	3.37
query30	0.25	0.09	0.07
query31	2.82	0.60	0.40
query32	3.22	0.55	0.46
query33	3.02	3.15	3.17
query34	15.84	5.11	4.51
query35	4.53	4.52	4.46
query36	0.69	0.48	0.48
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.82 s
Total hot run time: 29.33 s

@deardeng
Copy link
Contributor Author

deardeng commented May 29, 2025

run feut

dataroaring
dataroaring approved these changes May 29, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 29, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 29, 2025

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou merged commit 00999f1 into apache:master Jun 12, 2025
29 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Jun 12, 2025
@deardeng
[fix](auto bucket)Set the estimated partition size to 5G in non cloud (…
bd5e6d5 
…#51258)
@github-actions github-actions bot mentioned this pull request Jun 12, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jun 18, 2025
@deardeng @dataroaring
[fix](auto bucket)Set the estimated partition size to 5G in non cloud (…
43cf415 
…#51258)
dataroaring pushed a commit that referenced this pull request Jun 20, 2025
@github-actions @deardeng
branch-3.0: [fix](auto bucket)Set the estimated partition size to 5G …
a7dddf0 
…in non cloud #51258 (#51682)

Cherry-picked from #51258

Co-authored-by: deardeng <dengxin@selectdb.com>
github-actions bot pushed a commit that referenced this pull request Jun 20, 2025
@deardeng
[fix](auto bucket)Set the estimated partition size to 5G in non cloud (…
6d786ec 
…#51258)
@github-actions github-actions bot mentioned this pull request Jun 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 20, 2025
@github-actions @deardeng
branch-3.1: [fix](auto bucket)Set the estimated partition size to 5G …
b66feb9 
…in non cloud #51258 (#52024)

Cherry-picked from #51258

Co-authored-by: deardeng <dengxin@selectdb.com>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@koarz koarz koarz approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@deardeng @Thearas @doris-robot @gavinchou @koarz @dataroaring @morrySnow