Skip to content

Conversation

@Lchangliang
Copy link
Contributor

When doing schema change, it will create many shadow tablets. And many TCreateTabletReq will be new. So we can use object pool to reduct they memory usage.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Lchangliang
Copy link
Contributor Author

run buildall

@Lchangliang
Copy link
Contributor Author

run buildall

1 similar comment
@Lchangliang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40194 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2756c213e966368dac3200bc41ac9bea2f19d0f, data reload: false

------ Round 1 ----------------------------------
q1	17891	4441	4386	4386
q2	2686	198	206	198
q3	11684	1110	1095	1095
q4	10750	884	773	773
q5	7510	2744	2680	2680
q6	234	144	140	140
q7	967	623	610	610
q8	9259	2087	2069	2069
q9	8974	6533	6505	6505
q10	8945	3767	3738	3738
q11	460	240	238	238
q12	431	236	241	236
q13	17757	3005	2979	2979
q14	269	215	226	215
q15	518	477	472	472
q16	519	392	377	377
q17	977	670	705	670
q18	8175	7536	7394	7394
q19	5897	1523	1499	1499
q20	660	316	336	316
q21	4874	3264	3934	3264
q22	394	340	342	340
Total cold run time: 119831 ms
Total hot run time: 40194 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4280	4254	4254
q2	371	258	263	258
q3	2967	2764	2711	2711
q4	1870	1581	1637	1581
q5	5253	5281	5308	5281
q6	222	128	132	128
q7	2115	1770	1737	1737
q8	3213	3301	3290	3290
q9	8347	8211	8391	8211
q10	3888	3693	3673	3673
q11	593	472	487	472
q12	758	603	626	603
q13	17244	2952	2992	2952
q14	294	255	250	250
q15	527	481	483	481
q16	466	419	412	412
q17	1778	1489	1472	1472
q18	7647	7403	7452	7403
q19	1704	1516	1462	1462
q20	2007	1757	1771	1757
q21	4965	4857	4643	4643
q22	605	533	518	518
Total cold run time: 71217 ms
Total hot run time: 53549 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173595 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2756c213e966368dac3200bc41ac9bea2f19d0f, data reload: false

query1	940	387	368	368
query2	6649	2459	2339	2339
query3	6650	217	217	217
query4	19480	17215	17163	17163
query5	4125	489	476	476
query6	255	163	159	159
query7	4589	303	291	291
query8	340	285	281	281
query9	8768	2422	2404	2404
query10	608	321	281	281
query11	10480	10019	10006	10006
query12	140	87	84	84
query13	1639	364	368	364
query14	10136	7658	7790	7658
query15	236	192	184	184
query16	7816	268	261	261
query17	1908	552	530	530
query18	1974	270	272	270
query19	199	156	156	156
query20	93	83	80	80
query21	211	133	132	132
query22	4353	4138	4009	4009
query23	33718	33060	33149	33060
query24	11158	2821	2769	2769
query25	628	358	362	358
query26	1416	149	148	148
query27	3033	312	317	312
query28	7260	2043	2052	2043
query29	962	641	640	640
query30	290	153	150	150
query31	987	725	747	725
query32	94	53	59	53
query33	774	282	273	273
query34	973	454	454	454
query35	743	619	630	619
query36	1132	953	963	953
query37	168	70	69	69
query38	2939	2720	2722	2720
query39	848	788	823	788
query40	213	134	133	133
query41	57	53	53	53
query42	121	103	102	102
query43	604	503	574	503
query44	1216	738	745	738
query45	204	161	170	161
query46	1104	711	708	708
query47	1845	1802	1781	1781
query48	380	306	295	295
query49	1089	470	399	399
query50	761	390	371	371
query51	6786	6620	6667	6620
query52	108	96	96	96
query53	358	291	284	284
query54	956	449	438	438
query55	76	72	74	72
query56	295	262	252	252
query57	1130	1065	1048	1048
query58	255	262	248	248
query59	3442	3180	3094	3094
query60	294	279	280	279
query61	91	90	89	89
query62	631	436	432	432
query63	321	294	289	289
query64	8983	2235	1747	1747
query65	3196	3123	3117	3117
query66	939	346	335	335
query67	15463	14814	14902	14814
query68	4528	542	536	536
query69	458	304	328	304
query70	1136	1189	1191	1189
query71	391	280	282	280
query72	7251	5890	5699	5699
query73	737	322	324	322
query74	5891	5558	5468	5468
query75	3409	2604	2643	2604
query76	2725	1009	875	875
query77	457	310	311	310
query78	10478	9935	9858	9858
query79	2321	517	518	517
query80	2481	487	503	487
query81	609	223	226	223
query82	776	112	101	101
query83	312	175	181	175
query84	272	94	93	93
query85	2061	342	338	338
query86	495	331	317	317
query87	3291	3110	3071	3071
query88	4227	2383	2365	2365
query89	483	382	396	382
query90	1837	194	191	191
query91	143	112	113	112
query92	68	52	53	52
query93	2357	511	509	509
query94	1263	205	201	201
query95	419	329	333	329
query96	592	266	270	266
query97	3235	3111	3048	3048
query98	338	240	194	194
query99	1257	845	834	834
Total cold run time: 275928 ms
Total hot run time: 173595 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b2756c213e966368dac3200bc41ac9bea2f19d0f, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.07
query5	0.50	0.49	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.48
query10	0.53	0.54	0.54
query11	0.15	0.11	0.11
query12	0.14	0.13	0.12
query13	0.60	0.58	0.59
query14	0.79	0.79	0.78
query15	0.83	0.81	0.83
query16	0.36	0.37	0.37
query17	1.05	0.95	0.98
query18	0.23	0.25	0.24
query19	1.88	1.85	1.74
query20	0.01	0.01	0.00
query21	15.42	0.67	0.66
query22	4.22	6.74	2.20
query23	18.33	1.36	1.28
query24	2.13	0.22	0.22
query25	0.15	0.10	0.08
query26	0.26	0.18	0.17
query27	0.09	0.08	0.08
query28	13.30	1.02	1.00
query29	12.58	3.29	3.28
query30	0.26	0.07	0.06
query31	2.87	0.40	0.38
query32	3.25	0.48	0.47
query33	2.91	2.89	2.94
query34	16.99	4.40	4.45
query35	4.52	4.48	4.50
query36	0.65	0.46	0.46
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.18	0.14	0.15
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.5 s
Total hot run time: 30.88 s

@Lchangliang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39822 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7cae9b00c3df8633cc2d73f6e49cd7878680d567, data reload: false

------ Round 1 ----------------------------------
q1	18380	4652	4370	4370
q2	2612	207	199	199
q3	11673	1133	1195	1133
q4	10595	859	826	826
q5	7903	2690	2666	2666
q6	228	143	140	140
q7	951	610	603	603
q8	9232	2047	2084	2047
q9	8958	6426	6431	6426
q10	8896	3710	3751	3710
q11	459	244	237	237
q12	429	231	228	228
q13	17762	2966	2983	2966
q14	256	215	225	215
q15	519	487	473	473
q16	517	376	374	374
q17	970	727	705	705
q18	8013	7488	7345	7345
q19	5493	1355	1530	1355
q20	647	308	327	308
q21	4893	3157	3771	3157
q22	388	339	345	339
Total cold run time: 119774 ms
Total hot run time: 39822 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4378	4207	4247	4207
q2	362	259	258	258
q3	2987	2705	2698	2698
q4	1872	1613	1661	1613
q5	5246	5270	5255	5255
q6	210	130	129	129
q7	2072	1755	1719	1719
q8	3177	3341	3388	3341
q9	8352	8315	8333	8315
q10	3832	3635	3719	3635
q11	600	473	488	473
q12	787	587	605	587
q13	17411	2986	3021	2986
q14	299	265	281	265
q15	520	483	487	483
q16	466	411	409	409
q17	1782	1482	1454	1454
q18	7635	7568	7317	7317
q19	1757	1624	1461	1461
q20	1996	1780	1743	1743
q21	4848	4772	4857	4772
q22	622	548	541	541
Total cold run time: 71211 ms
Total hot run time: 53661 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171949 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7cae9b00c3df8633cc2d73f6e49cd7878680d567, data reload: false

query1	929	374	369	369
query2	6446	2475	2267	2267
query3	6672	208	208	208
query4	19216	17472	17185	17185
query5	4131	489	471	471
query6	250	148	154	148
query7	4582	289	287	287
query8	320	289	290	289
query9	8471	2393	2370	2370
query10	602	299	278	278
query11	10463	10029	9969	9969
query12	142	92	85	85
query13	1649	376	354	354
query14	9314	7562	7582	7562
query15	227	182	188	182
query16	8022	264	263	263
query17	1902	541	535	535
query18	2046	267	268	267
query19	187	162	158	158
query20	90	84	84	84
query21	206	125	123	123
query22	4317	3992	3997	3992
query23	33487	32983	33029	32983
query24	11918	2787	2795	2787
query25	650	353	354	353
query26	1744	147	147	147
query27	2922	312	313	312
query28	7517	2038	2009	2009
query29	1031	617	591	591
query30	292	146	150	146
query31	951	736	729	729
query32	95	55	54	54
query33	766	288	317	288
query34	908	477	463	463
query35	735	614	602	602
query36	1082	914	943	914
query37	176	68	68	68
query38	2862	2732	2728	2728
query39	842	781	769	769
query40	285	126	124	124
query41	55	56	53	53
query42	124	97	98	97
query43	582	557	544	544
query44	1272	711	727	711
query45	187	162	162	162
query46	1079	743	702	702
query47	1850	1797	1799	1797
query48	358	295	294	294
query49	1168	396	392	392
query50	749	387	386	386
query51	6741	6735	6603	6603
query52	106	89	90	89
query53	356	288	286	286
query54	992	465	436	436
query55	72	69	72	69
query56	269	246	252	246
query57	1127	1044	1045	1044
query58	243	247	241	241
query59	3348	3318	3061	3061
query60	285	263	265	263
query61	93	88	96	88
query62	645	438	453	438
query63	324	282	283	282
query64	9855	2201	1725	1725
query65	3162	3077	3116	3077
query66	1401	326	335	326
query67	15369	15074	14684	14684
query68	4557	530	532	530
query69	462	300	304	300
query70	1142	1116	1141	1116
query71	378	271	269	269
query72	7322	5323	5553	5323
query73	751	319	321	319
query74	5974	5494	5505	5494
query75	3370	2612	2655	2612
query76	2828	1042	850	850
query77	441	289	297	289
query78	10435	9768	9800	9768
query79	2350	499	505	499
query80	2489	453	443	443
query81	589	219	220	219
query82	856	101	96	96
query83	306	166	167	166
query84	258	85	92	85
query85	2251	282	261	261
query86	518	329	310	310
query87	3230	3083	3036	3036
query88	4254	2332	2339	2332
query89	478	373	378	373
query90	1871	193	194	193
query91	128	98	98	98
query92	64	50	50	50
query93	2424	506	496	496
query94	1310	242	180	180
query95	392	309	320	309
query96	594	271	265	265
query97	3219	3088	3046	3046
query98	217	194	197	194
query99	1197	819	819	819
Total cold run time: 276914 ms
Total hot run time: 171949 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7cae9b00c3df8633cc2d73f6e49cd7878680d567, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.52	0.49	0.49
query6	1.12	0.74	0.71
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.53	0.49	0.49
query10	0.55	0.56	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.58	0.60
query14	0.78	0.76	0.77
query15	0.82	0.81	0.83
query16	0.35	0.35	0.36
query17	0.97	1.02	0.98
query18	0.22	0.25	0.27
query19	1.81	1.68	1.72
query20	0.02	0.01	0.01
query21	15.41	0.66	0.65
query22	4.50	7.41	1.61
query23	18.28	1.31	1.26
query24	2.05	0.24	0.23
query25	0.15	0.08	0.08
query26	0.25	0.18	0.17
query27	0.08	0.08	0.07
query28	13.22	1.02	1.00
query29	12.63	3.32	3.26
query30	0.26	0.08	0.06
query31	2.83	0.39	0.38
query32	3.30	0.47	0.47
query33	2.86	2.87	2.97
query34	17.19	4.41	4.47
query35	4.43	4.50	4.45
query36	0.65	0.46	0.46
query37	0.18	0.15	0.14
query38	0.16	0.14	0.14
query39	0.04	0.04	0.03
query40	0.18	0.15	0.13
query41	0.10	0.04	0.05
query42	0.05	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.52 s
Total hot run time: 30.11 s

@Lchangliang
Copy link
Contributor Author

run p0

@Lchangliang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40324 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 465601be26e3777750b058d5af676d53591dc8a9, data reload: false

------ Round 1 ----------------------------------
q1	17620	4358	4213	4213
q2	2017	188	193	188
q3	10457	1092	1143	1092
q4	10195	770	781	770
q5	7430	2643	2646	2643
q6	217	133	133	133
q7	964	597	589	589
q8	9222	2027	2063	2027
q9	8954	6575	6558	6558
q10	8990	3706	3735	3706
q11	463	233	239	233
q12	448	230	227	227
q13	17769	2946	2961	2946
q14	266	225	229	225
q15	502	484	473	473
q16	519	375	377	375
q17	946	718	621	621
q18	8286	8032	7823	7823
q19	6198	1585	1487	1487
q20	696	338	321	321
q21	5184	3322	3332	3322
q22	416	352	352	352
Total cold run time: 117759 ms
Total hot run time: 40324 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4553	4419	4419	4419
q2	374	257	263	257
q3	3051	2903	2845	2845
q4	2002	1723	1731	1723
q5	5502	5505	5508	5505
q6	221	124	124	124
q7	2179	1922	1788	1788
q8	3257	3385	3384	3384
q9	8729	8676	8681	8676
q10	4038	3906	3806	3806
q11	585	488	516	488
q12	821	613	611	611
q13	15926	3019	3190	3019
q14	299	266	289	266
q15	511	481	505	481
q16	524	435	432	432
q17	1813	1472	1469	1469
q18	7963	7771	7747	7747
q19	1775	1676	1660	1660
q20	2146	1875	1828	1828
q21	8897	4762	4845	4762
q22	643	563	558	558
Total cold run time: 75809 ms
Total hot run time: 55848 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172218 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 465601be26e3777750b058d5af676d53591dc8a9, data reload: false

query1	807	388	368	368
query2	6407	2371	2265	2265
query3	6408	202	203	202
query4	19785	17264	17024	17024
query5	3795	465	454	454
query6	244	157	158	157
query7	4451	296	289	289
query8	331	300	294	294
query9	8377	2392	2361	2361
query10	566	342	292	292
query11	10580	10070	9990	9990
query12	139	84	83	83
query13	1580	358	356	356
query14	9967	6806	7388	6806
query15	227	188	197	188
query16	7729	261	261	261
query17	1517	547	508	508
query18	1845	273	265	265
query19	194	152	148	148
query20	98	81	84	81
query21	210	129	127	127
query22	4349	4130	3879	3879
query23	33593	34578	33612	33612
query24	11201	2803	2838	2803
query25	516	349	356	349
query26	706	148	147	147
query27	2756	314	317	314
query28	7254	2043	2017	2017
query29	849	622	638	622
query30	270	147	150	147
query31	934	733	721	721
query32	97	52	53	52
query33	760	287	276	276
query34	895	454	454	454
query35	722	628	624	624
query36	1088	944	927	927
query37	136	67	68	67
query38	2843	2730	2678	2678
query39	864	808	802	802
query40	201	124	138	124
query41	54	52	52	52
query42	126	93	98	93
query43	577	541	521	521
query44	1215	726	714	714
query45	191	165	164	164
query46	1068	700	722	700
query47	1868	1763	1775	1763
query48	374	299	302	299
query49	917	418	410	410
query50	764	385	374	374
query51	6731	6642	6654	6642
query52	135	87	87	87
query53	349	283	292	283
query54	868	440	433	433
query55	71	75	72	72
query56	270	257	264	257
query57	1138	1035	1043	1035
query58	266	243	247	243
query59	3330	3096	3226	3096
query60	289	293	272	272
query61	92	88	89	88
query62	635	465	433	433
query63	328	283	298	283
query64	8793	2216	1775	1775
query65	3174	3076	3095	3076
query66	762	325	343	325
query67	15347	14888	14909	14888
query68	5431	533	534	533
query69	590	428	385	385
query70	1187	1132	1130	1130
query71	431	263	262	262
query72	7123	5495	5488	5488
query73	766	321	319	319
query74	5822	5521	5526	5521
query75	3554	2610	2704	2610
query76	3289	918	901	901
query77	599	291	282	282
query78	10235	9881	9769	9769
query79	2166	503	502	502
query80	1327	447	444	444
query81	547	225	222	222
query82	732	99	101	99
query83	193	157	157	157
query84	277	86	84	84
query85	1326	287	302	287
query86	414	321	329	321
query87	3273	3074	3122	3074
query88	3372	2321	2315	2315
query89	469	378	382	378
query90	1919	185	188	185
query91	127	98	99	98
query92	57	48	48	48
query93	1662	504	496	496
query94	1228	193	182	182
query95	396	309	308	308
query96	579	266	257	257
query97	3198	3039	3023	3023
query98	205	197	193	193
query99	1285	865	833	833
Total cold run time: 268802 ms
Total hot run time: 172218 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 465601be26e3777750b058d5af676d53591dc8a9, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.07
query5	0.49	0.48	0.49
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.54	0.49	0.49
query10	0.55	0.56	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.58	0.58
query14	0.79	0.80	0.78
query15	0.84	0.81	0.82
query16	0.37	0.37	0.37
query17	1.02	0.96	1.04
query18	0.23	0.25	0.23
query19	1.86	1.70	1.67
query20	0.01	0.01	0.01
query21	15.40	0.65	0.66
query22	4.32	7.12	2.22
query23	18.28	1.31	1.24
query24	1.68	0.33	0.23
query25	0.15	0.08	0.07
query26	0.28	0.18	0.18
query27	0.10	0.08	0.08
query28	13.18	1.01	1.02
query29	12.63	3.33	3.32
query30	0.26	0.06	0.06
query31	2.87	0.39	0.39
query32	3.28	0.47	0.47
query33	2.89	2.97	2.91
query34	17.06	4.45	4.37
query35	4.50	4.48	4.46
query36	0.66	0.46	0.47
query37	0.19	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.19	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.13 s
Total hot run time: 30.83 s

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 15, 2024
try {
BinlogConfig binlogConfig = new BinlogConfig(tbl.getBinlogConfig());
Preconditions.checkState(tbl.getState() == OlapTableState.ROLLUP);
Map<Object, Object> objectPool = new HashMap<Object, Object>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to use a concurrent map.

@dataroaring dataroaring merged commit 3dd2733 into apache:master Jun 15, 2024
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
…dow tablets in schema change process (#36285)

When doing schema change, it will create many shadow tablets. And many
TCreateTabletReq will be new. So we can use object pool to reduct they
memory usage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants