Skip to content

Conversation

@Hastyshell
Copy link
Collaborator

Reverts #41360

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Hastyshell Hastyshell changed the title Revert "Revert "[enhancement](mem-tracker) Use thread local mem tracker to track s3 file buffer memory usage"" [DNM] test Sep 27, 2024
@Hastyshell
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9626/25811)
Line Coverage: 28.70% (79664/277607)
Region Coverage: 28.13% (41188/146441)
Branch Coverage: 24.75% (20972/84752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ac9a93db9314f970d2d7892469ebe124eac9c062_ac9a93db9314f970d2d7892469ebe124eac9c062/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40953 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac9a93db9314f970d2d7892469ebe124eac9c062, data reload: false

------ Round 1 ----------------------------------
q1	17571	7845	7260	7260
q2	2015	273	267	267
q3	12277	1071	1222	1071
q4	10563	786	744	744
q5	7774	2886	2804	2804
q6	245	154	151	151
q7	964	597	610	597
q8	9365	1988	1949	1949
q9	6569	6425	6431	6425
q10	7004	2271	2326	2271
q11	435	241	251	241
q12	404	213	221	213
q13	17771	3031	2980	2980
q14	249	210	203	203
q15	580	534	502	502
q16	637	576	603	576
q17	990	545	570	545
q18	7267	6781	6749	6749
q19	1406	967	1052	967
q20	471	207	198	198
q21	4022	3273	3323	3273
q22	1100	967	985	967
Total cold run time: 109679 ms
Total hot run time: 40953 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7248	7249	7203	7203
q2	328	235	226	226
q3	3104	2956	2998	2956
q4	2073	1867	1820	1820
q5	5810	5764	5808	5764
q6	230	148	148	148
q7	2271	1869	1821	1821
q8	3425	3518	3553	3518
q9	8986	8945	8908	8908
q10	3604	3578	3558	3558
q11	579	507	484	484
q12	826	610	649	610
q13	9429	3197	3188	3188
q14	317	283	274	274
q15	593	527	535	527
q16	675	640	647	640
q17	1859	1627	1611	1611
q18	8264	7693	7505	7505
q19	1739	1439	1575	1439
q20	2118	1861	1881	1861
q21	5579	5458	5367	5367
q22	1118	1071	1089	1071
Total cold run time: 70175 ms
Total hot run time: 60499 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192638 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac9a93db9314f970d2d7892469ebe124eac9c062, data reload: false

query1	919	396	403	396
query2	6283	2124	2084	2084
query3	8688	207	213	207
query4	33710	23586	23562	23562
query5	3950	475	473	473
query6	275	169	167	167
query7	4191	305	317	305
query8	279	224	225	224
query9	9401	2670	2660	2660
query10	496	282	285	282
query11	17885	15137	15337	15137
query12	147	102	100	100
query13	1530	419	405	405
query14	10151	7671	7544	7544
query15	254	175	175	175
query16	7934	515	487	487
query17	1687	619	603	603
query18	2064	325	319	319
query19	369	158	157	157
query20	143	115	122	115
query21	222	118	114	114
query22	4693	4446	4446	4446
query23	36189	34369	34024	34024
query24	10945	2855	2859	2855
query25	519	411	417	411
query26	711	164	172	164
query27	2163	299	308	299
query28	7529	2424	2444	2424
query29	677	448	446	446
query30	257	158	156	156
query31	1012	805	783	783
query32	94	59	55	55
query33	807	321	310	310
query34	911	546	493	493
query35	901	755	757	755
query36	1087	955	977	955
query37	147	87	91	87
query38	4090	3867	3964	3867
query39	1480	1460	1415	1415
query40	207	100	99	99
query41	53	47	54	47
query42	117	95	99	95
query43	553	514	502	502
query44	1274	791	801	791
query45	197	166	170	166
query46	1126	725	715	715
query47	1950	1820	1839	1820
query48	470	394	389	389
query49	912	413	427	413
query50	842	417	415	415
query51	7066	6882	6979	6882
query52	102	88	86	86
query53	253	186	191	186
query54	1196	475	489	475
query55	80	79	78	78
query56	284	273	276	273
query57	1201	1108	1120	1108
query58	241	237	267	237
query59	3122	3066	2934	2934
query60	287	269	269	269
query61	105	104	111	104
query62	836	691	653	653
query63	217	187	186	186
query64	3901	628	615	615
query65	3397	3260	3210	3210
query66	852	302	297	297
query67	16046	15584	15545	15545
query68	4735	569	583	569
query69	628	296	302	296
query70	1198	1133	1139	1133
query71	420	286	275	275
query72	7597	4103	4076	4076
query73	771	342	349	342
query74	10662	8928	9129	8928
query75	4617	2683	2693	2683
query76	3801	876	984	876
query77	743	290	303	290
query78	10466	9852	9556	9556
query79	4562	588	595	588
query80	2264	450	446	446
query81	600	242	243	242
query82	716	149	147	147
query83	320	139	143	139
query84	292	76	80	76
query85	1733	287	287	287
query86	432	300	309	300
query87	4430	4261	4368	4261
query88	4053	2427	2396	2396
query89	414	297	287	287
query90	2179	194	193	193
query91	183	145	145	145
query92	74	48	47	47
query93	3329	569	548	548
query94	1208	290	298	290
query95	360	260	260	260
query96	635	281	286	281
query97	3254	3143	3148	3143
query98	224	201	197	197
query99	1567	1290	1325	1290
Total cold run time: 310144 ms
Total hot run time: 192638 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.95 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ac9a93db9314f970d2d7892469ebe124eac9c062, data reload: false

query1	0.05	0.04	0.05
query2	0.07	0.02	0.02
query3	0.22	0.07	0.07
query4	1.65	0.11	0.10
query5	0.54	0.52	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.54	0.54	0.56
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.59	0.60
query14	2.77	2.78	2.77
query15	0.90	0.82	0.81
query16	0.37	0.41	0.39
query17	1.03	0.96	1.01
query18	0.20	0.20	0.20
query19	1.93	1.77	1.89
query20	0.01	0.01	0.01
query21	15.35	0.61	0.60
query22	2.59	2.57	1.77
query23	17.22	0.91	0.83
query24	2.96	0.31	1.28
query25	0.30	0.11	0.04
query26	0.49	0.14	0.14
query27	0.04	0.04	0.04
query28	11.20	1.11	1.08
query29	12.53	3.28	3.28
query30	0.26	0.06	0.07
query31	2.86	0.38	0.37
query32	3.28	0.47	0.46
query33	2.96	3.01	3.05
query34	17.03	4.46	4.44
query35	4.51	4.50	4.52
query36	0.67	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 107.61 s
Total hot run time: 31.95 s

@Hastyshell Hastyshell force-pushed the revert-41360-revert-40597-enhance-s3-file-buf-tracker branch from ac9a93d to 853298c Compare September 27, 2024 07:32
@Hastyshell
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.30% (9628/25811)
Line Coverage: 28.70% (79670/277603)
Region Coverage: 28.14% (41202/146440)
Branch Coverage: 24.75% (20974/84752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/853298c28aab8764e20f54cb84d52746bb21e220_853298c28aab8764e20f54cb84d52746bb21e220/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40939 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 853298c28aab8764e20f54cb84d52746bb21e220, data reload: false

------ Round 1 ----------------------------------
q1	17647	7363	7286	7286
q2	2023	272	269	269
q3	12213	1054	1135	1054
q4	10578	786	800	786
q5	7748	2882	2863	2863
q6	238	152	149	149
q7	987	623	612	612
q8	9387	1940	2033	1940
q9	6596	6381	6397	6381
q10	6982	2290	2289	2289
q11	426	239	247	239
q12	407	211	213	211
q13	17796	2973	3009	2973
q14	240	215	211	211
q15	636	534	508	508
q16	628	611	574	574
q17	960	544	517	517
q18	7237	6689	6761	6689
q19	1390	1002	1085	1002
q20	462	201	197	197
q21	3925	3313	3203	3203
q22	1093	986	993	986
Total cold run time: 109599 ms
Total hot run time: 40939 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7245	7236	7267	7236
q2	328	227	228	227
q3	2995	2919	2960	2919
q4	2093	1911	1835	1835
q5	5779	5746	5776	5746
q6	244	147	145	145
q7	2202	1865	1814	1814
q8	3346	3569	3443	3443
q9	8891	8973	8866	8866
q10	3575	3578	3532	3532
q11	591	491	511	491
q12	816	624	632	624
q13	10286	3173	3174	3173
q14	313	270	283	270
q15	584	518	528	518
q16	704	669	660	660
q17	1863	1636	1623	1623
q18	8240	7755	7505	7505
q19	1693	1540	1599	1540
q20	2119	1880	1887	1880
q21	5375	5275	5401	5275
q22	1164	1038	1040	1038
Total cold run time: 70446 ms
Total hot run time: 60360 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191431 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 853298c28aab8764e20f54cb84d52746bb21e220, data reload: false

query1	940	410	414	410
query2	6306	2060	1981	1981
query3	8685	203	206	203
query4	33584	23428	23545	23428
query5	3795	462	463	462
query6	301	177	168	168
query7	4199	309	301	301
query8	284	232	224	224
query9	9385	2642	2645	2642
query10	443	288	290	288
query11	17876	15203	15234	15203
query12	155	97	95	95
query13	1534	438	417	417
query14	9782	7512	7280	7280
query15	253	166	168	166
query16	7834	466	470	466
query17	1650	628	598	598
query18	1772	315	313	313
query19	305	158	157	157
query20	120	123	115	115
query21	213	109	109	109
query22	4793	4493	4450	4450
query23	36494	33975	34320	33975
query24	10937	2817	2831	2817
query25	535	407	412	407
query26	742	160	166	160
query27	2074	297	304	297
query28	6887	2402	2406	2402
query29	679	437	443	437
query30	282	154	169	154
query31	1059	806	771	771
query32	99	55	56	55
query33	777	307	301	301
query34	960	505	487	487
query35	860	755	720	720
query36	1118	935	959	935
query37	147	84	92	84
query38	4184	3902	3890	3890
query39	1478	1427	1416	1416
query40	207	96	97	96
query41	52	48	49	48
query42	115	103	100	100
query43	524	487	465	465
query44	1224	805	794	794
query45	196	163	173	163
query46	1130	741	728	728
query47	1941	1812	1837	1812
query48	464	370	380	370
query49	942	418	437	418
query50	816	413	430	413
query51	6985	6868	6911	6868
query52	100	85	89	85
query53	255	182	176	176
query54	1226	452	466	452
query55	78	77	77	77
query56	271	262	263	262
query57	1200	1117	1089	1089
query58	237	250	239	239
query59	3115	2864	2919	2864
query60	304	272	255	255
query61	103	103	97	97
query62	878	663	663	663
query63	225	194	187	187
query64	3627	638	612	612
query65	3328	3213	3215	3213
query66	773	337	297	297
query67	15926	15592	15431	15431
query68	4495	570	568	568
query69	579	294	294	294
query70	1182	1124	1144	1124
query71	425	273	272	272
query72	7418	4051	3944	3944
query73	777	347	338	338
query74	10366	8933	9060	8933
query75	3805	2672	2661	2661
query76	3672	810	983	810
query77	630	294	296	294
query78	11559	10102	9578	9578
query79	3633	597	580	580
query80	2217	466	440	440
query81	579	244	262	244
query82	668	142	135	135
query83	315	134	138	134
query84	290	76	77	76
query85	1013	286	289	286
query86	493	299	299	299
query87	4423	4294	4347	4294
query88	4456	2382	2390	2382
query89	430	288	283	283
query90	2185	188	185	185
query91	201	143	148	143
query92	70	49	46	46
query93	4818	550	545	545
query94	1115	282	287	282
query95	360	253	258	253
query96	637	288	285	285
query97	3304	3142	3173	3142
query98	225	211	200	200
query99	1725	1323	1335	1323
Total cold run time: 307828 ms
Total hot run time: 191431 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 853298c28aab8764e20f54cb84d52746bb21e220, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.67	0.11	0.10
query5	0.50	0.52	0.52
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.03	0.03	0.03
query9	0.56	0.49	0.50
query10	0.54	0.54	0.55
query11	0.13	0.10	0.10
query12	0.13	0.10	0.11
query13	0.62	0.59	0.60
query14	2.85	2.76	2.78
query15	0.92	0.82	0.84
query16	0.37	0.38	0.42
query17	1.02	0.98	1.00
query18	0.19	0.19	0.19
query19	1.87	1.78	2.00
query20	0.01	0.01	0.01
query21	15.36	0.62	0.59
query22	2.78	2.48	1.75
query23	16.68	1.27	0.77
query24	2.61	0.99	1.55
query25	0.29	0.08	0.06
query26	0.41	0.14	0.14
query27	0.05	0.04	0.04
query28	10.84	1.09	1.07
query29	12.54	3.32	3.26
query30	0.25	0.06	0.06
query31	2.90	0.39	0.38
query32	3.27	0.48	0.47
query33	2.97	3.03	3.00
query34	17.05	4.42	4.49
query35	4.50	4.51	4.47
query36	0.65	0.49	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 106.52 s
Total hot run time: 32.51 s

@Hastyshell Hastyshell force-pushed the revert-41360-revert-40597-enhance-s3-file-buf-tracker branch from 853298c to fbde507 Compare September 27, 2024 08:32
@Hastyshell
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41084 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fbde5076086c99f725894cf3b74c6a2f299b0a6a, data reload: false

------ Round 1 ----------------------------------
q1	18000	7526	7406	7406
q2	2455	159	180	159
q3	10553	1178	1165	1165
q4	10216	778	778	778
q5	7756	2900	2831	2831
q6	242	151	147	147
q7	1004	627	633	627
q8	9350	1987	1985	1985
q9	6601	6410	6460	6410
q10	6973	2308	2300	2300
q11	451	251	248	248
q12	415	225	215	215
q13	17776	2988	2960	2960
q14	247	210	223	210
q15	594	529	506	506
q16	634	607	567	567
q17	1004	577	549	549
q18	7401	6709	6658	6658
q19	1396	1069	1029	1029
q20	505	205	202	202
q21	4035	3201	3152	3152
q22	1108	980	1006	980
Total cold run time: 108716 ms
Total hot run time: 41084 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7485	7219	7216	7216
q2	344	233	228	228
q3	2927	2783	2781	2781
q4	1967	1783	1746	1746
q5	5446	5500	5514	5500
q6	225	143	138	138
q7	2124	1705	1740	1705
q8	3248	3404	3398	3398
q9	8534	8513	8534	8513
q10	3502	3429	3425	3425
q11	578	478	483	478
q12	791	586	593	586
q13	8968	2997	2981	2981
q14	298	254	263	254
q15	552	506	508	506
q16	678	614	622	614
q17	1809	1579	1583	1579
q18	7749	7486	7495	7486
q19	1657	1475	1484	1475
q20	2012	1793	1810	1793
q21	5498	5163	5155	5155
q22	1110	1003	991	991
Total cold run time: 67502 ms
Total hot run time: 58548 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192143 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fbde5076086c99f725894cf3b74c6a2f299b0a6a, data reload: false

query1	975	370	371	370
query2	6524	2078	2001	2001
query3	6701	217	219	217
query4	33966	23435	23496	23435
query5	4330	486	479	479
query6	281	190	171	171
query7	4628	315	310	310
query8	292	237	230	230
query9	9540	2670	2660	2660
query10	465	283	284	283
query11	17949	15157	15173	15157
query12	158	100	100	100
query13	1632	460	427	427
query14	10957	7184	7428	7184
query15	322	173	184	173
query16	8029	470	480	470
query17	1791	570	596	570
query18	2159	307	311	307
query19	376	156	156	156
query20	119	111	109	109
query21	219	105	106	105
query22	4370	4435	4131	4131
query23	34953	34205	35024	34205
query24	11163	2878	2838	2838
query25	617	426	420	420
query26	1167	163	163	163
query27	2770	301	298	298
query28	8054	2471	2450	2450
query29	711	444	443	443
query30	314	172	154	154
query31	1042	776	813	776
query32	99	59	58	58
query33	786	313	309	309
query34	932	505	504	504
query35	861	751	744	744
query36	1131	911	953	911
query37	154	87	87	87
query38	3992	3951	4062	3951
query39	1514	1469	1411	1411
query40	297	102	100	100
query41	54	46	53	46
query42	122	99	98	98
query43	538	483	492	483
query44	1301	845	814	814
query45	194	169	169	169
query46	1162	718	711	711
query47	1947	1813	1841	1813
query48	469	378	373	373
query49	1182	437	409	409
query50	839	413	405	405
query51	7169	6832	6937	6832
query52	104	93	91	91
query53	264	192	192	192
query54	1202	474	479	474
query55	84	80	86	80
query56	286	274	273	273
query57	1206	1112	1102	1102
query58	275	225	232	225
query59	3084	3002	3028	3002
query60	301	273	269	269
query61	110	112	108	108
query62	872	650	680	650
query63	234	196	194	194
query64	5147	665	634	634
query65	3329	3217	3189	3189
query66	1259	308	309	308
query67	16146	15641	15524	15524
query68	5170	575	553	553
query69	493	296	296	296
query70	1197	1113	1153	1113
query71	346	277	285	277
query72	6938	4046	4220	4046
query73	773	347	347	347
query74	10336	9061	9212	9061
query75	3454	2703	2683	2683
query76	3262	859	926	859
query77	531	313	309	309
query78	10642	9594	9619	9594
query79	4053	620	609	609
query80	1298	464	468	464
query81	565	237	246	237
query82	728	146	139	139
query83	259	144	139	139
query84	300	84	74	74
query85	1942	298	282	282
query86	499	314	306	306
query87	4481	4375	4374	4374
query88	4576	2374	2365	2365
query89	402	302	294	294
query90	2018	190	189	189
query91	183	144	145	144
query92	68	51	50	50
query93	2977	555	547	547
query94	988	284	300	284
query95	362	254	258	254
query96	630	280	292	280
query97	3263	3132	3140	3132
query98	224	198	209	198
query99	1640	1315	1309	1309
Total cold run time: 309818 ms
Total hot run time: 192143 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9626/25811)
Line Coverage: 28.69% (79650/277599)
Region Coverage: 28.12% (41185/146439)
Branch Coverage: 24.75% (20974/84752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fbde5076086c99f725894cf3b74c6a2f299b0a6a_fbde5076086c99f725894cf3b74c6a2f299b0a6a/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 32.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fbde5076086c99f725894cf3b74c6a2f299b0a6a, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.02	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.53	0.49	0.50
query6	1.13	0.73	0.72
query7	0.01	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.51
query10	0.56	0.57	0.57
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.59	0.60
query14	2.69	2.70	2.76
query15	0.89	0.81	0.83
query16	0.39	0.40	0.40
query17	1.04	1.01	1.01
query18	0.24	0.22	0.22
query19	1.92	1.85	2.01
query20	0.01	0.01	0.01
query21	15.38	0.58	0.58
query22	2.76	1.99	1.72
query23	16.99	1.09	0.73
query24	2.62	1.42	0.64
query25	0.22	0.16	0.04
query26	0.46	0.14	0.14
query27	0.05	0.04	0.04
query28	10.94	1.09	1.07
query29	12.82	3.31	3.28
query30	0.25	0.07	0.07
query31	2.86	0.37	0.38
query32	3.28	0.45	0.46
query33	2.98	2.99	2.99
query34	16.95	4.45	4.46
query35	4.45	4.53	4.54
query36	0.67	0.49	0.47
query37	0.08	0.06	0.05
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.06 s
Total hot run time: 32.21 s

@Hastyshell Hastyshell closed this Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants