[enhancement](mem-tracker) Use thread local mem tracker to track s3 file buffer memory usage #40597

Hastyshell · 2024-09-10T07:01:40Z

Proposed changes

Track s3 file buffer memory usage with thread local tracker, so that memory usage will be specified to detail.

…ile buffer memory usage

doris-robot · 2024-09-10T07:01:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Hastyshell · 2024-09-10T07:01:48Z

run buildall

doris-robot · 2024-09-10T07:46:14Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (9395/25508)
Line Coverage: 28.23% (77493/274521)
Region Coverage: 27.64% (40016/144801)
Branch Coverage: 24.26% (20355/83894)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e4d72ea783a7509f9ed383864fdf4e597d75cc2_0e4d72ea783a7509f9ed383864fdf4e597d75cc2/report/index.html

yiguolei

LGTM

github-actions · 2024-09-10T08:38:18Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-10T08:38:19Z

PR approved by anyone and no changes requested.

doris-robot · 2024-09-10T12:53:23Z

TPC-H: Total hot run time: 39643 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e4d72ea783a7509f9ed383864fdf4e597d75cc2, data reload: false

------ Round 1 ----------------------------------
q1	17901	5017	4503	4503
q2	2032	199	193	193
q3	11729	1018	1259	1018
q4	10538	773	864	773
q5	7795	2971	2974	2971
q6	249	147	144	144
q7	1046	648	633	633
q8	9349	2205	2200	2200
q9	7349	6831	6848	6831
q10	7098	2283	2264	2264
q11	470	255	249	249
q12	444	233	231	231
q13	17955	3109	3168	3109
q14	314	242	250	242
q15	567	515	502	502
q16	541	437	439	437
q17	1056	709	780	709
q18	7459	6967	6989	6967
q19	1431	1193	1103	1103
q20	713	342	356	342
q21	4241	3223	3305	3223
q22	1138	1018	999	999
Total cold run time: 111415 ms
Total hot run time: 39643 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4893	4501	4532	4501
q2	405	271	280	271
q3	2989	2772	2771	2771
q4	2117	1794	1769	1769
q5	5857	5902	5761	5761
q6	258	136	137	136
q7	2304	1829	1828	1828
q8	3563	3799	3805	3799
q9	8959	8968	9046	8968
q10	3785	3451	3520	3451
q11	666	517	534	517
q12	840	611	618	611
q13	14783	3346	3344	3344
q14	333	295	292	292
q15	572	505	503	503
q16	557	512	528	512
q17	2001	1650	1650	1650
q18	8289	8031	7931	7931
q19	1913	1661	1692	1661
q20	2183	1951	1966	1951
q21	6058	5756	5732	5732
q22	1147	1007	1049	1007
Total cold run time: 74472 ms
Total hot run time: 58966 ms

doris-robot · 2024-09-10T13:05:32Z

TPC-DS: Total hot run time: 198283 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e4d72ea783a7509f9ed383864fdf4e597d75cc2, data reload: false

query1	1277	889	934	889
query2	6457	1974	1893	1893
query3	10611	4019	4111	4019
query4	58387	25477	23667	23667
query5	4921	511	505	505
query6	382	174	174	174
query7	5562	295	292	292
query8	328	234	219	219
query9	7391	2521	2512	2512
query10	408	288	264	264
query11	16417	15143	15227	15143
query12	167	108	103	103
query13	1422	422	406	406
query14	10600	7444	7455	7444
query15	207	177	178	177
query16	6762	489	496	489
query17	1100	573	577	573
query18	1473	300	284	284
query19	192	144	147	144
query20	119	116	108	108
query21	214	102	104	102
query22	4724	4621	4326	4326
query23	34364	33808	33897	33808
query24	5968	2921	2832	2832
query25	495	378	380	378
query26	597	152	149	149
query27	1585	284	280	280
query28	3866	2074	2068	2068
query29	654	411	415	411
query30	241	151	157	151
query31	956	782	786	782
query32	68	55	52	52
query33	459	290	294	290
query34	888	472	468	468
query35	843	708	747	708
query36	1049	905	941	905
query37	143	89	84	84
query38	4017	3912	3934	3912
query39	1441	1420	1404	1404
query40	205	112	112	112
query41	45	46	45	45
query42	123	96	93	93
query43	516	493	463	463
query44	1087	762	749	749
query45	194	168	167	167
query46	1096	748	736	736
query47	1893	1780	1820	1780
query48	374	290	317	290
query49	769	450	463	450
query50	821	405	412	405
query51	6990	6945	6883	6883
query52	100	84	86	84
query53	246	176	180	176
query54	569	458	456	456
query55	75	77	74	74
query56	274	245	263	245
query57	1226	1071	1104	1071
query58	219	231	236	231
query59	2998	2871	2767	2767
query60	315	263	262	262
query61	103	97	102	97
query62	755	641	659	641
query63	219	190	183	183
query64	1356	679	668	668
query65	3219	3136	3165	3136
query66	651	337	346	337
query67	15939	15205	15202	15202
query68	1395	866	860	860
query69	423	318	319	318
query70	1185	1123	1205	1123
query71	350	341	330	330
query72	4658	3498	3505	3498
query73	596	574	578	574
query74	9191	8959	8847	8847
query75	2966	2942	3017	2942
query76	999	844	839	839
query77	507	409	402	402
query78	9310	9440	9186	9186
query79	898	883	866	866
query80	792	798	784	784
query81	450	260	263	260
query82	272	263	264	263
query83	192	189	188	188
query84	187	107	106	106
query85	628	393	389	389
query86	313	314	318	314
query87	4363	4406	4263	4263
query88	4327	4048	4066	4048
query89	372	356	362	356
query90	1432	310	309	309
query91	121	129	124	124
query92	79	83	80	80
query93	909	903	900	900
query94	528	376	374	374
query95	424	414	418	414
query96	473	463	462	462
query97	3112	3158	3144	3144
query98	231	231	230	230
query99	1296	1272	1281	1272
Total cold run time: 292111 ms
Total hot run time: 198283 ms

doris-robot · 2024-09-10T13:11:03Z

ClickBench: Total hot run time: 31.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e4d72ea783a7509f9ed383864fdf4e597d75cc2, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.22	0.06	0.07
query4	1.64	0.10	0.08
query5	0.51	0.50	0.50
query6	1.12	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.53	0.53
query11	0.16	0.13	0.12
query12	0.16	0.13	0.12
query13	0.62	0.59	0.59
query14	1.37	1.43	1.45
query15	0.86	0.83	0.82
query16	0.41	0.38	0.36
query17	1.05	0.97	0.98
query18	0.21	0.19	0.20
query19	1.80	1.79	1.79
query20	0.01	0.01	0.02
query21	15.41	0.66	0.67
query22	4.36	7.09	2.09
query23	18.28	1.44	1.34
query24	2.08	0.23	0.22
query25	0.16	0.09	0.09
query26	0.26	0.19	0.18
query27	0.08	0.08	0.07
query28	13.24	1.01	1.00
query29	12.58	3.36	3.36
query30	0.24	0.05	0.06
query31	2.86	0.41	0.40
query32	3.24	0.49	0.48
query33	3.00	2.97	3.06
query34	17.00	4.34	4.46
query35	4.46	4.49	4.51
query36	0.66	0.50	0.50
query37	0.19	0.17	0.16
query38	0.15	0.14	0.15
query39	0.06	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.04	0.04
query42	0.05	0.06	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.05 s
Total hot run time: 31.75 s

…ile buffer memory usage

Hastyshell · 2024-09-11T00:14:50Z

run buildall

doris-robot · 2024-09-11T00:48:13Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (9396/25510)
Line Coverage: 28.24% (77517/274536)
Region Coverage: 27.63% (40012/144808)
Branch Coverage: 24.26% (20352/83900)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6c64b0f1f985cc59f8878ff67656541bb60207c4_6c64b0f1f985cc59f8878ff67656541bb60207c4/report/index.html

doris-robot · 2024-09-11T00:53:28Z

TPC-H: Total hot run time: 37984 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c64b0f1f985cc59f8878ff67656541bb60207c4, data reload: false

------ Round 1 ----------------------------------
q1	17623	4358	4292	4292
q2	2017	188	183	183
q3	11682	953	1165	953
q4	10495	804	677	677
q5	7749	2859	2808	2808
q6	225	139	138	138
q7	949	618	614	614
q8	9314	2049	2047	2047
q9	7288	6539	6534	6534
q10	7009	2205	2161	2161
q11	449	244	242	242
q12	392	218	225	218
q13	17759	3098	3090	3090
q14	273	236	233	233
q15	536	502	488	488
q16	542	436	435	435
q17	985	678	778	678
q18	7260	6893	6932	6893
q19	1394	1007	1090	1007
q20	673	343	335	335
q21	3896	2947	2958	2947
q22	1103	1023	1011	1011
Total cold run time: 109613 ms
Total hot run time: 37984 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4275	4282	4275
q2	390	277	258	258
q3	2844	2614	2687	2614
q4	1942	1635	1678	1635
q5	5603	5733	5736	5733
q6	225	152	150	150
q7	2217	1830	1857	1830
q8	3303	3425	3415	3415
q9	8886	8925	8838	8838
q10	3548	3374	3400	3374
q11	619	525	526	525
q12	849	639	684	639
q13	12698	3313	3214	3214
q14	312	291	295	291
q15	545	480	506	480
q16	552	489	479	479
q17	1819	1518	1550	1518
q18	8231	8003	7837	7837
q19	1746	1660	1656	1656
q20	2148	1913	1927	1913
q21	5833	5473	5641	5473
q22	1139	1010	999	999
Total cold run time: 69806 ms
Total hot run time: 57146 ms

doris-robot · 2024-09-11T01:05:46Z

TPC-DS: Total hot run time: 196957 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c64b0f1f985cc59f8878ff67656541bb60207c4, data reload: false

query1	1257	870	875	870
query2	6346	1950	1881	1881
query3	10579	3900	3904	3900
query4	60276	29152	23221	23221
query5	5115	501	507	501
query6	398	166	165	165
query7	5648	308	296	296
query8	319	223	222	222
query9	7713	2514	2519	2514
query10	410	275	268	268
query11	17260	15163	15405	15163
query12	151	103	109	103
query13	1475	396	389	389
query14	10621	7276	7115	7115
query15	207	177	164	164
query16	6561	500	508	500
query17	1086	587	586	586
query18	1486	296	292	292
query19	197	150	146	146
query20	130	109	107	107
query21	208	102	107	102
query22	4638	4531	4562	4531
query23	34287	33470	33616	33470
query24	6025	2916	2905	2905
query25	490	384	381	381
query26	608	152	154	152
query27	1611	279	281	279
query28	3772	2031	2006	2006
query29	646	407	409	407
query30	221	156	153	153
query31	920	730	773	730
query32	67	53	51	51
query33	431	289	285	285
query34	869	462	484	462
query35	841	710	735	710
query36	1063	943	907	907
query37	143	87	85	85
query38	3996	3975	3871	3871
query39	1429	1427	1395	1395
query40	201	114	117	114
query41	48	49	46	46
query42	121	101	98	98
query43	501	481	481	481
query44	1102	763	751	751
query45	195	168	166	166
query46	1096	752	756	752
query47	1929	1806	1814	1806
query48	379	301	304	301
query49	785	451	456	451
query50	850	416	430	416
query51	7082	6839	6963	6839
query52	97	89	88	88
query53	254	181	183	181
query54	580	487	470	470
query55	80	74	77	74
query56	292	266	262	262
query57	1216	1055	1073	1055
query58	220	291	221	221
query59	3035	2835	2744	2744
query60	286	273	268	268
query61	104	100	101	100
query62	755	663	676	663
query63	215	185	182	182
query64	1323	670	695	670
query65	3243	3150	3155	3150
query66	661	336	357	336
query67	15740	15665	15147	15147
query68	1910	552	552	552
query69	402	269	272	269
query70	1143	1128	1140	1128
query71	332	277	267	267
query72	4620	4094	3921	3921
query73	756	328	332	328
query74	9171	8892	8906	8892
query75	3301	2768	2709	2709
query76	1356	1001	922	922
query77	519	330	326	326
query78	9789	9279	9212	9212
query79	944	902	860	860
query80	854	817	806	806
query81	532	256	261	256
query82	268	264	266	264
query83	192	186	188	186
query84	266	105	105	105
query85	759	404	402	402
query86	319	311	309	309
query87	4352	4460	4400	4400
query88	4848	4135	4144	4135
query89	371	366	363	363
query90	2015	309	312	309
query91	125	128	120	120
query92	79	73	75	73
query93	911	912	912	912
query94	723	359	382	359
query95	448	417	413	413
query96	472	473	473	473
query97	3109	3132	3092	3092
query98	231	241	235	235
query99	1447	1263	1242	1242
Total cold run time: 298454 ms
Total hot run time: 196957 ms

doris-robot · 2024-09-11T01:11:16Z

ClickBench: Total hot run time: 31.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c64b0f1f985cc59f8878ff67656541bb60207c4, data reload: false

query1	0.04	0.04	0.05
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.08	0.09
query5	0.50	0.50	0.50
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.53	0.50	0.48
query10	0.55	0.57	0.54
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.58
query14	1.39	1.39	1.39
query15	0.85	0.85	0.82
query16	0.35	0.36	0.38
query17	0.99	0.96	1.03
query18	0.22	0.21	0.21
query19	1.84	1.75	1.82
query20	0.02	0.01	0.01
query21	15.40	0.66	0.66
query22	4.72	6.49	2.18
query23	18.33	1.41	1.29
query24	2.12	0.23	0.22
query25	0.15	0.09	0.09
query26	0.26	0.18	0.18
query27	0.07	0.08	0.08
query28	13.23	1.03	1.00
query29	12.59	3.38	3.33
query30	0.24	0.06	0.05
query31	2.87	0.40	0.38
query32	3.27	0.48	0.48
query33	2.97	2.99	2.98
query34	17.10	4.43	4.48
query35	4.44	4.46	4.48
query36	0.65	0.47	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.51 s
Total hot run time: 31.72 s

xinyiZzz

LGTM

xinyiZzz · 2024-09-11T02:32:19Z

Hastyshell · 2024-09-11T07:29:03Z

run buildall

Hastyshell · 2024-09-11T07:56:31Z

run buildall

doris-robot · 2024-09-14T13:16:20Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 68aeb8f3aab502d49fd44ff2a6066b7c5f8d9fdf, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.51	0.49	0.50
query6	1.13	0.74	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.51
query10	0.56	0.57	0.57
query11	0.14	0.11	0.10
query12	0.14	0.10	0.11
query13	0.61	0.59	0.59
query14	1.41	1.43	1.46
query15	0.87	0.88	0.85
query16	0.36	0.37	0.39
query17	1.02	1.05	1.08
query18	0.19	0.18	0.20
query19	1.95	1.82	1.78
query20	0.01	0.01	0.01
query21	15.38	0.55	0.55
query22	3.17	4.06	2.69
query23	17.59	1.03	0.93
query24	2.33	0.27	0.21
query25	0.24	0.05	0.06
query26	0.16	0.13	0.14
query27	0.04	0.03	0.03
query28	12.45	1.02	0.99
query29	12.59	3.27	3.28
query30	0.25	0.06	0.05
query31	2.88	0.39	0.37
query32	3.32	0.47	0.46
query33	3.01	3.03	3.05
query34	15.42	4.31	4.26
query35	4.28	4.33	4.29
query36	0.71	0.48	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.78 s
Total hot run time: 31.11 s

github-actions · 2024-09-19T02:09:06Z

PR approved by at least one committer and no changes requested.

xinyiZzz

LGTM

Hastyshell · 2024-09-19T03:51:21Z

run buildall

doris-robot · 2024-09-19T04:39:20Z

TPC-H: Total hot run time: 42086 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7302a5c966cf23f047d874962a5e37e365b1a94c, data reload: false

------ Round 1 ----------------------------------
q1	18200	7437	7364	7364
q2	2815	161	170	161
q3	11360	1167	1272	1167
q4	10585	815	740	740
q5	8321	3165	3162	3162
q6	239	162	151	151
q7	1033	625	612	612
q8	9438	2076	2070	2070
q9	6952	6448	6435	6435
q10	7032	2283	2301	2283
q11	443	251	258	251
q12	417	213	212	212
q13	17785	3018	2991	2991
q14	261	212	219	212
q15	589	537	509	509
q16	666	601	613	601
q17	994	762	795	762
q18	7323	6815	6795	6795
q19	1407	1022	1004	1004
q20	610	299	288	288
q21	4161	3417	3319	3319
q22	1146	997	1022	997
Total cold run time: 111777 ms
Total hot run time: 42086 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7229	7270	7180	7180
q2	331	232	226	226
q3	2869	2764	2776	2764
q4	1922	1701	1668	1668
q5	5391	5411	5402	5402
q6	226	138	140	138
q7	2098	1701	1701	1701
q8	3211	3328	3363	3328
q9	8414	8390	8439	8390
q10	3407	3338	3357	3338
q11	576	469	478	469
q12	788	591	640	591
q13	5755	2984	2999	2984
q14	301	263	264	263
q15	575	526	517	517
q16	699	657	669	657
q17	1752	1543	1537	1537
q18	7771	7223	7425	7223
q19	1676	1700	1460	1460
q20	2052	1794	1798	1794
q21	5343	5268	5203	5203
q22	1090	1020	1028	1020
Total cold run time: 63476 ms
Total hot run time: 57853 ms

doris-robot · 2024-09-19T04:51:20Z

TPC-DS: Total hot run time: 194073 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7302a5c966cf23f047d874962a5e37e365b1a94c, data reload: false

query1	974	368	360	360
query2	6507	2112	2097	2097
query3	6694	217	224	217
query4	34132	23282	23469	23282
query5	4269	478	470	470
query6	261	163	157	157
query7	4637	294	302	294
query8	268	214	219	214
query9	9624	2667	2680	2667
query10	449	288	300	288
query11	18175	15067	15169	15067
query12	158	102	98	98
query13	1617	429	395	395
query14	10120	7111	7472	7111
query15	280	174	174	174
query16	8079	465	492	465
query17	1669	575	546	546
query18	2114	313	307	307
query19	349	145	148	145
query20	113	106	106	106
query21	215	108	103	103
query22	4481	4344	4396	4344
query23	34752	34110	33939	33939
query24	11234	2982	2924	2924
query25	653	398	404	398
query26	1347	167	158	158
query27	2819	289	289	289
query28	8349	2487	2481	2481
query29	885	422	420	420
query30	323	157	156	156
query31	1055	798	812	798
query32	94	53	57	53
query33	762	295	289	289
query34	965	480	499	480
query35	872	709	719	709
query36	1110	955	969	955
query37	154	91	79	79
query38	3944	3910	3926	3910
query39	1470	1422	1399	1399
query40	277	94	94	94
query41	49	47	46	46
query42	115	93	94	93
query43	526	485	494	485
query44	1277	832	800	800
query45	192	160	163	160
query46	1143	768	757	757
query47	1937	1812	1831	1812
query48	453	361	364	361
query49	1135	400	395	395
query50	821	398	398	398
query51	7116	6850	6900	6850
query52	97	85	82	82
query53	257	186	178	178
query54	1288	457	458	457
query55	79	77	76	76
query56	285	256	255	255
query57	1194	1129	1096	1096
query58	239	224	221	221
query59	3118	3092	3059	3059
query60	296	271	269	269
query61	105	103	100	100
query62	870	658	675	658
query63	214	189	182	182
query64	5271	630	619	619
query65	3268	3149	3162	3149
query66	1435	308	318	308
query67	16243	15806	15586	15586
query68	3102	873	876	873
query69	448	336	340	336
query70	1205	1152	1184	1152
query71	329	351	326	326
query72	5914	3384	2967	2967
query73	594	576	569	569
query74	9567	8899	8958	8899
query75	3094	2824	2875	2824
query76	1934	852	846	846
query77	395	353	363	353
query78	9396	9301	9921	9301
query79	906	890	866	866
query80	596	572	568	568
query81	455	250	253	250
query82	231	229	229	229
query83	157	158	166	158
query84	231	105	99	99
query85	671	370	344	344
query86	330	307	311	307
query87	4359	4272	4378	4272
query88	4336	4016	4003	4003
query89	367	354	350	350
query90	1515	303	303	303
query91	163	162	165	162
query92	81	73	70	70
query93	915	902	890	890
query94	574	371	356	356
query95	420	413	399	399
query96	481	481	481	481
query97	3107	3145	3119	3119
query98	219	229	225	225
query99	1409	1291	1286	1286
Total cold run time: 292774 ms
Total hot run time: 194073 ms

doris-robot · 2024-09-19T04:56:45Z

ClickBench: Total hot run time: 32.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7302a5c966cf23f047d874962a5e37e365b1a94c, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.49	0.52	0.52
query6	1.13	0.72	0.71
query7	0.02	0.02	0.02
query8	0.03	0.02	0.03
query9	0.56	0.52	0.50
query10	0.54	0.57	0.55
query11	0.14	0.10	0.10
query12	0.13	0.10	0.10
query13	0.59	0.60	0.59
query14	3.14	3.00	3.05
query15	0.88	0.82	0.81
query16	0.38	0.36	0.37
query17	1.00	1.00	0.98
query18	0.23	0.21	0.21
query19	1.97	1.89	1.86
query20	0.01	0.01	0.01
query21	15.35	0.58	0.57
query22	2.18	2.48	1.98
query23	17.30	0.88	0.72
query24	2.48	1.28	0.56
query25	0.16	0.18	0.10
query26	0.42	0.14	0.13
query27	0.04	0.05	0.03
query28	11.28	1.09	1.06
query29	12.58	3.20	3.19
query30	0.24	0.06	0.05
query31	2.88	0.39	0.38
query32	3.24	0.46	0.45
query33	2.99	2.99	3.04
query34	16.82	4.39	4.42
query35	4.46	4.45	4.44
query36	0.68	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.01
query43	0.03	0.03	0.02
Total cold run time: 106.77 s
Total hot run time: 32.37 s

doris-robot · 2024-09-19T05:14:54Z

TeamCity be ut coverage result:
Function Coverage: 37.33% (9584/25676)
Line Coverage: 28.71% (79202/275901)
Region Coverage: 28.18% (41013/145524)
Branch Coverage: 24.82% (20901/84224)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7302a5c966cf23f047d874962a5e37e365b1a94c_7302a5c966cf23f047d874962a5e37e365b1a94c/report/index.html

Hastyshell · 2024-09-19T07:36:02Z

run external

github-actions · 2024-09-20T02:33:53Z

PR approved by at least one committer and no changes requested.

xinyiZzz

LGTM

…ack s3 file buffer memory usage" (#41360) Reverts #40597 This PR cased memtable memtracker check in cloud_p0 to fail. ``` *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43518&logView=flowAware&focusLine=43518) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43519&logView=flowAware&focusLine=43519) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43520&logView=flowAware&focusLine=43520) *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43521&logView=flowAware&focusLine=43521) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43522&logView=flowAware&focusLine=43522) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43523&logView=flowAware&focusLine=43523) *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43524&logView=flowAware&focusLine=43524) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43525&logView=flowAware&focusLine=43525) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358175 15766 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43526&logView=flowAware&focusLine=43526) *** Check failure stack trace: *** ``` http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=25290&logView=flowAware&focusLine=44023

…ile buffer memory usage (#40597) Track s3 file buffer memory usage with thread local tracker, so that memory usage will be specified to detail.

…ack s3 file buffer memory usage" (#41360) Reverts #40597 This PR cased memtable memtracker check in cloud_p0 to fail. ``` *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43518&logView=flowAware&focusLine=43518) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43519&logView=flowAware&focusLine=43519) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43520&logView=flowAware&focusLine=43520) *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43521&logView=flowAware&focusLine=43521) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43522&logView=flowAware&focusLine=43522) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43523&logView=flowAware&focusLine=43523) *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43524&logView=flowAware&focusLine=43524) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43525&logView=flowAware&focusLine=43525) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358175 15766 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43526&logView=flowAware&focusLine=43526) *** Check failure stack trace: *** ``` http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=25290&logView=flowAware&focusLine=44023

…ile buffer memory usage (apache#40597) Track s3 file buffer memory usage with thread local tracker, so that memory usage will be specified to detail.

…ack s3 file buffer memory usage" (apache#41360) Reverts apache#40597 This PR cased memtable memtracker check in cloud_p0 to fail. ``` *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43518&logView=flowAware&focusLine=43518) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43519&logView=flowAware&focusLine=43519) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43520&logView=flowAware&focusLine=43520) *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43521&logView=flowAware&focusLine=43521) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43522&logView=flowAware&focusLine=43522) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43523&logView=flowAware&focusLine=43523) *** Check failure stack trace: *** [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43524&logView=flowAware&focusLine=43524) F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43525&logView=flowAware&focusLine=43525) mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358175 15766 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880 [17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43526&logView=flowAware&focusLine=43526) *** Check failure stack trace: *** ``` http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=25290&logView=flowAware&focusLine=44023

[enhancement](mem-tracker) Use thread local mem tracker to track s3 f…

0e4d72e

…ile buffer memory usage

Hastyshell added dev/2.1.x dev/2.0.x labels Sep 10, 2024

yiguolei previously approved these changes Sep 10, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 10, 2024

github-actions bot added the reviewed label Sep 10, 2024

[enhancement](mem-tracker) Use thread local mem tracker to track s3 f…

6c64b0f

…ile buffer memory usage

Hastyshell force-pushed the enhance-s3-file-buf-tracker branch from 0e4d72e to 6c64b0f Compare September 11, 2024 00:14

xinyiZzz previously approved these changes Sep 11, 2024

View reviewed changes

fix snapshot tracking

8c5844f

Hastyshell dismissed stale reviews from xinyiZzz and yiguolei via 5ca232e September 11, 2024 07:28

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 11, 2024

Hastyshell force-pushed the enhance-s3-file-buf-tracker branch from 5ca232e to 02a3fc1 Compare September 11, 2024 07:56

gavinchou previously approved these changes Sep 19, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024

xinyiZzz previously approved these changes Sep 19, 2024

View reviewed changes

Hastyshell added 2 commits September 19, 2024 11:49

fix orphan tracker

be9b825

merge

7302a5c

Hastyshell dismissed stale reviews from xinyiZzz and gavinchou via 7302a5c September 19, 2024 03:50

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 19, 2024

gavinchou approved these changes Sep 20, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2024

xinyiZzz approved these changes Sep 26, 2024

View reviewed changes

xinyiZzz merged commit 62a64d1 into apache:master Sep 26, 2024

kaijchen mentioned this pull request Sep 26, 2024

Revert "[enhancement](mem-tracker) Use thread local mem tracker to track s3 file buffer memory usage" #41360

Merged

Hastyshell removed dev/2.0.x dev/2.1.x labels Oct 9, 2024

dataroaring added the dev/3.0.3-merged label Oct 13, 2024

[enhancement](mem-tracker) Use thread local mem tracker to track s3 file buffer memory usage #40597

[enhancement](mem-tracker) Use thread local mem tracker to track s3 file buffer memory usage #40597

Uh oh!

Conversation

Hastyshell commented Sep 10, 2024

Proposed changes

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

Hastyshell commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 10, 2024

Uh oh!

github-actions bot commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

Hastyshell commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 11, 2024

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

xinyiZzz commented Sep 11, 2024

Uh oh!

Hastyshell commented Sep 11, 2024

Uh oh!

Hastyshell commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 14, 2024

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

Hastyshell commented Sep 19, 2024

Uh oh!

doris-robot commented Sep 19, 2024

Uh oh!

doris-robot commented Sep 19, 2024

Uh oh!

doris-robot commented Sep 19, 2024

Uh oh!

doris-robot commented Sep 19, 2024

Uh oh!

Hastyshell commented Sep 19, 2024

Uh oh!

github-actions bot commented Sep 20, 2024

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants