Skip to content

[fix](memory) Reflect Allocator memory tracker #40439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Sep 5, 2024
Merged

[fix](memory) Reflect Allocator memory tracker #40439

yiguolei merged 1 commit into from
Sep 5, 2024

Conversation

@xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Sep 5, 2024

fix:
img_v3_02e7_e9847bd3-89ad-4e9e-8fa1-76109a5386fg

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Sep 5, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

TPC-H: Total hot run time: 38612 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39e64edd1b974e60590e6198d4cc08d2d817280f, data reload: false

------ Round 1 ----------------------------------
q1	17625	4481	4315	4315
q2	2014	187	183	183
q3	10470	1178	1175	1175
q4	10137	725	742	725
q5	7949	2947	2885	2885
q6	231	142	142	142
q7	991	631	639	631
q8	9679	2078	2097	2078
q9	6979	6554	6550	6550
q10	7023	2153	2252	2153
q11	491	237	258	237
q12	392	232	229	229
q13	18804	3086	3085	3085
q14	272	231	240	231
q15	555	491	499	491
q16	502	400	394	394
q17	998	691	726	691
q18	7415	6908	6943	6908
q19	1392	1087	1088	1087
q20	678	324	334	324
q21	4196	3288	3106	3106
q22	1099	992	1042	992
Total cold run time: 109892 ms
Total hot run time: 38612 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5175	4301	4317	4301
q2	373	274	283	274
q3	2893	2668	2675	2668
q4	1998	1692	1688	1688
q5	5405	5451	5489	5451
q6	219	131	132	131
q7	2151	1774	1734	1734
q8	3231	3410	3360	3360
q9	8390	8457	8397	8397
q10	3454	3207	3196	3196
q11	606	494	507	494
q12	791	598	595	595
q13	13796	3136	3092	3092
q14	316	270	271	270
q15	524	484	488	484
q16	467	424	418	418
q17	1799	1510	1491	1491
q18	7883	7394	7479	7394
q19	1668	1571	1534	1534
q20	2057	1805	1828	1805
q21	5486	5338	5235	5235
q22	1111	1071	1027	1027
Total cold run time: 69793 ms
Total hot run time: 55039 ms

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

TPC-DS: Total hot run time: 188342 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39e64edd1b974e60590e6198d4cc08d2d817280f, data reload: false

query1	899	377	383	377
query2	6506	1918	1865	1865
query3	6647	211	216	211
query4	25375	23050	23202	23050
query5	4138	500	486	486
query6	277	172	167	167
query7	4601	299	297	297
query8	286	219	220	219
query9	8509	2464	2484	2464
query10	433	275	284	275
query11	16728	15120	15141	15120
query12	149	106	101	101
query13	1632	398	389	389
query14	9774	7166	7138	7138
query15	288	164	184	164
query16	8078	482	461	461
query17	1647	573	558	558
query18	2117	311	299	299
query19	343	152	156	152
query20	122	110	110	110
query21	211	110	106	106
query22	4505	4128	4076	4076
query23	34212	34189	33723	33723
query24	11377	2879	2913	2879
query25	546	402	408	402
query26	1127	159	156	156
query27	2525	280	287	280
query28	6883	2038	2006	2006
query29	680	430	425	425
query30	310	175	158	158
query31	994	753	787	753
query32	104	57	56	56
query33	738	311	298	298
query34	889	469	491	469
query35	871	740	700	700
query36	1081	900	925	900
query37	144	85	86	85
query38	4020	3858	3929	3858
query39	1445	1402	1390	1390
query40	275	120	120	120
query41	51	48	49	48
query42	115	100	100	100
query43	512	459	454	454
query44	1213	774	742	742
query45	198	171	170	170
query46	1105	716	772	716
query47	1885	1794	1807	1794
query48	372	307	297	297
query49	1193	462	459	459
query50	826	406	404	404
query51	6982	7003	6936	6936
query52	102	87	88	87
query53	264	186	182	182
query54	900	463	459	459
query55	77	77	75	75
query56	289	272	271	271
query57	1194	1063	1068	1063
query58	261	236	235	235
query59	2950	2887	2861	2861
query60	307	283	281	281
query61	126	123	123	123
query62	855	634	662	634
query63	228	186	187	186
query64	5093	692	637	637
query65	3219	3158	3124	3124
query66	1327	338	358	338
query67	15707	15250	15551	15250
query68	4922	574	566	566
query69	437	274	285	274
query70	1150	1142	1138	1138
query71	416	271	268	268
query72	7182	4049	4226	4049
query73	753	326	335	326
query74	9204	8811	8960	8811
query75	3591	2653	2694	2653
query76	3338	1006	998	998
query77	697	316	310	310
query78	9807	9122	9039	9039
query79	1466	533	528	528
query80	964	502	502	502
query81	573	237	229	229
query82	894	137	133	133
query83	322	153	160	153
query84	260	77	82	77
query85	1360	296	277	277
query86	381	299	300	299
query87	4434	4192	4240	4192
query88	3297	2355	2351	2351
query89	380	291	282	282
query90	1991	185	191	185
query91	127	110	99	99
query92	59	51	48	48
query93	1353	537	543	537
query94	988	298	282	282
query95	356	251	256	251
query96	590	262	266	262
query97	3160	3047	3075	3047
query98	228	200	200	200
query99	1575	1307	1318	1307
Total cold run time: 285092 ms
Total hot run time: 188342 ms

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

ClickBench: Total hot run time: 31.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39e64edd1b974e60590e6198d4cc08d2d817280f, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.08	0.07
query5	0.53	0.52	0.50
query6	1.13	0.74	0.74
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.53	0.47	0.49
query10	0.55	0.56	0.52
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.62	0.59	0.58
query14	0.77	0.80	0.78
query15	0.84	0.83	0.81
query16	0.38	0.37	0.38
query17	0.98	1.04	1.05
query18	0.20	0.21	0.20
query19	1.94	1.85	1.80
query20	0.01	0.02	0.01
query21	15.40	0.88	0.88
query22	4.24	7.03	2.13
query23	18.28	1.44	1.23
query24	2.12	0.23	0.24
query25	0.15	0.08	0.08
query26	0.26	0.18	0.17
query27	0.07	0.08	0.07
query28	13.19	1.01	0.99
query29	12.66	3.36	3.33
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.27	0.47	0.47
query33	2.92	3.02	3.01
query34	17.09	4.35	4.46
query35	4.45	4.45	4.40
query36	0.66	0.46	0.51
query37	0.19	0.15	0.16
query38	0.15	0.16	0.14
query39	0.05	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.05	0.04
Total cold run time: 109.5 s
Total hot run time: 31.14 s

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

TeamCity be ut coverage result:
Function Coverage: 36.83% (9391/25500)
Line Coverage: 28.26% (77407/273946)
Region Coverage: 27.65% (39943/144448)
Branch Coverage: 24.29% (20324/83658)
Coverage Report: http://coverage.selectdb-in.cc/coverage/39e64edd1b974e60590e6198d4cc08d2d817280f_39e64edd1b974e60590e6198d4cc08d2d817280f/report/index.html

@yiguolei yiguolei merged commit 5797814 into apache:master Sep 5, 2024
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024
@xinyiZzz
[fix](memory) Reflect Allocator memory tracker (apache#40439)
fcd9bb1 
fix:

![img_v3_02e7_e9847bd3-89ad-4e9e-8fa1-76109a5386fg](https://github.com/user-attachments/assets/0a7d627e-23d0-40f1-bec9-b9d9f64fed02)
@xinyiZzz xinyiZzz mentioned this pull request Sep 9, 2024
Closed
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024
@xinyiZzz
[fix](memory) Reflect Allocator memory tracker (apache#40439)
83d79e7 
fix:

![img_v3_02e7_e9847bd3-89ad-4e9e-8fa1-76109a5386fg](https://github.com/user-attachments/assets/0a7d627e-23d0-40f1-bec9-b9d9f64fed02)
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024
@xinyiZzz
[fix](memory) Reflect Allocator memory tracker (apache#40439)
eb4d37c 
fix:

![img_v3_02e7_e9847bd3-89ad-4e9e-8fa1-76109a5386fg](https://github.com/user-attachments/assets/0a7d627e-23d0-40f1-bec9-b9d9f64fed02)
yiguolei pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024
@xinyiZzz @yiguolei
[fix](memory) Reflect Allocator memory tracker (apache#40439)
618d14a 
fix:

![img_v3_02e7_e9847bd3-89ad-4e9e-8fa1-76109a5386fg](https://github.com/user-attachments/assets/0a7d627e-23d0-40f1-bec9-b9d9f64fed02)
yiguolei pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 11, 2024
@xinyiZzz @yiguolei
[fix](memory) Reflect Allocator memory tracker (apache#40439)
81dbc98 
fix:

![img_v3_02e7_e9847bd3-89ad-4e9e-8fa1-76109a5386fg](https://github.com/user-attachments/assets/0a7d627e-23d0-40f1-bec9-b9d9f64fed02)
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 14, 2024
@xinyiZzz
[fix](memory) Reflect Allocator memory tracker (apache#40439)
1a9ca61 
fix:

![img_v3_02e7_e9847bd3-89ad-4e9e-8fa1-76109a5386fg](https://github.com/user-attachments/assets/0a7d627e-23d0-40f1-bec9-b9d9f64fed02)
@xinyiZzz xinyiZzz mentioned this pull request Sep 14, 2024
Merged
yiguolei added a commit that referenced this pull request Sep 15, 2024
@xinyiZzz @sollhui @yiguolei
[branch-2.1](memory) When Load ends, check memory tracker value retur…
b52b572 
…ns is equal to 0 (#40850)

pick
#38960
#39908
#40043
#40092
#40016
#40439

---------

Co-authored-by: hui lai <1353307710@qq.com>
Co-authored-by: yiguolei <676222867@qq.com>
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@xinyiZzz @dataroaring
[fix](memory) Reflect Allocator memory tracker (#40439)
89cd9bf 
fix:

![img_v3_02e7_e9847bd3-89ad-4e9e-8fa1-76109a5386fg](https://github.com/user-attachments/assets/0a7d627e-23d0-40f1-bec9-b9d9f64fed02)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@Gabriel39 Gabriel39 Gabriel39 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xinyiZzz @doris-robot @yiguolei @Gabriel39 @dataroaring