[memtracker](accuracy) should not account resuable buffer to query memtracker #33933

yiguolei · 2024-04-21T03:27:49Z

Proposed changes

should only account resuable buffer to global compression tracker.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

…mtracker

doris-robot · 2024-04-21T03:27:55Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-04-21T03:34:01Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-21T03:47:16Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-21T03:49:13Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2024-04-21T05:20:10Z

run buildall

doris-robot · 2024-04-21T05:50:21Z

TeamCity be ut coverage result:
Function Coverage: 35.38% (8918/25204)
Line Coverage: 27.11% (73329/270530)
Region Coverage: 26.24% (37873/144352)
Branch Coverage: 23.05% (19285/83682)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9a07e9570a1a2b9ff163efd91373156ea5399832_9a07e9570a1a2b9ff163efd91373156ea5399832/report/index.html

be/src/util/block_compression.cpp

yiguolei · 2024-04-21T14:18:47Z

run buildall

github-actions · 2024-04-21T14:23:40Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-21T14:42:01Z

TeamCity be ut coverage result:
Function Coverage: 35.38% (8917/25204)
Line Coverage: 27.10% (73315/270526)
Region Coverage: 26.24% (37878/144350)
Branch Coverage: 23.05% (19286/83682)
Coverage Report: http://coverage.selectdb-in.cc/coverage/35b81fac6ab05eb20bc8bbf23c2d8bddba98cf4a_35b81fac6ab05eb20bc8bbf23c2d8bddba98cf4a/report/index.html

xinyiZzz

LGTM

github-actions · 2024-04-21T14:43:24Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-21T14:43:26Z

PR approved by anyone and no changes requested.

doris-robot · 2024-04-21T14:56:52Z

TPC-H: Total hot run time: 38443 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35b81fac6ab05eb20bc8bbf23c2d8bddba98cf4a, data reload: false

------ Round 1 ----------------------------------
q1	17596	4266	4273	4266
q2	2005	192	186	186
q3	10457	1178	1144	1144
q4	10195	681	816	681
q5	7501	2700	2568	2568
q6	219	134	137	134
q7	1024	602	589	589
q8	9222	2021	2008	2008
q9	7382	6559	6526	6526
q10	8556	3522	3496	3496
q11	435	234	235	234
q12	482	228	226	226
q13	18757	2956	2926	2926
q14	261	230	245	230
q15	532	479	496	479
q16	526	390	388	388
q17	966	706	739	706
q18	7435	6811	6673	6673
q19	3851	1528	1514	1514
q20	654	323	306	306
q21	3553	2862	2851	2851
q22	370	319	312	312
Total cold run time: 111979 ms
Total hot run time: 38443 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4423	4178	4218	4178
q2	365	272	259	259
q3	2954	2711	2723	2711
q4	1856	1559	1549	1549
q5	5246	5252	5284	5252
q6	212	124	127	124
q7	2223	1866	1862	1862
q8	3206	3321	3360	3321
q9	8537	8519	8484	8484
q10	4076	3866	3935	3866
q11	656	484	493	484
q12	789	609	659	609
q13	17532	3246	3124	3124
q14	324	299	285	285
q15	542	500	479	479
q16	489	462	439	439
q17	1832	1550	1512	1512
q18	8124	7891	7902	7891
q19	1689	1520	1559	1520
q20	2026	1876	1819	1819
q21	8139	4923	4999	4923
q22	543	469	472	469
Total cold run time: 75783 ms
Total hot run time: 55160 ms

doris-robot · 2024-04-21T15:08:32Z

TPC-DS: Total hot run time: 185404 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35b81fac6ab05eb20bc8bbf23c2d8bddba98cf4a, data reload: false

query1	875	374	364	364
query2	6188	2664	2368	2368
query3	6654	202	203	202
query4	22651	21265	21364	21265
query5	4117	400	404	400
query6	268	182	173	173
query7	4581	288	283	283
query8	240	194	191	191
query9	8575	2343	2312	2312
query10	414	245	248	245
query11	14584	14309	14174	14174
query12	143	89	87	87
query13	1647	370	371	370
query14	10109	7141	7906	7141
query15	254	176	182	176
query16	8162	265	256	256
query17	1907	573	536	536
query18	2087	274	264	264
query19	256	146	152	146
query20	91	82	85	82
query21	192	127	131	127
query22	5058	4947	4858	4858
query23	33733	33152	33169	33152
query24	10764	3034	3074	3034
query25	609	398	399	398
query26	704	166	150	150
query27	2268	375	387	375
query28	5776	2093	2041	2041
query29	840	645	614	614
query30	271	175	180	175
query31	962	788	782	782
query32	86	51	56	51
query33	694	265	256	256
query34	1125	511	522	511
query35	855	724	720	720
query36	1113	941	948	941
query37	113	71	83	71
query38	3526	3350	3311	3311
query39	1625	1585	1585	1585
query40	169	127	130	127
query41	44	41	45	41
query42	101	97	98	97
query43	586	557	535	535
query44	1169	761	767	761
query45	284	277	272	272
query46	1095	764	719	719
query47	1976	1911	1923	1911
query48	400	299	317	299
query49	843	405	398	398
query50	800	419	409	409
query51	6935	6808	6838	6808
query52	103	92	90	90
query53	354	282	274	274
query54	302	227	226	226
query55	77	70	69	69
query56	257	222	221	221
query57	1167	1121	1135	1121
query58	210	195	195	195
query59	3385	3394	3247	3247
query60	242	237	232	232
query61	88	86	89	86
query62	588	470	443	443
query63	307	279	280	279
query64	4706	4052	3976	3976
query65	3046	2996	3060	2996
query66	748	333	343	333
query67	15520	15023	15153	15023
query68	6359	534	532	532
query69	534	314	304	304
query70	1284	1174	1219	1174
query71	1461	1268	1276	1268
query72	6525	2614	2469	2469
query73	722	319	321	319
query74	6781	6373	6328	6328
query75	3767	2630	2656	2630
query76	4178	1020	973	973
query77	605	258	268	258
query78	11063	10155	10178	10155
query79	8078	518	520	518
query80	1541	441	449	441
query81	526	250	238	238
query82	1644	96	97	96
query83	205	171	168	168
query84	269	83	87	83
query85	1322	269	265	265
query86	466	303	295	295
query87	3518	3273	3308	3273
query88	5005	2325	2335	2325
query89	507	381	369	369
query90	1958	184	185	184
query91	125	96	94	94
query92	57	48	48	48
query93	6270	505	492	492
query94	1020	180	177	177
query95	386	364	310	310
query96	591	262	260	260
query97	3138	2921	2959	2921
query98	237	213	214	213
query99	1237	849	860	849
Total cold run time: 290998 ms
Total hot run time: 185404 ms

doris-robot · 2024-04-21T15:14:02Z

ClickBench: Total hot run time: 30.77 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35b81fac6ab05eb20bc8bbf23c2d8bddba98cf4a, data reload: false

query1	0.03	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.47	0.50
query6	1.47	0.72	0.71
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.54	0.49	0.48
query10	0.55	0.56	0.54
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.58	0.58
query14	0.76	0.76	0.79
query15	0.81	0.81	0.81
query16	0.37	0.36	0.38
query17	1.01	0.94	0.97
query18	0.19	0.25	0.24
query19	1.84	1.68	1.79
query20	0.02	0.01	0.01
query21	15.39	0.66	0.64
query22	3.69	7.44	2.58
query23	18.25	1.39	1.19
query24	1.37	0.39	0.20
query25	0.14	0.07	0.08
query26	0.24	0.15	0.16
query27	0.08	0.07	0.08
query28	13.42	1.00	0.98
query29	12.55	3.28	3.25
query30	0.25	0.06	0.05
query31	2.86	0.37	0.38
query32	3.28	0.45	0.46
query33	2.77	2.82	2.85
query34	16.85	4.40	4.48
query35	4.49	4.49	4.49
query36	0.65	0.46	0.46
query37	0.19	0.16	0.14
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 108.06 s
Total hot run time: 30.77 s

mrhhsg

LGTM

…mtracker (apache#33933) Co-authored-by: yiguolei <yiguolei@gmail.com>

pick #33396 #33862 #33853 #33732 #33841 #33933 #34901 #35014 --------- Co-authored-by: yiguolei <676222867@qq.com> Co-authored-by: yiguolei <yiguolei@gmail.com>

[memtracker](accuracy) should not account resuable buffer to query me…

4abf81f

…mtracker

Doris-Extras added 2 commits April 21, 2024 11:41

f

eaade95

f

9a07e95

xinyiZzz reviewed Apr 21, 2024

View reviewed changes

be/src/util/block_compression.cpp Show resolved Hide resolved

f

35b81fa

xinyiZzz approved these changes Apr 21, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 21, 2024

github-actions bot added the reviewed label Apr 21, 2024

mrhhsg approved these changes Apr 21, 2024

View reviewed changes

HappenLee merged commit af589c0 into apache:master Apr 21, 2024

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024

[memtracker](accuracy) should not account resuable buffer to query me…

23322b1

…mtracker (apache#33933) Co-authored-by: yiguolei <yiguolei@gmail.com>

xinyiZzz mentioned this pull request Sep 9, 2024

[branch-2.1](memory) Pick query memory tracker value is equal to 0 and exception handle #40564

Closed

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024

[memtracker](accuracy) should not account resuable buffer to query me…

f87b450

…mtracker (apache#33933) Co-authored-by: yiguolei <yiguolei@gmail.com>

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024

[memtracker](accuracy) should not account resuable buffer to query me…

8b13026

…mtracker (apache#33933) Co-authored-by: yiguolei <yiguolei@gmail.com>

yiguolei added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024

[memtracker](accuracy) should not account resuable buffer to query me…

faf326c

…mtracker (apache#33933) Co-authored-by: yiguolei <yiguolei@gmail.com>

yiguolei added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 11, 2024

[memtracker](accuracy) should not account resuable buffer to query me…

92766d6

…mtracker (apache#33933) Co-authored-by: yiguolei <yiguolei@gmail.com>

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 13, 2024

[memtracker](accuracy) should not account resuable buffer to query me…

d4fe9cd

…mtracker (apache#33933) Co-authored-by: yiguolei <yiguolei@gmail.com>

xinyiZzz mentioned this pull request Sep 13, 2024

[branch-2.1](memory) Allocator support address sanitizers #40836

Merged

xinyiZzz added the dev/2.1.7-merged label Sep 20, 2024

[memtracker](accuracy) should not account resuable buffer to query memtracker #33933

[memtracker](accuracy) should not account resuable buffer to query memtracker #33933

Uh oh!

Conversation

yiguolei commented Apr 21, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 21, 2024

Uh oh!

github-actions bot commented Apr 21, 2024

Uh oh!

github-actions bot commented Apr 21, 2024

Uh oh!

github-actions bot commented Apr 21, 2024

Uh oh!

yiguolei commented Apr 21, 2024

Uh oh!

doris-robot commented Apr 21, 2024

Uh oh!

Uh oh!

yiguolei commented Apr 21, 2024

Uh oh!

github-actions bot commented Apr 21, 2024

Uh oh!

doris-robot commented Apr 21, 2024

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 21, 2024

Uh oh!

github-actions bot commented Apr 21, 2024

Uh oh!

doris-robot commented Apr 21, 2024

Uh oh!

doris-robot commented Apr 21, 2024

Uh oh!

doris-robot commented Apr 21, 2024

Uh oh!

mrhhsg left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants