[Improvement]Add customStdAllocator for vector/map #41193

wangbo · 2024-09-24T03:48:15Z

Proposed changes

1 Add customStdAllocator for vector/map.
2 Manually count memory for delete bitmap.

doris-robot · 2024-09-24T03:48:20Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

wangbo · 2024-09-24T03:48:31Z

run buildall

xinyiZzz · 2024-09-24T04:01:55Z

be/src/olap/rowid_conversion.h

 public:
    RowIdConversion() = default;
-    ~RowIdConversion() = default;
+    ~RowIdConversion() { CONSUME_THREAD_MEM_TRACKER(-_seg_rowid_map_mem_used); }


RELEASE_THREAD_MEM_TRACKER

xinyiZzz · 2024-09-24T04:03:08Z

be/src/vec/common/custom_allocator.h

+using MyVector = std::vector<P, CustomStdAllocator<P>>;
+
+template <typename K, typename T>
+using MyMap = std::map<K, T, std::less<K>, CustomStdAllocator<std::pair<const K, T>>>;


maybe DorisMap sounds better

be/src/olap/rowid_conversion.h

doris-robot · 2024-09-24T04:44:02Z

TPC-H: Total hot run time: 41665 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 100d0a053351c850682d94e3ffa228f47257aae8, data reload: false

------ Round 1 ----------------------------------
q1	17571	7349	7235	7235
q2	2022	293	289	289
q3	12213	1064	1238	1064
q4	10550	762	787	762
q5	7771	3085	3081	3081
q6	239	155	154	154
q7	1016	612	650	612
q8	9455	2104	2030	2030
q9	6769	6376	6382	6376
q10	7010	2282	2275	2275
q11	432	249	254	249
q12	416	233	219	219
q13	17782	2973	2994	2973
q14	232	211	218	211
q15	586	532	508	508
q16	668	627	597	597
q17	983	873	873	873
q18	7285	6735	6750	6735
q19	1411	912	990	912
q20	576	292	296	292
q21	4017	3204	3356	3204
q22	1137	1022	1014	1014
Total cold run time: 110141 ms
Total hot run time: 41665 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7246	7265	7253	7253
q2	339	234	242	234
q3	3069	3010	2935	2935
q4	2076	1892	1785	1785
q5	5692	5562	5716	5562
q6	235	150	147	147
q7	2177	1813	1833	1813
q8	3291	3466	3465	3465
q9	8789	8805	8830	8805
q10	3482	3506	3467	3467
q11	600	482	480	480
q12	838	645	619	619
q13	9612	3196	3142	3142
q14	299	285	290	285
q15	584	550	529	529
q16	727	681	653	653
q17	1817	1620	1625	1620
q18	8161	7972	7866	7866
q19	1711	1530	1509	1509
q20	2117	1871	1919	1871
q21	5584	5485	5442	5442
q22	1186	1053	1055	1053
Total cold run time: 69632 ms
Total hot run time: 60535 ms

doris-robot · 2024-09-24T04:56:05Z

TPC-DS: Total hot run time: 191736 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 100d0a053351c850682d94e3ffa228f47257aae8, data reload: false

query1	958	402	393	393
query2	6629	2072	1989	1989
query3	8683	195	207	195
query4	34055	23517	23558	23517
query5	3446	467	465	465
query6	279	166	171	166
query7	4197	299	290	290
query8	286	220	233	220
query9	9694	2705	2678	2678
query10	486	284	265	265
query11	18125	15138	15276	15138
query12	162	98	96	96
query13	1532	423	396	396
query14	8951	7632	7280	7280
query15	244	169	173	169
query16	7564	491	462	462
query17	1684	627	599	599
query18	2070	325	319	319
query19	392	158	154	154
query20	126	118	114	114
query21	218	108	106	106
query22	4865	4544	4398	4398
query23	35237	34322	34525	34322
query24	11345	3025	2970	2970
query25	629	406	414	406
query26	1211	159	161	159
query27	2355	283	294	283
query28	7927	2476	2475	2475
query29	820	438	432	432
query30	261	163	160	160
query31	1032	800	802	800
query32	95	53	54	53
query33	747	308	310	308
query34	906	500	493	493
query35	868	729	749	729
query36	1094	939	971	939
query37	158	92	88	88
query38	4000	3969	3952	3952
query39	1480	1403	1410	1403
query40	209	98	97	97
query41	50	49	49	49
query42	121	96	99	96
query43	532	481	496	481
query44	1296	835	822	822
query45	200	167	163	163
query46	1141	754	740	740
query47	1915	1845	1838	1838
query48	481	371	369	369
query49	886	408	407	407
query50	809	414	415	414
query51	7043	6741	6992	6741
query52	100	111	86	86
query53	251	181	178	178
query54	1156	468	457	457
query55	82	75	77	75
query56	294	257	244	244
query57	1228	1091	1092	1091
query58	229	229	226	226
query59	3334	2820	2824	2820
query60	293	275	263	263
query61	112	101	104	101
query62	836	663	675	663
query63	221	186	186	186
query64	4095	669	638	638
query65	3237	3189	3175	3175
query66	863	306	322	306
query67	15894	15587	15637	15587
query68	4728	606	588	588
query69	535	303	308	303
query70	1226	1125	1065	1065
query71	377	281	274	274
query72	7587	4067	4050	4050
query73	759	327	330	327
query74	10277	9041	8972	8972
query75	3471	2677	2665	2665
query76	3152	967	933	933
query77	619	303	286	286
query78	10011	9212	9153	9153
query79	1531	553	559	553
query80	1168	445	440	440
query81	565	240	242	240
query82	691	146	143	143
query83	302	134	133	133
query84	289	87	83	83
query85	1351	320	277	277
query86	393	306	296	296
query87	4535	4365	4373	4365
query88	3174	2323	2337	2323
query89	388	283	290	283
query90	1903	188	186	186
query91	172	146	143	143
query92	60	47	49	47
query93	1809	563	550	550
query94	923	298	290	290
query95	366	258	266	258
query96	611	279	275	275
query97	3271	3148	3063	3063
query98	212	214	192	192
query99	1611	1286	1291	1286
Total cold run time: 299786 ms
Total hot run time: 191736 ms

doris-robot · 2024-09-24T05:01:32Z

ClickBench: Total hot run time: 32.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 100d0a053351c850682d94e3ffa228f47257aae8, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.64	0.09	0.10
query5	0.50	0.50	0.53
query6	1.14	0.73	0.71
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.50
query10	0.54	0.55	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.99	2.93	3.02
query15	0.89	0.80	0.83
query16	0.36	0.36	0.39
query17	1.07	1.03	1.02
query18	0.20	0.20	0.19
query19	1.93	1.84	1.95
query20	0.01	0.01	0.01
query21	15.37	0.56	0.56
query22	2.94	2.01	1.77
query23	17.32	0.80	0.93
query24	2.72	0.80	2.09
query25	0.24	0.06	0.07
query26	0.51	0.14	0.13
query27	0.03	0.04	0.04
query28	10.20	1.09	1.07
query29	12.58	3.26	3.22
query30	0.25	0.05	0.06
query31	2.89	0.39	0.37
query32	3.28	0.48	0.47
query33	2.95	2.95	3.08
query34	17.03	4.43	4.37
query35	4.43	4.41	4.45
query36	0.65	0.48	0.46
query37	0.08	0.05	0.07
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.96 s
Total hot run time: 32.31 s

zhannngchen

LGTM

github-actions · 2024-09-24T08:48:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-24T08:48:17Z

PR approved by anyone and no changes requested.

wangbo · 2024-09-24T09:02:10Z

run buildall

wangbo · 2024-09-24T10:49:00Z

run buildall

wangbo · 2024-09-24T11:23:00Z

run buildall

wangbo · 2024-09-25T03:14:44Z

run buildall

be/src/vec/common/custom_allocator.h

doris-robot · 2024-09-26T05:02:42Z

TPC-H: Total hot run time: 41015 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87e606d0bcccd09cbca56f91e480faf7b28bfd34, data reload: false

------ Round 1 ----------------------------------
q1	17576	8104	7257	7257
q2	2019	288	291	288
q3	11960	1093	1225	1093
q4	10564	782	730	730
q5	7785	2886	2856	2856
q6	235	149	148	148
q7	998	619	598	598
q8	9370	1976	1942	1942
q9	6622	6497	6390	6390
q10	6947	2325	2335	2325
q11	435	253	255	253
q12	419	223	223	223
q13	17773	3006	2976	2976
q14	238	208	220	208
q15	584	530	520	520
q16	646	598	581	581
q17	995	569	552	552
q18	7178	6620	6793	6620
q19	1402	1064	1067	1064
q20	484	208	209	208
q21	4105	3298	3214	3214
q22	1093	969	1016	969
Total cold run time: 109428 ms
Total hot run time: 41015 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7235	7226	7241	7226
q2	327	230	232	230
q3	3066	2940	2972	2940
q4	2110	1925	1874	1874
q5	5724	5677	5710	5677
q6	231	153	148	148
q7	2202	1783	1777	1777
q8	3411	3532	3556	3532
q9	8897	9134	8972	8972
q10	3581	3541	3566	3541
q11	581	512	490	490
q12	873	627	661	627
q13	8553	3161	3198	3161
q14	310	273	290	273
q15	573	530	531	530
q16	702	645	646	645
q17	1884	1653	1596	1596
q18	8325	7670	7873	7670
q19	1726	1643	1535	1535
q20	2139	1877	1888	1877
q21	5541	5386	5413	5386
q22	1144	1043	1051	1043
Total cold run time: 69135 ms
Total hot run time: 60750 ms

doris-robot · 2024-09-26T05:03:13Z

TeamCity be ut coverage result:
Function Coverage: 37.30% (9632/25822)
Line Coverage: 28.70% (79682/277619)
Region Coverage: 28.12% (41199/146492)
Branch Coverage: 24.76% (20981/84754)
Coverage Report: http://coverage.selectdb-in.cc/coverage/87e606d0bcccd09cbca56f91e480faf7b28bfd34_87e606d0bcccd09cbca56f91e480faf7b28bfd34/report/index.html

doris-robot · 2024-09-26T05:14:52Z

TPC-DS: Total hot run time: 192538 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87e606d0bcccd09cbca56f91e480faf7b28bfd34, data reload: false

query1	960	381	377	377
query2	6356	2118	2020	2020
query3	8681	199	208	199
query4	34297	23493	23429	23429
query5	3376	464	477	464
query6	270	164	160	160
query7	4193	301	302	301
query8	277	222	221	221
query9	9593	2672	2670	2670
query10	458	277	272	272
query11	17956	15204	15241	15204
query12	156	100	99	99
query13	1533	430	410	410
query14	9638	7266	7559	7266
query15	254	166	180	166
query16	8031	473	491	473
query17	1689	605	618	605
query18	2174	337	324	324
query19	375	148	154	148
query20	120	111	125	111
query21	216	119	105	105
query22	4665	4452	4696	4452
query23	35309	34684	34507	34507
query24	11171	2851	2843	2843
query25	620	406	424	406
query26	1167	180	159	159
query27	2610	304	291	291
query28	7497	2440	2443	2440
query29	847	440	442	440
query30	262	156	151	151
query31	993	786	808	786
query32	93	54	55	54
query33	747	301	294	294
query34	907	509	494	494
query35	877	718	735	718
query36	1093	946	915	915
query37	146	80	84	80
query38	3973	3879	3888	3879
query39	1516	1401	1422	1401
query40	203	100	100	100
query41	51	49	49	49
query42	115	94	94	94
query43	527	488	486	486
query44	1246	822	795	795
query45	197	169	167	167
query46	1150	714	732	714
query47	1924	1838	1825	1825
query48	469	372	377	372
query49	902	403	394	394
query50	841	409	412	409
query51	7038	6946	7014	6946
query52	101	89	89	89
query53	254	178	203	178
query54	1105	460	478	460
query55	75	76	75	75
query56	283	267	251	251
query57	1236	1091	1116	1091
query58	235	229	241	229
query59	3142	2878	2970	2878
query60	288	266	276	266
query61	103	106	105	105
query62	841	654	651	651
query63	226	183	180	180
query64	4424	642	636	636
query65	3251	3192	3194	3192
query66	889	298	305	298
query67	16042	15664	15613	15613
query68	5065	555	572	555
query69	602	301	301	301
query70	1188	1179	1143	1143
query71	434	271	274	271
query72	7608	4093	4098	4093
query73	763	348	355	348
query74	10440	9002	9173	9002
query75	3796	2689	2698	2689
query76	3537	897	840	840
query77	519	312	299	299
query78	10575	9597	9597	9597
query79	2252	584	608	584
query80	2695	438	456	438
query81	580	238	235	235
query82	708	140	140	140
query83	297	133	138	133
query84	282	81	74	74
query85	1775	299	282	282
query86	471	314	312	312
query87	4390	4313	4340	4313
query88	3806	2446	2419	2419
query89	406	282	294	282
query90	2163	185	191	185
query91	171	163	142	142
query92	66	47	46	46
query93	2400	545	556	545
query94	1269	306	294	294
query95	367	253	261	253
query96	633	283	288	283
query97	3257	3100	3088	3088
query98	222	214	194	194
query99	1523	1287	1281	1281
Total cold run time: 306514 ms
Total hot run time: 192538 ms

doris-robot · 2024-09-26T05:20:19Z

ClickBench: Total hot run time: 33.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 87e606d0bcccd09cbca56f91e480faf7b28bfd34, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.68	0.09	0.10
query5	0.51	0.50	0.52
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.56	0.51	0.50
query10	0.55	0.59	0.55
query11	0.13	0.11	0.11
query12	0.13	0.11	0.10
query13	0.61	0.59	0.60
query14	2.71	2.72	2.84
query15	0.91	0.82	0.83
query16	0.39	0.36	0.38
query17	1.03	0.97	1.04
query18	0.20	0.19	0.19
query19	1.94	1.88	2.04
query20	0.02	0.01	0.01
query21	15.35	0.60	0.57
query22	2.53	2.74	2.13
query23	17.56	0.83	0.83
query24	2.96	1.29	1.61
query25	0.18	0.12	0.06
query26	0.58	0.14	0.13
query27	0.04	0.04	0.04
query28	9.75	1.11	1.06
query29	12.53	3.20	3.17
query30	0.24	0.06	0.05
query31	2.89	0.37	0.38
query32	3.29	0.47	0.46
query33	2.97	2.98	3.03
query34	16.78	4.49	4.44
query35	4.48	4.53	4.47
query36	0.67	0.49	0.48
query37	0.08	0.05	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.16 s
Total hot run time: 33.11 s

github-actions · 2024-09-26T06:47:27Z

PR approved by at least one committer and no changes requested.

xinyiZzz

LGTM

pick #41193

xinyiZzz reviewed Sep 24, 2024

View reviewed changes

zhannngchen reviewed Sep 24, 2024

View reviewed changes

be/src/olap/rowid_conversion.h Show resolved Hide resolved

wangbo force-pushed the 0913_std_mem_local branch 2 times, most recently from 374e0cd to 4436141 Compare September 24, 2024 04:28

wangbo force-pushed the 0913_std_mem_local branch from 8b1cdad to afb520f Compare September 24, 2024 08:30

zhannngchen previously approved these changes Sep 24, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 24, 2024

github-actions bot added the reviewed label Sep 24, 2024

wangbo dismissed zhannngchen’s stale review via d827db5 September 24, 2024 09:01

wangbo force-pushed the 0913_std_mem_local branch 2 times, most recently from d827db5 to aabd735 Compare September 24, 2024 09:01

wangbo force-pushed the 0913_std_mem_local branch from 398cfee to f8b7132 Compare September 24, 2024 11:08

wangbo force-pushed the 0913_std_mem_local branch 4 times, most recently from 7542ad9 to e6f3319 Compare September 25, 2024 03:13

yiguolei reviewed Sep 25, 2024

View reviewed changes

be/src/vec/common/custom_allocator.h Outdated Show resolved Hide resolved

wangbo force-pushed the 0913_std_mem_local branch from e6f3319 to 71fa09c Compare September 25, 2024 03:41

yiguolei approved these changes Sep 26, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 26, 2024

xinyiZzz approved these changes Sep 26, 2024

View reviewed changes

wangbo merged commit dbfe839 into apache:master Sep 26, 2024

cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024

[Improvement]Add customStdAllocator for vector/map (apache#41193)

a27ef17

xinyiZzz added dev/2.1.x dev/3.0.x labels Dec 3, 2024

github-actions bot added dev/3.0.x-conflict dev/2.1.x-conflict labels Dec 3, 2024

wangbo added a commit to wangbo/incubator-doris that referenced this pull request Dec 6, 2024

[Improvement]Add customStdAllocator for vector/map (apache#41193)

06c792a

wangbo added a commit to wangbo/incubator-doris that referenced this pull request Dec 6, 2024

[Improvement]Add customStdAllocator for vector/map (apache#41193)

f89c41f

wangbo added a commit to wangbo/incubator-doris that referenced this pull request Dec 6, 2024

[Improvement]Add customStdAllocator for vector/map (apache#41193)

d2f98ff

wangbo mentioned this pull request Dec 6, 2024

[branch-2.1]Add customStdAllocator for vector/map (#41193) #45124

Merged

16 tasks

yiguolei pushed a commit that referenced this pull request Dec 9, 2024

[branch-2.1]Add customStdAllocator for vector/map (#41193) (#45124)

0bcb997

yiguolei added dev/2.1.8-merged and removed dev/2.1.x dev/2.1.x-conflict labels Dec 9, 2024

wangbo added a commit to wangbo/incubator-doris that referenced this pull request Jan 23, 2025

[Improvement]Add customStdAllocator for vector/map (apache#41193)

85bb3c0

wangbo added a commit to wangbo/incubator-doris that referenced this pull request Jan 23, 2025

[Improvement]Add customStdAllocator for vector/map (apache#41193)

96e87eb

wangbo mentioned this pull request Jan 23, 2025

[branch-3.0]Add customStdAllocator for vector/map (#41193) #47368

Merged

wangbo added a commit that referenced this pull request Jan 24, 2025

[branch-3.0]Add customStdAllocator for vector/map (#41193) (#47368)

110e0fc

pick #41193

dataroaring added dev/3.0.2-merged and removed dev/3.0.x dev/3.0.x-conflict labels Mar 20, 2025

xinyiZzz mentioned this pull request Apr 28, 2025

branch-2.1: [fix](memory) Fix thread_mem_tracker() performance #50524

Merged

16 tasks

[Improvement]Add customStdAllocator for vector/map #41193

[Improvement]Add customStdAllocator for vector/map #41193

Uh oh!

Conversation

wangbo commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Sep 24, 2024

Uh oh!

wangbo commented Sep 24, 2024

Uh oh!

xinyiZzz Sep 24, 2024

Choose a reason for hiding this comment

Uh oh!

xinyiZzz Sep 24, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

doris-robot commented Sep 24, 2024

Uh oh!

doris-robot commented Sep 24, 2024

Uh oh!

doris-robot commented Sep 24, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 24, 2024

Uh oh!

github-actions bot commented Sep 24, 2024

Uh oh!

wangbo commented Sep 24, 2024

Uh oh!

wangbo commented Sep 24, 2024

Uh oh!

wangbo commented Sep 24, 2024

Uh oh!

wangbo commented Sep 25, 2024

Uh oh!

Uh oh!

doris-robot commented Sep 26, 2024

Uh oh!

doris-robot commented Sep 26, 2024

Uh oh!

doris-robot commented Sep 26, 2024

Uh oh!

doris-robot commented Sep 26, 2024

Uh oh!

github-actions bot commented Sep 26, 2024

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wangbo commented Sep 24, 2024 •

edited

Loading