Skip to content

[fix](fe memory) Fix fe schema change high memory usage #30231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jan 23, 2024
Merged

[fix](fe memory) Fix fe schema change high memory usage #30231

dataroaring merged 1 commit into from
Jan 23, 2024

Conversation

@JackDrogon
Copy link
Contributor

@JackDrogon JackDrogon commented Jan 22, 2024

Proposed changes

Fix fe schema change high memory usage.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@JackDrogon
Fix fe schema change high memory usage
a931ab1 
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
@dataroaring
Copy link
Contributor

dataroaring commented Jan 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jan 22, 2024

TPC-H: Total hot run time: 38645 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a931ab17d5a8e5f826f3113228dd06f18df6ff20, data reload: false

------ Round 1 ----------------------------------
q1	17725	5209	5318	5209
q2	2040	138	135	135
q3	10708	1173	1169	1169
q4	10221	764	821	764
q5	7784	3100	3108	3100
q6	193	126	126	126
q7	839	493	484	484
q8	9227	1870	1928	1870
q9	7177	6330	6308	6308
q10	8175	3078	3009	3009
q11	420	217	197	197
q12	352	187	190	187
q13	18010	3373	3391	3373
q14	246	217	218	217
q15	549	507	495	495
q16	449	372	374	372
q17	932	544	494	494
q18	7320	6900	6759	6759
q19	1568	1435	1387	1387
q20	606	290	286	286
q21	2763	2434	2404	2404
q22	365	300	307	300
Total cold run time: 107669 ms
Total hot run time: 38645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5352	5326	5173	5173
q2	323	219	210	210
q3	3309	3134	3204	3134
q4	2052	2055	1974	1974
q5	6020	5864	5706	5706
q6	204	122	122	122
q7	2304	1931	1892	1892
q8	3218	3379	3359	3359
q9	8960	8897	8750	8750
q10	3931	3822	3813	3813
q11	558	462	466	462
q12	791	611	614	611
q13	16944	3145	3151	3145
q14	295	249	272	249
q15	556	507	500	500
q16	494	471	479	471
q17	1866	1846	1898	1846
q18	9444	18395	9556	9556
q19	24227	1574	1528	1528
q20	4549	1950	1911	1911
q21	14392	5440	5512	5440
q22	982	549	528	528
Total cold run time: 110771 ms
Total hot run time: 60380 ms

@doris-robot
Copy link

doris-robot commented Jan 22, 2024

TPC-DS: Total hot run time: 176527 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a931ab17d5a8e5f826f3113228dd06f18df6ff20, data reload: false

query1	948	331	329	329
query2	6553	1945	1866	1866
query3	6704	201	193	193
query4	33102	22620	22371	22371
query5	4447	363	408	363
query6	245	170	174	170
query7	5081	261	256	256
query8	231	173	173	173
query9	8337	2518	2502	2502
query10	411	243	245	243
query11	16474	15458	15445	15445
query12	119	72	69	69
query13	1680	391	382	382
query14	10515	7008	6971	6971
query15	223	183	182	182
query16	5789	256	246	246
query17	949	466	465	465
query18	1792	260	256	256
query19	180	133	132	132
query20	68	70	71	70
query21	198	124	129	124
query22	5166	5018	5052	5018
query23	31402	30679	30762	30679
query24	7534	2823	2774	2774
query25	492	310	314	310
query26	873	146	143	143
query27	2646	280	289	280
query28	4118	1849	1817	1817
query29	937	626	610	610
query30	235	133	144	133
query31	912	726	771	726
query32	95	51	50	50
query33	419	213	205	205
query34	824	440	450	440
query35	893	783	812	783
query36	1341	1255	1209	1209
query37	90	65	58	58
query38	3386	3279	3262	3262
query39	1305	1258	1274	1258
query40	195	86	83	83
query41	39	34	35	34
query42	89	82	85	82
query43	481	456	464	456
query44	1056	692	692	692
query45	192	177	173	173
query46	1052	638	659	638
query47	1550	1553	1558	1553
query48	408	308	299	299
query49	730	280	281	280
query50	680	310	298	298
query51	5289	5261	5217	5217
query52	96	74	76	74
query53	323	255	244	244
query54	213	196	188	188
query55	90	72	80	72
query56	175	177	159	159
query57	976	946	942	942
query58	185	158	155	155
query59	2803	2646	2501	2501
query60	203	198	193	193
query61	84	83	81	81
query62	495	359	352	352
query63	273	249	253	249
query64	3648	1787	1760	1760
query65	3322	3242	3250	3242
query66	1157	320	316	316
query67	15776	14951	15149	14951
query68	11053	534	510	510
query69	587	321	299	299
query70	1697	1586	1538	1538
query71	508	217	210	210
query72	5075	2849	2851	2849
query73	2183	319	314	314
query74	7115	6332	6382	6332
query75	4822	2340	2297	2297
query76	6449	1010	980	980
query77	672	228	227	227
query78	9098	8753	8594	8594
query79	1012	489	489	489
query80	594	324	316	316
query81	467	208	204	204
query82	182	80	78	78
query83	141	115	120	115
query84	274	67	71	67
query85	1079	337	317	317
query86	398	377	409	377
query87	3541	3388	3363	3363
query88	3024	2176	2188	2176
query89	425	351	355	351
query90	2207	185	185	185
query91	153	121	125	121
query92	52	43	45	43
query93	1742	428	415	415
query94	1411	161	154	154
query95	518	450	451	450
query96	631	309	312	309
query97	4284	4130	4184	4130
query98	196	186	188	186
query99	998	693	684	684
Total cold run time: 280882 ms
Total hot run time: 176527 ms

@doris-robot
Copy link

doris-robot commented Jan 22, 2024

ClickBench: Total hot run time: 29.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a931ab17d5a8e5f826f3113228dd06f18df6ff20, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.02	0.02
query3	0.22	0.06	0.07
query4	1.68	0.08	0.10
query5	0.54	0.52	0.52
query6	1.24	0.65	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.49	0.50
query10	0.55	0.55	0.54
query11	0.12	0.09	0.08
query12	0.11	0.10	0.09
query13	0.59	0.60	0.61
query14	0.78	0.79	0.80
query15	0.78	0.77	0.77
query16	0.36	0.37	0.37
query17	0.98	0.97	1.01
query18	0.24	0.24	0.24
query19	1.87	1.75	1.81
query20	0.01	0.01	0.01
query21	15.41	0.54	0.56
query22	3.02	2.43	1.34
query23	17.27	0.92	0.82
query24	2.28	1.18	0.70
query25	0.38	0.31	0.06
query26	0.51	0.14	0.15
query27	0.04	0.05	0.06
query28	12.52	0.78	0.77
query29	12.93	3.19	3.15
query30	0.54	0.46	0.48
query31	2.78	0.35	0.35
query32	3.37	0.47	0.48
query33	3.24	3.22	3.18
query34	15.68	4.31	4.28
query35	4.33	4.31	4.24
query36	1.10	1.07	1.07
query37	0.07	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.01	0.01
query40	0.16	0.12	0.13
query41	0.08	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 106.58 s
Total hot run time: 29.95 s

@doris-robot
Copy link

doris-robot commented Jan 22, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a931ab17d5a8e5f826f3113228dd06f18df6ff20 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.6 seconds inserted 10000000 Rows, about 684K ops/s

@XueYuhai
Copy link
Contributor

XueYuhai commented Jan 23, 2024

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jan 23, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jan 23, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 23, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jan 23, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 00283a5 into apache:master Jan 23, 2024
yiguolei pushed a commit that referenced this pull request Jan 24, 2024
@JackDrogon @Doris-Extras
[fix](fe-memory) Fix fe schema change high memory usage (#30231)
dde5ed5 
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
This was referenced Jun 24, 2024
Merged
Merged
dataroaring pushed a commit that referenced this pull request Jun 25, 2024
@Lchangliang
[fix](schema change) reduce memory usage in schema change process (#3…
39a8600 
…6751)

pick
#36285
#30231
#33073
dataroaring pushed a commit that referenced this pull request Jun 25, 2024
@Lchangliang
[fix](schema change) reduce memory usage in schema change process #30231
07ce9cf 
 #36285 #33073 (#36756)

pick
#30231
#36285
#33073
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@Lchangliang
[fix](schema change) reduce memory usage in schema change process (ap…
e5d126b 
…ache#36751)

pick
apache#36285
apache#30231
apache#33073
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@w41ter w41ter w41ter approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.0-merged need_more_review reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@JackDrogon @dataroaring @doris-robot @XueYuhai @w41ter @xiaokang @wm1581066