Skip to content

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jun 17, 2024

Sometime when using Hudi Catalog, there will be lots of objects like
org.apache.hadoop.hdfs.client.impl.DfsClientConf in FE memory,
which belongs to the Hadoop FileSystem's Cache.

Disable this Cache to avoid FE OOM by setting
fs.xxx.impl.disable.cache to true

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 17, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39792 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d197ef1cdebcaaa71913fbc8304036bf6b53e973, data reload: false

------ Round 1 ----------------------------------
q1	18041	4456	4407	4407
q2	2554	200	191	191
q3	11024	1109	1193	1109
q4	10316	761	747	747
q5	8432	2735	2668	2668
q6	221	139	140	139
q7	963	627	613	613
q8	9261	2067	2049	2049
q9	9001	6475	6493	6475
q10	8847	3747	3681	3681
q11	452	234	233	233
q12	431	234	228	228
q13	18732	2988	2948	2948
q14	273	218	223	218
q15	529	477	491	477
q16	527	376	375	375
q17	965	698	709	698
q18	7901	7292	7326	7292
q19	3475	1517	1353	1353
q20	654	318	332	318
q21	4918	3244	3234	3234
q22	381	343	339	339
Total cold run time: 117898 ms
Total hot run time: 39792 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4189	4318	4189
q2	369	274	283	274
q3	2991	2776	2750	2750
q4	1902	1583	1582	1582
q5	5209	5313	5302	5302
q6	219	129	131	129
q7	2103	1704	1733	1704
q8	3151	3324	3346	3324
q9	8399	8349	8358	8349
q10	3890	3675	3661	3661
q11	578	516	501	501
q12	759	595	603	595
q13	17342	2941	2969	2941
q14	291	254	263	254
q15	524	475	480	475
q16	485	419	404	404
q17	1766	1490	1462	1462
q18	7645	7377	7268	7268
q19	1704	1594	1549	1549
q20	1959	1812	1779	1779
q21	4773	4628	4549	4549
q22	601	552	557	552
Total cold run time: 71065 ms
Total hot run time: 53593 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172752 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d197ef1cdebcaaa71913fbc8304036bf6b53e973, data reload: false

query1	915	378	379	378
query2	6449	2336	2289	2289
query3	6650	202	204	202
query4	19707	17327	17234	17234
query5	4139	471	456	456
query6	256	166	158	158
query7	4600	296	289	289
query8	320	287	287	287
query9	8518	2400	2381	2381
query10	601	304	284	284
query11	10734	9965	9899	9899
query12	136	89	85	85
query13	1682	366	357	357
query14	8726	7487	7560	7487
query15	229	193	189	189
query16	7815	264	255	255
query17	1922	529	525	525
query18	1944	275	263	263
query19	193	148	153	148
query20	96	86	78	78
query21	211	125	123	123
query22	4325	4184	4028	4028
query23	33700	33189	33111	33111
query24	10766	2880	2732	2732
query25	636	347	353	347
query26	1620	149	149	149
query27	3061	314	312	312
query28	7390	2038	2032	2032
query29	1032	614	595	595
query30	283	156	154	154
query31	957	728	763	728
query32	95	57	55	55
query33	788	325	272	272
query34	954	468	456	456
query35	743	601	625	601
query36	1074	928	938	928
query37	152	73	69	69
query38	2862	2744	2746	2744
query39	825	789	800	789
query40	282	125	126	125
query41	57	52	54	52
query42	127	97	99	97
query43	568	532	545	532
query44	1200	715	727	715
query45	191	171	168	168
query46	1071	717	730	717
query47	1852	1725	1745	1725
query48	378	297	299	297
query49	1178	408	402	402
query50	753	375	382	375
query51	6801	6673	6672	6672
query52	103	93	88	88
query53	355	296	285	285
query54	1054	431	439	431
query55	76	75	72	72
query56	270	264	263	263
query57	1177	1085	1021	1021
query58	255	255	244	244
query59	3424	3294	3132	3132
query60	293	268	269	268
query61	97	91	93	91
query62	653	442	466	442
query63	313	293	284	284
query64	9990	2279	1752	1752
query65	3144	3133	3129	3129
query66	1374	337	332	332
query67	15322	14857	14812	14812
query68	4737	527	542	527
query69	621	445	380	380
query70	1116	1228	1171	1171
query71	414	274	263	263
query72	8055	5386	5839	5386
query73	759	325	323	323
query74	5909	5530	5413	5413
query75	3671	2666	2679	2666
query76	3348	948	957	948
query77	669	309	306	306
query78	10508	9667	9743	9667
query79	2686	510	509	509
query80	1890	487	479	479
query81	583	220	229	220
query82	1307	105	106	105
query83	340	183	177	177
query84	276	88	97	88
query85	1629	326	321	321
query86	487	329	323	323
query87	3301	3085	3076	3076
query88	4047	2366	2361	2361
query89	463	379	381	379
query90	1781	197	195	195
query91	139	111	111	111
query92	65	51	50	50
query93	2393	522	502	502
query94	1275	200	193	193
query95	412	318	317	317
query96	605	269	267	267
query97	3227	3058	2992	2992
query98	220	203	201	201
query99	1293	845	844	844
Total cold run time: 277690 ms
Total hot run time: 172752 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d197ef1cdebcaaa71913fbc8304036bf6b53e973, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.48	0.48
query6	1.12	0.72	0.72
query7	0.03	0.02	0.02
query8	0.06	0.04	0.04
query9	0.54	0.49	0.48
query10	0.55	0.54	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.61
query14	0.75	0.77	0.80
query15	0.86	0.82	0.81
query16	0.37	0.37	0.36
query17	1.02	0.96	1.04
query18	0.23	0.23	0.25
query19	1.92	1.81	1.81
query20	0.01	0.00	0.01
query21	15.43	0.65	0.65
query22	3.70	7.49	2.49
query23	18.29	1.50	1.26
query24	2.07	0.23	0.22
query25	0.14	0.09	0.08
query26	0.25	0.17	0.17
query27	0.08	0.09	0.07
query28	13.31	1.00	0.99
query29	12.64	3.29	3.31
query30	0.26	0.06	0.06
query31	2.88	0.38	0.38
query32	3.26	0.47	0.46
query33	2.90	2.91	2.93
query34	17.16	4.39	4.39
query35	4.44	4.43	4.45
query36	0.65	0.45	0.47
query37	0.19	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.07 s
Total hot run time: 31.14 s

@morningman morningman merged commit 7b6db96 into apache:master Jun 17, 2024
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
Sometime when using Hudi Catalog, there will be lots of objects like
`org.apache.hadoop.hdfs.client.impl.DfsClientConf` in FE memory,
which belongs to the Hadoop FileSystem's Cache.

Disable this Cache to avoid FE OOM by setting
`fs.xxx.impl.disable.cache` to true
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants