[fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP #40773

zy-kkk · 2024-09-12T15:55:29Z

When FE collects statistics, its ThreadLocal will hold the ThreadLocal of HikariCP in JdbcClient, making it difficult for HikariCP to be gc, so this problem can be alleviated by enabling weak references of HikariCP.
Introduced from #38889

…in HikariCP

doris-robot · 2024-09-12T15:55:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zy-kkk · 2024-09-12T15:55:37Z

run buildall

doris-robot · 2024-09-12T16:32:09Z

TPC-H: Total hot run time: 42779 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ab17b22f2df0ba4f129f5ca5f6c4f3879f82713c, data reload: false

------ Round 1 ----------------------------------
q1	17595	7293	7197	7197
q2	2014	183	204	183
q3	10457	1266	1336	1266
q4	10321	932	1016	932
q5	7722	3148	3139	3139
q6	240	152	149	149
q7	1035	632	622	622
q8	9446	2037	2018	2018
q9	6769	6266	6313	6266
q10	7027	2503	2503	2503
q11	433	248	254	248
q12	411	227	229	227
q13	17756	3015	3002	3002
q14	286	250	255	250
q15	578	532	521	521
q16	515	440	426	426
q17	981	936	956	936
q18	7432	6836	6845	6836
q19	1386	1233	1221	1221
q20	626	330	334	330
q21	3906	3496	3490	3490
q22	1082	1054	1017	1017
Total cold run time: 108018 ms
Total hot run time: 42779 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7183	7151	7150	7150
q2	334	240	244	240
q3	3060	3119	3033	3033
q4	2081	2138	2078	2078
q5	5688	5580	5683	5580
q6	238	147	148	147
q7	2118	1798	1753	1753
q8	3392	3439	3413	3413
q9	8872	9020	8814	8814
q10	3499	3612	3628	3612
q11	588	493	496	493
q12	832	605	610	605
q13	8721	3242	3234	3234
q14	321	292	289	289
q15	608	541	556	541
q16	537	456	462	456
q17	1819	1757	1752	1752
q18	8538	8115	8139	8115
q19	1779	1771	1751	1751
q20	2136	1903	1904	1903
q21	5884	5617	5716	5617
q22	1149	1089	1141	1089
Total cold run time: 69377 ms
Total hot run time: 61665 ms

morningman

LGTM

github-actions · 2024-09-13T03:16:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-13T03:16:27Z

PR approved by anyone and no changes requested.

doris-robot · 2024-09-13T11:57:23Z

TPC-H: Total hot run time: 42729 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ab17b22f2df0ba4f129f5ca5f6c4f3879f82713c, data reload: false

------ Round 1 ----------------------------------
q1	17573	7367	7175	7175
q2	2023	183	176	176
q3	10809	1272	1359	1272
q4	10327	1004	1051	1004
q5	7712	3165	3130	3130
q6	233	148	150	148
q7	1031	627	611	611
q8	9471	2006	2040	2006
q9	6728	6295	6287	6287
q10	7063	2526	2500	2500
q11	444	260	254	254
q12	400	227	224	224
q13	17778	3035	2978	2978
q14	292	258	257	257
q15	578	521	541	521
q16	561	452	424	424
q17	999	952	956	952
q18	7537	6747	6884	6747
q19	1395	1233	1230	1230
q20	606	329	323	323
q21	3865	3528	3516	3516
q22	1065	994	1002	994
Total cold run time: 108490 ms
Total hot run time: 42729 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7147	7135	7178	7135
q2	348	241	230	230
q3	3101	3127	3028	3028
q4	2094	2112	2024	2024
q5	5679	5582	5671	5582
q6	243	146	148	146
q7	2154	1761	1766	1761
q8	3354	3453	3381	3381
q9	8867	8808	8776	8776
q10	3431	3575	3602	3575
q11	587	485	481	481
q12	807	601	629	601
q13	9046	3226	3212	3212
q14	328	274	294	274
q15	591	541	548	541
q16	502	467	485	467
q17	1805	1741	1734	1734
q18	8575	8081	7949	7949
q19	1775	1739	1739	1739
q20	2139	1900	1875	1875
q21	5885	5324	5690	5324
q22	1216	1047	1021	1021
Total cold run time: 69674 ms
Total hot run time: 60856 ms

doris-robot · 2024-09-13T12:09:34Z

TPC-DS: Total hot run time: 199688 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ab17b22f2df0ba4f129f5ca5f6c4f3879f82713c, data reload: false

query1	1274	886	887	886
query2	6347	1795	1762	1762
query3	10643	3784	3764	3764
query4	54620	26021	23980	23980
query5	5007	542	529	529
query6	325	168	159	159
query7	5642	304	294	294
query8	277	223	220	220
query9	6115	2593	2590	2590
query10	413	300	267	267
query11	15952	15773	15441	15441
query12	150	101	99	99
query13	1481	398	395	395
query14	10383	8011	7075	7075
query15	216	178	184	178
query16	6635	513	485	485
query17	1156	638	618	618
query18	1217	323	310	310
query19	201	162	157	157
query20	120	117	114	114
query21	211	106	109	106
query22	4606	4553	4477	4477
query23	35052	33991	33855	33855
query24	6123	3087	3132	3087
query25	523	406	442	406
query26	617	162	163	162
query27	1550	286	283	283
query28	2818	2107	2085	2085
query29	712	437	434	434
query30	227	155	155	155
query31	946	768	763	763
query32	77	57	56	56
query33	475	304	304	304
query34	884	493	481	481
query35	878	731	748	731
query36	1040	921	912	912
query37	136	80	80	80
query38	3980	3884	4056	3884
query39	1481	1391	1392	1391
query40	213	116	116	116
query41	50	49	47	47
query42	123	100	107	100
query43	481	432	463	432
query44	1156	799	777	777
query45	197	172	173	172
query46	1117	857	824	824
query47	1908	1789	1829	1789
query48	397	307	305	305
query49	743	465	464	464
query50	938	445	449	445
query51	7108	6806	7073	6806
query52	100	89	89	89
query53	255	181	182	181
query54	569	474	466	466
query55	76	78	76	76
query56	288	267	269	267
query57	1202	1075	1088	1075
query58	228	255	281	255
query59	2801	2704	2507	2507
query60	288	269	274	269
query61	102	112	96	96
query62	792	670	678	670
query63	225	182	184	182
query64	1594	703	671	671
query65	3280	3135	3154	3135
query66	638	289	301	289
query67	16244	15764	15776	15764
query68	2032	859	853	853
query69	436	324	331	324
query70	1157	1144	1156	1144
query71	360	348	367	348
query72	5972	3324	3423	3324
query73	597	587	582	582
query74	9228	9227	9157	9157
query75	3129	2987	2986	2986
query76	970	842	832	832
query77	535	400	398	398
query78	9456	9343	9140	9140
query79	937	890	857	857
query80	836	808	806	806
query81	466	267	270	267
query82	268	260	261	260
query83	187	188	186	186
query84	313	107	103	103
query85	658	411	401	401
query86	320	309	315	309
query87	4391	4307	4303	4303
query88	4130	4058	4068	4058
query89	378	364	367	364
query90	1830	312	306	306
query91	125	123	124	123
query92	78	78	73	73
query93	1026	1026	1022	1022
query94	655	366	384	366
query95	457	436	418	418
query96	467	467	467	467
query97	3120	3120	3127	3120
query98	236	232	228	228
query99	1677	1319	1326	1319
Total cold run time: 290050 ms
Total hot run time: 199688 ms

doris-robot · 2024-09-13T12:15:03Z

ClickBench: Total hot run time: 31.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ab17b22f2df0ba4f129f5ca5f6c4f3879f82713c, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.03
query3	0.22	0.05	0.05
query4	1.68	0.06	0.06
query5	0.50	0.50	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.57	0.53	0.52
query10	0.57	0.58	0.56
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.62	0.61	0.60
query14	1.48	1.49	1.46
query15	0.90	0.88	0.87
query16	0.35	0.35	0.35
query17	1.04	1.06	1.02
query18	0.16	0.16	0.19
query19	1.87	1.84	1.88
query20	0.01	0.01	0.01
query21	15.39	0.69	0.68
query22	4.04	6.69	1.85
query23	17.93	1.37	1.41
query24	2.20	0.24	0.22
query25	0.18	0.08	0.08
query26	0.29	0.18	0.18
query27	0.07	0.08	0.08
query28	13.18	1.11	1.10
query29	12.51	3.35	3.30
query30	0.25	0.05	0.05
query31	2.87	0.42	0.41
query32	3.23	0.49	0.49
query33	2.98	3.05	3.06
query34	15.46	4.33	4.29
query35	4.36	4.34	4.33
query36	0.68	0.49	0.48
query37	0.20	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 108 s
Total hot run time: 31.65 s

…in HikariCP (apache#40773) When FE collects statistics, its ThreadLocal will hold the ThreadLocal of HikariCP in JdbcClient, making it difficult for HikariCP to be gc, so this problem can be alleviated by enabling weak references of HikariCP.

… references in HikariCP (#40923) pick (#40773) When FE collects statistics, its ThreadLocal will hold the ThreadLocal of HikariCP in JdbcClient, making it difficult for HikariCP to be gc, so this problem can be alleviated by enabling weak references of HikariCP.

…in HikariCP (apache#40773) When FE collects statistics, its ThreadLocal will hold the ThreadLocal of HikariCP in JdbcClient, making it difficult for HikariCP to be gc, so this problem can be alleviated by enabling weak references of HikariCP.

…in HikariCP (#40773) (#41503) bp #40773 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

… references in HikariCP (apache#40923) pick (apache#40773) When FE collects statistics, its ThreadLocal will hold the ThreadLocal of HikariCP in JdbcClient, making it difficult for HikariCP to be gc, so this problem can be alleviated by enabling weak references of HikariCP.

[fix](jdbc catalog) Fixed FE memory leak by enabling weak references …

ab17b22

…in HikariCP

zy-kkk added usercase Important user case type label dev/2.0.x dev/2.1.x dev/3.0.x labels Sep 12, 2024

morningman approved these changes Sep 13, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 13, 2024

github-actions bot added the reviewed label Sep 13, 2024

BePPPower approved these changes Sep 13, 2024

View reviewed changes

yiguolei merged commit 10ba034 into apache:master Sep 13, 2024

zy-kkk deleted the fix_fe_hikari branch September 18, 2024 08:34

zy-kkk mentioned this pull request Sep 18, 2024

[branch-2.1][fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP #40923

Merged

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Sep 19, 2024

morningman added p0_b and removed dev/2.0.x labels Sep 23, 2024

morningman mentioned this pull request Sep 30, 2024

[fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP (#40773) #41503

Merged

morningman added a commit that referenced this pull request Sep 30, 2024

[fix](jdbc catalog) Fixed FE memory leak by enabling weak references …

009115c

…in HikariCP (#40773) (#41503) bp #40773 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

morningman removed the dev/3.0.x label Sep 30, 2024

morningman added the dev/3.0.2-merged label Sep 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP #40773

[fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP #40773

Uh oh!

zy-kkk commented Sep 12, 2024 •

edited by morningman

Loading

Uh oh!

doris-robot commented Sep 12, 2024

Uh oh!

zy-kkk commented Sep 12, 2024

Uh oh!

doris-robot commented Sep 12, 2024

Uh oh!

morningman left a comment

Uh oh!

github-actions bot commented Sep 13, 2024

Uh oh!

github-actions bot commented Sep 13, 2024

Uh oh!

doris-robot commented Sep 13, 2024

Uh oh!

doris-robot commented Sep 13, 2024

Uh oh!

doris-robot commented Sep 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP #40773

[fix](jdbc catalog) Fixed FE memory leak by enabling weak references in HikariCP #40773

Uh oh!

Conversation

zy-kkk commented Sep 12, 2024 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Sep 12, 2024

Uh oh!

zy-kkk commented Sep 12, 2024

Uh oh!

doris-robot commented Sep 12, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 13, 2024

Uh oh!

github-actions bot commented Sep 13, 2024

Uh oh!

doris-robot commented Sep 13, 2024

Uh oh!

doris-robot commented Sep 13, 2024

Uh oh!

doris-robot commented Sep 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zy-kkk commented Sep 12, 2024 •

edited by morningman

Loading