Skip to content

[fix](jdbc catalog) Fix connection leak in PostgreSQL JDBC client #49568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Apr 2, 2025
Merged

[fix](jdbc catalog) Fix connection leak in PostgreSQL JDBC client #49568

morningman merged 2 commits into from
Apr 2, 2025

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Mar 27, 2025

What problem does this PR solve?

Connection leak in JdbcPostgreSQLClient when handling array types, causing PostgreSQL server error "too many clients already".

Solution

1.Use explicit resource management instead of try-with-resources
2. Add exception handling and properly close all database resources

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 27, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk
Copy link
Member Author

zy-kkk commented Mar 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-H: Total hot run time: 33939 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5444e6a53e5af08912d61b129966c756a21f5ed, data reload: false

------ Round 1 ----------------------------------
q1	25909	5051	5003	5003
q2	2097	293	192	192
q3	10410	1263	690	690
q4	10217	986	524	524
q5	7558	2350	2359	2350
q6	191	169	131	131
q7	891	749	612	612
q8	9302	1268	1054	1054
q9	6762	5107	5093	5093
q10	6823	2283	1908	1908
q11	453	271	259	259
q12	337	353	217	217
q13	17796	3703	3067	3067
q14	227	229	214	214
q15	560	503	497	497
q16	611	613	585	585
q17	568	845	339	339
q18	7662	7179	7057	7057
q19	1231	965	579	579
q20	328	330	194	194
q21	3854	2626	2376	2376
q22	1066	1065	998	998
Total cold run time: 114853 ms
Total hot run time: 33939 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5156	5141	5114	5114
q2	229	321	219	219
q3	2141	2658	2308	2308
q4	1436	1862	1422	1422
q5	4508	4406	4420	4406
q6	217	167	126	126
q7	1939	1879	1745	1745
q8	2542	2546	2511	2511
q9	7237	7091	7264	7091
q10	2989	3192	2761	2761
q11	579	497	475	475
q12	677	764	632	632
q13	3558	3922	3271	3271
q14	278	298	297	297
q15	550	502	496	496
q16	639	679	639	639
q17	1150	1579	1342	1342
q18	7629	7595	7536	7536
q19	802	809	1005	809
q20	1946	2038	1893	1893
q21	5095	4780	4670	4670
q22	1053	1010	1010	1010
Total cold run time: 52350 ms
Total hot run time: 50773 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-DS: Total hot run time: 186944 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f5444e6a53e5af08912d61b129966c756a21f5ed, data reload: false

query1	1041	496	477	477
query2	6533	1909	1880	1880
query3	6793	222	216	216
query4	26183	23885	23291	23291
query5	4371	664	499	499
query6	312	217	202	202
query7	4610	481	276	276
query8	311	250	238	238
query9	8650	2563	2529	2529
query10	472	306	264	264
query11	15771	15294	14854	14854
query12	177	111	112	111
query13	1658	518	403	403
query14	9928	6250	6258	6250
query15	216	187	168	168
query16	7664	636	468	468
query17	1197	734	572	572
query18	2014	407	313	313
query19	206	190	162	162
query20	124	117	119	117
query21	208	129	108	108
query22	4374	4408	4238	4238
query23	33909	33225	33120	33120
query24	8442	2411	2414	2411
query25	523	468	388	388
query26	1215	260	149	149
query27	2737	501	334	334
query28	4337	2438	2427	2427
query29	707	568	426	426
query30	284	217	188	188
query31	975	858	773	773
query32	73	68	63	63
query33	570	391	318	318
query34	794	884	497	497
query35	814	822	760	760
query36	948	987	897	897
query37	115	98	78	78
query38	4205	4068	4028	4028
query39	1479	1402	1411	1402
query40	207	115	100	100
query41	56	53	52	52
query42	120	105	104	104
query43	515	525	473	473
query44	1347	817	814	814
query45	177	176	169	169
query46	841	1028	661	661
query47	1794	1843	1722	1722
query48	373	422	299	299
query49	785	519	439	439
query50	676	725	433	433
query51	4176	4226	4145	4145
query52	123	103	100	100
query53	226	256	180	180
query54	489	495	421	421
query55	82	77	84	77
query56	270	296	257	257
query57	1117	1147	1093	1093
query58	250	240	239	239
query59	2638	2871	2778	2778
query60	286	278	258	258
query61	127	124	133	124
query62	801	732	656	656
query63	230	192	188	188
query64	4262	1040	673	673
query65	4355	4214	4283	4214
query66	1075	454	305	305
query67	16023	15959	15513	15513
query68	8710	884	517	517
query69	471	305	266	266
query70	1240	1094	1097	1094
query71	482	298	277	277
query72	5534	4843	4906	4843
query73	764	655	349	349
query74	8991	9162	8665	8665
query75	3944	3258	2675	2675
query76	3706	1190	756	756
query77	785	378	343	343
query78	9915	10215	9397	9397
query79	2636	811	572	572
query80	679	520	441	441
query81	477	257	225	225
query82	470	133	102	102
query83	210	177	165	165
query84	300	105	82	82
query85	783	363	298	298
query86	334	308	287	287
query87	4435	4538	4321	4321
query88	3252	2247	2205	2205
query89	391	320	281	281
query90	1987	210	213	210
query91	143	196	112	112
query92	78	60	62	60
query93	1318	1076	586	586
query94	682	415	289	289
query95	361	279	267	267
query96	480	567	276	276
query97	3240	3268	3125	3125
query98	225	211	215	211
query99	1474	1398	1265	1265
Total cold run time: 276858 ms
Total hot run time: 186944 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

ClickBench: Total hot run time: 31.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f5444e6a53e5af08912d61b129966c756a21f5ed, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.24	0.19	0.19
query4	1.61	0.19	0.20
query5	0.59	0.56	0.60
query6	1.19	0.71	0.71
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.53	0.52
query10	0.58	0.58	0.57
query11	0.15	0.11	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.67	2.85	2.70
query15	0.92	0.84	0.84
query16	0.38	0.37	0.38
query17	1.04	1.06	1.07
query18	0.21	0.20	0.20
query19	1.94	1.98	1.84
query20	0.01	0.01	0.00
query21	15.36	0.87	0.55
query22	0.77	1.22	0.98
query23	14.72	1.38	0.69
query24	6.69	0.82	1.19
query25	0.49	0.17	0.09
query26	0.59	0.16	0.13
query27	0.04	0.05	0.05
query28	10.09	0.88	0.44
query29	12.56	4.01	3.30
query30	0.24	0.08	0.06
query31	2.84	0.59	0.38
query32	3.23	0.54	0.49
query33	3.06	3.09	3.17
query34	15.79	5.15	4.53
query35	4.56	4.61	4.55
query36	0.67	0.49	0.49
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.25 s
Total hot run time: 31.67 s

@github-actions
Copy link
Contributor

github-actions bot commented Apr 2, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 2, 2025

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 1fc222d into apache:master Apr 2, 2025
29 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Apr 2, 2025
@zy-kkk
[fix](jdbc catalog) Fix connection leak in PostgreSQL JDBC client (#4…
b6ba44a 
…9568)

### What problem does this PR solve?

Connection leak in JdbcPostgreSQLClient when handling array types,
causing PostgreSQL server error "too many clients already".

### Solution

1.Use explicit resource management instead of try-with-resources
2. Add exception handling and properly close all database resources
@github-actions github-actions bot mentioned this pull request Apr 2, 2025
Merged
dataroaring pushed a commit that referenced this pull request Apr 7, 2025
@github-actions @zy-kkk
branch-3.0: [fix](jdbc catalog) Fix connection leak in PostgreSQL JDB…
a7ab767 
…C client #49568 (#49760)

Cherry-picked from #49568

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
Closed
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request Apr 27, 2025
@zy-kkk
[fix](jdbc catalog) Fix connection leak in PostgreSQL JDBC client apa…
966fb83 
…che#49568 (apache#3848)

pick: apache#49568
@zy-kkk zy-kkk deleted the fix_pg_connection branch May 16, 2025 03:09
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@zy-kkk @koarz
[fix](jdbc catalog) Fix connection leak in PostgreSQL JDBC client (ap…
62ea92e 
…ache#49568)

### What problem does this PR solve?

Connection leak in JdbcPostgreSQLClient when handling array types,
causing PostgreSQL server error "too many clients already".

### Solution

1.Use explicit resource management instead of try-with-resources
2. Add exception handling and properly close all database resources
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

+1 more reviewer

@hubgeter hubgeter hubgeter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zy-kkk @hello-stephen @doris-robot @morningman @CalvinKirs @hubgeter @dataroaring