Skip to content

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Dec 1, 2025

Problem

JNI Global References may not be deleted when JdbcConnector::close() fails
midway due to early returns from RETURN_IF_ERROR macros. This prevents Java
GC from collecting BaseJdbcExecutor and its associated ResultSet objects,
causing old generation heap to grow to 99%+.

Solution

  • C++: Always delete all three Global References regardless of Java close() failure
  • Java: Always clear member variable references (resultSet, stmt, conn) in finally block

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk
Copy link
Member Author

zy-kkk commented Dec 1, 2025

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 1, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 33974 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 983ba63cdcd897904ceb31939bfb2407d9a2ac51, data reload: false

------ Round 1 ----------------------------------
q1	17627	5181	4972	4972
q2	2076	309	212	212
q3	10242	1288	735	735
q4	10210	833	337	337
q5	7528	2466	2139	2139
q6	191	167	137	137
q7	956	787	649	649
q8	9359	1369	1022	1022
q9	7124	5251	5318	5251
q10	6795	2258	1830	1830
q11	526	323	291	291
q12	340	373	238	238
q13	17797	3671	3012	3012
q14	234	228	219	219
q15	608	535	508	508
q16	913	873	819	819
q17	586	761	523	523
q18	7484	7005	7101	7005
q19	1239	979	586	586
q20	346	343	226	226
q21	4052	3278	2343	2343
q22	1006	999	920	920
Total cold run time: 107239 ms
Total hot run time: 33974 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4966	4972	4961	4961
q2	337	383	322	322
q3	2165	2689	2319	2319
q4	1319	1768	1316	1316
q5	4241	4448	4633	4448
q6	226	181	128	128
q7	2043	2051	1867	1867
q8	2698	2502	2525	2502
q9	7393	7635	7294	7294
q10	3059	3280	2840	2840
q11	592	531	511	511
q12	675	768	626	626
q13	3555	3894	3390	3390
q14	310	303	318	303
q15	596	538	505	505
q16	923	965	901	901
q17	1156	1361	1393	1361
q18	7978	7621	7595	7595
q19	825	788	850	788
q20	2052	2091	1929	1929
q21	4943	4580	4167	4167
q22	1085	1040	979	979
Total cold run time: 53137 ms
Total hot run time: 51052 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181893 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 983ba63cdcd897904ceb31939bfb2407d9a2ac51, data reload: false

query1	1023	421	428	421
query2	6589	1216	1193	1193
query3	6741	229	238	229
query4	25572	23267	22938	22938
query5	4423	683	482	482
query6	337	253	250	250
query7	4643	522	311	311
query8	299	257	278	257
query9	8702	2602	2658	2602
query10	512	352	312	312
query11	15165	15283	14838	14838
query12	190	120	118	118
query13	1713	602	455	455
query14	9613	5977	6057	5977
query15	218	204	189	189
query16	7489	687	490	490
query17	1254	790	633	633
query18	2016	436	347	347
query19	215	210	184	184
query20	130	128	123	123
query21	223	139	120	120
query22	3946	4037	4027	4027
query23	33020	31886	32251	31886
query24	8623	2407	2429	2407
query25	640	568	504	504
query26	1252	278	172	172
query27	2733	502	351	351
query28	4336	2199	2181	2181
query29	853	643	534	534
query30	317	243	212	212
query31	835	718	625	625
query32	85	78	76	76
query33	610	390	346	346
query34	821	883	552	552
query35	842	841	748	748
query36	888	921	829	829
query37	136	115	88	88
query38	3885	3936	3762	3762
query39	1474	1387	1413	1387
query40	230	132	122	122
query41	64	61	62	61
query42	125	119	114	114
query43	459	439	415	415
query44	1284	760	748	748
query45	199	192	184	184
query46	877	982	644	644
query47	1675	1732	1630	1630
query48	395	415	331	331
query49	786	510	437	437
query50	682	707	410	410
query51	3882	3922	3845	3845
query52	111	114	109	109
query53	241	254	193	193
query54	309	298	282	282
query55	92	98	96	96
query56	342	340	330	330
query57	1138	1177	1113	1113
query58	297	284	272	272
query59	2293	2397	2306	2306
query60	375	364	331	331
query61	165	157	162	157
query62	811	706	667	667
query63	234	199	202	199
query64	4570	1239	891	891
query65	4041	3960	3986	3960
query66	1151	447	353	353
query67	15279	15172	14815	14815
query68	8252	980	620	620
query69	512	347	311	311
query70	1077	971	1010	971
query71	481	348	313	313
query72	5993	4977	4860	4860
query73	686	585	353	353
query74	8887	8865	8601	8601
query75	3666	3052	2530	2530
query76	3732	1156	759	759
query77	804	427	328	328
query78	9472	9613	8842	8842
query79	2825	847	593	593
query80	697	573	479	479
query81	511	280	245	245
query82	510	141	112	112
query83	303	269	245	245
query84	306	125	97	97
query85	905	487	452	452
query86	397	304	292	292
query87	4043	4067	3960	3960
query88	4054	2285	2269	2269
query89	394	339	296	296
query90	1925	224	233	224
query91	172	172	147	147
query92	89	73	69	69
query93	2165	988	649	649
query94	748	466	346	346
query95	511	421	413	413
query96	504	549	285	285
query97	2649	2681	2567	2567
query98	238	220	220	220
query99	1408	1389	1274	1274
Total cold run time: 273386 ms
Total hot run time: 181893 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 983ba63cdcd897904ceb31939bfb2407d9a2ac51, data reload: false

query1	0.05	0.05	0.05
query2	0.11	0.06	0.06
query3	0.26	0.10	0.09
query4	1.61	0.12	0.11
query5	0.28	0.27	0.27
query6	1.16	0.64	0.64
query7	0.03	0.03	0.03
query8	0.06	0.05	0.04
query9	0.58	0.52	0.50
query10	0.56	0.56	0.57
query11	0.16	0.11	0.12
query12	0.15	0.11	0.13
query13	0.62	0.60	0.59
query14	0.99	0.98	0.98
query15	0.82	0.80	0.80
query16	0.39	0.39	0.44
query17	1.03	1.10	1.06
query18	0.23	0.21	0.21
query19	1.96	1.87	1.89
query20	0.01	0.02	0.02
query21	15.44	0.27	0.15
query22	4.62	0.05	0.05
query23	15.93	0.27	0.10
query24	1.38	0.49	0.26
query25	0.08	0.07	0.05
query26	0.15	0.14	0.14
query27	0.08	0.05	0.06
query28	3.18	1.21	1.02
query29	12.62	3.91	3.23
query30	0.29	0.14	0.11
query31	2.82	0.63	0.38
query32	3.24	0.55	0.46
query33	2.97	3.04	3.06
query34	16.64	5.21	4.50
query35	4.53	4.53	4.59
query36	0.66	0.49	0.48
query37	0.10	0.07	0.06
query38	0.07	0.05	0.04
query39	0.05	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.27 s
Total hot run time: 27.4 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 6.25% (2/32) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.82% (18492/35009)
Line Coverage 38.35% (169362/441577)
Region Coverage 33.14% (131451/396621)
Branch Coverage 34.10% (56612/166017)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 59.38% (19/32) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.64% (24591/34326)
Line Coverage 58.17% (256632/441183)
Region Coverage 53.27% (213834/401432)
Branch Coverage 54.76% (91399/166914)

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 99427c7 into apache:master Dec 5, 2025
28 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 5, 2025
…8574)

## Problem
JNI Global References may not be deleted when `JdbcConnector::close()`
fails
midway due to early returns from `RETURN_IF_ERROR` macros. This prevents
Java
GC from collecting `BaseJdbcExecutor` and its associated `ResultSet`
objects,
  causing old generation heap to grow to 99%+.

  ## Solution
- C++: Always delete all three Global References regardless of Java
close() failure
- Java: Always clear member variable references (`resultSet`, `stmt`,
`conn`) in finally block
github-actions bot pushed a commit that referenced this pull request Dec 5, 2025
…8574)

## Problem
JNI Global References may not be deleted when `JdbcConnector::close()`
fails
midway due to early returns from `RETURN_IF_ERROR` macros. This prevents
Java
GC from collecting `BaseJdbcExecutor` and its associated `ResultSet`
objects,
  causing old generation heap to grow to 99%+.

  ## Solution
- C++: Always delete all three Global References regardless of Java
close() failure
- Java: Always clear member variable references (`resultSet`, `stmt`,
`conn`) in finally block
@zy-kkk zy-kkk deleted the fix_jdbc_mem branch December 5, 2025 07:20
morrySnow pushed a commit that referenced this pull request Dec 5, 2025
…close #58574 (#58752)

Cherry-picked from #58574

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
yiguolei pushed a commit that referenced this pull request Dec 11, 2025
…close #58574 (#58753)

Cherry-picked from #58574

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
…ache#58574)

## Problem
JNI Global References may not be deleted when `JdbcConnector::close()`
fails
midway due to early returns from `RETURN_IF_ERROR` macros. This prevents
Java
GC from collecting `BaseJdbcExecutor` and its associated `ResultSet`
objects,
  causing old generation heap to grow to 99%+.

  ## Solution
- C++: Always delete all three Global References regardless of Java
close() failure
- Java: Always clear member variable references (`resultSet`, `stmt`,
`conn`) in finally block
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants