Skip to content

branch-3.0: [fix](hudi) Fix Memory Leak in BitCaskDiskMap Due to Circular Reference #48955 #49114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Mar 19, 2025
Merged

branch-3.0: [fix](hudi) Fix Memory Leak in BitCaskDiskMap Due to Circular Reference #48955 #49114

dataroaring merged 1 commit into from
Mar 19, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Mar 16, 2025

Cherry-picked from #48955

@github-actions github-actions bot requested a review from dataroaring as a code owner March 16, 2025 09:10
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 16, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Mar 16, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 16, 2025

run buildall

@suxiaogang223 @dataroaring
[fix](hudi) Fix Memory Leak in BitCaskDiskMap Due to Circular Referen…
7271668 
…ce (#48955)

### What problem does this PR solve?

Problem Summary:
As this issue describes apache/hudi#12918
While debugging a memory leak issue, I noticed a steady increase in the
number of BitCaskDiskMap instances through jmap logs. Using VisualVM, I
found that a memory leak may exist in BitCaskDiskMap due to a circular
reference.

![image](https://github.com/user-attachments/assets/0af6fa66-4c61-46dc-9c67-df1ae18c6607)
- The cleanup method is an instance method that implicitly depends on
the this pointer
- When cleanup is called from shutdownThread, it holds a reference to
this
- Meanwhile, the DiskMap class holds a reference to shutdownThread

![image](https://github.com/user-attachments/assets/3a762e6d-4ebe-4177-a6ff-c01afa5b8180)

This creates a circular reference:
```text
  DiskMap (this) -> shutdownThread -> cleanup -> DiskMap (this)
```
@dataroaring dataroaring force-pushed the auto-pick-48955-branch-3.0 branch from 8bde275 to 7271668 Compare March 17, 2025 07:42
@dataroaring
Copy link
Contributor

dataroaring commented Mar 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

TPC-H: Total hot run time: 40181 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72716689f0512f3dbbc39c3a6886e38f8c1e6170, data reload: false

------ Round 1 ----------------------------------
q1	17582	7031	6590	6590
q2	2072	181	184	181
q3	10683	1070	1159	1070
q4	10431	761	810	761
q5	7754	2868	2797	2797
q6	220	134	133	133
q7	976	615	605	605
q8	9360	1946	2066	1946
q9	6624	6454	6450	6450
q10	7052	2266	2277	2266
q11	464	269	260	260
q12	406	206	205	205
q13	17762	2946	2998	2946
q14	239	202	214	202
q15	509	467	457	457
q16	662	568	574	568
q17	993	631	585	585
q18	7408	6753	6708	6708
q19	1389	1087	1012	1012
q20	462	207	198	198
q21	4008	3248	3242	3242
q22	1121	1010	999	999
Total cold run time: 108177 ms
Total hot run time: 40181 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6584	6553	6567	6553
q2	331	238	230	230
q3	2919	2766	2952	2766
q4	2009	1751	1771	1751
q5	5794	5811	5749	5749
q6	212	126	131	126
q7	2179	1853	1790	1790
q8	3419	3585	3548	3548
q9	8800	8887	8956	8887
q10	3574	3564	3494	3494
q11	593	492	491	491
q12	792	613	617	613
q13	8547	3269	3129	3129
q14	308	272	266	266
q15	498	476	472	472
q16	683	658	646	646
q17	1880	1619	1588	1588
q18	8360	7804	7620	7620
q19	1678	1617	1624	1617
q20	2127	1877	1891	1877
q21	5483	5386	5346	5346
q22	1134	1101	1003	1003
Total cold run time: 67904 ms
Total hot run time: 59562 ms

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

TPC-DS: Total hot run time: 196989 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 72716689f0512f3dbbc39c3a6886e38f8c1e6170, data reload: false

query1	1330	940	926	926
query2	6237	2056	2018	2018
query3	10946	4432	4357	4357
query4	66114	28952	23222	23222
query5	5034	468	435	435
query6	406	174	167	167
query7	5632	310	303	303
query8	293	216	210	210
query9	9167	2618	2584	2584
query10	466	289	260	260
query11	17724	15072	15596	15072
query12	164	108	105	105
query13	1550	442	441	441
query14	10969	7287	7434	7287
query15	229	185	187	185
query16	7165	461	490	461
query17	1080	574	584	574
query18	1875	309	314	309
query19	210	163	166	163
query20	118	112	113	112
query21	206	105	101	101
query22	4995	4421	4544	4421
query23	34359	34092	34010	34010
query24	6147	2919	2926	2919
query25	533	434	436	434
query26	657	173	172	172
query27	1793	359	365	359
query28	4495	2485	2431	2431
query29	712	494	457	457
query30	236	161	159	159
query31	1018	863	840	840
query32	76	55	56	55
query33	445	293	311	293
query34	918	517	514	514
query35	844	735	738	735
query36	1105	945	969	945
query37	125	66	66	66
query38	4133	3962	4023	3962
query39	1539	1583	1461	1461
query40	201	98	96	96
query41	49	46	47	46
query42	119	105	105	105
query43	538	491	478	478
query44	1172	822	829	822
query45	192	171	168	168
query46	1128	739	711	711
query47	2022	1936	1936	1936
query48	496	407	405	405
query49	730	398	382	382
query50	839	426	433	426
query51	7525	7205	7106	7106
query52	96	84	86	84
query53	257	178	185	178
query54	569	459	454	454
query55	76	78	73	73
query56	269	234	246	234
query57	1258	1110	1148	1110
query58	207	205	200	200
query59	3096	2962	2897	2897
query60	282	260	238	238
query61	108	109	111	109
query62	790	665	670	665
query63	224	188	186	186
query64	1388	671	627	627
query65	3273	3183	3183	3183
query66	705	299	301	299
query67	15993	15657	15639	15639
query68	3237	597	597	597
query69	428	269	260	260
query70	1200	1132	1127	1127
query71	355	263	261	261
query72	6196	4109	4143	4109
query73	746	353	351	351
query74	10077	8926	9295	8926
query75	3349	2643	2652	2643
query76	1954	1063	1009	1009
query77	538	280	275	275
query78	10535	9740	9639	9639
query79	1260	606	586	586
query80	856	433	427	427
query81	514	246	240	240
query82	1238	88	84	84
query83	202	140	142	140
query84	284	78	79	78
query85	862	295	293	293
query86	337	283	267	267
query87	4369	4286	4240	4240
query88	3762	2373	2366	2366
query89	418	294	295	294
query90	1994	184	190	184
query91	177	150	149	149
query92	63	47	57	47
query93	1559	554	564	554
query94	772	302	285	285
query95	359	261	260	260
query96	613	284	284	284
query97	3300	3180	3140	3140
query98	230	209	198	198
query99	1542	1294	1296	1294
Total cold run time: 317973 ms
Total hot run time: 196989 ms

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

ClickBench: Total hot run time: 31.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 72716689f0512f3dbbc39c3a6886e38f8c1e6170, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.62	0.10	0.10
query5	0.53	0.52	0.51
query6	1.14	0.72	0.73
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.56	0.49	0.50
query10	0.56	0.55	0.54
query11	0.14	0.10	0.10
query12	0.17	0.11	0.10
query13	0.61	0.59	0.58
query14	2.73	2.74	2.78
query15	0.89	0.83	0.82
query16	0.39	0.38	0.39
query17	1.06	1.04	1.07
query18	0.24	0.21	0.22
query19	1.81	1.82	1.99
query20	0.01	0.01	0.01
query21	15.39	0.58	0.58
query22	2.41	2.39	0.92
query23	17.02	0.94	0.85
query24	3.25	1.97	0.78
query25	0.30	0.12	0.04
query26	0.52	0.14	0.13
query27	0.05	0.03	0.04
query28	9.73	0.54	0.48
query29	12.60	3.28	3.29
query30	0.25	0.06	0.06
query31	2.85	0.38	0.39
query32	3.25	0.46	0.47
query33	2.94	2.99	3.05
query34	17.30	4.51	4.51
query35	4.60	4.56	4.54
query36	0.70	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.11	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.49 s
Total hot run time: 31.13 s

dataroaring
dataroaring approved these changes Mar 19, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 29d0ca8 into branch-3.0 Mar 19, 2025
26 of 29 checks passed
@github-actions github-actions bot deleted the auto-pick-48955-branch-3.0 branch March 19, 2025 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @dataroaring @doris-robot @suxiaogang223