[Fix](compaction) Fix nullptr when executing cold compaction #48329

Yukang-Lian · 2025-02-25T13:49:45Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

The cold compaction did not set the compaction policy in the initial phase, causing a null pointer exception when logging later. This issue occurs infrequently; if the tablet has previously done any compaction or calculated a compaction score, the compaction policy will be set automatically, allowing cold compaction to run normally. However, if a newly created tablet undergoes cold compaction immediately, this problem will arise. This PR fixes the issue.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-02-25T13:49:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Yukang-Lian · 2025-02-25T13:49:54Z

run buildall

doris-robot · 2025-02-25T15:39:25Z

TPC-H: Total hot run time: 33194 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 082cc4b2b4359667cbe59d64b8c65b6d7c1e078f, data reload: false

------ Round 1 ----------------------------------
q1	17610	5590	5384	5384
q2	2061	321	188	188
q3	10447	1435	809	809
q4	10230	1142	580	580
q5	7584	2728	2758	2728
q6	218	184	143	143
q7	1025	811	665	665
q8	9320	1467	1258	1258
q9	5179	4817	4875	4817
q10	6927	2409	1947	1947
q11	526	281	264	264
q12	361	400	238	238
q13	17796	3895	3178	3178
q14	260	234	215	215
q15	538	472	476	472
q16	647	647	586	586
q17	610	938	365	365
q18	6810	6334	6146	6146
q19	1695	1088	599	599
q20	342	348	195	195
q21	3178	2457	2105	2105
q22	378	367	312	312
Total cold run time: 103742 ms
Total hot run time: 33194 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5582	5564	5526	5526
q2	259	356	234	234
q3	2329	2872	2440	2440
q4	1544	2030	1412	1412
q5	4445	4241	4459	4241
q6	260	192	131	131
q7	2198	2007	1844	1844
q8	2973	3000	2971	2971
q9	7280	7144	7202	7144
q10	3242	3406	2830	2830
q11	635	517	486	486
q12	670	823	636	636
q13	3690	4084	3364	3364
q14	302	322	293	293
q15	545	493	473	473
q16	674	734	662	662
q17	1255	1794	1407	1407
q18	7726	7498	7222	7222
q19	903	899	1003	899
q20	2049	2136	1887	1887
q21	5890	5035	4987	4987
q22	730	641	554	554
Total cold run time: 55181 ms
Total hot run time: 51643 ms

doris-robot · 2025-02-25T15:51:32Z

TPC-DS: Total hot run time: 191817 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 082cc4b2b4359667cbe59d64b8c65b6d7c1e078f, data reload: false

query1	1313	962	935	935
query2	6265	1880	1870	1870
query3	11145	4657	4562	4562
query4	53814	24961	23345	23345
query5	5057	578	512	512
query6	346	223	200	200
query7	4879	527	300	300
query8	325	256	245	245
query9	5501	2639	2628	2628
query10	425	327	275	275
query11	15104	15227	14883	14883
query12	162	109	110	109
query13	1027	530	373	373
query14	10913	7133	6957	6957
query15	214	206	203	203
query16	7091	664	484	484
query17	1087	722	595	595
query18	1521	418	322	322
query19	200	205	158	158
query20	126	121	115	115
query21	211	131	109	109
query22	4376	4420	4311	4311
query23	34220	33495	33475	33475
query24	5850	2428	2460	2428
query25	490	474	408	408
query26	670	285	161	161
query27	1663	513	343	343
query28	2923	2469	2480	2469
query29	580	582	470	470
query30	228	186	166	166
query31	912	893	836	836
query32	77	71	64	64
query33	477	401	322	322
query34	790	889	521	521
query35	800	844	804	804
query36	981	1001	908	908
query37	125	110	85	85
query38	4263	4136	4039	4039
query39	1500	1439	1531	1439
query40	217	133	114	114
query41	60	68	58	58
query42	132	108	115	108
query43	526	542	479	479
query44	1412	845	811	811
query45	190	183	173	173
query46	925	1101	681	681
query47	1872	1855	1799	1799
query48	403	435	322	322
query49	720	523	441	441
query50	722	771	430	430
query51	4315	4307	4185	4185
query52	109	103	97	97
query53	240	274	190	190
query54	495	510	443	443
query55	90	87	80	80
query56	274	294	276	276
query57	1151	1167	1122	1122
query58	258	248	250	248
query59	2836	2790	2687	2687
query60	293	280	281	280
query61	130	118	125	118
query62	753	766	686	686
query63	248	198	192	192
query64	1647	1051	726	726
query65	3343	3263	3261	3261
query66	770	401	329	329
query67	15728	15270	15197	15197
query68	7006	911	505	505
query69	544	304	279	279
query70	1179	1134	1122	1122
query71	498	304	282	282
query72	5891	3755	3514	3514
query73	1295	773	356	356
query74	8950	9192	8927	8927
query75	3746	3141	2759	2759
query76	4206	1188	800	800
query77	632	373	292	292
query78	10008	10091	9390	9390
query79	2148	849	597	597
query80	683	545	460	460
query81	499	276	244	244
query82	659	126	100	100
query83	293	176	157	157
query84	282	95	78	78
query85	805	366	340	340
query86	357	312	282	282
query87	4380	4414	4497	4414
query88	2895	2229	2200	2200
query89	411	311	284	284
query90	1927	208	192	192
query91	135	143	107	107
query92	78	67	58	58
query93	1278	1070	583	583
query94	686	420	306	306
query95	349	272	255	255
query96	487	576	271	271
query97	3300	3392	3225	3225
query98	246	221	210	210
query99	1402	1436	1266	1266
Total cold run time: 294765 ms
Total hot run time: 191817 ms

doris-robot · 2025-02-25T15:56:50Z

ClickBench: Total hot run time: 31.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 082cc4b2b4359667cbe59d64b8c65b6d7c1e078f, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.23	0.07	0.06
query4	1.61	0.10	0.10
query5	0.55	0.54	0.55
query6	1.17	0.71	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.54	0.52
query10	0.58	0.59	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.78	2.72	2.65
query15	0.92	0.84	0.85
query16	0.37	0.37	0.38
query17	1.03	1.04	1.02
query18	0.21	0.19	0.19
query19	1.87	1.77	1.96
query20	0.02	0.02	0.02
query21	15.35	0.89	0.55
query22	0.77	1.21	0.65
query23	14.93	1.36	0.66
query24	7.29	1.34	1.12
query25	0.49	0.25	0.14
query26	0.70	0.18	0.15
query27	0.05	0.06	0.05
query28	9.19	0.81	0.42
query29	12.60	3.92	3.25
query30	0.25	0.10	0.06
query31	2.81	0.59	0.37
query32	3.22	0.55	0.46
query33	3.06	3.08	3.10
query34	15.81	5.17	4.54
query35	4.54	4.56	4.56
query36	0.67	0.49	0.51
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.21 s
Total hot run time: 31.16 s

Yukang-Lian · 2025-02-26T02:42:56Z

run buildall

doris-robot · 2025-02-26T03:20:26Z

TPC-H: Total hot run time: 31594 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 082cc4b2b4359667cbe59d64b8c65b6d7c1e078f, data reload: false

------ Round 1 ----------------------------------
q1	17628	5265	5107	5107
q2	2044	299	171	171
q3	10397	1223	756	756
q4	10208	1014	545	545
q5	7520	2278	2334	2278
q6	192	166	134	134
q7	897	760	601	601
q8	9297	1281	1108	1108
q9	4872	4755	4611	4611
q10	6814	2309	1912	1912
q11	480	284	261	261
q12	340	349	218	218
q13	17770	3663	3142	3142
q14	231	222	213	213
q15	509	470	451	451
q16	616	610	604	604
q17	579	855	342	342
q18	6937	6123	6263	6123
q19	1215	957	561	561
q20	334	321	192	192
q21	2819	2180	1964	1964
q22	371	334	300	300
Total cold run time: 102070 ms
Total hot run time: 31594 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5152	5220	5154	5154
q2	242	330	235	235
q3	2183	2671	2329	2329
q4	1465	1803	1393	1393
q5	4329	4171	4119	4119
q6	214	167	126	126
q7	1868	1830	1686	1686
q8	2592	2546	2589	2546
q9	7221	7217	7214	7214
q10	2998	3214	2743	2743
q11	576	533	488	488
q12	684	806	645	645
q13	3506	3788	3284	3284
q14	289	305	272	272
q15	507	462	464	462
q16	666	698	628	628
q17	1156	1592	1334	1334
q18	7612	7224	7313	7224
q19	831	878	1042	878
q20	1989	2043	1916	1916
q21	5410	4972	4901	4901
q22	606	586	554	554
Total cold run time: 52096 ms
Total hot run time: 50131 ms

doris-robot · 2025-02-26T03:32:35Z

TPC-DS: Total hot run time: 190623 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 082cc4b2b4359667cbe59d64b8c65b6d7c1e078f, data reload: false

query1	1332	982	955	955
query2	6309	1970	1923	1923
query3	11028	4441	4346	4346
query4	54065	25483	23299	23299
query5	5247	519	490	490
query6	369	214	183	183
query7	5079	502	292	292
query8	330	244	237	237
query9	6443	2629	2629	2629
query10	412	307	272	272
query11	15541	15050	14886	14886
query12	157	103	107	103
query13	1130	506	381	381
query14	10615	6307	6854	6307
query15	218	211	189	189
query16	7047	673	504	504
query17	1092	740	579	579
query18	1514	427	364	364
query19	194	187	161	161
query20	131	128	129	128
query21	212	122	110	110
query22	4229	4253	4129	4129
query23	33925	33509	33272	33272
query24	5681	2425	2425	2425
query25	468	466	404	404
query26	730	270	157	157
query27	1845	529	330	330
query28	2910	2460	2412	2412
query29	587	578	426	426
query30	215	193	153	153
query31	898	883	804	804
query32	72	64	61	61
query33	465	349	311	311
query34	745	871	515	515
query35	831	843	773	773
query36	956	1024	898	898
query37	113	92	72	72
query38	4108	4184	4155	4155
query39	1558	1483	1463	1463
query40	208	124	106	106
query41	54	51	51	51
query42	125	106	101	101
query43	514	527	493	493
query44	1334	862	830	830
query45	184	182	173	173
query46	887	1046	654	654
query47	1834	1899	1792	1792
query48	398	419	320	320
query49	694	510	430	430
query50	719	735	431	431
query51	4369	4262	4229	4229
query52	111	110	97	97
query53	231	250	186	186
query54	500	495	422	422
query55	78	79	80	79
query56	268	265	262	262
query57	1162	1192	1119	1119
query58	267	235	251	235
query59	2924	3042	2887	2887
query60	312	287	281	281
query61	147	152	142	142
query62	745	756	711	711
query63	229	202	197	197
query64	1568	1166	691	691
query65	3277	3267	3251	3251
query66	842	417	304	304
query67	15898	15627	15253	15253
query68	7035	876	508	508
query69	552	308	258	258
query70	1221	1122	1121	1121
query71	522	350	257	257
query72	5806	3558	3737	3558
query73	1263	732	351	351
query74	9308	8893	8784	8784
query75	3687	3196	2755	2755
query76	4335	1189	738	738
query77	601	364	292	292
query78	10058	10164	9394	9394
query79	3132	846	591	591
query80	698	519	446	446
query81	500	282	235	235
query82	653	129	94	94
query83	295	169	154	154
query84	291	88	67	67
query85	790	346	320	320
query86	380	306	279	279
query87	4639	4618	4325	4325
query88	2939	2221	2212	2212
query89	418	311	286	286
query90	1967	260	188	188
query91	135	137	108	108
query92	71	59	57	57
query93	1260	1035	599	599
query94	731	444	327	327
query95	370	275	264	264
query96	502	583	270	270
query97	3523	3550	3420	3420
query98	246	231	216	216
query99	1472	1444	1324	1324
Total cold run time: 298080 ms
Total hot run time: 190623 ms

doris-robot · 2025-02-26T03:38:01Z

ClickBench: Total hot run time: 30.68 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 082cc4b2b4359667cbe59d64b8c65b6d7c1e078f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.56	0.55	0.56
query6	1.18	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.61	0.52	0.53
query10	0.57	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.62	0.60	0.59
query14	2.79	2.83	2.67
query15	0.91	0.86	0.85
query16	0.37	0.37	0.38
query17	0.99	1.02	1.07
query18	0.21	0.19	0.19
query19	1.94	1.89	1.89
query20	0.01	0.01	0.01
query21	15.36	0.94	0.54
query22	0.76	1.19	0.66
query23	14.93	1.40	0.58
query24	6.66	1.79	0.75
query25	0.48	0.29	0.07
query26	0.58	0.15	0.14
query27	0.05	0.04	0.04
query28	9.82	0.86	0.41
query29	12.52	4.06	3.38
query30	0.24	0.09	0.06
query31	2.82	0.59	0.38
query32	3.23	0.56	0.46
query33	2.98	3.04	3.01
query34	15.65	5.11	4.51
query35	4.52	4.53	4.47
query36	0.66	0.51	0.47
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.18	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.84 s
Total hot run time: 30.68 s

doris-robot · 2025-02-26T03:54:29Z

TeamCity be ut coverage result:
Function Coverage: 44.76% (11932/26658)
Line Coverage: 34.21% (99881/291964)
Region Coverage: 33.36% (51106/153205)
Branch Coverage: 28.92% (25669/88770)
Coverage Report: http://coverage.selectdb-in.cc/coverage/082cc4b2b4359667cbe59d64b8c65b6d7c1e078f_082cc4b2b4359667cbe59d64b8c65b6d7c1e078f/report/index.html
Increment Report: http://coverage.selectdb-in.cc/coverage/082cc4b2b4359667cbe59d64b8c65b6d7c1e078f_082cc4b2b4359667cbe59d64b8c65b6d7c1e078f/increment_report/index.html

Yukang-Lian · 2025-02-26T07:17:29Z

run cloud_p0

github-actions · 2025-02-26T08:22:17Z

PR approved by anyone and no changes requested.

github-actions · 2025-02-26T09:46:23Z

PR approved by at least one committer and no changes requested.

The cold compaction did not set the compaction policy in the initial phase, causing a null pointer exception when logging later. This issue occurs infrequently; if the tablet has previously done any compaction or calculated a compaction score, the compaction policy will be set automatically, allowing cold compaction to run normally. However, if a newly created tablet undergoes cold compaction immediately, this problem will arise. This PR fixes the issue.

…ion #48329 (#48376) Cherry-picked from #48329 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

…48329) The cold compaction did not set the compaction policy in the initial phase, causing a null pointer exception when logging later. This issue occurs infrequently; if the tablet has previously done any compaction or calculated a compaction score, the compaction policy will be set automatically, allowing cold compaction to run normally. However, if a newly created tablet undergoes cold compaction immediately, this problem will arise. This PR fixes the issue.

…ecuting cold compaction (#48329)" (#48886)

…48329) The cold compaction did not set the compaction policy in the initial phase, causing a null pointer exception when logging later. This issue occurs infrequently; if the tablet has previously done any compaction or calculated a compaction score, the compaction policy will be set automatically, allowing cold compaction to run normally. However, if a newly created tablet undergoes cold compaction immediately, this problem will arise. This PR fixes the issue.

1

082cc4b

Hastyshell approved these changes Feb 26, 2025

View reviewed changes

github-actions bot added the reviewed label Feb 26, 2025

Hastyshell approved these changes Feb 26, 2025

View reviewed changes

gavinchou approved these changes Feb 26, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 26, 2025

gavinchou approved these changes Feb 26, 2025

View reviewed changes

gavinchou added p0_c dev/3.0.x labels Feb 26, 2025

gavinchou merged commit a5fa85b into apache:master Feb 26, 2025
31 of 35 checks passed

github-actions bot mentioned this pull request Feb 26, 2025

branch-3.0: [Fix](compaction) Fix nullptr when executing cold compaction #48329 #48376

Merged

dataroaring pushed a commit that referenced this pull request Feb 27, 2025

branch-3.0: [Fix](compaction) Fix nullptr when executing cold compact…

c406aa9

…ion #48329 (#48376) Cherry-picked from #48329 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Feb 27, 2025

gavinchou added the dev/2.1.x label Mar 10, 2025

github-actions bot added the dev/2.1.x-conflict label Mar 10, 2025

Yukang-Lian mentioned this pull request Mar 10, 2025

[cherry-pick](branch-2.1) Pick "[Fix](compaction) Fix nullptr when executing cold compaction (#48329)" #48886

Merged

16 tasks

yiguolei pushed a commit that referenced this pull request Mar 20, 2025

[cherry-pick](branch-2.1) Pick "[Fix](compaction) Fix nullptr when ex…

5ecfee5

…ecuting cold compaction (#48329)" (#48886)

yiguolei added dev/2.1.9-merged and removed dev/2.1.x dev/2.1.x-conflict labels Mar 20, 2025

gavinchou mentioned this pull request Apr 23, 2025

Release Notes 3.0.5 #50333

Closed

[Fix](compaction) Fix nullptr when executing cold compaction #48329

[Fix](compaction) Fix nullptr when executing cold compaction #48329

Uh oh!

Conversation

Yukang-Lian commented Feb 25, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 25, 2025

Uh oh!

Yukang-Lian commented Feb 25, 2025

Uh oh!

doris-robot commented Feb 25, 2025

Uh oh!

doris-robot commented Feb 25, 2025

Uh oh!

doris-robot commented Feb 25, 2025

Uh oh!

Yukang-Lian commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

Yukang-Lian commented Feb 26, 2025

Uh oh!

github-actions bot commented Feb 26, 2025

Uh oh!

github-actions bot commented Feb 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants