Skip to content

branch-3.1: [fix](inverted index) catch IO exception to avoid coredump in inverted index string reader #51844 #52867

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 2 commits into from
Jul 8, 2025
Merged

branch-3.1: [fix](inverted index) catch IO exception to avoid coredump in inverted index string reader #51844 #52867

airborne12 merged 2 commits into from
Jul 8, 2025

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Jul 7, 2025

cherry pick from #51844

@airborne12
[fix](inverted index) catch IO exception to avoid coredump in inverte…
df5311f 
…d index string reader (apache#51844)

Problem Summary:
This PR adds error handling around CLucene interactions in the string
inverted index reader to prevent core dumps on IO failures and
introduces.
@airborne12 airborne12 requested a review from morrySnow as a code owner July 7, 2025 06:28
@airborne12
Copy link
Member Author

airborne12 commented Jul 7, 2025

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jul 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12
Copy link
Member Author

airborne12 commented Jul 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 7, 2025

BE UT Coverage Report

Increment line coverage 86.11% (31/36) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.22% (12451/27533)
Line Coverage 36.18% (110378/305112)
Region Coverage 35.34% (57273/162083)
Branch Coverage 32.42% (31063/95800)

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-H: Total hot run time: 40018 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8816cfdc917e7e2f454397897cce3633a911b45b, data reload: false

------ Round 1 ----------------------------------
q1	17600	6857	6723	6723
q2	2064	194	175	175
q3	10650	1136	1194	1136
q4	10534	704	677	677
q5	7769	2911	2807	2807
q6	222	137	136	136
q7	998	630	599	599
q8	9355	1982	1985	1982
q9	6658	6427	6378	6378
q10	7054	2329	2317	2317
q11	461	268	263	263
q12	406	214	214	214
q13	17771	2985	2961	2961
q14	226	205	210	205
q15	503	463	462	462
q16	484	381	374	374
q17	995	624	560	560
q18	7277	6702	6678	6678
q19	1321	1105	1099	1099
q20	495	202	198	198
q21	4232	3097	3150	3097
q22	1097	977	985	977
Total cold run time: 108172 ms
Total hot run time: 40018 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6627	6575	6764	6575
q2	324	228	237	228
q3	2933	2949	2942	2942
q4	2056	1815	1810	1810
q5	5722	5718	5708	5708
q6	211	126	129	126
q7	2212	1775	1807	1775
q8	3367	3526	3557	3526
q9	8823	8893	8876	8876
q10	3556	3556	3518	3518
q11	603	499	491	491
q12	798	648	624	624
q13	6450	3232	3138	3138
q14	317	283	267	267
q15	519	480	466	466
q16	507	438	434	434
q17	1854	1629	1632	1629
q18	8248	7864	7905	7864
q19	1726	1440	1596	1440
q20	2064	1829	1846	1829
q21	5295	5043	5030	5030
q22	1136	1028	1004	1004
Total cold run time: 65348 ms
Total hot run time: 59300 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-H: Total hot run time: 39699 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8816cfdc917e7e2f454397897cce3633a911b45b, data reload: false

------ Round 1 ----------------------------------
q1	17613	6761	6563	6563
q2	2073	219	174	174
q3	10486	1121	1145	1121
q4	10208	735	737	735
q5	7730	2861	2831	2831
q6	211	134	132	132
q7	1001	614	594	594
q8	9346	1921	1971	1921
q9	6622	6391	6395	6391
q10	7010	2239	2323	2239
q11	473	265	257	257
q12	421	211	207	207
q13	17789	2990	2982	2982
q14	236	204	207	204
q15	515	471	483	471
q16	470	387	374	374
q17	965	586	582	582
q18	7120	6777	6656	6656
q19	1315	1023	906	906
q20	471	199	204	199
q21	3857	3165	3172	3165
q22	1115	998	995	995
Total cold run time: 107047 ms
Total hot run time: 39699 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6626	6640	6547	6547
q2	330	234	238	234
q3	2891	2824	2958	2824
q4	2024	1751	1822	1751
q5	5807	5806	5818	5806
q6	216	134	131	131
q7	2261	1889	1834	1834
q8	3451	3515	3538	3515
q9	8908	8925	8976	8925
q10	3610	3460	3527	3460
q11	588	519	483	483
q12	802	581	625	581
q13	8380	3175	3146	3146
q14	308	280	290	280
q15	513	459	470	459
q16	477	444	447	444
q17	1818	1601	1599	1599
q18	8314	7822	7600	7600
q19	1682	1556	1586	1556
q20	2033	1790	1804	1790
q21	5138	5083	4896	4896
q22	1128	1088	1043	1043
Total cold run time: 67305 ms
Total hot run time: 58904 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-DS: Total hot run time: 196376 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8816cfdc917e7e2f454397897cce3633a911b45b, data reload: false

query1	1284	916	912	912
query2	6328	1954	1871	1871
query3	10831	4322	4222	4222
query4	32803	23590	23488	23488
query5	3485	474	437	437
query6	277	177	178	177
query7	3993	315	328	315
query8	290	233	233	233
query9	9552	2587	2571	2571
query10	471	271	265	265
query11	18001	15046	15171	15046
query12	160	104	105	104
query13	1565	438	423	423
query14	9581	7073	6692	6692
query15	264	183	194	183
query16	8071	457	504	457
query17	1605	611	602	602
query18	2157	315	331	315
query19	242	164	168	164
query20	130	121	114	114
query21	212	108	112	108
query22	4727	4532	4632	4532
query23	35198	34079	33755	33755
query24	12483	2992	2900	2900
query25	694	430	433	430
query26	1760	174	178	174
query27	2837	350	352	350
query28	8099	2200	2227	2200
query29	1049	486	454	454
query30	264	160	167	160
query31	1076	824	831	824
query32	91	57	59	57
query33	784	296	306	296
query34	1158	512	525	512
query35	928	744	761	744
query36	1118	937	951	937
query37	261	70	67	67
query38	4132	3959	3970	3959
query39	1531	1468	1461	1461
query40	285	104	109	104
query41	48	47	55	47
query42	109	103	107	103
query43	535	474	474	474
query44	1288	825	818	818
query45	180	171	171	171
query46	1187	753	736	736
query47	2015	1920	1905	1905
query48	470	361	358	358
query49	1051	403	408	403
query50	847	419	427	419
query51	7508	7302	7259	7259
query52	106	89	92	89
query53	259	189	187	187
query54	1329	472	483	472
query55	84	79	83	79
query56	275	249	254	249
query57	1336	1213	1208	1208
query58	231	228	218	218
query59	3278	3127	3061	3061
query60	305	278	254	254
query61	125	116	110	110
query62	878	710	707	707
query63	225	190	207	190
query64	5485	710	665	665
query65	3376	3285	3224	3224
query66	1315	294	295	294
query67	15870	15542	15615	15542
query68	5526	590	561	561
query69	439	267	269	267
query70	1147	1083	1088	1083
query71	344	264	255	255
query72	6229	4120	3977	3977
query73	775	359	346	346
query74	10363	9229	8992	8992
query75	3365	2656	2652	2652
query76	3205	1133	1022	1022
query77	380	263	274	263
query78	10675	9663	9647	9647
query79	2449	603	621	603
query80	1148	436	428	428
query81	567	222	224	222
query82	728	89	88	88
query83	238	148	140	140
query84	231	81	84	81
query85	1483	307	292	292
query86	448	298	295	295
query87	4431	4305	4220	4220
query88	4229	2397	2372	2372
query89	417	308	302	302
query90	2069	183	185	183
query91	141	113	109	109
query92	71	51	53	51
query93	2203	559	557	557
query94	900	312	302	302
query95	354	255	260	255
query96	612	285	285	285
query97	3306	3169	3237	3169
query98	230	218	202	202
query99	1473	1298	1298	1298
Total cold run time: 309668 ms
Total hot run time: 196376 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

ClickBench: Total hot run time: 29.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8816cfdc917e7e2f454397897cce3633a911b45b, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.11
query5	0.53	0.51	0.53
query6	1.15	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.57	0.56	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.59	0.60
query14	0.78	0.78	0.78
query15	0.84	0.83	0.83
query16	0.39	0.38	0.38
query17	1.10	1.08	1.06
query18	0.23	0.22	0.22
query19	1.92	1.89	1.89
query20	0.01	0.01	0.01
query21	15.39	0.60	0.60
query22	2.18	1.90	1.26
query23	16.97	0.86	0.84
query24	2.63	0.71	2.00
query25	0.21	0.11	0.07
query26	0.53	0.13	0.13
query27	0.06	0.04	0.04
query28	10.26	0.49	0.46
query29	12.63	3.21	3.22
query30	0.24	0.06	0.06
query31	2.86	0.37	0.39
query32	3.26	0.46	0.46
query33	2.95	2.97	3.08
query34	17.09	4.49	4.52
query35	4.55	4.51	4.50
query36	0.66	0.47	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.92 s
Total hot run time: 29.48 s

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-DS: Total hot run time: 196464 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8816cfdc917e7e2f454397897cce3633a911b45b, data reload: false

query1	1283	930	907	907
query2	6365	1951	1816	1816
query3	10929	4442	4511	4442
query4	33223	23284	23546	23284
query5	3587	455	449	449
query6	264	188	176	176
query7	3994	317	324	317
query8	295	241	233	233
query9	9494	2576	2559	2559
query10	492	268	249	249
query11	18105	15580	15168	15168
query12	158	107	103	103
query13	1564	439	414	414
query14	8698	6472	6912	6472
query15	221	195	190	190
query16	7939	495	506	495
query17	1584	595	619	595
query18	2106	319	342	319
query19	213	164	169	164
query20	130	114	113	113
query21	204	113	108	108
query22	4954	4578	4430	4430
query23	34920	34511	34710	34511
query24	11448	2927	3020	2927
query25	633	453	434	434
query26	1000	183	177	177
query27	2784	354	368	354
query28	7587	2181	2139	2139
query29	717	464	449	449
query30	277	166	167	166
query31	1042	827	850	827
query32	98	57	58	57
query33	805	293	299	293
query34	1046	514	525	514
query35	872	734	720	720
query36	1127	958	953	953
query37	111	68	69	68
query38	4066	3932	3916	3916
query39	1502	1461	1449	1449
query40	201	103	104	103
query41	50	51	48	48
query42	119	107	108	107
query43	516	480	478	478
query44	1282	823	834	823
query45	187	170	167	167
query46	1154	749	710	710
query47	2029	1898	1894	1894
query48	437	340	341	340
query49	1011	391	389	389
query50	853	438	428	428
query51	7329	7332	7377	7332
query52	112	94	92	92
query53	267	183	183	183
query54	1205	474	497	474
query55	80	77	78	77
query56	266	250	237	237
query57	1347	1220	1215	1215
query58	231	210	214	210
query59	3252	2980	3015	2980
query60	281	263	252	252
query61	112	112	111	111
query62	866	705	686	686
query63	225	188	199	188
query64	3877	659	653	653
query65	3347	3301	3256	3256
query66	887	293	301	293
query67	16154	15585	15487	15487
query68	4416	597	584	584
query69	445	260	263	260
query70	1154	1113	1111	1111
query71	341	264	262	262
query72	6161	4110	3978	3978
query73	753	343	353	343
query74	10562	8902	9183	8902
query75	3375	2620	2657	2620
query76	2762	1178	1080	1080
query77	367	270	269	269
query78	10647	9579	9559	9559
query79	2309	612	606	606
query80	1208	433	420	420
query81	556	226	221	221
query82	909	89	85	85
query83	222	142	143	142
query84	241	77	75	75
query85	1311	312	293	293
query86	461	295	296	295
query87	4340	4201	4218	4201
query88	4143	2406	2380	2380
query89	415	283	295	283
query90	1887	182	185	182
query91	149	105	109	105
query92	66	49	53	49
query93	2443	551	556	551
query94	832	290	304	290
query95	355	251	254	251
query96	610	276	287	276
query97	3268	3145	3212	3145
query98	225	202	195	195
query99	1529	1293	1281	1281
Total cold run time: 301976 ms
Total hot run time: 196464 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

ClickBench: Total hot run time: 29.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8816cfdc917e7e2f454397897cce3633a911b45b, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.11
query5	0.52	0.52	0.49
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.49
query10	0.56	0.54	0.57
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	0.76	0.79	0.79
query15	0.84	0.82	0.81
query16	0.39	0.38	0.38
query17	1.07	1.07	1.05
query18	0.23	0.21	0.22
query19	1.88	1.78	1.84
query20	0.01	0.02	0.01
query21	15.41	0.59	0.60
query22	2.71	1.62	1.61
query23	17.07	1.03	0.86
query24	3.02	0.44	1.58
query25	0.26	0.21	0.04
query26	0.33	0.14	0.13
query27	0.05	0.05	0.06
query28	10.50	0.52	0.48
query29	12.59	3.20	3.19
query30	0.25	0.06	0.07
query31	2.85	0.37	0.37
query32	3.26	0.46	0.45
query33	2.99	3.01	2.96
query34	17.20	4.52	4.49
query35	4.49	4.58	4.57
query36	0.65	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.04
Total cold run time: 104.99 s
Total hot run time: 29.45 s

@airborne12 airborne12 merged commit d854139 into apache:branch-3.1 Jul 8, 2025
19 of 21 checks passed
@airborne12 airborne12 deleted the pick_51844_to_origin_branch-3.1 branch July 8, 2025 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@airborne12 @Thearas @doris-robot @morrySnow