Skip to content

Conversation

@AshinGau
Copy link
Member

@AshinGau AshinGau commented May 14, 2024

Proposed changes

Prevent core dump if read position delete file failed:

hdfsOpenFile(/usr/hive/warehouse/hadoop_catalog/multi_catalog/iceberg_position_gen_data/data/00000-17-9c155cb4-dc13-459a-92ff-bbc959cd29cb-00001.orc): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error:
NoRouteToHostException: No route to hostjava.net.NoRouteToHostException: No Route to Host from  VM-0-105-ubuntu/127.0.1.1 to 172.21.16.47:4007 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see:  http://wiki.apache.org/hadoop/NoRouteToHost
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

5# doris::vectorized::OrcReader::set_fill_columns(std::unordered_map, std::allocator >, std::tuple, std::allocator >, doris::SlotDescriptor const*>, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, std::tuple, std::allocator >, doris::SlotDescriptor const*> > > > const&, std::unordered_map, std::allocator >, std::shared_ptr, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, std::shared_ptr > > > const&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/format/orc/vorc_reader.cpp:857
 6# doris::vectorized::IcebergOrcReader::_read_position_delete_file(doris::TFileRangeDesc const*, phmap::parallel_flat_hash_map, std::allocator >, std::unique_ptr >, std::default_delete > > >, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, std::unique_ptr >, std::default_delete > > > > >, 8ul, std::mutex>*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/format/table/iceberg_reader.cpp:629
 7# doris::vectorized::IcebergTableReader::_position_delete_base(std::vector > const&)::$_0::operator()[abi:cxx11]() const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/format/table/iceberg_reader.cpp:301

Should return the error status if initialize the position file reader failed.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@AshinGau
Copy link
Member Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 14, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-H: Total hot run time: 41939 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e107841ecaae694099ca23bf97303a78c53430db, data reload: false

------ Round 1 ----------------------------------
q1	17991	4373	4266	4266
q2	2021	193	191	191
q3	10536	1271	1201	1201
q4	10168	885	788	788
q5	7573	2710	2720	2710
q6	220	134	133	133
q7	1051	598	593	593
q8	9346	2166	2113	2113
q9	10210	7164	7255	7164
q10	9153	3967	3906	3906
q11	472	250	236	236
q12	485	219	228	219
q13	17924	3084	3145	3084
q14	264	222	215	215
q15	508	472	478	472
q16	559	398	402	398
q17	1017	650	711	650
q18	8583	7535	7714	7535
q19	4819	1562	1566	1562
q20	667	316	316	316
q21	5170	3991	3920	3920
q22	340	267	271	267
Total cold run time: 119077 ms
Total hot run time: 41939 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4330	4217	4198	4198
q2	358	267	270	267
q3	2962	2752	2729	2729
q4	1868	1611	1580	1580
q5	5318	5316	5309	5309
q6	210	120	124	120
q7	2258	1866	1899	1866
q8	3200	3344	3343	3343
q9	8998	8959	9020	8959
q10	3897	3670	3725	3670
q11	580	477	481	477
q12	746	627	596	596
q13	17460	2940	3020	2940
q14	297	249	268	249
q15	508	484	472	472
q16	477	417	409	409
q17	1772	1472	1462	1462
q18	7654	7448	7401	7401
q19	1659	1575	1543	1543
q20	1952	1758	1820	1758
q21	7376	4943	4785	4785
q22	540	471	500	471
Total cold run time: 74420 ms
Total hot run time: 54604 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.67% (8982/25180)
Line Coverage: 27.33% (74263/271747)
Region Coverage: 26.56% (38372/144455)
Branch Coverage: 23.39% (19574/83702)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e107841ecaae694099ca23bf97303a78c53430db_e107841ecaae694099ca23bf97303a78c53430db/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 187545 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e107841ecaae694099ca23bf97303a78c53430db, data reload: false

query1	901	377	349	349
query2	6463	2401	2341	2341
query3	6648	208	207	207
query4	25534	21502	21229	21229
query5	4122	432	422	422
query6	276	170	178	170
query7	4598	297	286	286
query8	244	192	195	192
query9	8582	2451	2432	2432
query10	441	250	255	250
query11	14744	14184	14269	14184
query12	144	95	89	89
query13	1640	387	388	387
query14	9826	7727	7361	7361
query15	232	168	195	168
query16	7363	272	260	260
query17	1743	559	545	545
query18	1315	276	276	276
query19	203	155	148	148
query20	95	86	88	86
query21	202	136	135	135
query22	5016	4855	4841	4841
query23	34056	33507	33496	33496
query24	10533	2878	2866	2866
query25	576	370	361	361
query26	702	160	157	157
query27	2227	323	339	323
query28	5971	2114	2069	2069
query29	846	619	603	603
query30	291	158	155	155
query31	1009	749	751	749
query32	90	51	54	51
query33	635	267	243	243
query34	877	495	489	489
query35	805	676	667	667
query36	1064	906	948	906
query37	101	65	67	65
query38	2906	2779	2769	2769
query39	1633	1602	1587	1587
query40	211	128	125	125
query41	41	38	37	37
query42	107	96	100	96
query43	579	537	547	537
query44	1125	727	743	727
query45	298	242	251	242
query46	1070	729	717	717
query47	1940	1893	1887	1887
query48	397	310	300	300
query49	952	398	409	398
query50	783	392	406	392
query51	6930	6768	6768	6768
query52	99	96	89	89
query53	352	289	286	286
query54	860	438	425	425
query55	77	74	74	74
query56	242	237	220	220
query57	1222	1148	1136	1136
query58	223	202	203	202
query59	3537	3231	3225	3225
query60	252	236	240	236
query61	117	87	85	85
query62	648	461	496	461
query63	313	290	282	282
query64	8536	7384	7417	7384
query65	3148	3084	3100	3084
query66	814	342	353	342
query67	15724	15192	15217	15192
query68	4522	543	537	537
query69	482	319	315	315
query70	1198	1145	1108	1108
query71	414	275	275	275
query72	8046	2597	2394	2394
query73	705	334	331	331
query74	6616	6407	6189	6189
query75	3353	2648	2553	2553
query76	2836	1039	995	995
query77	448	266	268	266
query78	10756	10048	10089	10048
query79	2041	523	527	523
query80	885	448	448	448
query81	508	219	220	219
query82	874	98	97	97
query83	235	165	174	165
query84	251	83	85	83
query85	1447	271	334	271
query86	469	304	322	304
query87	3302	3082	3038	3038
query88	4102	2461	2446	2446
query89	476	385	389	385
query90	1984	196	188	188
query91	125	96	97	96
query92	60	50	48	48
query93	2256	519	500	500
query94	1204	183	191	183
query95	408	307	301	301
query96	593	282	270	270
query97	3171	3013	2993	2993
query98	234	228	217	217
query99	1234	906	896	896
Total cold run time: 280486 ms
Total hot run time: 187545 ms

@yiguolei yiguolei merged commit 82596e6 into apache:master May 14, 2024
morningman pushed a commit that referenced this pull request May 22, 2024
…oid> (#34873)

Followup #34797
`static_cast<void>` has ignored the wrong status, some of them should make the query finished with error status, so replace `static_cast<void>`  with `RETURN_IF_ERROR`.

### Remaining Works
The following three scenarios need to be handled separately and cannot be simply replaced:
1. The outer function returns void;
2. Call status function inner constructors or destructors;
3. Call status function with best effort, and should ignore the wrong status.
yiguolei pushed a commit that referenced this pull request May 23, 2024
…oid> (#34873)

Followup #34797
`static_cast<void>` has ignored the wrong status, some of them should make the query finished with error status, so replace `static_cast<void>`  with `RETURN_IF_ERROR`.

The following three scenarios need to be handled separately and cannot be simply replaced:
1. The outer function returns void;
2. Call status function inner constructors or destructors;
3. Call status function with best effort, and should ignore the wrong status.
dataroaring pushed a commit that referenced this pull request May 26, 2024
…oid> (#34873)

Followup #34797
`static_cast<void>` has ignored the wrong status, some of them should make the query finished with error status, so replace `static_cast<void>`  with `RETURN_IF_ERROR`.

### Remaining Works
The following three scenarios need to be handled separately and cannot be simply replaced:
1. The outer function returns void;
2. Call status function inner constructors or destructors;
3. Call status function with best effort, and should ignore the wrong status.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.3-merged dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants