Skip to content

Conversation

@mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Mar 21, 2024

Proposed changes

*** Check failure stack trace: ***
F20240321 11:59:37.800169 12514 column.h:607] Column Nullable(Int64) is not a contiguous block of memory
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:421
 1# 0x00007F6B6DC38090 in /lib/x86_64-linux-gnu/libc.so.6
 2# raise at ../sysdeps/unix/sysv/linux/raise.c:51
 3# abort at /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81
 4# 0x000055623D6DBC5D in /root/deploy/be/lib/doris_be
 5# 0x000055623D6CE2FA in /root/deploy/be/lib/doris_be
 6# google::LogMessage::SendToLog() in /root/deploy/be/lib/doris_be
 7# google::LogMessage::Flush() in /root/deploy/be/lib/doris_be
 8# google::LogMessageFatal::~LogMessageFatal() in /root/deploy/be/lib/doris_be
 9# doris::vectorized::IColumn::get_raw_data() const in /root/deploy/be/lib/doris_be
10# void doris::vectorized::MethodKeysFixed >, false>::pack_fixeds(unsigned long, std::vector > const&, std::vector > const&, std::vector >&) at /root/doris/be/src/vec/common/hash_table/hash_map_context.h:404
11# doris::vectorized::MethodKeysFixed >, false>::init_serialized_keys(std::vector > const&, unsigned long, unsigned char const*, bool, bool, unsigned int) at /root/doris/be/src/vec/common/hash_table/hash_map_context.h:469
12# doris::Status doris::vectorized::ProcessHashTableBuild >, false>, doris::vectorized::HashJoinNode>::run<1, false, false, true>(doris::vectorized::MethodKeysFixed >, false>&, doris::vectorized::PODArray, 15ul, 16ul> const*, bool*) at /root/doris/be/src/vec/exec/join/vhash_join_node.h:129
13# doris::Status doris::vectorized::HashJoinNode::_process_build_block(doris::RuntimeState*, doris::vectorized::Block&)::$_1::operator() >, false>&, std::integral_constant&, std::integral_constant, std::integral_constant, std::integral_constant >(doris::vectorized::MethodKeysFixed >, false>&, std::integral_constant&, std::integral_constant, std::integral_constant, std::integral_constant) const at /root/doris/be/src/vec/exec/join/vhash_join_node.cpp:988
14# doris::Status std::__invoke_impl, doris::vectorized::MethodKeysFixed >, false>&, std::integral_constant&, std::integral_constant, std::integral_constant, std::integral_constant >(std::__invoke_other, doris::vectorized::Overload&&, doris::vectorized::MethodKeysFixed >, false>&, std::integral_constant&, std::integral_constant&&, std::integral_constant&&, std::integral_constant&&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
15# std::__invoke_result, doris::vectorized::MethodKeysFixed >, false>&, std::integral_constant&, std::integral_constant, std::integral_constant, std::integral_constant >::type std::__invoke, doris::vectorized::MethodKeysFixed >, false>&, std::integral_constant&, std::integral_constant, std::integral_constant, std::integral_constant >(doris::vectorized::Overload&&, doris::vectorized::MethodKeysFixed >, false>&, std::integral_constant&, std::integral_constant&&, std::integral_constant&&, std::integral_constant&&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96
16# _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIN5doris6StatusEEEONS4_10vectorized8OverloadIJZNS7_12HashJoinNode20_process_build_blockEPNS4_12RuntimeStateERNS7_5BlockEE3$_0ZNS9_20_process_build_blockESB_SD_E3$_1EEERSt7variantIJSt9monostateNS7_16MethodSerializedINS4_13JoinHashTableINS4_9StringRefE11DefaultHashISM_vEEEEENS7_15MethodOneNumberIhNSL_Ih9HashCRC32IhEEEEENSR_ItNSL_ItSS_ItEEEEENSR_IjNSL_IjSS_IjEEEEENSR_ImNSL_ImSS_ImEEEEENSR_INS7_7UInt128ENSL_IS15_SS_IS15_EEEEENSR_INS7_7UInt256ENSL_IS19_SS_IS19_EEEEENS7_15MethodKeysFixedIS13_Lb1EEENS1D_IS13_Lb0EEENS1D_IS17_Lb1EEENS1D_IS17_Lb0EEENS1D_IS1B_Lb1EEENS1D_IS1B_Lb0EEENS1D_INSL_INS7_7UInt136ESS_IS1K_EEELb1EEENS1D_IS1M_Lb0EEEEERSI_IJSt17integral_constantINS4_7TJoinOp4typeELS1T_0EES1R_IS1T_LS1T_2EES1R_IS1T_LS1T_8EES1R_IS1T_LS1T_1EES1R_IS1T_LS1T_4EES1R_IS1T_LS1T_3EES1R_IS1T_LS1T_5EES1R_IS1T_LS1T_7EES1R_IS1T_LS1T_9EES1R_IS1T_LS1T_10EES1R_IS1T_LS1T_11EEEEOSI_IJS1R_IbLb0EES1R_IbLb1EEEES2A_S2A_EJEEESt16integer_sequenceImJLm9ELm3ELm0ELm0ELm1EEEE14__visit_invokeESH_S1Q_S26_S2A_S2A_S2A_ at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013
17# decltype(auto) std::__do_visit, doris::vectorized::Overload, std::variant > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false> >&, std::variant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant >&, std::variant, std::integral_constant >, std::variant, std::integral_constant >, std::variant, std::integral_constant > >(doris::vectorized::Overload&&, std::variant > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false> >&, std::variant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant >&, std::variant, std::integral_constant >&&, std::variant, std::integral_constant >&&, std::variant, std::integral_constant >&&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1714
18# decltype(auto) std::visit, std::variant > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false> >&, std::variant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant >&, std::variant, std::integral_constant >, std::variant, std::integral_constant >, std::variant, std::integral_constant > >(doris::vectorized::Overload&&, std::variant > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodOneNumber > >, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false>, doris::vectorized::MethodKeysFixed >, true>, doris::vectorized::MethodKeysFixed >, false> >&, std::variant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant, std::integral_constant >&, std::variant, std::integral_constant >&&, std::variant, std::integral_constant >&&, std::variant, std::integral_constant >&&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1769
19# doris::vectorized::HashJoinNode::_process_build_block(doris::RuntimeState*, doris::vectorized::Block&) at /root/doris/be/src/vec/exec/join/vhash_join_node.cpp:971
20# doris::vectorized::HashJoinNode::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /root/doris/be/src/vec/exec/join/vhash_join_node.cpp:759
21# doris::pipeline::StreamingOperator::sink(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState) at /root/doris/be/src/pipeline/exec/operator.h:334
22# doris::pipeline::PipelineTask::execute(bool*) at /root/doris/be/src/pipeline/pipeline_task.cpp:304
23# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/doris/be/src/pipeline/task_scheduler.cpp:339
24# doris::pipeline::TaskScheduler::start()::$_0::operator()() const at /root/doris/be/src/pipeline/task_scheduler.cpp:218
25# void std::__invoke_impl(std::__invoke_other, doris::pipeline::TaskScheduler::start()::$_0&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
26# std::enable_if, void>::type std::__invoke_r(doris::pipeline::TaskScheduler::start()::$_0&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:117
27# std::_Function_handler::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
28# std::function::operator()() const at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
29# doris::FunctionRunnable::run() at /root/doris/be/src/util/threadpool.cpp:48
30# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:543
31# void std::__invoke_impl(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74
32# std::__invoke_result::type std::__invoke(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96
33# void std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420
34# void std::_Bind::operator()<, void>() at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503
35# void std::__invoke_impl&>(std::__invoke_other, std::_Bind&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
36# std::enable_if&>, void>::type std::__invoke_r&>(std::_Bind&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:117
37# std::_Function_handler >::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
38# std::function::operator()() const at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
39# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:498
40# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
41# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mrhhsg
Copy link
Member Author

mrhhsg commented Mar 21, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

return;
}

std::vector<vectorized::DataTypePtr> data_types;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comment for the change

@mrhhsg
Copy link
Member Author

mrhhsg commented Mar 21, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 21, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37659 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86d5fc4f5eb7cf77fa5708189ef09fac92ba975c, data reload: false

------ Round 1 ----------------------------------
q1	17639	4240	4128	4128
q2	2034	157	146	146
q3	10622	1078	1190	1078
q4	10643	785	800	785
q5	7778	3019	2978	2978
q6	206	123	123	123
q7	1027	590	560	560
q8	9463	2015	2029	2015
q9	7294	6514	6451	6451
q10	8384	3288	3516	3288
q11	424	224	219	219
q12	363	195	194	194
q13	17802	2864	2871	2864
q14	244	204	199	199
q15	516	457	467	457
q16	459	355	358	355
q17	967	503	554	503
q18	7253	6546	6380	6380
q19	3554	1491	1415	1415
q20	532	264	266	264
q21	3544	3021	2958	2958
q22	355	299	309	299
Total cold run time: 111103 ms
Total hot run time: 37659 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4125	4103	4054	4054
q2	327	230	221	221
q3	3001	2868	2850	2850
q4	1869	1569	1553	1553
q5	5228	5235	5253	5235
q6	194	119	118	118
q7	2263	1875	1888	1875
q8	3158	3317	3274	3274
q9	8550	8543	8559	8543
q10	3718	3670	3649	3649
q11	558	438	440	438
q12	743	540	542	540
q13	16916	2840	2847	2840
q14	279	250	252	250
q15	476	443	455	443
q16	465	424	410	410
q17	1738	1497	1476	1476
q18	7635	7115	7252	7115
q19	1630	1514	1527	1514
q20	1916	1705	1726	1705
q21	4909	4852	4738	4738
q22	521	479	466	466
Total cold run time: 70219 ms
Total hot run time: 53307 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.27% (8730/24750)
Line Coverage: 27.09% (71499/263931)
Region Coverage: 26.33% (37094/140891)
Branch Coverage: 23.24% (18972/81636)
Coverage Report: http://coverage.selectdb-in.cc/coverage/86d5fc4f5eb7cf77fa5708189ef09fac92ba975c_86d5fc4f5eb7cf77fa5708189ef09fac92ba975c/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 180634 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86d5fc4f5eb7cf77fa5708189ef09fac92ba975c, data reload: false

query1	917	357	360	357
query2	7752	2099	1990	1990
query3	6715	210	208	208
query4	25442	20743	20701	20701
query5	4373	444	410	410
query6	275	177	171	171
query7	4625	286	278	278
query8	223	162	164	162
query9	9952	2293	2277	2277
query10	466	236	248	236
query11	17130	14201	14151	14151
query12	145	89	89	89
query13	1636	425	417	417
query14	11324	10842	10185	10185
query15	258	204	193	193
query16	8067	264	260	260
query17	1925	574	541	541
query18	2075	293	286	286
query19	215	162	158	158
query20	92	85	93	85
query21	199	133	124	124
query22	4649	4437	4438	4437
query23	31723	31081	31298	31081
query24	12203	2899	2788	2788
query25	673	372	372	372
query26	1876	151	145	145
query27	3267	350	340	340
query28	8156	1875	1858	1858
query29	1051	634	595	595
query30	309	144	140	140
query31	939	728	740	728
query32	101	60	56	56
query33	765	251	245	245
query34	1128	483	481	481
query35	817	641	610	610
query36	979	888	887	887
query37	281	80	78	78
query38	3550	3450	3397	3397
query39	1439	1364	1373	1364
query40	290	124	107	107
query41	48	43	54	43
query42	109	93	94	93
query43	500	454	465	454
query44	1329	732	719	719
query45	271	252	265	252
query46	1073	711	695	695
query47	1668	1577	1581	1577
query48	463	341	352	341
query49	1229	333	319	319
query50	737	374	359	359
query51	6765	6701	6638	6638
query52	107	97	90	90
query53	337	278	268	268
query54	326	253	244	244
query55	84	79	76	76
query56	240	232	232	232
query57	1087	996	997	996
query58	221	202	217	202
query59	2719	2681	2630	2630
query60	274	246	258	246
query61	101	98	112	98
query62	601	429	388	388
query63	300	269	279	269
query64	6346	3854	3812	3812
query65	3136	3024	3019	3019
query66	1414	365	354	354
query67	14842	14234	14162	14162
query68	6999	524	534	524
query69	599	383	382	382
query70	1209	1179	1157	1157
query71	531	284	281	281
query72	6598	2805	2659	2659
query73	739	326	320	320
query74	7322	6561	6568	6561
query75	4300	2845	2871	2845
query76	4859	890	933	890
query77	639	263	268	263
query78	10334	9663	9494	9494
query79	12337	532	507	507
query80	1862	380	380	380
query81	499	208	214	208
query82	773	207	185	185
query83	215	145	143	143
query84	283	77	82	77
query85	1484	330	318	318
query86	436	306	283	283
query87	3761	3568	3518	3518
query88	5555	2288	2282	2282
query89	510	358	365	358
query90	2005	173	171	171
query91	163	140	142	140
query92	62	47	48	47
query93	7501	513	493	493
query94	1223	178	170	170
query95	426	325	323	323
query96	618	267	269	267
query97	3014	2873	2867	2867
query98	237	219	206	206
query99	1261	755	751	751
Total cold run time: 312683 ms
Total hot run time: 180634 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 86d5fc4f5eb7cf77fa5708189ef09fac92ba975c with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@yiguolei yiguolei merged commit 4127f45 into apache:master Mar 21, 2024
@mrhhsg mrhhsg deleted the fix_join_core branch March 22, 2024 01:54
seawinde pushed a commit to seawinde/doris that referenced this pull request Mar 22, 2024
@yiguolei yiguolei mentioned this pull request Mar 24, 2024
mrhhsg added a commit to mrhhsg/doris that referenced this pull request Mar 26, 2024
mrhhsg added a commit to mrhhsg/doris that referenced this pull request Mar 26, 2024
mrhhsg added a commit to mrhhsg/doris that referenced this pull request Mar 26, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.8-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants