Skip to content

[fix](bitmap) incorrect type of BitmapValue with fastunion #36834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
HappenLee merged 1 commit into from
Jun 27, 2024
Merged

[fix](bitmap) incorrect type of BitmapValue with fastunion #36834

HappenLee merged 1 commit into from
Jun 27, 2024

Conversation

@mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Jun 26, 2024 •
edited
Loading

Proposed changes

*** Query id: de527a3dee814cda-84ccf1d0532e016c ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1719287583 (unix time) try "date -d @1719287583" if you are using GNU date ***
*** Current BE git commitID: d2b0cd5c18 ***
*** SIGABRT unknown detail explain (@0x19a1) received by PID 6561 (TID 8834 OR 0x7f5135bb2640) from PID 6561; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:421
 1# 0x00007F5BF6477520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill at ./nptl/pthread_kill.c:89
 3# raise at ../sysdeps/posix/raise.c:27
 4# abort at ./stdlib/abort.c:81
 5# 0x000055F384EFC2FD in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
 6# 0x000055F384EEE93A in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
 7# google::LogMessage::SendToLog() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
 8# google::LogMessage::Flush() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
 9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
10# doris::BitmapValue::deserialize(char const*) at /root/doris/be/src/util/bitmap_value.h:1916
11# doris::vectorized::DataTypeBitMap::deserialize(char const*, doris::vectorized::IColumn*, int) const at /root/doris/be/src/vec/data_types/data_type_bitmap.cpp:87
12# doris::vectorized::Block::deserialize(doris::PBlock const&) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
13# doris::vectorized::VDataStreamRecvr::SenderQueue::add_block(doris::PBlock const&, int, long, google::protobuf::Closure**) at /root/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:167
14# doris::vectorized::VDataStreamRecvr::add_block(doris::PBlock const&, int, int, long, google::protobuf::Closure**) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
15# doris::vectorized::VDataStreamMgr::transmit_block(doris::PTransmitDataParams const*, google::protobuf::Closure**) at /root/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:139
16# doris::PInternalService::_transmit_block(google::protobuf::RpcController*, doris::PTransmitDataParams const*, doris::PTransmitDataResult*, google::protobuf::Closure*, doris::Status const&) at /root/doris/be/src/service/internal_service.cpp:1542
17# doris::PInternalService::transmit_block(google::protobuf::RpcController*, doris::PTransmitDataParams const*, doris::PTransmitDataResult*, google::protobuf::Closure*) at /root/doris/be/src/service/internal_service.cpp:1491
18# doris::PBackendService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) at /root/doris/gensrc/build/gen_cpp/internal_service.pb.cc:42040
19# brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
20# brpc::ProcessInputMessage(void*) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
21# brpc::InputMessenger::InputMessageClosure::~InputMessageClosure() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
22# brpc::InputMessenger::OnNewMessages(brpc::Socket*) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
23# brpc::Socket::ProcessEvent(void*) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
24# bthread::TaskGroup::task_runner(long) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
25# bthread_make_fcontext in /mnt/hdd01/ci/master-deploy/be/lib/doris_be

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mrhhsg
Copy link
Member Author

mrhhsg commented Jun 26, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 26, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-H: Total hot run time: 40093 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit df970856a954cfbef7c5e28696847e8c1d3110d5, data reload: false

------ Round 1 ----------------------------------
q1	17926	4589	4419	4419
q2	2608	194	197	194
q3	11124	1107	1106	1106
q4	10237	848	853	848
q5	7685	2661	2731	2661
q6	225	139	138	138
q7	988	605	613	605
q8	9276	2114	2093	2093
q9	9159	6497	6497	6497
q10	8977	3726	3737	3726
q11	455	237	226	226
q12	450	223	220	220
q13	18858	2948	2998	2948
q14	258	213	224	213
q15	513	483	478	478
q16	530	377	370	370
q17	979	685	667	667
q18	8097	7602	7364	7364
q19	7263	1501	1474	1474
q20	644	316	322	316
q21	4847	3196	3357	3196
q22	380	334	334	334
Total cold run time: 121479 ms
Total hot run time: 40093 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4379	4286	4256	4256
q2	356	260	251	251
q3	2958	2722	2706	2706
q4	1874	1615	1624	1615
q5	5237	5319	5349	5319
q6	213	125	128	125
q7	2139	1726	1689	1689
q8	3184	3350	3335	3335
q9	8396	8387	8333	8333
q10	3842	3697	3667	3667
q11	584	491	483	483
q12	785	630	634	630
q13	17483	2945	2973	2945
q14	281	263	264	263
q15	512	473	479	473
q16	481	410	409	409
q17	1766	1438	1476	1438
q18	7543	7354	7360	7354
q19	1720	1484	1615	1484
q20	1974	1778	1767	1767
q21	4765	4702	4606	4606
q22	621	541	555	541
Total cold run time: 71093 ms
Total hot run time: 53689 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-DS: Total hot run time: 172771 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit df970856a954cfbef7c5e28696847e8c1d3110d5, data reload: false

query1	944	403	379	379
query2	6487	2479	2168	2168
query3	6635	203	207	203
query4	19534	17264	17353	17264
query5	4141	476	451	451
query6	246	162	158	158
query7	4587	299	290	290
query8	320	293	273	273
query9	8728	2390	2370	2370
query10	594	274	285	274
query11	10542	10047	9946	9946
query12	131	84	86	84
query13	1643	389	392	389
query14	10285	7045	7279	7045
query15	229	187	188	187
query16	7815	279	269	269
query17	1884	555	540	540
query18	1971	288	293	288
query19	198	159	166	159
query20	90	91	85	85
query21	208	134	131	131
query22	4249	4085	4084	4084
query23	33762	32982	33222	32982
query24	12045	2772	2821	2772
query25	695	384	391	384
query26	1789	154	159	154
query27	3009	311	311	311
query28	7472	2109	2088	2088
query29	1162	627	624	624
query30	275	154	150	150
query31	957	771	746	746
query32	98	54	57	54
query33	782	299	298	298
query34	940	460	473	460
query35	749	641	612	612
query36	1099	939	948	939
query37	297	75	74	74
query38	2912	2762	2718	2718
query39	861	787	841	787
query40	279	133	128	128
query41	63	53	53	53
query42	120	98	101	98
query43	615	554	565	554
query44	1240	743	750	743
query45	198	161	171	161
query46	1069	707	728	707
query47	1841	1785	1750	1750
query48	380	307	302	302
query49	1208	410	417	410
query50	789	378	385	378
query51	6905	6769	6801	6769
query52	107	90	92	90
query53	361	302	290	290
query54	941	450	443	443
query55	78	75	73	73
query56	306	290	273	273
query57	1177	1053	1096	1053
query58	260	245	275	245
query59	3461	3121	3400	3121
query60	302	286	298	286
query61	111	112	105	105
query62	657	452	549	452
query63	313	290	286	286
query64	9856	2213	1761	1761
query65	3244	3097	3086	3086
query66	1360	350	324	324
query67	15831	15099	14781	14781
query68	8650	564	555	555
query69	665	483	417	417
query70	1191	1074	1091	1074
query71	546	280	280	280
query72	9047	5597	5546	5546
query73	1443	322	328	322
query74	5963	5505	5454	5454
query75	5521	2660	2632	2632
query76	5035	930	921	921
query77	781	299	304	299
query78	10392	9989	9704	9704
query79	8240	523	514	514
query80	981	463	464	463
query81	546	224	220	220
query82	549	102	112	102
query83	354	171	164	164
query84	274	82	85	82
query85	1136	272	269	269
query86	382	308	330	308
query87	3318	3076	3080	3076
query88	4489	2381	2410	2381
query89	542	381	379	379
query90	2001	190	189	189
query91	128	103	99	99
query92	66	51	47	47
query93	6892	517	512	512
query94	1285	182	186	182
query95	410	318	303	303
query96	604	269	265	265
query97	3208	3068	3100	3068
query98	235	207	202	202
query99	1208	830	852	830
Total cold run time: 299519 ms
Total hot run time: 172771 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit df970856a954cfbef7c5e28696847e8c1d3110d5, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.21	0.05	0.05
query4	1.68	0.06	0.07
query5	0.50	0.49	0.50
query6	1.14	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.58	0.58	0.57
query14	0.75	0.78	0.79
query15	0.82	0.82	0.81
query16	0.37	0.37	0.37
query17	0.95	0.98	1.03
query18	0.21	0.26	0.23
query19	1.83	1.70	1.72
query20	0.01	0.00	0.00
query21	15.44	0.65	0.64
query22	4.43	6.67	1.98
query23	18.34	1.33	1.25
query24	2.06	0.22	0.22
query25	0.15	0.08	0.08
query26	0.26	0.18	0.17
query27	0.08	0.08	0.08
query28	13.28	1.01	0.99
query29	12.62	3.34	3.34
query30	0.27	0.06	0.06
query31	2.90	0.38	0.38
query32	3.30	0.46	0.47
query33	2.86	2.94	2.87
query34	17.32	4.38	4.44
query35	4.48	4.45	4.47
query36	0.65	0.46	0.50
query37	0.19	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.09	0.04	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.78 s
Total hot run time: 30.53 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 27, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2024

PR approved by anyone and no changes requested.

@mrhhsg mrhhsg changed the title [fix](bitmap) incorrect type of bitmap with fastunion [fix](bitmap) incorrect type of BitmapValue with fastunion Jun 27, 2024
HappenLee
HappenLee approved these changes Jun 27, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee HappenLee merged commit 233b0b5 into apache:master Jun 27, 2024
@mrhhsg mrhhsg deleted the fix_bitmap branch June 27, 2024 03:19
mrhhsg added a commit to mrhhsg/doris that referenced this pull request Jun 27, 2024
@mrhhsg
[fix](bitmap) incorrect type of BitmapValue with fastunion (apache#36834
35b5b8b 
)
@mrhhsg mrhhsg mentioned this pull request Jun 27, 2024
Merged
mrhhsg added a commit to mrhhsg/doris that referenced this pull request Jun 27, 2024
@mrhhsg
[fix](bitmap) incorrect type of BitmapValue with fastunion (apache#36834
5a3bdc2 
)
@mrhhsg mrhhsg mentioned this pull request Jun 27, 2024
Merged
HappenLee pushed a commit that referenced this pull request Jun 28, 2024
@mrhhsg
[fix](bitmap) incorrect type of BitmapValue with fastunion (#36834) (#…
f27ae8f 
…36896)
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@mrhhsg @dataroaring
[fix](bitmap) incorrect type of BitmapValue with fastunion (#36834)
1256101
yiguolei pushed a commit that referenced this pull request Jul 1, 2024
@mrhhsg
[fix](bitmap) incorrect type of BitmapValue with fastunion (#36834) (#…
c44d158 
…36898)

## Proposed changes

pick #36834

<!--Describe your changes.-->
BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Jul 10, 2024
@mrhhsg @BiteTheDDDDt
[fix](bitmap) incorrect type of BitmapValue with fastunion (apache#36834
42af471 
) (apache#36896)
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@mrhhsg
[fix](bitmap) incorrect type of BitmapValue with fastunion (apache#36834
583794f 
) (apache#36898)

## Proposed changes

pick apache#36834

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@HappenLee HappenLee HappenLee approved these changes

@zhangstar333 zhangstar333 zhangstar333 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@mrhhsg @doris-robot @yiguolei @HappenLee @zhangstar333 @wm1581066 @dataroaring