-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[fix](ip) fix datatype serde for ipv6 with rowstore #43065
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
|
run buildall |
TPC-H: Total hot run time: 41763 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 192044 ms |
ClickBench: Total hot run time: 32.66 s |
|
run buildall |
xiaokang
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 41834 ms |
TPC-DS: Total hot run time: 191761 ms |
ClickBench: Total hot run time: 32.1 s |
eldenmoon
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
TeamCity be ut coverage result: |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 41612 ms |
TPC-DS: Total hot run time: 196919 ms |
ClickBench: Total hot run time: 32.51 s |
|
run buildall |
eldenmoon
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
PR approved by at least one committer and no changes requested. |
TPC-H: Total hot run time: 41409 ms |
TPC-DS: Total hot run time: 195896 ms |
ClickBench: Total hot run time: 32.41 s |
|
TeamCity be ut coverage result: |
|
run p0 |
xiaokang
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
before this pr: if we has ipv6 column in table which is support store_row_column we insert some data and then make sql with update we will meet core ``` *** Aborted at 1730367188 (unix time) try "date -d @1730367188" if you are using GNU date *** *** Current BE git commitID: face753 *** *** SIGSEGV invalid permissions for mapped object (@0x60c000a339ae) received by PID 4176451 (TID 4187168 OR 0x7ff774f5a700) from PID 10697134; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so 2# JVM_handle_linux_signal in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so 3# 0x00007FFFF63AEB50 in /lib64/libc.so.6 4# doris::IPv6Value::from_string(unsigned __int128&, char const*, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/runtime/ipv6_value.h:55 5# bool doris::vectorized::read_ipv6_text_impl<unsigned __int128>(unsigned __int128&, doris::vectorized::ReadBuffer&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/io/io_helper.h:316 6# doris::vectorized::DataTypeIPv6SerDe::read_one_cell_from_jsonb(doris::vectorized::IColumn&, doris::JsonbValue const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_ipv6_serde.cpp:76 7# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, char const*, unsigned long, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/jsonb/serialize.cpp:104 8# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, doris::vectorized::ColumnStr<unsigned int> const&, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/jsonb/serialize.cpp:83 9# doris::BaseTablet::fetch_value_through_row_column(std::shared_ptr<doris::Rowset>, doris::TabletSchema const&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::Block&) at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/base_tablet.cpp:885 10# doris::FixedReadPlan::read_columns_by_plan(doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int> >, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::vectorized::Block&, std::map<unsigned int, unsigned int, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >*, signed char const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/partial_update_info.cpp:295 11# doris::FixedReadPlan::fill_missing_columns(doris::RowsetWriterContext*, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::TabletSchema const&, doris::vectorized::Block&, std::vector<bool, std::allocator<bool> > const&, bool, unsigned long const&, doris::vectorized::Block const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/partial_update_info.cpp:332 12# doris::segment_v2::VerticalSegmentWriter::_append_block_with_partial_content(doris::segment_v2::RowsInBlock&, doris::vectorized::Block&) at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:502 ```
before this pr: if we has ipv6 column in table which is support store_row_column we insert some data and then make sql with update we will meet core ``` *** Aborted at 1730367188 (unix time) try "date -d @1730367188" if you are using GNU date *** *** Current BE git commitID: face753 *** *** SIGSEGV invalid permissions for mapped object (@0x60c000a339ae) received by PID 4176451 (TID 4187168 OR 0x7ff774f5a700) from PID 10697134; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so 2# JVM_handle_linux_signal in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so 3# 0x00007FFFF63AEB50 in /lib64/libc.so.6 4# doris::IPv6Value::from_string(unsigned __int128&, char const*, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/runtime/ipv6_value.h:55 5# bool doris::vectorized::read_ipv6_text_impl<unsigned __int128>(unsigned __int128&, doris::vectorized::ReadBuffer&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/io/io_helper.h:316 6# doris::vectorized::DataTypeIPv6SerDe::read_one_cell_from_jsonb(doris::vectorized::IColumn&, doris::JsonbValue const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_ipv6_serde.cpp:76 7# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, char const*, unsigned long, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/jsonb/serialize.cpp:104 8# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, doris::vectorized::ColumnStr<unsigned int> const&, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/jsonb/serialize.cpp:83 9# doris::BaseTablet::fetch_value_through_row_column(std::shared_ptr<doris::Rowset>, doris::TabletSchema const&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::Block&) at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/base_tablet.cpp:885 10# doris::FixedReadPlan::read_columns_by_plan(doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int> >, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::vectorized::Block&, std::map<unsigned int, unsigned int, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >*, signed char const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/partial_update_info.cpp:295 11# doris::FixedReadPlan::fill_missing_columns(doris::RowsetWriterContext*, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::TabletSchema const&, doris::vectorized::Block&, std::vector<bool, std::allocator<bool> > const&, bool, unsigned long const&, doris::vectorized::Block const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/partial_update_info.cpp:332 12# doris::segment_v2::VerticalSegmentWriter::_append_block_with_partial_content(doris::segment_v2::RowsInBlock&, doris::vectorized::Block&) at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:502 ```
before this pr: if we has ipv6 column in table which is support store_row_column we insert some data and then make sql with update we will meet core ``` *** Aborted at 1730367188 (unix time) try "date -d @1730367188" if you are using GNU date *** *** Current BE git commitID: face753 *** *** SIGSEGV invalid permissions for mapped object (@0x60c000a339ae) received by PID 4176451 (TID 4187168 OR 0x7ff774f5a700) from PID 10697134; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so 2# JVM_handle_linux_signal in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so 3# 0x00007FFFF63AEB50 in /lib64/libc.so.6 4# doris::IPv6Value::from_string(unsigned __int128&, char const*, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/runtime/ipv6_value.h:55 5# bool doris::vectorized::read_ipv6_text_impl<unsigned __int128>(unsigned __int128&, doris::vectorized::ReadBuffer&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/io/io_helper.h:316 6# doris::vectorized::DataTypeIPv6SerDe::read_one_cell_from_jsonb(doris::vectorized::IColumn&, doris::JsonbValue const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_ipv6_serde.cpp:76 7# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, char const*, unsigned long, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/jsonb/serialize.cpp:104 8# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, doris::vectorized::ColumnStr<unsigned int> const&, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/jsonb/serialize.cpp:83 9# doris::BaseTablet::fetch_value_through_row_column(std::shared_ptr<doris::Rowset>, doris::TabletSchema const&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::Block&) at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/base_tablet.cpp:885 10# doris::FixedReadPlan::read_columns_by_plan(doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int> >, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::vectorized::Block&, std::map<unsigned int, unsigned int, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >*, signed char const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/partial_update_info.cpp:295 11# doris::FixedReadPlan::fill_missing_columns(doris::RowsetWriterContext*, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::TabletSchema const&, doris::vectorized::Block&, std::vector<bool, std::allocator<bool> > const&, bool, unsigned long const&, doris::vectorized::Block const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/partial_update_info.cpp:332 12# doris::segment_v2::VerticalSegmentWriter::_append_block_with_partial_content(doris::segment_v2::RowsInBlock&, doris::vectorized::Block&) at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:502 ```
Cherry-picked from #43065 Co-authored-by: amory <wangqiannan@selectdb.com>
before this pr: if we has ipv6 column in table which is support store_row_column we insert some data and then make sql with update we will meet core ``` *** Aborted at 1730367188 (unix time) try "date -d @1730367188" if you are using GNU date *** *** Current BE git commitID: face753 *** *** SIGSEGV invalid permissions for mapped object (@0x60c000a339ae) received by PID 4176451 (TID 4187168 OR 0x7ff774f5a700) from PID 10697134; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so 2# JVM_handle_linux_signal in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so 3# 0x00007FFFF63AEB50 in /lib64/libc.so.6 4# doris::IPv6Value::from_string(unsigned __int128&, char const*, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/runtime/ipv6_value.h:55 5# bool doris::vectorized::read_ipv6_text_impl<unsigned __int128>(unsigned __int128&, doris::vectorized::ReadBuffer&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/io/io_helper.h:316 6# doris::vectorized::DataTypeIPv6SerDe::read_one_cell_from_jsonb(doris::vectorized::IColumn&, doris::JsonbValue const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_ipv6_serde.cpp:76 7# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, char const*, unsigned long, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/jsonb/serialize.cpp:104 8# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, doris::vectorized::ColumnStr<unsigned int> const&, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/jsonb/serialize.cpp:83 9# doris::BaseTablet::fetch_value_through_row_column(std::shared_ptr<doris::Rowset>, doris::TabletSchema const&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::Block&) at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/base_tablet.cpp:885 10# doris::FixedReadPlan::read_columns_by_plan(doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int> >, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::vectorized::Block&, std::map<unsigned int, unsigned int, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >*, signed char const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/partial_update_info.cpp:295 11# doris::FixedReadPlan::fill_missing_columns(doris::RowsetWriterContext*, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::TabletSchema const&, doris::vectorized::Block&, std::vector<bool, std::allocator<bool> > const&, bool, unsigned long const&, doris::vectorized::Block const*) const at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/partial_update_info.cpp:332 12# doris::segment_v2::VerticalSegmentWriter::_append_block_with_partial_content(doris::segment_v2::RowsInBlock&, doris::vectorized::Block&) at /mnt/disk1/wangqiannan/amory/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:502 ```
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
before this pr:
if we has ipv6 column in table which is support store_row_column
we insert some data
and then make sql with update we will meet core
Check List (For Committer)
Test
Behavior changed:
Does this need documentation?
Release note
None
Check List (For Reviewer who merge this PR)