[Bug] string column is too large:total_length=4297083314,element_number=162668

[wuweiqing-dev]

Search before asking

• I had searched in the issues and found no similar issues.

Version

doris-1.2.4-rc01

What's Wrong?

The BE node is inactivated and the OUT log reports an error:string column is too large:total_length=4297083314,element_number=162668

What You Expected?

In response to this situation, it is possible to make alarms, rather than direct inactivation of the BE node, which will cause serious consequences and the entire cluster will not work for a short time

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wuweiqing-dev]

F0519 22:01:45.518649 71394 column_string.h:66] string column length is too large: total_length=4297083314 ,element_number=162668
*** Check failure stack trace: ***
@ 0x55b7b57ca2ad google::LogMessage::Fail()
@ 0x55b7b57cc7e9 google::LogMessage::SendToLog()
@ 0x55b7b57c9e16 google::LogMessage::Flush()
@ 0x55b7b57cce59 google::LogMessageFatal::~LogMessageFatal()
@ 0x55b7b13db145 doris::vectorized::ColumnString::insert_from()
@ 0x55b7b13d188b doris::vectorized::ColumnString::insert_indices_from()
@ 0x55b7b13c5408 doris::vectorized::ColumnNullable::insert_indices_from()
@ 0x55b7b146badd doris::vectorized::MutableBlock::add_rows()
@ 0x55b7afe7860c doris::MemTable::insert()
@ 0x55b7aff5cd6e doris::DeltaWriter::write()
@ 0x55b7b0450fff doris::TabletsChannel::add_batch<>()
@ 0x55b7b0525963 doris::LoadChannel::add_batch<>()
@ 0x55b7b0525b80 doris::LoadChannelMgr::add_batch<>()
@ 0x55b7b051745a _ZNSt17_Function_handlerIFvvEZN5doris20PInternalServiceImpl24_tablet_writer_add_blockEPN6google8protobuf13RpcControllerEPKNS1_28PTabletWriterAddBlockRequestEPNS1_27PTabletWriterAddBlockResultEPNS4_7ClosureEEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x55b7b03309e5 doris::PriorityThreadPool::work_thread()
@ 0x55b7b8208b10 execute_native_thread_routine
@ 0x7f0059a4adc5 start_thread
@ 0x7f0059d5673d __clone
@ (nil) (unknown)
*** Query id: 0-0 ***
*** Aborted at 1684504906 (unix time) try "date -d @1684504906" if you are using GNU date ***
*** Current BE git commitID: Unknown ***
*** SIGABRT unkown detail explain (@0x3e800010e4b) received by PID 69195 (TID 0x7efbb3f54700) from PID 69195; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# 0x00007F0059C94250 in /lib64/libc.so.6
2# raise in /lib64/libc.so.6
3# __GI_abort in /lib64/libc.so.6
4# 0x000055B7B57D4C99 in /opt/appdata/disk01/doris/be/lib/doris_be
5# 0x000055B7B57CA2AD in /opt/appdata/disk01/doris/be/lib/doris_be
6# google::LogMessage::SendToLog() in /opt/appdata/disk01/doris/be/lib/doris_be
7# google::LogMessage::Flush() in /opt/appdata/disk01/doris/be/lib/doris_be
8# google::LogMessageFatal::~LogMessageFatal() in /opt/appdata/disk01/doris/be/lib/doris_be
9# doris::vectorized::ColumnString::insert_from(doris::vectorized::IColumn const&, unsigned long) at /root/doris/be/src/vec/columns/column_string.h:151
10# doris::vectorized::ColumnString::insert_indices_from(doris::vectorized::IColumn const&, int const*, int const*) at /root/doris/be/src/vec/columns/column_string.cpp:106
11# doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, int const*, int const*) at /root/doris/be/src/vec/columns/column_nullable.cpp:259
12# doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, int const*, int const*) at /root/doris/be/src/vec/core/block.cpp:1008
13# doris::MemTable::insert(doris::vectorized::Block const*, std::vector<int, std::allocator > const&) at /root/doris/be/src/olap/memtable.cpp:202
14# doris::DeltaWriter::write(doris::vectorized::Block const*, std::vector<int, std::allocator > const&) at /root/doris/be/src/olap/delta_writer.cpp:233
15# doris::Status doris::TabletsChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequestconst&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/tablets_channel.cpp:489
16# doris::Status doris::LoadChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel.h:163
17# doris::Status doris::LoadChannelMgr::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequestconst&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel_mgr.h:168
18# std::_Function_handler<void (), doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*, doris::PTabletWriterAddBlockRequestconst*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
19# doris::PriorityThreadPool::work_thread(int) at /root/doris/be/src/util/priority_thread_pool.hpp:140
20# execute_native_thread_routine in /opt/appdata/disk01/doris/be/lib/doris_be
21# start_thread in /lib64/libpthread.so.0
22# clone in /lib64/libc.so.6

[dataroaring]

How to reproduce it?

[JJJzheng]

遇到了同样的问题，随机出现的。
出现了 crash 之后，be 节点就挂了。

[michael1991]

Facing same issue here ...

[davycun]

@xiaokang
1、下面是外部数据源表的信息(是postgresql)，doris的版本是2.0.0-rc4
CatalogId | 0
+-------------+------------------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+------+---------+-------+
| account_id | INT | No | true | | |
| create_time | DATETIME(6) | No | true | | |
| update_time | DATETIME(6) | No | true | | |
| lng | DOUBLE | Yes | true | | |
| lat | DOUBLE | Yes | true | | |
| ip | TEXT | Yes | true | | |
| id | INT | No | true | | |
| category | TEXT | No | true | | |
| deleted | BOOLEAN | No | true | | |
| extra | JSON | No | true | | |
| geometry | UNSUPPORTED_TYPE | Yes | true | | |
+-------------+------------------+------+------+---------+-------+

2、 查询这个表的json 字段的时候就报错 select extra from my_catalog.my_schema.my_table limit 1;

3、下面执行sql后，backend的异常信息（string column length is too large）
doris-be01-1 | terminate called after throwing an instance of 'doris::Exception'
doris-be01-1 | what(): [E-3113] string column length is too large: total_length=94751273517062, element_number=1
doris-be01-1 | 0. /root/src/doris/be/src/common/stack_trace.cpp:298: StackTrace::tryCapture() @ 0x000000000b36ece7 in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 1. /root/src/doris/be/src/common/stack_trace.h:0: doris::get_stack_traceabi:cxx11 @ 0x000000000b36d2bd in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 2. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173: doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits >) @ 0x000000000ae1c7ae in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 3. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::Exception::Exception<unsigned long&, unsigned long&>(int, std::basic_string_view<char, std::char_traits >, unsigned long&, unsigned long&) @ 0x000000000a8e2818 in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 4. /root/src/doris/be/src/vec/columns/column_string.h:71: doris::vectorized::ColumnString::insert(doris::vectorized::Field const&) @ 0x000000000c3c725b in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 5. /root/src/doris/be/src/vec/common/cow.h:402: doris::vectorized::IDataType::create_column_const(unsigned long, doris::vectorized::Field const&) const @ 0x000000000d1004a2 in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 6. /root/src/doris/be/src/vec/data_types/data_type.cpp:0: doris::vectorized::IDataType::create_column_const_with_default_value(unsigned long) const @ 0x000000000d100596 in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 7. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:98: doris::vectorized::JdbcConnector::_cast_string_to_json(doris::SlotDescriptor const*, doris::vectorized::Block*, int, int) @ 0x000000000e7c2028 in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 8. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360: doris::vectorized::JdbcConnector::get_next(bool*, std::vector<COWdoris::vectorized::IColumn::mutable_ptrdoris::vectorized::IColumn, std::allocator<COWdoris::vectorized::IColumn::mutable_ptrdoris::vectorized::IColumn > >&, doris::vectorized::Block*, int) @ 0x000000000e7bf7ef in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 9. /root/src/doris/be/src/common/status.h:414: doris::vectorized::NewJdbcScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) @ 0x000000000e5f3caa in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 10. /root/src/doris/be/src/vec/exec/scan/vscanner.cpp:0: doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) @ 0x000000000e6f8584 in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 11. /root/src/doris/be/src/common/status.h:335: doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptrdoris::vectorized::VScanner) @ 0x000000000e658723 in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 12. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701: std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::{lambda()#3}>::_M_invoke(std::_Any_data const&) @ 0x000000000e6599a1 in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 13. /root/src/doris/be/src/util/threadpool.cpp:0: doris::ThreadPool::dispatch_thread() @ 0x000000000b3ab4bf in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 14. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void*) @ 0x000000000b3a153c in /opt/apache-doris/be/lib/doris_be
doris-be01-1 | 15. start_thread @ 0x0000000000008ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so
doris-be01-1 | 16. clone @ 0x00000000000fddef in /lib/x86_64-linux-gnu/libc-2.31.so

[haoshuai5358]

我在1.2.6也遇到了同样的问题
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# 0x00007F0059C94250 in /lib64/libc.so.6
2# raise in /lib64/libc.so.6
3# __GI_abort in /lib64/libc.so.6
4# 0x000055B7B57D4C99 in /opt/appdata/disk01/doris/be/lib/doris_be
5# 0x000055B7B57CA2AD in /opt/appdata/disk01/doris/be/lib/doris_be
6# google::LogMessage::SendToLog() in /opt/appdata/disk01/doris/be/lib/doris_be
7# google::LogMessage::Flush() in /opt/appdata/disk01/doris/be/lib/doris_be
8# google::LogMessageFatal::~LogMessageFatal() in /opt/appdata/disk01/doris/be/lib/doris_be
9# doris::vectorized::ColumnString::insert_from(doris::vectorized::IColumn const&, unsigned long) at /root/doris/be/src/vec/columns/column_string.h:151
10# doris::vectorized::ColumnString::insert_indices_from(doris::vectorized::IColumn const&, int const*, int const*) at /root/doris/be/src/vec/columns/column_string.cpp:106
11# doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, int const*, int const*) at /root/doris/be/src/vec/columns/column_nullable.cpp:259
12# doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, int const*, int const*) at /root/doris/be/src/vec/core/block.cpp:1008
13# doris::MemTable::insert(doris::vectorized::Block const*, std::vector<int, std::allocator > const&) at /root/doris/be/src/olap/memtable.cpp:202

[megarobo-caiweimin]

遇到了同样的问题，有任何进展么？

[lynchlee]

囧，生产大故障

[megarobo-caiweimin]

Can anyone help resolve the bug? Doris 1.2.7.1 has same problem

[megarobo-caiweimin]

The bug seems to be fixed in 2.0.4. I find following pages.
#28584
#28625

[xiaodouxia]

任何人都可以帮助解决该错误吗？ Doris 1.2.7.1 有同样的问题
Have you solved this problem now? I have the same problem with the same version