Skip to content

[Bug]If the key column of the table is a double / float type, it will crash when checksum #3735

@marising

Description

@marising

Describe the bug
If the key column of the table is double/float type,it will crash when checksum.

  1. be/src/olap/task/engine_checksum_task.cpp
    return_columns do not contain column of double/float type
    // ignore float and double type considering to precision lose
    for (size_t i = 0; i < tablet->tablet_schema().num_columns(); ++i) {
        FieldType type = tablet->tablet_schema().column(i).type();
        if (type == OLAP_FIELD_TYPE_FLOAT || type == OLAP_FIELD_TYPE_DOUBLE) {
            continue;
        }

        reader_params.return_columns.push_back(i);
    }
  1. be/src/olap/row.h Use the all columns
    use num_key_columns from Schema,num_key_columns include the column of double/float type
template<typename LhsRowType, typename RhsRowType>
int compare_row(const LhsRowType& lhs, const RhsRowType& rhs) {
    for (uint32_t cid = 0; cid < lhs.schema()->num_key_columns(); ++cid) {
        auto res = lhs.schema()->column(cid)->compare_cell(lhs.cell(cid), rhs.cell(cid));
        if (res != 0) {
            return res;
        }
    }
    return 0;
}

// Used to compare row with input scan key. Scan key only contains key columns,
// row contains all key columns, which is superset of key columns.
// So we should compare the common prefix columns of lhs and rhs.
//
// NOTE: if you are not sure if you can use it, please don't use this function.
template<typename LhsRowType, typename RhsRowType>
int compare_row_key(const LhsRowType& lhs, const RhsRowType& rhs) {
    auto cmp_cids = std::min(lhs.schema()->num_column_ids(), rhs.schema()->num_column_ids());
    for (uint32_t cid = 0; cid < cmp_cids; ++cid) {
        auto res = lhs.schema()->column(cid)->compare_cell(lhs.cell(cid), rhs.cell(cid));
        if (res != 0) {
            return res;
        }
    }
    return 0;
}
  1. Core file

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/export/home/lihaibo/deploy_jd/be/lib/palo_be'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000d6b838 in compare_cell<doris::RowCursorCell, doris::RowCursorCell> (this=, rhs=..., lhs=...) at /root/map_doris/be/src/olap/field.h:138
138 /root/map_doris/be/src/olap/field.h: No such file or directory.
Missing separate debuginfos, use: debuginfo-install libgcc-4.8.5-28.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0 0x0000000000d6b838 in compare_cell<doris::RowCursorCell, doris::RowCursorCell> (this=, rhs=..., lhs=...) at /root/map_doris/be/src/olap/field.h:138
#1 compare_row<doris::RowCursor, doris::RowCursor> (rhs=..., lhs=...) at /root/map_doris/be/src/olap/row.h:62
#2 operator() (this=, b=0x58dee40, a=0x5df3b00) at /root/map_doris/be/src/olap/reader.cpp:269
#3 operator()<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vectordoris::CollectIterator::ChildCtx* >, doris::CollectIterator::ChildCtx*> (__val=, __it=..., this=)
at /usr/include/c++/7.3.0/bits/predefined_ops.h:177
#4 std::__push_heap<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vector<doris::CollectIterator::ChildCtx*, std::allocatordoris::CollectIterator::ChildCtx* > >, long, doris::CollectIterator::ChildCtx*, __gnu_cxx::__ops::_Iter_comp_valdoris::CollectIterator::ChildCtxComparator > (__first=..., __holeIndex=1, __topIndex=0, __value=0x58dee40, __comp=...) at /usr/include/c++/7.3.0/bits/stl_heap.h:133
#5 0x0000000000d66cd1 in push_heap<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vectordoris::CollectIterator::ChildCtx* >, doris::CollectIterator::ChildCtxComparator> (__last=..., __first=..., __comp=...)
at /usr/include/c++/7.3.0/bits/stl_heap.h:207
#6 push (__x=@0x58df878: 0x58dee40, this=0x58df888) at /usr/include/c++/7.3.0/bits/stl_queue.h:598
#7 _merge_next (delete_flag=, row=, this=0x58df860) at /root/map_doris/be/src/olap/reader.cpp:227
#8 next (delete_flag=, row=, this=0x58df860) at /root/map_doris/be/src/olap/reader.cpp:68
#9 doris::Reader::_agg_key_next_row (this=0x7ffe9d5a1920, row_cursor=0x7ffe9d5a1800, mem_pool=0x5db2fc0, agg_pool=, eof=) at /root/map_doris/be/src/olap/reader.cpp:371
#10 0x0000000000d2c617 in next_row_with_aggregation (eof=0x7ffe9d5a1770, agg_pool=, mem_pool=, row_cursor=0x7ffe9d5a1800, this=0x7ffe9d5a1920) at /root/map_doris/be/src/olap/reader.h:130
#11 doris::EngineChecksumTask::_compute_checksum (this=this@entry=0x7ffe9d5a45d0) at /root/map_doris/be/src/olap/task/engine_checksum_task.cpp:120
#12 0x0000000000d2d2ba in doris::EngineChecksumTask::execute (this=this@entry=0x7ffe9d5a45d0) at /root/map_doris/be/src/olap/task/engine_checksum_task.cpp:36
#13 0x0000000000aa462e in main (argc=1, argv=0x7ffe9d5a4708) at /root/map_doris/be/src/service/doris_main.cpp:183

Metadata

Metadata

Assignees

Labels

area/storageIssues or PRs related to storage enginekind/fixCategorizes issue or PR as related to a bug.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions