Adding ROLLUP result in Doris All BE core dump

[kangkaisen]

Describe the bug
when our user add a ROLLUP, our doris cluster all BEs core dump.
Our Doris cluster is the latest version.

ALTER TABLE cs_db.table_test_palo_detail ADD ROLLUP table_test_palo_detail_rollup(user_id,dat
e,cost)

The table table_test_palo_detail createStmt is normal:

CREATE TABLE `table_test_palo_detail` (
  `user_id` int(11) ,
  `date` varchar(256)  ,
  `timestamp` varchar(256),
  `age` int(11),
  `sex` int(11) ,
  `last_visit_date` varchar(256) REPLACE ,
  `cost` int(11) SUM,
  `max_dwell_time` int(11) MAX ,
  `min_dwell_time` int(11) MIN
) ENGINE=OLAP
AGGREGATE KEY(`user_id`, `date`, `timestamp`, `age`, `sex`)
DISTRIBUTED BY HASH(`user_id`) BUCKETS 10
PROPERTIES (
"storage_type" = "COLUMN",
 "replication_num" = "3"
);

The following is the log:

tcmalloc: large alloc 1952546816 bytes == 0x2fbd4000 @  0x242a493 0x258b61b 0xcf2e83 0xcc710f 0xcd009c 0xcdafbe 0xcde21a 0xcdf4e0
0xc5bdb6 0x111b9e5 0x112620d 0x7fb19e923e25
tcmalloc: large alloc 1952546816 bytes == 0xa49ec000 @  0x242a493 0x258b61b 0xcf2e83 0xcc710f 0xcd009c 0xcdafbe 0xcde21a 0xcdf4e0
0xc5bdb6 0x111b9e5 0x112620d 0x7fb19e923e25
tcmalloc: large alloc 1952546816 bytes == 0x119908000 @  0x242a493 0x258b61b 0xcf2e83 0xcc710f 0xcd009c 0xcdafbe 0xcde21a 0xcdf4e0
 0xc5bdb6 0x111b9e5 0x112620d 0x7fb19e923e25
tcmalloc: large alloc 1952546816 bytes == 0x18df20000 @  0x242a3a3 0x258b61b 0xcf2e83 0xcc7158 0xcd009c 0xcdafbe 0xcde21a 0xcdf4e0
 0xc5bdb6 0x111b9e5 0x112620d 0x7fb19e923e25
tcmalloc: large alloc 1952546816 bytes == 0x202538000 @  0x242a3a3 0x258b61b 0xcf2e83 0xcc7158 0xcd009c 0xcdafbe 0xcde21a 0xcdf4e0
 0xc5bdb6 0x111b9e5 0x112620d 0x7fb19e923e25
tcmalloc: large alloc 1952546816 bytes == 0x276b50000 @  0x242a3a3 0x258b61b 0xcf2e83 0xcc7158 0xcd009c 0xcdafbe 0xcde21a 0xcdf4e0
 0xc5bdb6 0x111b9e5 0x112620d 0x7fb19e923e25
PC: @           0xd436c0 doris::get_type_info()
*** SIGSEGV (@0x10) received by PID 28194 (TID 0x7f953bd7e700) from PID 16; stack trace: ***
    @     0x7f9593294270 (unknown)
    @           0xd436c0 doris::get_type_info()
    @           0xd1eba4 doris::Field::create()
    @           0xcf2e0d doris::WrapperField::create()
    @           0xcc710f doris::Rowset::add_column_statistics()
    @           0xcd009c doris::LinkedSchemaChange::process()
    @           0xcdafbe doris::SchemaChangeHandler::_alter_table()
    @           0xcde21a doris::SchemaChangeHandler::_do_alter_table()
    @           0xcdf4e0 doris::SchemaChangeHandler::process_alter_table()
    @           0xc5bdb6 doris::OLAPEngine::create_rollup_table()
    @          0x111b9e5 doris::TaskWorkerPool::_alter_table()
    @          0x112620d doris::TaskWorkerPool::_alter_table_worker_thread_callback()
    @     0x7f959304ae25 start_thread
    @     0x7f959335734d __clone

The following is the stack

   (gdb) bt
#0  0x0000000000d436c0 in doris::get_type_info (field_type=0) at /home/kangkaisen/palo/be/src/olap/types.cpp:79
#1  0x0000000000d1eba4 in Field (field_info=..., this=0x2df95230) at /home/kangkaisen/palo/be/src/olap/field.cpp:58
#2  doris::Field::create (field_info=...) at /home/kangkaisen/palo/be/src/olap/field.cpp:32
#3  0x0000000000cf2e0d in doris::WrapperField::create (info=..., len=len@entry=0)
    at /home/kangkaisen/palo/be/src/olap/wrapper_field.cpp:31
#4  0x0000000000cc710f in doris::Rowset::add_column_statistics (this=this@entry=0x2feae380, column_statistic_fields=...)
    at /home/kangkaisen/palo/be/src/olap/rowset.cpp:205
#5  0x0000000000cd009c in doris::LinkedSchemaChange::process (this=0x2fada080, olap_data=0x2df5fb00, new_rowset=0x2feae380)
    at /home/kangkaisen/palo/be/src/olap/schema_change.cpp:718
#6  0x0000000000cdafbe in doris::SchemaChangeHandler::_alter_table (sc_params=sc_params@entry=0x7fe170d675f0)
    at /home/kangkaisen/palo/be/src/olap/schema_change.cpp:2080
#7  0x0000000000cde21a in doris::SchemaChangeHandler::_do_alter_table (this=this@entry=0x7fe170d6c768,
    type=type@entry=doris::ALTER_TABLET_CREATE_ROLLUP_TABLE, ref_olap_table=..., request=...)
    at /home/kangkaisen/palo/be/src/olap/schema_change.cpp:1624
#8  0x0000000000cdf4e0 in doris::SchemaChangeHandler::process_alter_table (this=this@entry=0x7fe170d6c768,
    type=type@entry=doris::ALTER_TABLET_CREATE_ROLLUP_TABLE, request=...)
    at /home/kangkaisen/palo/be/src/olap/schema_change.cpp:1397
#9  0x0000000000c5bdb6 in doris::OLAPEngine::create_rollup_table (this=<optimized out>, request=...)
    at /home/kangkaisen/palo/be/src/olap/olap_engine.cpp:2262
#10 0x000000000111b9e5 in doris::TaskWorkerPool::_alter_table (this=this@entry=0x2da95d40, alter_tablet_request=...,
    signature=signature@entry=1428627, task_type=doris::TTaskType::ROLLUP,
    finish_task_request=finish_task_request@entry=0x7fe170d71fa0) at /home/kangkaisen/palo/be/src/agent/task_worker_pool.cpp:623
#11 0x000000000112620d in doris::TaskWorkerPool::_alter_table_worker_thread_callback (arg_this=0x2da95d40)
    at /home/kangkaisen/palo/be/src/agent/task_worker_pool.cpp:541
#12 0x00007fe1c8061e25 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fe1c836e34d in clone () from /lib64/libc.so.6

[chaoyli]

You can add log if rowset.cpp:205 to get more information. Field_type should not be 0，because the min is 1.

enum FieldType {
    OLAP_FIELD_TYPE_TINYINT = 1,           // MYSQL_TYPE_TINY
    OLAP_FIELD_TYPE_UNSIGNED_TINYINT = 2,
    OLAP_FIELD_TYPE_SMALLINT = 3,          // MYSQL_TYPE_SHORT
    OLAP_FIELD_TYPE_UNSIGNED_SMALLINT = 4,
    OLAP_FIELD_TYPE_INT = 5,            // MYSQL_TYPE_LONG
    OLAP_FIELD_TYPE_UNSIGNED_INT = 6,
    OLAP_FIELD_TYPE_BIGINT = 7,           // MYSQL_TYPE_LONGLONG
    OLAP_FIELD_TYPE_UNSIGNED_BIGINT = 8,
    OLAP_FIELD_TYPE_LARGEINT = 9,
    OLAP_FIELD_TYPE_FLOAT = 10,          // MYSQL_TYPE_FLOAT
    OLAP_FIELD_TYPE_DOUBLE = 11,        // MYSQL_TYPE_DOUBLE
    OLAP_FIELD_TYPE_DISCRETE_DOUBLE = 12,
    OLAP_FIELD_TYPE_CHAR = 13,        // MYSQL_TYPE_STRING
    OLAP_FIELD_TYPE_DATE = 14,          // MySQL_TYPE_NEWDATE
    OLAP_FIELD_TYPE_DATETIME = 15,      // MySQL_TYPE_DATETIME
    OLAP_FIELD_TYPE_DECIMAL = 16,       // DECIMAL, using different store format against MySQL
    OLAP_FIELD_TYPE_VARCHAR = 17,
    OLAP_FIELD_TYPE_STRUCT = 18,        // Struct
    OLAP_FIELD_TYPE_LIST = 19,          // LIST
    OLAP_FIELD_TYPE_MAP = 20,           // Map
    OLAP_FIELD_TYPE_UNKNOWN = 21,       // UNKNOW Type
    OLAP_FIELD_TYPE_NONE = 22,
    OLAP_FIELD_TYPE_HLL = 23
};

[kangkaisen]

OK, Thanks you！ I will add more log and test.

[chaoyli]

Also you can debug the be process in you own machine. The procedure is like this:

1. Start the be process using gdb. You can replace the last line of start_be.sh by
   gdb ${DORIS_HOME}/lib/palo_be "$@"
2. Run the be process in gdb.
3. Connect to doris cluster, send schema change task.
   Also if you want to be more accurate, you can compile doris be in DEBUG mode in O0 compile optimization level.
   You can set BUILD_TYPE environment to taking it into effect.

[kangkaisen]

OK, Thanks you very much. I will learn gdb and try to debug BE.

[kangkaisen]

This bug could reproduce stably.

1 create table:

CREATE TABLE `table_test_palo_detail` (
  `user_id` int(11) ,
  `date` varchar(256)  ,
  `timestamp` varchar(256),
  `age` int(11),
  `sex` int(11) ,
  `last_visit_date` varchar(256) REPLACE ,
  `cost` int(11) SUM,
  `max_dwell_time` int(11) MAX ,
  `min_dwell_time` int(11) MIN
) ENGINE=OLAP
AGGREGATE KEY(`user_id`, `date`, `timestamp`, `age`, `sex`)
DISTRIBUTED BY HASH(`user_id`) BUCKETS 10
PROPERTIES (
"storage_type" = "COLUMN",
 "replication_num" = "3"
);

2 load some data

1003	2017-10-02	2017-10-02 13:15:00	32	0	2017-10-02 11:20:00	30	11	11
1002	2017-10-02	2017-10-02 13:10:00	20	1	2017-10-02 12:59:12	200	5	5
1004	2017-10-01	2017-10-01 12:12:48	35	0	2017-10-01 10:00:15	100	3	3
1004	2017-10-03	2017-10-03 12:38:20	35	0	2017-10-03 10:20:22	11	6	6
1000	2017-10-01	2017-10-01 08:00:05	20	0	2017-10-11 06:00:00	20	10	10
1000	2017-10-01	2017-10-01 09:00:05	20	0	2017-10-11 07:00:01	15	2	2
1001	2017-10-01	2017-10-01 18:12:10	30	1	2017-10-11 05:05:43	2	22	22

3 add roll up

ALTER TABLE table_test_palo_detail ADD ROLLUP table_test_palo_detail_rollup(user_id,dat
e,cost)