Routine load problem

[luoyafei]

Q: create routine load, but table no data to load.

resource:

Only 1 FE, 1 BE, 1 KFK, 1 ZK
FE, BE, KFK, ZK all on the same machine

kafka version:

0.10.1.0

create topic:

bin/kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper 10.39.59.92:2181

create table:

CREATE TABLE IF NOT EXISTS test_table
(
`id` INT NOT NULL,
`name` VARCHAR(20) NOT NULL,
`timestamp` DATETIME NOT NULL,
`count` INT SUM DEFAULT "0"
)
ENGINE=olap
AGGREGATE KEY(`id`,`name`,`timestamp`)
PARTITION BY RANGE(`timestamp`)
(
PARTITION `p201906` VALUES LESS THAN ("2019-07-01 00:00:00"),
PARTITION `p201907` VALUES LESS THAN ("2019-08-01 00:00:00")
) DISTRIBUTED BY HASH(`id`) BUCKETS 8
PROPERTIES
(
"replication_num" = "1"
);

create routine:

CREATE ROUTINE LOAD feed.load_test ON test_table
COLUMNS(id, name, timestamp, count)
PROPERTIES
(
"desired_concurrent_number"="1"
)
FROM KAFKA
(
"kafka_broker_list" = "10.39.59.92:9092",
"kafka_topic" = "test"
);

There appears to be an import operation, but there is no data in the table test

Long agter then execution, I found these logs in FE

And BE also has a large number of such logs

[imay]

Please use 0.10.2

[luoyafei]

There are no changes after the revision 0.10.2

What causes this exception?

[imay]

It seems that FE is failed to submit load task to BE. You should look BE's INFO log to see if there is some problem?

[luoyafei]

like this?

[imay]

you should look backendId 10002's LOG, and you can pstack doris_be process to see what are the threads doing.

[luoyafei]

all most threads seem to be waiting for lock?

be_stack.log

[imay]

@luoyafei

Thanks for your help. I think the problem is that you have too many concurrent routine load job. I'm not sure your case. In this stack trace I found that you have more than 100 concurrent load task in this backend.

However it is bad for us to let submit_task stall, we will fix it later

[morningman]

@luoyafei How many routine load job have you been created in FE? And what is their concurrent task number?

[luoyafei]

this my routine load config;

mysql> show routine load;
+-------+------------------+---------------------+-----------+---------+--------------+------------+---------+----------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------+
| Id    | Name             | CreateTime          | PauseTime | EndTime | DbName       | TableName  | State   | DataSourceType | CurrentTaskNum | JobProperties                                                                                                                                                                                                                              | DataSourceProperties                                                          | Statistic                                                                                                                                                                              | Progress           | ReasonOfStateChanged | ErrorLogUrls |
+-------+------------------+---------------------+-----------+---------+--------------+------------+---------+----------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------+
| 11093 | load_test_0_10_2 | 2019-06-05 22:44:46 | N/A       | N/A     | yafei10:feed | test_table | RUNNING | KAFKA          | 1              | {"partitions":"*","columnToColumnExpr":"id,name,timestamp,count","maxBatchIntervalS":"10","whereExpr":"*","maxBatchSizeBytes":"104857600","columnSeparator":"\t","maxErrorNum":"0","currentTaskConcurrentNum":"1","maxBatchRows":"200000"} | {"topic":"test","currentKafkaPartitions":"0","brokerList":"10.39.59.92:9092"} | {"receivedBytes":0,"errorRows":0,"committedTaskNum":0,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":0,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":1} | {"0":"OFFSET_END"} |                      |              |
+-------+------------------+---------------------+-----------+---------+--------------+------------+---------+----------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+--------------+
1 row in set (0.00 sec)

[luoyafei]

concurrent task num set only 1
but i can see BE log, always submit new tasks
my doris version is 0.10.0-rc01

[morningman]

That's weird. Could you please send a be.INFO and fe.log file to my mail box: morningman@163.com

[luoyafei]

The log file has been sent via email.
thx :)

[morningman]

Sorry I didn't see your mail.
we can talk on WeChat: 13001006668