-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Description
Doris' load performance is very bad in some cases. Especially when using broker load to load large quantities of data, the system resources may be idle, but the load is very slow.
Currently I found the following problems:
-
The execution plan of Broker Load has only one concurrency on a single node, so even if the system resources are idle, the load speed cannot be improved by increasing the concurrency.
-
Memtable is written to disk serially. By default, a disk has two flush threads, and the current flush thread pool strategy is serial execution. In fact, each memtable can be written to disk in parallel.
-
A loadChannel may correspond to data sent by multiple OlapTableSinks, and because of the problem of LoadChannel lock granularity, the data receiving is a serial operation. Therefore, in some cases, even if the BE node is added, the load speed cannot be improved.
I am trying to solve these problem.