Skip to content

[Performance] Import the load performace #5281

@morningman

Description

@morningman

Doris' load performance is very bad in some cases. Especially when using broker load to load large quantities of data, the system resources may be idle, but the load is very slow.

Currently I found the following problems:

  1. The execution plan of Broker Load has only one concurrency on a single node, so even if the system resources are idle, the load speed cannot be improved by increasing the concurrency.

  2. Memtable is written to disk serially. By default, a disk has two flush threads, and the current flush thread pool strategy is serial execution. In fact, each memtable can be written to disk in parallel.

  3. A loadChannel may correspond to data sent by multiple OlapTableSinks, and because of the problem of LoadChannel lock granularity, the data receiving is a serial operation. Therefore, in some cases, even if the BE node is added, the load speed cannot be improved.

I am trying to solve these problem.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions