Skip to content

[Proposal] Support read and write lock in table level to reduce lock competition  #3743

@caiconghui

Description

@caiconghui

Now, Doris still get bad performance for High concurrency stream load, and I add some metric to trace the some bottleneck for issue #3696, and modify some be config to make it a little better. However, the locked range in db level is too big, when one slow task thread get db lock, it can easily block other tasks. I do some test for stream load, sadly, for db level lock, the total time cost for 5000 stream load in one java client is not stable, sometimes about 15s, sometimes need 30s.

By analyzing the return result info, I find some commit task performance are poor, because commit task can only executed one by one when get db write lock. And I try to use table lock instead of db lock to do some testing, now, the whole time cost is much more stable , about 11s for whole 5000 stream load(test cluster is combined by 1 fe with 5 be, total 500 tables, 10 stream load for 1 table).

Although some publish task still need 1s to complete, but table lock can significantly reduce the lock
competition. So I make this proposal to make some improvement to stream load performance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/loadIssues or PRs related to all kinds of loadkind/improvementproposalCategorizes an issue is a proposal

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions