-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Description
Now, Doris still get bad performance for High concurrency stream load, and I add some metric to trace the some bottleneck for issue #3696, and modify some be config to make it a little better. However, the locked range in db level is too big, when one slow task thread get db lock, it can easily block other tasks. I do some test for stream load, sadly, for db level lock, the total time cost for 5000 stream load in one java client is not stable, sometimes about 15s, sometimes need 30s.
By analyzing the return result info, I find some commit task performance are poor, because commit task can only executed one by one when get db write lock. And I try to use table lock instead of db lock to do some testing, now, the whole time cost is much more stable , about 11s for whole 5000 stream load(test cluster is combined by 1 fe with 5 be, total 500 tables, 10 stream load for 1 table).
Although some publish task still need 1s to complete, but table lock can significantly reduce the lock
competition. So I make this proposal to make some improvement to stream load performance.