(feature)(cloud) Use dynamic allocator instead of static buffer pool for better elasticity. #28983
Conversation
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
run buildall |
|
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
ByteYue
force-pushed
the
use_allocator
branch
from
e3658a1 to
4ca0f38
Compare
|
run buildall |
|
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
ByteYue
force-pushed
the
use_allocator
branch
from
4ca0f38 to
936e802
Compare
|
run buildall |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
PR approved by anyone and no changes requested. |
|
run buildall |
TPC-H: Total hot run time: 38772 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 184171 ms |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
dataroaring
force-pushed
the
use_allocator
branch
from
78e364e to
995b21e
Compare
|
run buildall |
TPC-H: Total hot run time: 38676 ms |
TPC-DS: Total hot run time: 184357 ms |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
TeamCity be ut coverage result: |
…for better elasticity. (apache#28983)
…for better elasticity. (apache#28983)
…#28983 #30703 #31169 (#31213) * (feature)(cloud) Use dynamic allocator instead of static buffer pool for better elasticity. (#28983) * [fix](outfile) Fix unable to export empty data (#30703) Issue Number: close #30600 Fix unable to export empty data to hdfs / S3, this behavior is inconsistent with version 1.2.7, version 1.2.7 can export empty data to hdfs/ S3, and there will be exported files on S3/HDFS. * [fix](file-writer) avoid empty file for segment writer (#31169) --------- Co-authored-by: AlexYue <yj976240184@gmail.com> Co-authored-by: zxealous <zhouchangyue@baidu.com>
5 participants
Proposed changes
Issue Number: close #xxx
The former implementation of the S3 buffer pool is not elastic, especially on cloud mode where all the load operation would access S3 then the static memory allocation strategy might block the load operation or it might cause dead lock in situation like vertical compaction.
TODO:
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...