[feature](file cache) Support in-memory filecache for no-disk/slow-disk system #38811
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| namespace doris::io { | ||
|
|
||
| MemFileCacheStorage::~MemFileCacheStorage() {} |
Contributor
There was a problem hiding this comment.
warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
Suggested change
| MemFileCacheStorage::~MemFileCacheStorage() {} | |
| MemFileCacheStorage::~MemFileCacheStorage() = default; |
gavinchou
changed the title
gavinchou
requested changes
Aug 19, 2024
| namespace doris::io { | ||
|
|
||
| struct MemBlock { | ||
| std::shared_ptr<char> addr; |
Contributor
There was a problem hiding this comment.
why not just a std::string
Contributor
Author
There was a problem hiding this comment.
It is a memory address pointing to a char array (a continous memory block), so I think it should be std::shared_ptr<char[]> instead?
Contributor
Author
|
TODO: config: for extensibility, use another json field instead of 'memory' in path |
Contributor
Author
|
Sry for this hanging WIP PR, I've been dealing with more urgent tasks. I will return to this ASAP! |
freemandealer
changed the title
| LOG_ERROR("").tag("file cache path", cache_base_path).tag("error", strerror(errno)); | ||
| return Status::IOError("{} statfs error {}", cache_base_path, strerror(errno)); | ||
| } | ||
| size_t disk_capacity = static_cast<size_t>( |
Contributor
There was a problem hiding this comment.
warning: use auto when initializing with a cast to avoid duplicating the type name [modernize-use-auto]
Suggested change
| size_t disk_capacity = static_cast<size_t>( | |
| auto disk_capacity = static_cast<size_t>( |
| } | ||
| } | ||
|
|
||
| void test_file_cache_memory_storage(io::FileCacheType cache_type) { |
Contributor
There was a problem hiding this comment.
warning: function 'test_file_cache_memory_storage' exceeds recommended size/complexity thresholds [readability-function-size]
void test_file_cache_memory_storage(io::FileCacheType cache_type) {
^Additional context
be/test/io/cache/block_file_cache_test.cpp:659: 397 lines including whitespace and comments (threshold 80)
void test_file_cache_memory_storage(io::FileCacheType cache_type) {
^|
|
||
| TEST_F(BlockFileCacheTest, ttl_modify_memory_storage) { | ||
| test_file_cache_memory_storage(io::FileCacheType::NORMAL); | ||
| auto sp = SyncPoint::get_instance(); |
Contributor
There was a problem hiding this comment.
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
Suggested change
| auto sp = SyncPoint::get_instance(); | |
| L);auto * |
| } | ||
| } | ||
|
|
||
| void test_file_cache_memory_storage(io::FileCacheType cache_type) { |
Contributor
There was a problem hiding this comment.
warning: function 'test_file_cache_memory_storage' exceeds recommended size/complexity thresholds [readability-function-size]
void test_file_cache_memory_storage(io::FileCacheType cache_type) {
^Additional context
be/test/io/cache/block_file_cache_test.cpp:659: 389 lines including whitespace and comments (threshold 80)
void test_file_cache_memory_storage(io::FileCacheType cache_type) {
^
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 38581 ms |
TPC-DS: Total hot run time: 187710 ms |
ClickBench: Total hot run time: 31.52 s |
|
TeamCity be ut coverage result: |
freemandealer
force-pushed
the
in-memory-file-cache
branch
from
a70c120 to
342316b
Compare
Contributor
Author
|
run buildall |
gavinchou
reviewed
Sep 23, 2024
gavinchou
reviewed
Sep 23, 2024
gavinchou
reviewed
Sep 23, 2024
gavinchou
reviewed
Sep 23, 2024
gavinchou
reviewed
Sep 23, 2024
gavinchou
reviewed
Sep 23, 2024
gavinchou
reviewed
Sep 23, 2024
TPC-H: Total hot run time: 41403 ms |
Contributor
Author
|
run buildall |
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
TPC-DS: Total hot run time: 191044 ms |
ClickBench: Total hot run time: 32.17 s |
TODO: 1. 优化 内存池 2. 写入、读取数据内存零拷贝 3. 原始一些代码的重构（很多fs相关的内容散布在 block_file_cache.cpp 里, 需要进行额外的清理） Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
freemandealer
force-pushed
the
in-memory-file-cache
branch
from
2b80049 to
b78918a
Compare
Contributor
Author
|
run buildall |
gavinchou
approved these changes
Sep 24, 2024
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 41701 ms |
TPC-DS: Total hot run time: 191397 ms |
ClickBench: Total hot run time: 32.42 s |
gavinchou
pushed a commit
that referenced
this pull request
Sep 24, 2024
…sk system (#38811) User can set file_cache_path=[{..., "storage":"memory"}] to use RAM as file cache storage.
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User can set file_cache_path=[{..., "storage":"memory"}] to use RAM as file cache storage.