[opt](memtracker) Optimize PODArray memory tracking accuracy
#50549
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
run buildall |
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
TPC-H: Total hot run time: 34128 ms |
TPC-DS: Total hot run time: 198589 ms |
ClickBench: Total hot run time: 29.21 s |
xinyiZzz
force-pushed
the
20250429_fix_podarray
branch
from
e48af9d to
1524e5b
Compare
Contributor
Author
|
run buildall |
xinyiZzz
force-pushed
the
20250429_fix_podarray
branch
from
1524e5b to
06383a4
Compare
yiguolei
reviewed
May 7, 2025
yiguolei
reviewed
May 7, 2025
Contributor
Author
|
run buildall |
xinyiZzz
force-pushed
the
20250429_fix_podarray
branch
from
e0fdf9c to
83987d6
Compare
Contributor
Author
|
run buildall |
xinyiZzz
force-pushed
the
20250429_fix_podarray
branch
from
83987d6 to
7c479b7
Compare
Contributor
Author
|
run buildall |
yiguolei
reviewed
May 8, 2025
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
xinyiZzz
force-pushed
the
20250429_fix_podarray
branch
from
7c479b7 to
4256478
Compare
Contributor
Author
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
xinyiZzz
force-pushed
the
20250429_fix_podarray
branch
from
4256478 to
82236d4
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33949 ms |
TPC-DS: Total hot run time: 186179 ms |
ClickBench: Total hot run time: 29.09 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
May 9, 2025
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
…e#50549) ### What problem does this PR solve? Based on apache#11740 The memory size tracked in Allocator is virtual memory. If the allocated memory is not fully used, the tracked virtual memory may be larger than the actual physical memory. `PODArray` does not memset 0 when allocated memory, the memory blocks allocated by `PODArray` (such as VOlapScanNode::_free_blocks) are usually used for memory reuse and will not be fully used. After this PR, the unused memory in `PODArray` is recorded in reserve to indicate that this part of the memory is not actually used.
16 tasks
yiguolei
pushed a commit
that referenced
this pull request
Jul 22, 2025
### What problem does this PR solve? #50549 may cause performance loss If capacity_bytes <= 256K, res_mem_growth = c_end_of_storage - c_res_mem. If capacity >= 512K: - If `c_end_of_storage - c_res_mem < TrackingGrowthMinSize`, then tracking to c_end_of_storage. - `c_end_new - c_res_mem` is the size of the physical memory growth, which is also the minimum tracking size of this time, `(((c_end_new - c_res_mem) >> 16) << 16)` is aligned down to 64K, assuming `capacity_bytes >= 512K`, so `(capacity_bytes >> 3)` is at least 64K, so `(((c_end_new - c_res_mem) >> 16) << 16) + (capacity_bytes() >> 3)` must be greater than `c_end_new - c_res_mem`. For example: - 256K < capacity <= 512K, it will only tracking twice, the second time `c_end_of_storage - c_res_mem < TrackingGrowthMinSize` is true, so it will tracking to c_end_of_storage. - capacity > 32M, `(((c_end_new - c_res_mem) >> 16) << 16)` align the increased physical memory size down to 64k, then add `(capacity_bytes() >> 3)` equals 2M, so `reset_resident_memory` is tracking an additional 2M, after that, physical memory growth within 2M does not need to reset_resident_memory again. so, when PODArray is expanded by power of 2, the memory is checked and tracked up to 8 times between each expansion, because each time additional tracking `(capacity_bytes() >> 3)`. after each reset_resident_memory, tracking_res_memory >= used_bytes;
w41ter
pushed a commit
to w41ter/incubator-doris
that referenced
this pull request
Jul 30, 2025
…2553) ### What problem does this PR solve? apache#50549 may cause performance loss If capacity_bytes <= 256K, res_mem_growth = c_end_of_storage - c_res_mem. If capacity >= 512K: - If `c_end_of_storage - c_res_mem < TrackingGrowthMinSize`, then tracking to c_end_of_storage. - `c_end_new - c_res_mem` is the size of the physical memory growth, which is also the minimum tracking size of this time, `(((c_end_new - c_res_mem) >> 16) << 16)` is aligned down to 64K, assuming `capacity_bytes >= 512K`, so `(capacity_bytes >> 3)` is at least 64K, so `(((c_end_new - c_res_mem) >> 16) << 16) + (capacity_bytes() >> 3)` must be greater than `c_end_new - c_res_mem`. For example: - 256K < capacity <= 512K, it will only tracking twice, the second time `c_end_of_storage - c_res_mem < TrackingGrowthMinSize` is true, so it will tracking to c_end_of_storage. - capacity > 32M, `(((c_end_new - c_res_mem) >> 16) << 16)` align the increased physical memory size down to 64k, then add `(capacity_bytes() >> 3)` equals 2M, so `reset_resident_memory` is tracking an additional 2M, after that, physical memory growth within 2M does not need to reset_resident_memory again. so, when PODArray is expanded by power of 2, the memory is checked and tracked up to 8 times between each expansion, because each time additional tracking `(capacity_bytes() >> 3)`. after each reset_resident_memory, tracking_res_memory >= used_bytes;
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Based on #11740
The memory size tracked in Allocator is virtual memory. If the allocated memory is not fully used, the tracked virtual memory may be larger than the actual physical memory.
PODArraydoes not memset 0 when allocated memory, the memory blocks allocated byPODArray(such as VOlapScanNode::_free_blocks) are usually used for memory reuse and will not be fully used.After this PR, the unused memory in
PODArrayis recorded in reserve to indicate that this part of the memory is not actually used.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)