[opt](memory) Optimize PODArray memory tracking performance #52553
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
yiguolei
reviewed
Jul 1, 2025
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33872 ms |
TPC-DS: Total hot run time: 184665 ms |
ClickBench: Total hot run time: 29.96 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33616 ms |
TPC-DS: Total hot run time: 185452 ms |
ClickBench: Total hot run time: 29.28 s |
xinyiZzz
force-pushed
the
20250630_fix_podarray
branch
4 times, most recently
from
3082a19 to
0e4882b
Compare
Contributor
Author
|
run buildall |
xinyiZzz
force-pushed
the
20250630_fix_podarray
branch
from
0e4882b to
3f2b45e
Compare
TPC-H: Total hot run time: 42597 ms |
xinyiZzz
force-pushed
the
20250630_fix_podarray
branch
from
3f2b45e to
d4ba0ed
Compare
Contributor
Author
|
run buildall |
xinyiZzz
force-pushed
the
20250630_fix_podarray
branch
from
d4ba0ed to
084ea2e
Compare
TPC-H: Total hot run time: 33403 ms |
TPC-DS: Total hot run time: 186616 ms |
ClickBench: Total hot run time: 29.33 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33819 ms |
TPC-DS: Total hot run time: 186104 ms |
ClickBench: Total hot run time: 32.85 s |
TPC-DS: Total hot run time: 189793 ms |
ClickBench: Total hot run time: 32.77 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34417 ms |
TPC-DS: Total hot run time: 189413 ms |
ClickBench: Total hot run time: 32.49 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
xinyiZzz
force-pushed
the
20250630_fix_podarray
branch
from
08920ac to
6b6ea35
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34599 ms |
TPC-DS: Total hot run time: 191119 ms |
ClickBench: Total hot run time: 33.03 s |
Contributor
Author
|
run buildall |
xinyiZzz
force-pushed
the
20250630_fix_podarray
branch
from
702549d to
48eb3f2
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34541 ms |
TPC-DS: Total hot run time: 189136 ms |
ClickBench: Total hot run time: 32.86 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Jul 22, 2025
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
w41ter
pushed a commit
to w41ter/incubator-doris
that referenced
this pull request
Jul 30, 2025
…2553) ### What problem does this PR solve? apache#50549 may cause performance loss If capacity_bytes <= 256K, res_mem_growth = c_end_of_storage - c_res_mem. If capacity >= 512K: - If `c_end_of_storage - c_res_mem < TrackingGrowthMinSize`, then tracking to c_end_of_storage. - `c_end_new - c_res_mem` is the size of the physical memory growth, which is also the minimum tracking size of this time, `(((c_end_new - c_res_mem) >> 16) << 16)` is aligned down to 64K, assuming `capacity_bytes >= 512K`, so `(capacity_bytes >> 3)` is at least 64K, so `(((c_end_new - c_res_mem) >> 16) << 16) + (capacity_bytes() >> 3)` must be greater than `c_end_new - c_res_mem`. For example: - 256K < capacity <= 512K, it will only tracking twice, the second time `c_end_of_storage - c_res_mem < TrackingGrowthMinSize` is true, so it will tracking to c_end_of_storage. - capacity > 32M, `(((c_end_new - c_res_mem) >> 16) << 16)` align the increased physical memory size down to 64k, then add `(capacity_bytes() >> 3)` equals 2M, so `reset_resident_memory` is tracking an additional 2M, after that, physical memory growth within 2M does not need to reset_resident_memory again. so, when PODArray is expanded by power of 2, the memory is checked and tracked up to 8 times between each expansion, because each time additional tracking `(capacity_bytes() >> 3)`. after each reset_resident_memory, tracking_res_memory >= used_bytes;
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
#50549 may cause performance loss
If capacity_bytes <= 256K, res_mem_growth = c_end_of_storage - c_res_mem.
If capacity >= 512K:
- If
c_end_of_storage - c_res_mem < TrackingGrowthMinSize, then tracking to c_end_of_storage.-
c_end_new - c_res_memis the size of the physical memory growth,which is also the minimum tracking size of this time,
(((c_end_new - c_res_mem) >> 16) << 16)is aligned down to 64K,assuming
capacity_bytes >= 512K, so(capacity_bytes >> 3)is at least 64K,so
(((c_end_new - c_res_mem) >> 16) << 16) + (capacity_bytes() >> 3)must be greater than
c_end_new - c_res_mem.so, when PODArray is expanded by power of 2,
the memory is checked and tracked up to 8 times between each expansion,
because each time additional tracking
(capacity_bytes() >> 3).after each reset_resident_memory, tracking_res_memory >= used_bytes;
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)