[improve](load) change memtable memory limiter policy #41018

kaijchen · 2024-09-19T12:25:19Z

Proposed changes

#40912 has changed meaning of write_mem in memtable memory limiter.
This PR is a followup to change the active memtable flush policy accordingly.

It also changed:

The amount of memtable writers selected in one flush.
The memtable writers are selected in orders of its size.

doris-robot · 2024-09-19T12:25:24Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

kaijchen · 2024-09-19T12:25:27Z

run buildall

doris-robot · 2024-09-19T14:07:22Z

TeamCity be ut coverage result:
Function Coverage: 37.32% (9591/25699)
Line Coverage: 28.71% (79276/276109)
Region Coverage: 28.17% (41038/145657)
Branch Coverage: 24.80% (20910/84318)
Coverage Report: http://coverage.selectdb-in.cc/coverage/44b63b632c25b76a91fa28f5d4e687dad4dd33cb_44b63b632c25b76a91fa28f5d4e687dad4dd33cb/report/index.html

xinyiZzz

LGTM

github-actions · 2024-09-19T17:05:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-19T17:05:28Z

PR approved by anyone and no changes requested.

yiguolei · 2024-09-20T03:11:09Z

be/src/olap/memtable_memory_limiter.cpp

+        }
+        int64_t mem = w->active_memtable_mem_consumption();
+        if (mem < sort_mem * 0.9) {
+            // if the memtable writer just got flushed, don't flush it again


when will reach this code？

## Proposed changes #40912 has changed meaning of `write_mem` in memtable memory limiter. This PR is a followup to change the active memtable flush policy accordingly. It also changed: 1. The amount of memtable writers selected in one flush. 2. The memtable writers are selected in orders of its size.

…#41018)" This reverts commit 34ab55f.

## Proposed changes #41018 used priority queue when selecting memtables to flush. But the compare function is wrong and causing the order to be the opposite. > Note that the Compare parameter is defined such that it returns true if its first argument comes before its second argument in a weak ordering. But because the priority queue outputs largest elements first, the elements that "come before" are actually output last. That is, the front of the queue contains the "last" element according to the weak ordering imposed by Compare. This PR fixes the compare function to make larger memtables come front.

## Proposed changes apache#40912 has changed meaning of `write_mem` in memtable memory limiter. This PR is a followup to change the active memtable flush policy accordingly. It also changed: 1. The amount of memtable writers selected in one flush. 2. The memtable writers are selected in orders of its size.

…he#41278) ## Proposed changes apache#41018 used priority queue when selecting memtables to flush. But the compare function is wrong and causing the order to be the opposite. > Note that the Compare parameter is defined such that it returns true if its first argument comes before its second argument in a weak ordering. But because the priority queue outputs largest elements first, the elements that "come before" are actually output last. That is, the front of the queue contains the "last" element according to the weak ordering imposed by Compare. This PR fixes the compare function to make larger memtables come front.

…41018) (#42144) backport #41018, #41278, #41245

This PR is a followup to change the active memtable flush policy accordingly. It also changed: 1. The amount of memtable writers selected in one flush. 2. The memtable writers are selected in orders of its size.

…he#41278) ## Proposed changes apache#41018 used priority queue when selecting memtables to flush. But the compare function is wrong and causing the order to be the opposite. > Note that the Compare parameter is defined such that it returns true if its first argument comes before its second argument in a weak ordering. But because the priority queue outputs largest elements first, the elements that "come before" are actually output last. That is, the front of the queue contains the "last" element according to the weak ordering imposed by Compare. This PR fixes the compare function to make larger memtables come front.

This PR is a followup to change the active memtable flush policy accordingly. It also changed: 1. The amount of memtable writers selected in one flush. 2. The memtable writers are selected in orders of its size.

…he#41278) ## Proposed changes apache#41018 used priority queue when selecting memtables to flush. But the compare function is wrong and causing the order to be the opposite. > Note that the Compare parameter is defined such that it returns true if its first argument comes before its second argument in a weak ordering. But because the priority queue outputs largest elements first, the elements that "come before" are actually output last. That is, the front of the queue contains the "last" element according to the weak ordering imposed by Compare. This PR fixes the compare function to make larger memtables come front.

This PR is a followup to change the active memtable flush policy accordingly. It also changed: 1. The amount of memtable writers selected in one flush. 2. The memtable writers are selected in orders of its size.

…he#41278) ## Proposed changes apache#41018 used priority queue when selecting memtables to flush. But the compare function is wrong and causing the order to be the opposite. > Note that the Compare parameter is defined such that it returns true if its first argument comes before its second argument in a weak ordering. But because the priority queue outputs largest elements first, the elements that "come before" are actually output last. That is, the front of the queue contains the "last" element according to the weak ordering imposed by Compare. This PR fixes the compare function to make larger memtables come front.

…52906) Related PR: #41018 Problem Summary: Update the `_need_flush()` function to subtract both `_queue_mem_usage` and `_flush_mem_usage` when deciding how much memory needs to be flushed. Previously, we only subtracted `_queue_mem_usage`, which could lead to flushing more active memtables than necessary. This change ensures that only the required amount of active memtable memory is flushed, avoiding premature flushes. This helps prevent creating small segments, which can hurt performance and storage efficiency.

…pache#52906) Related PR: apache#41018 Problem Summary: Update the `_need_flush()` function to subtract both `_queue_mem_usage` and `_flush_mem_usage` when deciding how much memory needs to be flushed. Previously, we only subtracted `_queue_mem_usage`, which could lead to flushing more active memtables than necessary. This change ensures that only the required amount of active memtable memory is flushed, avoiding premature flushes. This helps prevent creating small segments, which can hurt performance and storage efficiency.

[improve](load) change memtable memory limiter policy

44b63b6

xinyiZzz approved these changes Sep 19, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024

github-actions bot added the reviewed label Sep 19, 2024

yiguolei reviewed Sep 20, 2024

View reviewed changes

yiguolei approved these changes Sep 20, 2024

View reviewed changes

yiguolei merged commit 34ab55f into apache:master Sep 20, 2024

kaijchen added a commit to kaijchen/doris that referenced this pull request Sep 24, 2024

Revert "[improve](load) change memtable memory limiter policy (apache…

87eca6a

…#41018)" This reverts commit 34ab55f.

kaijchen mentioned this pull request Sep 25, 2024

[fix](load) fix priority queue order in memtable memory limiter #41278

Merged

kaijchen mentioned this pull request Oct 21, 2024

[fix](load) memtable memory limiter policy changes followup #40912 (#41018) #42144

Merged

liaoxin01 pushed a commit that referenced this pull request Oct 21, 2024

[fix](load) memtable memory limiter policy changes followup #40912 (#…

5285cbc

…41018) (#42144) backport #41018, #41278, #41245

liaoxin01 added the dev/3.0.3-merged label Oct 21, 2024

kaijchen mentioned this pull request Nov 18, 2024

[improve](load) more accurate memtable memory usage and new flush policy #44135

Closed

kaijchen mentioned this pull request Jul 7, 2025

[opt](load) avoid over-flushing active memtables in memory limiter #52906

Merged

16 tasks

kaijchen deleted the limiter-policy branch July 8, 2025 02:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve](load) change memtable memory limiter policy #41018

[improve](load) change memtable memory limiter policy #41018

Uh oh!

kaijchen commented Sep 19, 2024

Uh oh!

doris-robot commented Sep 19, 2024

Uh oh!

kaijchen commented Sep 19, 2024

Uh oh!

doris-robot commented Sep 19, 2024

Uh oh!

xinyiZzz left a comment

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

yiguolei Sep 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[improve](load) change memtable memory limiter policy #41018

[improve](load) change memtable memory limiter policy #41018

Uh oh!

Conversation

kaijchen commented Sep 19, 2024

Proposed changes

Uh oh!

doris-robot commented Sep 19, 2024

Uh oh!

kaijchen commented Sep 19, 2024

Uh oh!

doris-robot commented Sep 19, 2024

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

github-actions bot commented Sep 19, 2024

Uh oh!

yiguolei Sep 20, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants