[feature-wip] (memory tracker) (step3) Switch TLS mem tracker to separate more detailed memory usage #8605

xinyiZzz · 2022-03-22T18:07:43Z

Proposed changes

Issue Number: close #7196 (step 3.1/3)

Problem Summary:

In pr #8476, all memory usage of a process is recorded in the process mem tracker, and all memory usage of a query is recorded in the query mem tracker, and it is still necessary to manually call transfer to to track the cached memory size.

We hope to separate out more detailed memory usage based on Hook TCMalloc new/delete + TLS mem tracker.

In this pr, the more detailed mem tracker is switched to TLS, which automatically and accurately counts more detailed memory usage than before.

Checklist(Required)

Does it affect the original behavior: (Yes)
Has unit tests been added: (Yes)
Has document been added or modified: (No)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments

Added the method of switching mem tracker to TLS;

More accurate statistics of cache memory size. It is no longer necessary to manually calculate the memory size of each add/free cache, which can easily miss memory such as vector resize.
(TODO) Separate out the memory usage of each operator from the query mem tracker.

Added callback behavior when switching TLS mem tracker consume fails;

Replace all methods of manually determining whether the instance mem tracker has exceeded limit in the loop, such as join node and agg node.

Statistical effect:

Initially (after BE restart)
Query is executing
After Query is over
Load is executing
Note: LoadChannelMgr consumption is 0, because this pr has not really called switch tls mem tracker, and the next pr will be modified in batches.
After Load is over

Note: If QueryPool and LoadPool are found to have negative values for a short time, this is normal, refer to the comments in MemTrackerTaskPool::logout_task_mem_tracker.

be/src/exec/cross_join_node.cpp

be/src/exec/except_node.cpp

be/src/exec/intersect_node.cpp

be/src/runtime/thread_mem_tracker_mgr.h

be/src/vec/exec/vcross_join_node.cpp

be/src/vec/exec/vset_operation_node.cpp

yangzhg · 2022-03-23T07:21:42Z

be/src/runtime/thread_mem_tracker_mgr.cpp

-            nullptr, "In TCMalloc Hook, " + _consume_err_call_back.action_type, mem_usage, st);
-    if (_consume_err_call_back.call_back_func != nullptr) {
-        _consume_err_call_back.call_back_func();
+            nullptr, "In TCMalloc Hook, " + _consume_err_cb.cancel_msg, mem_usage, st);


Suggested change

nullptr, "In TCMalloc Hook, " + _consume_err_cb.cancel_msg, mem_usage, st);

nullptr, fmt::format("In TCMalloc Hook, {}", _consume_err_cb.cancel_msg), mem_usage, st);

fix~, for faster.

morningman · 2022-03-23T08:28:15Z

be/src/olap/lru_cache.cpp

 }

 void ShardedLRUCache::release(Handle* handle) {
+    SCOPED_SWITCH_THREAD_LOCAL_MEM_TRACKER(_mem_tracker);


Can we add a metric counter for this SWITCH operation?
I want to know how many SWITCH may be called.

ok, I try to add.

morningman · 2022-03-24T03:23:27Z

be/src/olap/lru_cache.cpp

        _shards[s] = new LRUCache(type);
        _shards[s]->set_capacity(per_shard);
    }
+    thread_local_ctx.get()->_thread_mem_tracker_mgr->clear_untracked_mems();


Why clear untracked mem here?
you can add some comment.

After the lru cache is created in the main thread, the main thread will not switch the lru cache mem tracker again.

After the non-query thread switches the mem tracker, if the thread will not switch the mem tracker again in the short term, can consider manually clear_untracked_mems.

The query thread will automatically clear_untracked_mems when detach_task.

morningman

LGTM

github-actions · 2022-03-24T03:51:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2022-03-24T03:51:58Z

PR approved by anyone and no changes requested.

morningman · 2022-03-24T06:29:02Z

merge for quick test

…rate more detailed memory usage (#8669) Based on #8605, Separate out the memory usage of each operator from the Query/Load/StorageEngine mem tracker.

…rate more detailed memory usage (apache#8669) Based on apache#8605, Separate out the memory usage of each operator from the Query/Load/StorageEngine mem tracker.

switch_tls_tracker

133986a

morningman added the dev/backlog waiting to be merged in future dev branch label Mar 23, 2022

yangzhg reviewed Mar 23, 2022

View reviewed changes

be/src/exec/cross_join_node.cpp Outdated Show resolved Hide resolved

yangzhg reviewed Mar 23, 2022

View reviewed changes

fix details

c3c98c6

morningman reviewed Mar 23, 2022

View reviewed changes

morningman added the area/memory-consumption label Mar 23, 2022

fix comment

b814868

xinyiZzz force-pushed the switch_tls_tracker branch from 33e77da to b814868 Compare March 23, 2022 13:46

fix buffer_allocator

019eb46

morningman reviewed Mar 24, 2022

View reviewed changes

add clear commet

eb818f8

xinyiZzz force-pushed the switch_tls_tracker branch from 0ac01eb to eb818f8 Compare March 24, 2022 03:48

morningman approved these changes Mar 24, 2022

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 24, 2022

github-actions bot added the reviewed label Mar 24, 2022

morningman merged commit aaaaae5 into apache:master Mar 24, 2022

xinyiZzz mentioned this pull request Mar 25, 2022

[feature-wip] (memory tracker) (step4) Switch TLS mem tracker to separate more detailed memory usage #8669

Merged

xinyiZzz changed the title ~~[feature] (memory) Switch TLS mem tracker to separate more detailed memory usage~~ [feature-wip] (memory) Switch TLS mem tracker to separate more detailed memory usage (Part1) Mar 30, 2022

xinyiZzz changed the title ~~[feature-wip] (memory) Switch TLS mem tracker to separate more detailed memory usage (Part1)~~ [feature-wip] (memory tracker) (step3) Switch TLS mem tracker to separate more detailed memory usage Apr 7, 2022

	nullptr, "In TCMalloc Hook, " + _consume_err_cb.cancel_msg, mem_usage, st);
	nullptr, fmt::format("In TCMalloc Hook, {}", _consume_err_cb.cancel_msg), mem_usage, st);

[feature-wip] (memory tracker) (step3) Switch TLS mem tracker to separate more detailed memory usage #8605

[feature-wip] (memory tracker) (step3) Switch TLS mem tracker to separate more detailed memory usage #8605

Uh oh!

Conversation

xinyiZzz commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem Summary:

Checklist(Required)

Further comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yangzhg Mar 23, 2022

Choose a reason for hiding this comment

Uh oh!

xinyiZzz Mar 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morningman Mar 23, 2022

Choose a reason for hiding this comment

Uh oh!

xinyiZzz Mar 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xinyiZzz Mar 24, 2022

Choose a reason for hiding this comment

Uh oh!

morningman Mar 24, 2022

Choose a reason for hiding this comment

Uh oh!

xinyiZzz Mar 24, 2022

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 24, 2022

Uh oh!

github-actions bot commented Mar 24, 2022

Uh oh!

morningman commented Mar 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xinyiZzz commented Mar 22, 2022 •

edited

Loading

xinyiZzz Mar 23, 2022 •

edited

Loading

xinyiZzz Mar 23, 2022 •

edited

Loading