-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[MemTracker] make all MemTrackers shared #4135
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This reverts commit 360deb3
|
Hello, I have two questions to ask。
|
As metioned in the first comment, this pr is for showing all MemTracker on BE's website #3714 , like kudu. So we need to make all MemTrackers not owned by anyone. If we get it |
So the mainly reason use |
Yes. We had met a lot of problems of memory, they usually are accompanied with large MemPool consumption. But we can't figure out, MemPool consumption is the total metric. I think showing MemTracker is valuable. |
The lifecycle of MemTracker for a query is clear。Now RunningProfile don't use shared ptr, also can show running statistics of query. So it 's really necessary to shared_ptr to manage lifecycles?Can it be a tree structure like the runningprofile? |
It's not for the lifecycle of MemTracker. How do you know the exact mem consumption of one query? What if the MemTracker node destoryed, when we are traversing the tree? |
after making MemTracker shared(#4135), some code haven't been fixed, and add some useless ut back to build. Fixed in this pr.
…g usage (apache#4345) After PR: apache#4135, If a mem tracker has parent, it should be created by 'CreateTracker'. So I removed other unused constructors. And also fix the bug described in apache#4344
…g usage (apache#4345) After PR: apache#4135, If a mem tracker has parent, it should be created by 'CreateTracker'. So I removed other unused constructors. And also fix the bug described in apache#4344
…g usage (apache#4345) After PR: apache#4135, If a mem tracker has parent, it should be created by 'CreateTracker'. So I removed other unused constructors. And also fix the bug described in apache#4344
We make all MemTrackers shared, in order to show MemTracker real-time consumptions on the web.
As follows:
1. nearly all MemTracker raw ptr -> shared_ptr
2. Use CreateTracker() to create new MemTracker(in order to add itself to its parent)
3. RowBatch & MemPool still use raw ptrs of MemTracker, it's easy to ensure RowBatch & MemPool destructor exec
before MemTracker's destructor. So we don't change these code.
4. MemTracker can use RuntimeProfile's counter to calc consumption. So RuntimeProfile's counter need to be shared
too. We add a shared counter pool to store the shared counter, don't change other counters of RuntimeProfile.
Note that, this PR doesn't change the MemTracker tree structure. So there still have some orphan trackers, e.g. RowBlockV2's MemTracker. If you find some shared MemTrackers are little memory consumption & too time-consuming, you could make them be the orphan, then it's fine to use the raw ptr.
Ref #3714
We make all MemTrackers shared, in order to show MemTracker real-time consumptions on the web.
As follows:
Note that, this PR doesn't change the MemTracker tree structure. So there still have some orphan trackers, e.g. RowBlockV2's MemTracker. If you find some shared MemTrackers are little mem consumption & too time-consuming, you could make them be the orphan, then it's fine to use the raw ptr.