[BUGFIX] Free up executor memory when reassigning new executor #19214
Conversation
|
Hey @access2rohit , Thanks for submitting the PR
CI supported jobs: [clang, windows-cpu, centos-gpu, website, windows-gpu, sanity, unix-cpu, edge, centos-cpu, unix-gpu, miscellaneous] Note: |
|
Looks like we only instantiate the object the first time because we want to check whether subgraph property is set, which has nothing to do with the executor really. We can define a variable like this and use it in the condition: https://github.com/apache/incubator-mxnet/blob/07e65e5cf3c629c78f206798f33a3529e95c0fc4/src/executor/graph_executor.cc#L58 |
access2rohit
force-pushed
the
fix_mem_leak
branch
from
07e65e5 to
3b0b9dc
Compare
|
@mseth10 Can you review again ? |
src/executor/graph_executor.cc
Outdated
| if (!exec->subgraph_property().empty()) { | ||
| const auto& backend_name = exec->subgraph_property(); | ||
| const auto& backend_name = dmlc::GetEnv("MXNET_SUBGRAPH_BACKEND", exec::GetDefaultSubgraphBackend()); | ||
| if (!backend_name.empty()) { |
There was a problem hiding this comment.
we should also include the check backend_name == "None" or we can do something like this before the condition https://github.com/apache/incubator-mxnet/blob/3b0b9dc9d865ee0b1456e8f400f6f4a25c77b5db/src/executor/graph_executor.cc#L59-L60
There was a problem hiding this comment.
I prefer we dont duplicate code/functionality if at all possible. Since we create exec with new, and then use it to get the backend_name, why dont we just call delete on exec before line 2080 when creating a new executor from the partitioned symbol?
There was a problem hiding this comment.
yeah I agree, we can just use delete exec for both functions
|
We also need to make this change for the SimpleBind API, it is implemented in a similar fashion https://github.com/apache/incubator-mxnet/blob/3b0b9dc9d865ee0b1456e8f400f6f4a25c77b5db/src/executor/graph_executor.cc#L2009 |
access2rohit
force-pushed
the
fix_mem_leak
branch
from
3b0b9dc to
9c9baaf
Compare
access2rohit
force-pushed
the
fix_mem_leak
branch
from
9c9baaf to
2cbef0a
Compare
samskalicky
left a comment
samskalicky
left a comment
There was a problem hiding this comment.
Thanks for the delete change, this looks very succinct!
4 participants
Description
Fixes memory leak caused due to not freeing up of graph executor object.
Checklist
Essentials