[Feature] Limit Doris's memory usage more accurately and reduce the occurrence of OOM

[yangzhg]

Search before asking

• I had searched in the issues and found no similar issues.

Description

Currently, the main reason that causes doris to exit abnormally is that it occupies too much memory and OOM is killed by the system. Although doris has a mem_limit configuration, the memory usage limit is not strict. As a result, some memory is not recorded and consumed in doris. There are two main parts of the rough:

1. The memory that can be tracked by memtracker is mainly occupied by various mempools
2. Memory that cannot be tracked through memtracker, mainly memory requested through new/malloc
   The current problem is as follows:
3. The memory that can be tracked by memtracker does not consider the total memory occupied by the current system when consuming, but only considers the mem limit of the current tracker. This situation is particularly prominent on the memtracker without memory display.
4. There are currently no restrictions on the memory requested through new/malloc

Use case

In order to solve the above two problems, what we need to do is as follows:

1. For the memory that can be tracked through the mem tracker, when you consume, check whether the used memory exceeds the system memory limit. If it exceeds the system memory limit, the application will fail directly.
2. For memory that cannot be tracked by the system, a hook for memory allocation needs to be set. Establish a mapping relationship between memory application and query. If the memory allocation exceeds the limit, the corresponding query will be cancelled.
3. Organize internal allocation relationships to reduce unnecessary memory allocation

Related issues

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct