[fix](memory) Fix jdk17 and jemalloc hook not compatible on some envs #34578

xinyiZzz · 2024-05-09T05:26:46Z

Proposed changes

Now in order to support malloc hook, compiling jemalloc will add tow compilation options --with-jemalloc-prefix=je and --disable-cxx, then overwrite malloc/free as follows to allow BE to use jemalloc:

#define ALIAS(doris_fn) __attribute__((alias(#doris_fn), used))
void* malloc(size_t size) __THROW ALIAS(doris_malloc);
void free(void* p) __THROW ALIAS(doris_free);
void* realloc(void* p, size_t size) __THROW ALIAS(doris_realloc);
void* calloc(size_t n, size_t size) __THROW ALIAS(doris_calloc);
void cfree(void* ptr) __THROW ALIAS(doris_cfree);

but after such overwrite like this, Doris BE on jdk17 cannot be started in some environments.
there are three solutions:

Modify overwrite malloc/free, fixed in Use jemalloc's experimental features to hook jemalloc #33897
Modify BE dynamically loads JVM, fixed in [Fix](jdk) Fix jdk17 crash on some envs. #32865
Commit of this PR, modify jemalloc compilation options, after remove --with-jemalloc-prefix=je, there is not need to overwrite malloc/free, BE will use Jemalloc by default.

So, if need to use memory hook, this PR still cannot solve this problem.
Looking forward to modify BE dynamically loads JVM, which will not only solve this problem, also fix other crashes caused by BE use JVM.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-09T05:26:51Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xinyiZzz · 2024-05-09T07:03:09Z

run buildall

xinyiZzz · 2024-05-09T08:18:23Z

run buildall

github-actions · 2024-05-09T08:18:47Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In build.sh line 364:
if [ ! -f "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" ];then
   ^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ ! -f "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" ]];then

For more information:
  https://www.shellcheck.net/wiki/SC2292 -- Prefer [[ ]] over [ ] for tests i...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- build.sh.orig
+++ build.sh
@@ -361,35 +361,35 @@
 if [[ -z "${USE_JEMALLOC}" ]]; then
     USE_JEMALLOC='ON'
 fi
-if [ ! -f "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" ];then
-# compatible with old thirdparty
-if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
-    USE_JEMALLOC_HOOK='ON'
-fi
+if [ ! -f "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" ]; then
+    # compatible with old thirdparty
+    if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
+        USE_JEMALLOC_HOOK='ON'
+    fi
 else
-if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
-    USE_JEMALLOC_HOOK='OFF'
+    if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
+        USE_JEMALLOC_HOOK='OFF'
+    fi
+    # update jemalloc prefix
+    rm -rf "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
+    rm -rf "${TP_LIB_DIR}/libjemalloc_doris.a"
+    rm -rf "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
+    rm -rf "${TP_INCLUDE_DIR}/rocksdb"
+    rm -rf "${TP_LIB_DIR}/librocksdb.a"
+    if [[ "${USE_JEMALLOC_HOOK}" == "ON" ]]; then
+        cp "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
+        cp "${TP_LIB_DIR}/libjemalloc_doris_with_prefix.a" "${TP_LIB_DIR}/libjemalloc_doris.a"
+        cp "${TP_LIB_DIR}/libjemalloc_doris_with_prefix_pic.a" "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
+        cp "${TP_LIB_DIR}/librocksdb_jemalloc_with_prefix.a" "${TP_LIB_DIR}/librocksdb.a"
+        cp -r "${TP_INCLUDE_DIR}/rocksdb_jemalloc_with_prefix" "${TP_INCLUDE_DIR}/rocksdb"
+    else
+        cp "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_no_prefix.h" "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
+        cp "${TP_LIB_DIR}/libjemalloc_doris_no_prefix.a" "${TP_LIB_DIR}/libjemalloc_doris.a"
+        cp "${TP_LIB_DIR}/libjemalloc_doris_no_prefix_pic.a" "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
+        cp "${TP_LIB_DIR}/librocksdb_jemalloc_no_prefix.a" "${TP_LIB_DIR}/librocksdb.a"
+        cp -r "${TP_INCLUDE_DIR}/rocksdb_jemalloc_no_prefix" "${TP_INCLUDE_DIR}/rocksdb"
+    fi
 fi
-# update jemalloc prefix
-rm -rf "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
-rm -rf "${TP_LIB_DIR}/libjemalloc_doris.a"
-rm -rf "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
-rm -rf "${TP_INCLUDE_DIR}/rocksdb"
-rm -rf "${TP_LIB_DIR}/librocksdb.a"
-if [[ "${USE_JEMALLOC_HOOK}" == "ON" ]]; then
-    cp "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
-    cp "${TP_LIB_DIR}/libjemalloc_doris_with_prefix.a" "${TP_LIB_DIR}/libjemalloc_doris.a"
-    cp "${TP_LIB_DIR}/libjemalloc_doris_with_prefix_pic.a" "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
-    cp "${TP_LIB_DIR}/librocksdb_jemalloc_with_prefix.a" "${TP_LIB_DIR}/librocksdb.a"
-    cp -r "${TP_INCLUDE_DIR}/rocksdb_jemalloc_with_prefix" "${TP_INCLUDE_DIR}/rocksdb"
-else
-    cp "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_no_prefix.h" "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
-    cp "${TP_LIB_DIR}/libjemalloc_doris_no_prefix.a" "${TP_LIB_DIR}/libjemalloc_doris.a"
-    cp "${TP_LIB_DIR}/libjemalloc_doris_no_prefix_pic.a" "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
-    cp "${TP_LIB_DIR}/librocksdb_jemalloc_no_prefix.a" "${TP_LIB_DIR}/librocksdb.a"
-    cp -r "${TP_INCLUDE_DIR}/rocksdb_jemalloc_no_prefix" "${TP_INCLUDE_DIR}/rocksdb"
-fi
-fi
 if [[ -z "${USE_BTHREAD_SCANNER}" ]]; then
     USE_BTHREAD_SCANNER='OFF'
 fi
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2024-05-09T08:24:23Z

clang-tidy review says "All clean, LGTM! 👍"

be/src/runtime/fragment_mgr.cpp

github-actions · 2024-05-09T08:45:17Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- build.sh.orig
+++ build.sh
@@ -362,34 +362,34 @@
     USE_JEMALLOC='ON'
 fi
 if [[ ! -f "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" ]]; then
-# compatible with old thirdparty
-if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
-    USE_JEMALLOC_HOOK='ON'
-fi
+    # compatible with old thirdparty
+    if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
+        USE_JEMALLOC_HOOK='ON'
+    fi
 else
-if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
-    USE_JEMALLOC_HOOK='OFF'
+    if [[ -z "${USE_JEMALLOC_HOOK}" ]]; then
+        USE_JEMALLOC_HOOK='OFF'
+    fi
+    # update jemalloc prefix
+    rm -rf "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
+    rm -rf "${TP_LIB_DIR}/libjemalloc_doris.a"
+    rm -rf "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
+    rm -rf "${TP_INCLUDE_DIR}/rocksdb"
+    rm -rf "${TP_LIB_DIR}/librocksdb.a"
+    if [[ "${USE_JEMALLOC_HOOK}" == "ON" ]]; then
+        cp "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
+        cp "${TP_LIB_DIR}/libjemalloc_doris_with_prefix.a" "${TP_LIB_DIR}/libjemalloc_doris.a"
+        cp "${TP_LIB_DIR}/libjemalloc_doris_with_prefix_pic.a" "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
+        cp "${TP_LIB_DIR}/librocksdb_jemalloc_with_prefix.a" "${TP_LIB_DIR}/librocksdb.a"
+        cp -r "${TP_INCLUDE_DIR}/rocksdb_jemalloc_with_prefix" "${TP_INCLUDE_DIR}/rocksdb"
+    else
+        cp "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_no_prefix.h" "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
+        cp "${TP_LIB_DIR}/libjemalloc_doris_no_prefix.a" "${TP_LIB_DIR}/libjemalloc_doris.a"
+        cp "${TP_LIB_DIR}/libjemalloc_doris_no_prefix_pic.a" "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
+        cp "${TP_LIB_DIR}/librocksdb_jemalloc_no_prefix.a" "${TP_LIB_DIR}/librocksdb.a"
+        cp -r "${TP_INCLUDE_DIR}/rocksdb_jemalloc_no_prefix" "${TP_INCLUDE_DIR}/rocksdb"
+    fi
 fi
-# update jemalloc prefix
-rm -rf "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
-rm -rf "${TP_LIB_DIR}/libjemalloc_doris.a"
-rm -rf "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
-rm -rf "${TP_INCLUDE_DIR}/rocksdb"
-rm -rf "${TP_LIB_DIR}/librocksdb.a"
-if [[ "${USE_JEMALLOC_HOOK}" == "ON" ]]; then
-    cp "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_with_prefix.h" "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
-    cp "${TP_LIB_DIR}/libjemalloc_doris_with_prefix.a" "${TP_LIB_DIR}/libjemalloc_doris.a"
-    cp "${TP_LIB_DIR}/libjemalloc_doris_with_prefix_pic.a" "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
-    cp "${TP_LIB_DIR}/librocksdb_jemalloc_with_prefix.a" "${TP_LIB_DIR}/librocksdb.a"
-    cp -r "${TP_INCLUDE_DIR}/rocksdb_jemalloc_with_prefix" "${TP_INCLUDE_DIR}/rocksdb"
-else
-    cp "${TP_INCLUDE_DIR}/jemalloc/jemalloc_doris_no_prefix.h" "${TP_INCLUDE_DIR}/jemalloc/jemalloc.h"
-    cp "${TP_LIB_DIR}/libjemalloc_doris_no_prefix.a" "${TP_LIB_DIR}/libjemalloc_doris.a"
-    cp "${TP_LIB_DIR}/libjemalloc_doris_no_prefix_pic.a" "${TP_LIB_DIR}/libjemalloc_doris_pic.a"
-    cp "${TP_LIB_DIR}/librocksdb_jemalloc_no_prefix.a" "${TP_LIB_DIR}/librocksdb.a"
-    cp -r "${TP_INCLUDE_DIR}/rocksdb_jemalloc_no_prefix" "${TP_INCLUDE_DIR}/rocksdb"
-fi
-fi
 if [[ -z "${USE_BTHREAD_SCANNER}" ]]; then
     USE_BTHREAD_SCANNER='OFF'
 fi
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

xinyiZzz · 2024-05-09T08:45:30Z

run buildall

github-actions · 2024-05-09T08:50:23Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-05-09T09:14:51Z

run buildall

github-actions · 2024-05-09T09:21:09Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-05-09T10:02:31Z

TPC-H: Total hot run time: 40982 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a860a9fe4d4162fe726a0c7d5b5c2d5f9d4471b, data reload: false

------ Round 1 ----------------------------------
q1	18025	4422	4226	4226
q2	2029	186	191	186
q3	10478	1153	1153	1153
q4	10149	757	711	711
q5	7514	2678	2639	2639
q6	215	128	129	128
q7	1012	566	562	562
q8	9220	2115	2060	2060
q9	9556	7137	7141	7137
q10	8860	3698	3730	3698
q11	454	243	240	240
q12	520	224	215	215
q13	17779	2971	2959	2959
q14	259	217	219	217
q15	511	466	479	466
q16	513	381	380	380
q17	972	680	795	680
q18	8014	7292	7413	7292
q19	3018	1572	1497	1497
q20	659	313	307	307
q21	5061	3983	3942	3942
q22	344	295	287	287
Total cold run time: 115162 ms
Total hot run time: 40982 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4292	4194	4182	4182
q2	365	254	267	254
q3	2966	2740	2721	2721
q4	1858	1595	1575	1575
q5	5299	5287	5288	5287
q6	212	121	124	121
q7	2253	1876	1851	1851
q8	3200	3343	3365	3343
q9	9066	9066	9040	9040
q10	3883	3686	3698	3686
q11	589	494	485	485
q12	782	611	578	578
q13	17450	3001	2930	2930
q14	276	271	271	271
q15	518	479	471	471
q16	457	420	417	417
q17	1773	1506	1471	1471
q18	7648	7599	7400	7400
q19	1696	1565	1560	1560
q20	1962	1779	1745	1745
q21	5105	4786	4897	4786
q22	576	485	502	485
Total cold run time: 72226 ms
Total hot run time: 54659 ms

doris-robot · 2024-05-09T10:14:33Z

TPC-DS: Total hot run time: 186689 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a860a9fe4d4162fe726a0c7d5b5c2d5f9d4471b, data reload: false

query1	922	363	349	349
query2	6445	2301	2416	2301
query3	6651	200	202	200
query4	24658	21267	21206	21206
query5	4143	411	418	411
query6	276	182	177	177
query7	4593	287	293	287
query8	259	189	185	185
query9	8655	2408	2408	2408
query10	438	239	255	239
query11	14631	14280	14274	14274
query12	136	91	86	86
query13	1639	362	371	362
query14	10526	8443	6677	6677
query15	254	158	180	158
query16	8165	261	258	258
query17	1875	554	552	552
query18	2118	285	280	280
query19	339	151	151	151
query20	94	85	89	85
query21	204	128	134	128
query22	5122	4968	4888	4888
query23	34146	33676	33638	33638
query24	11833	2948	2893	2893
query25	645	369	379	369
query26	1739	163	151	151
query27	2986	321	318	318
query28	7683	2054	2042	2042
query29	1029	642	616	616
query30	293	151	157	151
query31	985	763	750	750
query32	97	52	55	52
query33	755	254	256	254
query34	1118	485	490	485
query35	820	726	652	652
query36	1106	898	922	898
query37	280	67	70	67
query38	2969	2832	2821	2821
query39	1665	1608	1566	1566
query40	277	130	124	124
query41	44	41	42	41
query42	106	95	96	95
query43	595	520	541	520
query44	1198	722	733	722
query45	272	262	250	250
query46	1079	729	701	701
query47	1981	1889	1922	1889
query48	374	294	304	294
query49	1201	401	425	401
query50	776	386	395	386
query51	6789	6796	6737	6737
query52	102	93	89	89
query53	352	286	284	284
query54	1051	452	437	437
query55	74	73	74	73
query56	255	233	233	233
query57	1274	1179	1150	1150
query58	223	202	197	197
query59	3441	3269	3349	3269
query60	264	303	239	239
query61	89	87	84	84
query62	672	483	465	465
query63	302	277	276	276
query64	9741	7318	7365	7318
query65	3649	3077	3065	3065
query66	1361	342	334	334
query67	15772	14976	15041	14976
query68	8716	540	558	540
query69	571	330	313	313
query70	1426	1107	1112	1107
query71	507	273	266	266
query72	8539	2545	2359	2359
query73	1603	324	324	324
query74	6599	6254	6138	6138
query75	4718	2645	2593	2593
query76	5363	965	959	959
query77	621	258	260	258
query78	10797	10249	10408	10249
query79	7683	509	510	509
query80	1215	422	426	422
query81	494	220	221	220
query82	301	95	87	87
query83	197	163	164	163
query84	267	86	87	86
query85	954	264	258	258
query86	329	316	299	299
query87	3285	3103	3084	3084
query88	4495	2409	2413	2409
query89	543	379	381	379
query90	2048	183	188	183
query91	124	98	96	96
query92	57	47	47	47
query93	5928	526	511	511
query94	1263	180	197	180
query95	391	305	305	305
query96	600	268	266	266
query97	3137	2959	3000	2959
query98	235	216	209	209
query99	1248	877	885	877
Total cold run time: 309860 ms
Total hot run time: 186689 ms

doris-robot · 2024-05-09T10:37:13Z

TeamCity be ut coverage result:
Function Coverage: 35.69% (8983/25172)
Line Coverage: 27.34% (74214/271479)
Region Coverage: 26.58% (38375/144385)
Branch Coverage: 23.39% (19567/83664)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6a860a9fe4d4162fe726a0c7d5b5c2d5f9d4471b_6a860a9fe4d4162fe726a0c7d5b5c2d5f9d4471b/report/index.html

morningman

LGTM

github-actions · 2024-05-09T12:43:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-09T12:43:43Z

PR approved by anyone and no changes requested.

…apache#34578) Now in order to support malloc hook, compiling jemalloc will add tow compilation options --with-jemalloc-prefix=je and --disable-cxx, then overwrite malloc/free as follows to allow BE to use jemalloc: void* malloc(size_t size) __THROW ALIAS(doris_malloc); void free(void* p) __THROW ALIAS(doris_free); void* realloc(void* p, size_t size) __THROW ALIAS(doris_realloc); void* calloc(size_t n, size_t size) __THROW ALIAS(doris_calloc); void cfree(void* ptr) __THROW ALIAS(doris_cfree); but after such overwrite like this, Doris BE on jdk17 cannot be started in some environments. there are three solutions: Modify overwrite malloc/free, fixed in Use jemalloc's experimental features to hook jemalloc apache#33897 Modify BE dynamically loads JVM, fixed in [Fix](jdk) Fix jdk17 crash on some envs. apache#32865 Commit of this PR, modify jemalloc compilation options, after remove --with-jemalloc-prefix=je, there is not need to overwrite malloc/free, BE will use Jemalloc by default. So, if need to use memory hook, this PR still cannot solve this problem. Looking forward to modify BE dynamically loads JVM, which will not only solve this problem, also fix other crashes caused by BE use JVM.

…apache#34578) Now in order to support malloc hook, compiling jemalloc will add tow compilation options --with-jemalloc-prefix=je and --disable-cxx, then overwrite malloc/free as follows to allow BE to use jemalloc: #define ALIAS(doris_fn) __attribute__((alias(#doris_fn), used)) void* malloc(size_t size) __THROW ALIAS(doris_malloc); void free(void* p) __THROW ALIAS(doris_free); void* realloc(void* p, size_t size) __THROW ALIAS(doris_realloc); void* calloc(size_t n, size_t size) __THROW ALIAS(doris_calloc); void cfree(void* ptr) __THROW ALIAS(doris_cfree); but after such overwrite like this, Doris BE on jdk17 cannot be started in some environments. there are three solutions: Modify overwrite malloc/free, fixed in Use jemalloc's experimental features to hook jemalloc apache#33897 Modify BE dynamically loads JVM, fixed in [Fix](jdk) Fix jdk17 crash on some envs. apache#32865 Commit of this PR, modify jemalloc compilation options, after remove --with-jemalloc-prefix=je, there is not need to overwrite malloc/free, BE will use Jemalloc by default. So, if need to use memory hook, this PR still cannot solve this problem. Looking forward to modify BE dynamically loads JVM, which will not only solve this problem, also fix other crashes caused by BE use JVM.

## Proposed changes Revert #13660 `-l` and `dlopen` load Dynamic Link Library, will use the Allocator of main program, Doris use Jemalloc by default. but, `dlopen` load libjvm.so and use Jemalloc compiled with prefix, overwriting malloc/free will incompatible with libjvm.so. see #34578 for details. In addition, jemalloc not recommend `dlopen` to load Dynamic Link Library, `dlclose` will memory leak in that case. jemalloc/jemalloc#2404 jemalloc/jemalloc#1321 jemalloc/jemalloc#1890 `export LD_PRELOAD` to force `libjvm.so` use a separate `jemalloc.so` will not solve the problem, but may cause a new crash. ``` received by PID 2368 (TID 2391 OR 0x7f445cafb700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/selectdb-core/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/jdk/lib/server/libjvm.so 2# JVM_handle_linux_signal in /opt/jdk/lib/server/libjvm.so 3# 0x00007F448DB40400 in /lib64/libc.so.6 4# je_arena_dalloc_promoted at /root/selectdb-core/thirdparty/src/jemalloc-5.3.0/doris_build/../src/arena.c:1277 5# je_free_default at /root/selectdb-core/thirdparty/src/jemalloc-5.3.0/doris_build/../src/jemalloc.c:3014 6# __pthread_create_2_1 in /lib64/libpthread.so.0 7# os::create_thread(Thread*, os::ThreadType, unsigned long) in /opt/jdk/lib/server/libjvm.so 8# CompilerThread::CompilerThread(CompileQueue*, CompilerCounters*) in /opt/jdk/lib/server/libjvm.so 9# CompileBroker::make_thread(CompileBroker::ThreadType, _jobject*, CompileQueue*, AbstractCompiler*, JavaThread*) [clone .constprop.0] in /opt/jdk/lib/server/libjvm.so 10# CompileBroker::possibly_add_compiler_threads(JavaThread*) in /opt/jdk/lib/server/libjvm.so 11# CompileBroker::compiler_thread_loop() in /opt/jdk/lib/server/libjvm.so 12# JavaThread::thread_main_inner() in /opt/jdk/lib/server/libjvm.so 13# Thread::call_run() in /opt/jdk/lib/server/libjvm.so 14# thread_native_entry(Thread*) in /opt/jdk/lib/server/libjvm.so 15# start_thread in /lib64/libpthread.so.0 16# clone in /lib64/libc.so.6 ```

…ome envs (apache#34578)" This reverts commit 673c61a.

## Proposed changes Revert apache#13660 `-l` and `dlopen` load Dynamic Link Library, will use the Allocator of main program, Doris use Jemalloc by default. but, `dlopen` load libjvm.so and use Jemalloc compiled with prefix, overwriting malloc/free will incompatible with libjvm.so. see apache#34578 for details. In addition, jemalloc not recommend `dlopen` to load Dynamic Link Library, `dlclose` will memory leak in that case. jemalloc/jemalloc#2404 jemalloc/jemalloc#1321 jemalloc/jemalloc#1890 `export LD_PRELOAD` to force `libjvm.so` use a separate `jemalloc.so` will not solve the problem, but may cause a new crash. ``` received by PID 2368 (TID 2391 OR 0x7f445cafb700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/selectdb-core/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/jdk/lib/server/libjvm.so 2# JVM_handle_linux_signal in /opt/jdk/lib/server/libjvm.so 3# 0x00007F448DB40400 in /lib64/libc.so.6 4# je_arena_dalloc_promoted at /root/selectdb-core/thirdparty/src/jemalloc-5.3.0/doris_build/../src/arena.c:1277 5# je_free_default at /root/selectdb-core/thirdparty/src/jemalloc-5.3.0/doris_build/../src/jemalloc.c:3014 6# __pthread_create_2_1 in /lib64/libpthread.so.0 7# os::create_thread(Thread*, os::ThreadType, unsigned long) in /opt/jdk/lib/server/libjvm.so 8# CompilerThread::CompilerThread(CompileQueue*, CompilerCounters*) in /opt/jdk/lib/server/libjvm.so 9# CompileBroker::make_thread(CompileBroker::ThreadType, _jobject*, CompileQueue*, AbstractCompiler*, JavaThread*) [clone .constprop.0] in /opt/jdk/lib/server/libjvm.so 10# CompileBroker::possibly_add_compiler_threads(JavaThread*) in /opt/jdk/lib/server/libjvm.so 11# CompileBroker::compiler_thread_loop() in /opt/jdk/lib/server/libjvm.so 12# JavaThread::thread_main_inner() in /opt/jdk/lib/server/libjvm.so 13# Thread::call_run() in /opt/jdk/lib/server/libjvm.so 14# thread_native_entry(Thread*) in /opt/jdk/lib/server/libjvm.so 15# start_thread in /lib64/libpthread.so.0 16# clone in /lib64/libc.so.6 ```

…ome envs (apache#34578)" This reverts commit 673c61a.

Revert apache#13660 `-l` and `dlopen` load Dynamic Link Library, will use the Allocator of main program, Doris use Jemalloc by default. but, `dlopen` load libjvm.so and use Jemalloc compiled with prefix, overwriting malloc/free will incompatible with libjvm.so. see apache#34578 for details. In addition, jemalloc not recommend `dlopen` to load Dynamic Link Library, `dlclose` will memory leak in that case. jemalloc/jemalloc#2404 jemalloc/jemalloc#1321 jemalloc/jemalloc#1890 `export LD_PRELOAD` to force `libjvm.so` use a separate `jemalloc.so` will not solve the problem, but may cause a new crash. ``` received by PID 2368 (TID 2391 OR 0x7f445cafb700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/selectdb-core/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/jdk/lib/server/libjvm.so 2# JVM_handle_linux_signal in /opt/jdk/lib/server/libjvm.so 3# 0x00007F448DB40400 in /lib64/libc.so.6 4# je_arena_dalloc_promoted at /root/selectdb-core/thirdparty/src/jemalloc-5.3.0/doris_build/../src/arena.c:1277 5# je_free_default at /root/selectdb-core/thirdparty/src/jemalloc-5.3.0/doris_build/../src/jemalloc.c:3014 6# __pthread_create_2_1 in /lib64/libpthread.so.0 7# os::create_thread(Thread*, os::ThreadType, unsigned long) in /opt/jdk/lib/server/libjvm.so 8# CompilerThread::CompilerThread(CompileQueue*, CompilerCounters*) in /opt/jdk/lib/server/libjvm.so 9# CompileBroker::make_thread(CompileBroker::ThreadType, _jobject*, CompileQueue*, AbstractCompiler*, JavaThread*) [clone .constprop.0] in /opt/jdk/lib/server/libjvm.so 10# CompileBroker::possibly_add_compiler_threads(JavaThread*) in /opt/jdk/lib/server/libjvm.so 11# CompileBroker::compiler_thread_loop() in /opt/jdk/lib/server/libjvm.so 12# JavaThread::thread_main_inner() in /opt/jdk/lib/server/libjvm.so 13# Thread::call_run() in /opt/jdk/lib/server/libjvm.so 14# thread_native_entry(Thread*) in /opt/jdk/lib/server/libjvm.so 15# start_thread in /lib64/libpthread.so.0 16# clone in /lib64/libc.so.6 ```

1

9ead678

xinyiZzz force-pushed the 20240428_fix_jemalloc branch from 9d7f140 to 9ead678 Compare May 9, 2024 07:03

2

5483a64

xinyiZzz requested a review from morningman May 9, 2024 08:18

xinyiZzz changed the title ~~[fix](memory) Fix jdk17 and jemalloc not compatible on some envs~~ [fix](memory) Fix jdk17 and jemalloc hook not compatible on some envs May 9, 2024

yiguolei reviewed May 9, 2024

View reviewed changes

be/src/runtime/fragment_mgr.cpp Outdated Show resolved Hide resolved

3

e990f30

5

6a860a9

morningman approved these changes May 9, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 9, 2024

github-actions bot added the reviewed label May 9, 2024

yiguolei approved these changes May 9, 2024

View reviewed changes

xinyiZzz merged commit 673c61a into apache:master May 10, 2024

xinyiZzz mentioned this pull request May 28, 2024

[fix](libjdk) Revert support loading libjvm at runtime #35557

Merged

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request May 31, 2024

Revert "[fix](memory) Fix jdk17 and jemalloc hook not compatible on s…

8651751

…ome envs (apache#34578)" This reverts commit 673c61a.

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request May 31, 2024

Revert "[fix](memory) Fix jdk17 and jemalloc hook not compatible on s…

a0e78b1

…ome envs (apache#34578)" This reverts commit 673c61a.

xinyiZzz mentioned this pull request Jul 22, 2024

[Bug] BE crash at JEMALLOC free #36641

Closed

3 tasks

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

xinyiZzz mentioned this pull request Dec 17, 2024

[revert](memory) Revert disable Jemalloc Hook #45507

Closed

16 tasks

[fix](memory) Fix jdk17 and jemalloc hook not compatible on some envs #34578

[fix](memory) Fix jdk17 and jemalloc hook not compatible on some envs #34578

Uh oh!

Conversation

xinyiZzz commented May 9, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented May 9, 2024

Uh oh!

xinyiZzz commented May 9, 2024

Uh oh!

xinyiZzz commented May 9, 2024

Uh oh!

github-actions bot commented May 9, 2024

sh-checker report

Uh oh!

github-actions bot commented May 9, 2024

Uh oh!

Uh oh!

github-actions bot commented May 9, 2024

sh-checker report

Uh oh!

xinyiZzz commented May 9, 2024

Uh oh!

github-actions bot commented May 9, 2024

Uh oh!

xinyiZzz commented May 9, 2024

Uh oh!

github-actions bot commented May 9, 2024

Uh oh!

doris-robot commented May 9, 2024

Uh oh!

doris-robot commented May 9, 2024

Uh oh!

doris-robot commented May 9, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 9, 2024

Uh oh!

github-actions bot commented May 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

`sh-checker report`

`sh-checker report`