[pick20][opt](mow) reduce memory usage for mow table compaction

[cambyzju]

cherry-pick #36865 to branch-2.0

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[cambyzju]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[doris-robot]

TPC-H: Total hot run time: 50189 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 56be53894a29899bd98c38a11c58e7cbf59977cb, data reload: false

------ Round 1 ----------------------------------
q1	18023	4440	4376	4376
q2	2067	158	149	149
q3	10428	1908	1918	1908
q4	10188	1296	1362	1296
q5	8502	4017	3910	3910
q6	234	125	150	125
q7	2050	1618	1619	1618
q8	9308	2783	2765	2765
q9	10807	10352	10332	10332
q10	8638	3511	3552	3511
q11	426	253	243	243
q12	479	303	301	301
q13	18338	4010	3976	3976
q14	349	323	333	323
q15	505	465	456	456
q16	699	568	585	568
q17	1163	969	978	969
q18	7362	6956	6842	6842
q19	1822	1695	1652	1652
q20	553	312	297	297
q21	4494	4125	4162	4125
q22	525	450	447	447
Total cold run time: 116960 ms
Total hot run time: 50189 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4312	4327	4314	4314
q2	323	233	228	228
q3	4239	4170	4158	4158
q4	2774	2753	2751	2751
q5	7171	7088	7057	7057
q6	239	120	119	119
q7	3248	2863	2847	2847
q8	4372	4489	4511	4489
q9	17013	17004	16770	16770
q10	4298	4270	4301	4270
q11	745	705	697	697
q12	1032	855	865	855
q13	7309	3710	3765	3710
q14	453	431	422	422
q15	518	453	467	453
q16	739	684	679	679
q17	3872	3900	3815	3815
q18	8855	8799	8843	8799
q19	1747	1723	1663	1663
q20	2432	2162	2139	2139
q21	8591	8567	8433	8433
q22	1042	1036	972	972
Total cold run time: 85324 ms
Total hot run time: 79640 ms

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.88% (8112/21417)
Line Coverage: 29.54% (66432/224861)
Region Coverage: 29.00% (34228/118009)
Branch Coverage: 24.86% (17567/70658)
Coverage Report: http://coverage.selectdb-in.cc/coverage/56be53894a29899bd98c38a11c58e7cbf59977cb_56be53894a29899bd98c38a11c58e7cbf59977cb/report/index.html

[doris-robot]

TPC-DS: Total hot run time: 204334 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 56be53894a29899bd98c38a11c58e7cbf59977cb, data reload: false

query1	941	426	380	380
query2	6524	2717	2668	2668
query3	6923	211	206	206
query4	21417	18121	18092	18092
query5	19730	6587	6543	6543
query6	299	223	227	223
query7	4162	293	296	293
query8	437	432	446	432
query9	3109	2701	2608	2608
query10	422	296	301	296
query11	11342	10805	10700	10700
query12	124	84	73	73
query13	5600	698	681	681
query14	17419	13586	13407	13407
query15	419	242	252	242
query16	6462	296	258	258
query17	1743	1427	902	902
query18	2284	415	412	412
query19	204	155	151	151
query20	80	77	80	77
query21	191	93	102	93
query22	5218	5110	5051	5051
query23	32717	31940	32127	31940
query24	7041	6548	6538	6538
query25	527	418	426	418
query26	517	162	162	162
query27	1817	301	297	297
query28	6164	2370	2322	2322
query29	2906	2713	2671	2671
query30	237	164	169	164
query31	880	740	745	740
query32	71	63	60	60
query33	401	265	249	249
query34	848	467	480	467
query35	1103	921	894	894
query36	1259	1095	1290	1095
query37	89	62	59	59
query38	3123	2917	2932	2917
query39	1372	1349	1341	1341
query40	209	95	92	92
query41	47	45	44	44
query42	88	90	83	83
query43	928	681	651	651
query44	1110	729	738	729
query45	248	235	234	234
query46	1231	978	958	958
query47	1834	1845	1691	1691
query48	1023	708	697	697
query49	625	368	368	368
query50	887	611	589	589
query51	4697	4639	4714	4639
query52	92	84	90	84
query53	448	332	327	327
query54	2723	2454	2437	2437
query55	92	82	87	82
query56	239	225	201	201
query57	1156	1050	1076	1050
query58	218	199	205	199
query59	4251	4113	4104	4104
query60	211	201	206	201
query61	97	94	97	94
query62	829	458	463	458
query63	486	354	341	341
query64	2559	1536	1481	1481
query65	3642	3578	3560	3560
query66	797	386	375	375
query67	17181	15850	15536	15536
query68	8392	653	641	641
query69	583	352	340	340
query70	1600	1437	1305	1305
query71	397	308	339	308
query72	6547	3532	3520	3520
query73	736	317	313	313
query74	6250	5888	5819	5819
query75	4615	3662	3678	3662
query76	4784	1182	1185	1182
query77	661	253	251	251
query78	13093	11851	12764	11851
query79	12135	626	621	621
query80	1180	391	402	391
query81	503	243	231	231
query82	642	100	101	100
query83	167	134	128	128
query84	264	72	69	69
query85	960	325	326	325
query86	337	291	287	287
query87	3322	3013	3068	3013
query88	5657	2310	2304	2304
query89	353	282	264	264
query90	1980	201	220	201
query91	177	141	142	141
query92	60	53	51	51
query93	3478	568	584	568
query94	700	206	201	201
query95	1103	1070	1061	1061
query96	624	334	327	327
query97	6516	6354	6360	6354
query98	196	173	172	172
query99	2964	940	906	906
Total cold run time: 315046 ms
Total hot run time: 204334 ms

[doris-robot]

ClickBench: Total hot run time: 30.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 56be53894a29899bd98c38a11c58e7cbf59977cb, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.26	0.05	0.05
query4	1.77	0.08	0.07
query5	0.53	0.53	0.53
query6	1.24	0.61	0.60
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.50	0.47
query10	0.53	0.54	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.08
query13	0.62	0.61	0.62
query14	0.78	0.79	0.79
query15	0.77	0.75	0.78
query16	0.37	0.38	0.38
query17	1.01	0.99	1.03
query18	0.22	0.25	0.23
query19	1.97	1.78	1.86
query20	0.02	0.01	0.02
query21	15.45	0.56	0.54
query22	2.01	1.94	1.80
query23	17.29	1.08	1.03
query24	4.66	0.64	1.32
query25	0.33	0.08	0.04
query26	0.57	0.15	0.14
query27	0.05	0.03	0.04
query28	8.52	0.73	0.70
query29	12.71	2.37	2.35
query30	0.62	0.54	0.53
query31	2.81	0.39	0.37
query32	3.38	0.49	0.48
query33	3.09	3.11	3.08
query34	15.23	4.79	4.78
query35	4.84	4.84	4.85
query36	1.06	1.01	1.01
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.01	0.01
Total cold run time: 103.98 s
Total hot run time: 30.65 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 56be53894a29899bd98c38a11c58e7cbf59977cb with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

[cambyzju]

run external

[cambyzju]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[doris-robot]

TPC-H: Total hot run time: 50341 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f618f87adc4860245daf2f3d0402294526fd903, data reload: false

------ Round 1 ----------------------------------
q1	17697	4980	4392	4392
q2	2076	164	151	151
q3	10529	1897	1939	1897
q4	10347	1274	1338	1274
q5	8460	3984	3965	3965
q6	261	126	125	125
q7	2063	1606	1591	1591
q8	9559	2761	2728	2728
q9	10980	10612	10488	10488
q10	8645	3537	3547	3537
q11	424	247	253	247
q12	470	304	308	304
q13	18346	3959	3995	3959
q14	374	329	328	328
q15	505	464	460	460
q16	653	566	569	566
q17	1148	967	972	967
q18	7295	6987	6963	6963
q19	1800	1616	1630	1616
q20	529	306	317	306
q21	4517	4149	4053	4053
q22	519	440	424	424
Total cold run time: 117197 ms
Total hot run time: 50341 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4316	4274	4299	4274
q2	319	226	220	220
q3	4192	4141	4168	4141
q4	2739	2737	2742	2737
q5	7255	7189	7171	7171
q6	244	121	121	121
q7	3296	2849	2800	2800
q8	4370	4493	4540	4493
q9	17467	17058	17270	17058
q10	4254	4272	4256	4256
q11	738	693	688	688
q12	1044	864	869	864
q13	7645	3716	3731	3716
q14	455	429	427	427
q15	505	455	454	454
q16	736	678	667	667
q17	3907	3806	3930	3806
q18	8924	8765	8751	8751
q19	1703	1745	1653	1653
q20	2383	2167	2146	2146
q21	8534	8459	8498	8459
q22	1084	990	953	953
Total cold run time: 86110 ms
Total hot run time: 79855 ms

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.89% (8115/21417)
Line Coverage: 29.56% (66467/224862)
Region Coverage: 29.03% (34257/118012)
Branch Coverage: 24.90% (17591/70660)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3f618f87adc4860245daf2f3d0402294526fd903_3f618f87adc4860245daf2f3d0402294526fd903/report/index.html

[doris-robot]

TPC-DS: Total hot run time: 203146 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f618f87adc4860245daf2f3d0402294526fd903, data reload: false

query1	932	419	379	379
query2	6524	2840	2668	2668
query3	6920	207	201	201
query4	20421	18090	18015	18015
query5	19743	6525	6528	6525
query6	299	220	233	220
query7	4152	304	318	304
query8	414	471	395	395
query9	3083	2649	2594	2594
query10	405	315	311	311
query11	11434	10621	10701	10621
query12	122	77	75	75
query13	5594	703	698	698
query14	17289	13454	13240	13240
query15	370	240	249	240
query16	6482	287	263	263
query17	1719	1483	869	869
query18	2330	414	405	405
query19	209	151	151	151
query20	80	78	84	78
query21	187	97	94	94
query22	5239	5084	5050	5050
query23	32484	32063	31914	31914
query24	7042	6534	6579	6534
query25	520	428	417	417
query26	541	158	157	157
query27	1879	288	294	288
query28	6158	2371	2308	2308
query29	2922	2711	2663	2663
query30	244	166	167	166
query31	922	709	736	709
query32	67	67	61	61
query33	412	263	242	242
query34	843	480	501	480
query35	1144	959	949	949
query36	1396	1055	1056	1055
query37	91	58	58	58
query38	3129	2939	2982	2939
query39	1375	1335	1324	1324
query40	209	97	92	92
query41	45	44	43	43
query42	80	86	79	79
query43	664	870	659	659
query44	1113	724	731	724
query45	254	235	237	235
query46	1235	947	942	942
query47	1800	1626	1708	1626
query48	1028	718	700	700
query49	617	367	367	367
query50	873	609	622	609
query51	4781	4674	4774	4674
query52	95	72	80	72
query53	443	328	325	325
query54	2651	2490	2486	2486
query55	86	87	82	82
query56	242	218	228	218
query57	1194	1136	1109	1109
query58	226	207	214	207
query59	4216	3767	3915	3767
query60	218	203	203	203
query61	101	96	101	96
query62	889	499	461	461
query63	485	350	340	340
query64	2601	1555	1496	1496
query65	3631	3555	3556	3555
query66	783	379	382	379
query67	18439	15310	15757	15310
query68	8465	686	640	640
query69	576	344	351	344
query70	1563	1481	1339	1339
query71	409	311	324	311
query72	6497	3484	3527	3484
query73	740	324	314	314
query74	6367	5800	5793	5793
query75	4594	3749	3701	3701
query76	4737	1163	1099	1099
query77	623	258	253	253
query78	12721	11718	11958	11718
query79	8819	650	637	637
query80	2763	410	401	401
query81	514	235	230	230
query82	1709	102	97	97
query83	179	135	131	131
query84	265	71	75	71
query85	1361	319	327	319
query86	363	287	291	287
query87	3217	2996	3064	2996
query88	5173	2277	2314	2277
query89	389	303	273	273
query90	1745	209	218	209
query91	204	139	165	139
query92	58	53	52	52
query93	6200	538	552	538
query94	913	218	216	216
query95	1140	1072	1076	1072
query96	649	322	326	322
query97	6557	6354	6398	6354
query98	203	182	178	178
query99	2949	835	955	835
Total cold run time: 316778 ms
Total hot run time: 203146 ms

[doris-robot]

ClickBench: Total hot run time: 31.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f618f87adc4860245daf2f3d0402294526fd903, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.04
query4	1.80	0.08	0.08
query5	0.54	0.53	0.52
query6	1.32	0.64	0.60
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.48	0.49
query10	0.55	0.54	0.55
query11	0.12	0.08	0.09
query12	0.11	0.08	0.08
query13	0.62	0.62	0.62
query14	0.79	0.78	0.79
query15	0.78	0.76	0.76
query16	0.36	0.36	0.37
query17	0.97	1.01	1.01
query18	0.23	0.24	0.24
query19	1.91	1.85	1.83
query20	0.01	0.00	0.01
query21	15.46	0.57	0.53
query22	2.34	2.04	1.65
query23	17.17	1.10	1.14
query24	6.07	1.50	1.34
query25	0.40	0.07	0.06
query26	0.74	0.16	0.15
query27	0.05	0.03	0.04
query28	6.45	0.77	0.78
query29	12.61	2.19	2.19
query30	0.66	0.53	0.53
query31	2.80	0.39	0.38
query32	3.36	0.49	0.50
query33	3.06	3.06	3.11
query34	15.29	4.81	4.80
query35	4.84	4.87	4.88
query36	1.04	1.03	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.15	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.76 s
Total hot run time: 31.31 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3f618f87adc4860245daf2f3d0402294526fd903 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

[zhannngchen]

LGTM