branch-3.0 [fix] (inverted index) Fix UTF-8 4-byte truncation issue and add configuration to control correct term writing (#48657)

[airborne12]

cherry pick from #48657

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[airborne12]

run buildall

[doris-robot]

TPC-H: Total hot run time: 41039 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0faccd4d28c4c04cb15c4a8b16fe3d51fa58f414, data reload: false

------ Round 1 ----------------------------------
q1	17555	6947	6823	6823
q2	2083	170	167	167
q3	10645	1137	1229	1137
q4	10523	837	783	783
q5	7761	2992	2913	2913
q6	232	138	137	137
q7	1004	639	611	611
q8	9368	2057	2085	2057
q9	6702	6578	6588	6578
q10	7059	2295	2281	2281
q11	463	265	264	264
q12	425	217	215	215
q13	17797	2997	2997	2997
q14	242	205	214	205
q15	512	472	449	449
q16	662	577	585	577
q17	1026	573	693	573
q18	7556	6871	6697	6697
q19	1395	1083	1160	1083
q20	496	205	202	202
q21	4230	3554	3298	3298
q22	1089	994	992	992
Total cold run time: 108825 ms
Total hot run time: 41039 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6770	6769	6753	6753
q2	340	235	245	235
q3	3028	3079	3063	3063
q4	2114	1907	1884	1884
q5	5798	5887	5835	5835
q6	219	135	134	134
q7	2331	1819	1863	1819
q8	3532	3653	3669	3653
q9	8871	9000	9015	9000
q10	3659	3614	3649	3614
q11	597	503	500	500
q12	827	610	594	594
q13	9557	3197	3155	3155
q14	298	269	280	269
q15	523	473	484	473
q16	726	631	652	631
q17	1952	1693	1683	1683
q18	8462	8139	7964	7964
q19	1849	1651	1782	1651
q20	2079	1843	1872	1843
q21	5666	5609	5544	5544
q22	1146	1054	1014	1014
Total cold run time: 70344 ms
Total hot run time: 61311 ms

[doris-robot]

TPC-DS: Total hot run time: 197820 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0faccd4d28c4c04cb15c4a8b16fe3d51fa58f414, data reload: false

query1	1281	912	912	912
query2	6242	2137	2083	2083
query3	10949	4501	4406	4406
query4	61777	31551	23212	23212
query5	4846	458	472	458
query6	394	168	170	168
query7	5479	316	321	316
query8	314	229	220	220
query9	8665	2593	2600	2593
query10	438	282	264	264
query11	16038	15323	16078	15323
query12	158	105	103	103
query13	1444	446	442	442
query14	10420	7277	6972	6972
query15	212	178	181	178
query16	7256	515	540	515
query17	1150	586	578	578
query18	1859	338	312	312
query19	201	165	165	165
query20	119	113	113	113
query21	206	103	103	103
query22	4681	4707	4605	4605
query23	34576	33843	33713	33713
query24	6361	2923	2897	2897
query25	547	437	432	432
query26	653	174	168	168
query27	1956	359	372	359
query28	4205	2485	2478	2478
query29	720	491	425	425
query30	242	160	164	160
query31	1023	836	825	825
query32	65	55	58	55
query33	418	296	293	293
query34	943	510	511	510
query35	864	744	742	742
query36	1107	980	1004	980
query37	117	68	72	68
query38	4142	4095	4106	4095
query39	1510	1486	1507	1486
query40	200	100	96	96
query41	49	48	47	47
query42	113	104	104	104
query43	555	505	493	493
query44	1192	819	833	819
query45	193	164	170	164
query46	1129	740	716	716
query47	2069	1929	1947	1929
query48	483	385	390	385
query49	714	402	398	398
query50	850	431	423	423
query51	7393	7357	7315	7315
query52	102	88	94	88
query53	266	192	189	189
query54	581	458	488	458
query55	80	78	79	78
query56	303	248	261	248
query57	1277	1165	1157	1157
query58	223	211	211	211
query59	3430	2957	2993	2957
query60	273	249	249	249
query61	104	102	109	102
query62	756	664	704	664
query63	213	185	181	181
query64	1424	698	650	650
query65	3280	3219	3230	3219
query66	673	295	297	295
query67	16121	15567	15715	15567
query68	4139	599	582	582
query69	431	269	259	259
query70	1202	1160	1133	1133
query71	350	284	263	263
query72	6339	4149	4128	4128
query73	777	348	354	348
query74	10506	9181	9290	9181
query75	3366	2632	2675	2632
query76	1840	1171	1088	1088
query77	489	265	269	265
query78	10887	9629	9554	9554
query79	1892	599	598	598
query80	832	464	430	430
query81	537	249	236	236
query82	189	86	86	86
query83	169	148	151	148
query84	289	78	89	78
query85	909	302	286	286
query86	389	292	279	279
query87	4412	4234	4249	4234
query88	4511	2392	2382	2382
query89	403	296	288	288
query90	2097	189	189	189
query91	184	149	149	149
query92	67	49	47	47
query93	1829	562	559	559
query94	840	304	289	289
query95	362	255	250	250
query96	616	281	288	281
query97	3375	3169	3176	3169
query98	214	208	197	197
query99	1560	1283	1296	1283
Total cold run time: 313626 ms
Total hot run time: 197820 ms

[doris-robot]

ClickBench: Total hot run time: 32.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0faccd4d28c4c04cb15c4a8b16fe3d51fa58f414, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.02
query3	0.24	0.06	0.06
query4	1.63	0.11	0.10
query5	0.53	0.51	0.49
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.02	0.03
query9	0.57	0.50	0.50
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.58	0.60
query14	2.85	2.84	2.87
query15	0.89	0.82	0.83
query16	0.38	0.38	0.38
query17	1.09	1.07	1.06
query18	0.23	0.22	0.21
query19	1.97	1.91	2.01
query20	0.01	0.01	0.01
query21	15.36	0.58	0.56
query22	2.92	2.33	2.06
query23	17.01	1.08	1.05
query24	3.34	0.94	1.19
query25	0.29	0.07	0.07
query26	0.49	0.14	0.13
query27	0.05	0.05	0.04
query28	10.10	0.48	0.46
query29	12.63	3.24	3.27
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.26	0.46	0.46
query33	2.97	2.98	3.01
query34	16.93	4.50	4.53
query35	4.54	4.50	4.59
query36	0.68	0.50	0.49
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.26 s
Total hot run time: 32.75 s

[airborne12]

run buildall

[doris-robot]

TPC-H: Total hot run time: 39838 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a55145e208902a662e1abbbba7f634724a960257, data reload: false

------ Round 1 ----------------------------------
q1	17574	6703	6583	6583
q2	2079	166	161	161
q3	10612	1101	1156	1101
q4	10564	760	712	712
q5	7730	2876	2865	2865
q6	221	137	134	134
q7	975	613	609	609
q8	9397	1982	2057	1982
q9	6638	6403	6346	6346
q10	7028	2269	2298	2269
q11	467	257	262	257
q12	400	209	210	209
q13	17800	2995	3001	2995
q14	239	203	210	203
q15	499	451	492	451
q16	684	579	591	579
q17	954	568	610	568
q18	7263	6541	6690	6541
q19	1379	1073	988	988
q20	469	211	197	197
q21	3957	3111	3294	3111
q22	1063	977	995	977
Total cold run time: 107992 ms
Total hot run time: 39838 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6613	6597	6677	6597
q2	326	233	227	227
q3	2866	2697	2876	2697
q4	2025	1792	1813	1792
q5	5773	5754	5713	5713
q6	208	130	129	129
q7	2213	1820	1848	1820
q8	3386	3556	3492	3492
q9	8925	8739	8913	8739
q10	3562	3518	3526	3518
q11	575	485	491	485
q12	801	596	605	596
q13	8263	3170	3208	3170
q14	288	266	284	266
q15	507	471	448	448
q16	714	649	627	627
q17	1836	1616	1625	1616
q18	8229	7706	7811	7706
q19	1695	1619	1517	1517
q20	2092	1826	1807	1807
q21	5398	5203	5387	5203
q22	1119	1048	1017	1017
Total cold run time: 67414 ms
Total hot run time: 59182 ms

[doris-robot]

TPC-DS: Total hot run time: 196403 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a55145e208902a662e1abbbba7f634724a960257, data reload: false

query1	1278	910	897	897
query2	6237	2186	2044	2044
query3	10833	4298	4252	4252
query4	60898	30363	23567	23567
query5	5182	448	442	442
query6	403	178	165	165
query7	5469	318	324	318
query8	309	234	224	224
query9	8765	2581	2584	2581
query10	466	272	279	272
query11	17665	15159	16027	15159
query12	170	104	105	104
query13	1425	459	438	438
query14	10933	7228	6750	6750
query15	211	179	191	179
query16	7217	479	486	479
query17	1159	600	589	589
query18	1611	331	321	321
query19	220	167	163	163
query20	116	111	110	110
query21	207	104	109	104
query22	4558	4348	4690	4348
query23	34573	33600	35048	33600
query24	6356	2922	2979	2922
query25	527	405	416	405
query26	657	172	168	168
query27	1745	352	363	352
query28	4055	2483	2446	2446
query29	673	440	445	440
query30	242	170	157	157
query31	1014	827	841	827
query32	69	54	59	54
query33	447	290	290	290
query34	933	524	516	516
query35	829	731	738	731
query36	1077	980	980	980
query37	115	70	69	69
query38	4115	4022	4004	4004
query39	1513	1462	1487	1462
query40	202	103	103	103
query41	52	50	51	50
query42	120	101	102	101
query43	541	499	504	499
query44	1184	826	821	821
query45	184	167	166	166
query46	1139	739	757	739
query47	2005	1912	1935	1912
query48	487	393	387	387
query49	719	399	395	395
query50	865	432	450	432
query51	7242	7188	7179	7179
query52	103	90	100	90
query53	285	184	183	183
query54	605	476	466	466
query55	81	79	78	78
query56	278	260	254	254
query57	1277	1186	1164	1164
query58	219	210	219	210
query59	3256	2964	2834	2834
query60	285	260	275	260
query61	127	151	105	105
query62	753	688	649	649
query63	218	190	186	186
query64	1316	665	633	633
query65	3295	3187	3164	3164
query66	686	295	326	295
query67	15907	15641	15570	15570
query68	3882	588	580	580
query69	429	271	272	271
query70	1178	1155	1136	1136
query71	352	259	251	251
query72	6146	4244	4036	4036
query73	779	359	356	356
query74	10289	8924	9026	8924
query75	3341	2642	2691	2642
query76	1883	1059	1032	1032
query77	540	274	272	272
query78	10505	9612	9552	9552
query79	1497	603	597	597
query80	1166	435	439	435
query81	518	237	238	237
query82	316	87	86	86
query83	161	139	141	139
query84	279	77	80	77
query85	986	290	285	285
query86	368	273	292	273
query87	4432	4309	4445	4309
query88	3865	2410	2380	2380
query89	421	294	287	287
query90	1973	193	189	189
query91	184	148	148	148
query92	61	52	52	52
query93	1608	566	559	559
query94	779	303	276	276
query95	363	258	249	249
query96	628	285	277	277
query97	3368	3160	3167	3160
query98	217	207	199	199
query99	1536	1303	1283	1283
Total cold run time: 311753 ms
Total hot run time: 196403 ms

[doris-robot]

ClickBench: Total hot run time: 32.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a55145e208902a662e1abbbba7f634724a960257, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.02
query3	0.23	0.07	0.06
query4	1.63	0.11	0.10
query5	0.52	0.51	0.52
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.49
query10	0.54	0.55	0.55
query11	0.13	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.59	0.59
query14	2.89	2.75	2.87
query15	0.90	0.82	0.82
query16	0.39	0.37	0.37
query17	1.06	1.02	1.02
query18	0.23	0.21	0.22
query19	1.88	1.77	2.01
query20	0.02	0.01	0.01
query21	15.36	0.59	0.57
query22	3.33	1.98	1.90
query23	16.90	0.93	0.95
query24	3.26	1.66	1.19
query25	0.19	0.20	0.13
query26	0.38	0.14	0.14
query27	0.04	0.04	0.04
query28	9.41	0.48	0.45
query29	12.61	3.25	3.23
query30	0.25	0.06	0.06
query31	2.87	0.37	0.38
query32	3.27	0.46	0.46
query33	2.97	3.00	2.96
query34	16.97	4.44	4.48
query35	4.56	4.49	4.47
query36	0.70	0.48	0.49
query37	0.10	0.06	0.07
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.59 s
Total hot run time: 32.39 s

[dataroaring]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.