[fix] (inverted index) Fix UTF-8 4-byte truncation issue and add configuration to control correct term writing

[airborne12]

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Added the enable_inverted_index_correct_term_write configuration as a compatibility option to fix the issue of term truncation caused by special characters. In some situations, a rollback may be necessary after an upgrade. To accommodate this, the initial goal is to set enable_inverted_index_correct_term_write=false, prioritizing compatibility with correct term reading before enabling correct term writing. Once reading is confirmed to be working correctly, the configuration can be switched to enable_inverted_index_correct_term_write=true to write non-truncated terms properly.

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[airborne12]

run buildall

[qidaye]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[zzzxl1993]

LGTM

[doris-robot]

TPC-H: Total hot run time: 32700 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 322080b0c9dc0e8337a22542c6fb759330fd5b34, data reload: false

------ Round 1 ----------------------------------
q1	17605	5290	5114	5114
q2	2054	312	169	169
q3	10462	1302	725	725
q4	10238	1022	528	528
q5	7880	2460	2360	2360
q6	187	173	133	133
q7	933	748	607	607
q8	9327	1293	1112	1112
q9	5031	4787	4915	4787
q10	6819	2320	1878	1878
q11	467	280	258	258
q12	346	349	220	220
q13	17780	3672	3131	3131
q14	237	243	217	217
q15	526	499	470	470
q16	643	601	578	578
q17	576	879	361	361
q18	6984	6414	6362	6362
q19	2635	963	538	538
q20	304	311	186	186
q21	2738	2171	1988	1988
q22	1080	1017	978	978
Total cold run time: 104852 ms
Total hot run time: 32700 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5310	5133	5151	5133
q2	245	328	239	239
q3	2163	2701	2325	2325
q4	1492	1822	1396	1396
q5	4265	4143	4127	4127
q6	206	164	123	123
q7	2003	1965	1751	1751
q8	2627	2636	2594	2594
q9	7261	7142	7244	7142
q10	3024	3180	2765	2765
q11	566	519	509	509
q12	653	755	629	629
q13	3496	3870	3275	3275
q14	290	284	268	268
q15	528	491	469	469
q16	623	683	647	647
q17	1133	1626	1310	1310
q18	7758	7504	7380	7380
q19	822	823	833	823
q20	1964	2010	1903	1903
q21	5503	5065	4906	4906
q22	1114	1043	1004	1004
Total cold run time: 53046 ms
Total hot run time: 50718 ms

[doris-robot]

TPC-DS: Total hot run time: 185277 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 322080b0c9dc0e8337a22542c6fb759330fd5b34, data reload: false

query1	994	426	383	383
query2	6527	1860	1867	1860
query3	6787	210	209	209
query4	26532	23757	23316	23316
query5	4367	671	519	519
query6	301	214	197	197
query7	4616	502	292	292
query8	295	252	235	235
query9	8615	2494	2498	2494
query10	462	314	256	256
query11	15696	15100	14930	14930
query12	160	115	110	110
query13	1645	499	401	401
query14	9428	6189	6662	6189
query15	203	188	174	174
query16	7132	642	477	477
query17	921	739	576	576
query18	1938	388	287	287
query19	189	181	154	154
query20	118	111	114	111
query21	211	119	101	101
query22	4187	4303	4259	4259
query23	34044	33089	33164	33089
query24	7736	2350	2386	2350
query25	526	456	390	390
query26	1238	272	152	152
query27	2120	501	324	324
query28	3971	2402	2402	2402
query29	750	534	428	428
query30	283	220	195	195
query31	927	864	765	765
query32	77	69	65	65
query33	583	360	323	323
query34	787	851	507	507
query35	785	838	764	764
query36	952	962	905	905
query37	121	98	77	77
query38	4134	4166	4102	4102
query39	1444	1430	1419	1419
query40	213	120	103	103
query41	58	58	74	58
query42	132	105	109	105
query43	498	509	480	480
query44	1278	812	796	796
query45	190	174	166	166
query46	828	1019	615	615
query47	1743	1786	1684	1684
query48	387	411	302	302
query49	811	510	436	436
query50	700	721	415	415
query51	4139	4207	4191	4191
query52	106	108	101	101
query53	230	273	189	189
query54	514	496	484	484
query55	80	81	75	75
query56	270	257	252	252
query57	1114	1167	1081	1081
query58	244	232	234	232
query59	2520	2617	2743	2617
query60	290	277	318	277
query61	118	115	114	114
query62	816	751	681	681
query63	233	193	185	185
query64	4341	1010	666	666
query65	4376	4333	4354	4333
query66	1141	407	303	303
query67	15977	15549	15123	15123
query68	8154	875	509	509
query69	473	295	250	250
query70	1235	1151	1085	1085
query71	458	300	263	263
query72	5757	3665	3830	3665
query73	797	749	356	356
query74	9302	9079	8764	8764
query75	3727	3199	2701	2701
query76	3606	1191	744	744
query77	787	366	288	288
query78	9982	10180	9236	9236
query79	2625	824	592	592
query80	620	517	446	446
query81	487	263	266	263
query82	692	125	96	96
query83	177	174	155	155
query84	286	89	73	73
query85	765	354	309	309
query86	384	295	287	287
query87	4358	4484	4324	4324
query88	3549	2215	2209	2209
query89	394	321	294	294
query90	1836	201	199	199
query91	143	138	114	114
query92	75	65	60	60
query93	1712	1038	575	575
query94	672	422	307	307
query95	351	262	258	258
query96	479	563	268	268
query97	3346	3378	3267	3267
query98	235	204	204	204
query99	1411	1531	1265	1265
Total cold run time: 274024 ms
Total hot run time: 185277 ms

[doris-robot]

ClickBench: Total hot run time: 31.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 322080b0c9dc0e8337a22542c6fb759330fd5b34, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.57	0.54	0.57
query6	1.22	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.03
query9	0.60	0.53	0.53
query10	0.57	0.57	0.58
query11	0.15	0.11	0.11
query12	0.15	0.12	0.13
query13	0.62	0.60	0.60
query14	2.70	2.69	2.70
query15	0.93	0.84	0.85
query16	0.38	0.38	0.38
query17	1.05	1.00	1.03
query18	0.22	0.20	0.19
query19	1.90	2.02	1.78
query20	0.01	0.01	0.02
query21	15.35	0.94	0.55
query22	0.75	1.21	0.74
query23	14.86	1.39	0.59
query24	6.76	1.44	1.32
query25	0.52	0.25	0.14
query26	0.67	0.16	0.13
query27	0.05	0.05	0.06
query28	9.73	0.86	0.43
query29	12.60	3.84	3.25
query30	0.25	0.09	0.07
query31	2.81	0.59	0.38
query32	3.27	0.55	0.47
query33	3.08	3.09	3.02
query34	15.78	5.15	4.52
query35	4.55	4.62	4.51
query36	0.68	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 105.32 s
Total hot run time: 31.29 s

[doris-robot]

BE UT Coverage Report

Increment line coverage 33.33% (1/3) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.95% (12280/26727)
Line Coverage	35.39% (103650/292888)
Region Coverage	34.56% (53107/153667)
Branch Coverage	30.24% (26886/88908)

[airborne12]

run buildall

[doris-robot]

TPC-H: Total hot run time: 32677 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8bc1ae63337a1898648841b09775cdffcfa62383, data reload: false

------ Round 1 ----------------------------------
q1	17609	5244	5125	5125
q2	2057	312	171	171
q3	10383	1326	721	721
q4	10220	1038	523	523
q5	7539	2398	2405	2398
q6	181	167	136	136
q7	926	740	632	632
q8	9303	1357	1130	1130
q9	4995	4757	4667	4667
q10	6832	2321	1901	1901
q11	490	276	273	273
q12	345	355	214	214
q13	17753	3694	3098	3098
q14	222	241	209	209
q15	559	491	481	481
q16	621	628	601	601
q17	585	879	335	335
q18	7261	6411	6471	6411
q19	1356	972	534	534
q20	319	322	198	198
q21	2903	2179	1938	1938
q22	1079	1033	981	981
Total cold run time: 103538 ms
Total hot run time: 32677 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5195	5160	5159	5159
q2	241	329	239	239
q3	2203	2688	2289	2289
q4	1428	1853	1360	1360
q5	4286	4130	4180	4130
q6	207	162	123	123
q7	1927	1945	1766	1766
q8	2636	2648	2590	2590
q9	7289	7221	7168	7168
q10	2961	3111	2675	2675
q11	557	508	489	489
q12	660	722	569	569
q13	3298	3741	3088	3088
q14	260	283	264	264
q15	520	478	471	471
q16	643	709	635	635
q17	1124	1602	1280	1280
q18	7369	7347	7091	7091
q19	792	851	971	851
q20	1936	2058	1872	1872
q21	5477	4938	5090	4938
q22	1155	1123	1030	1030
Total cold run time: 52164 ms
Total hot run time: 50077 ms

[doris-robot]

TPC-DS: Total hot run time: 186628 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8bc1ae63337a1898648841b09775cdffcfa62383, data reload: false

query1	997	398	399	398
query2	6532	1965	1941	1941
query3	6791	210	210	210
query4	26639	23477	23672	23477
query5	4330	655	521	521
query6	307	206	190	190
query7	4606	505	296	296
query8	305	261	250	250
query9	8632	2535	2531	2531
query10	486	327	261	261
query11	15726	15301	14990	14990
query12	176	111	109	109
query13	1668	513	407	407
query14	10403	6117	6342	6117
query15	210	189	178	178
query16	7646	634	489	489
query17	1195	716	582	582
query18	2024	405	311	311
query19	201	199	159	159
query20	129	122	117	117
query21	207	123	104	104
query22	4275	4396	4204	4204
query23	34550	33583	33406	33406
query24	7731	2409	2454	2409
query25	565	474	417	417
query26	1234	268	153	153
query27	2672	475	336	336
query28	4970	2453	2436	2436
query29	717	558	423	423
query30	283	224	192	192
query31	976	867	805	805
query32	74	68	70	68
query33	582	391	315	315
query34	789	871	508	508
query35	854	852	771	771
query36	985	1037	905	905
query37	120	98	75	75
query38	4192	4173	4306	4173
query39	1522	1429	1438	1429
query40	206	122	108	108
query41	54	53	54	53
query42	128	111	121	111
query43	521	546	510	510
query44	1339	812	828	812
query45	185	178	177	177
query46	867	1056	638	638
query47	1738	1766	1713	1713
query48	372	395	297	297
query49	778	499	423	423
query50	675	739	400	400
query51	4206	4179	4171	4171
query52	104	109	96	96
query53	230	255	181	181
query54	477	475	403	403
query55	83	78	87	78
query56	282	288	258	258
query57	1127	1099	1075	1075
query58	252	242	237	237
query59	2664	2690	2534	2534
query60	275	274	274	274
query61	126	122	129	122
query62	793	697	670	670
query63	223	187	185	185
query64	4262	1012	710	710
query65	4433	4345	4320	4320
query66	1053	402	313	313
query67	15630	15554	15199	15199
query68	8604	874	504	504
query69	458	297	266	266
query70	1253	1135	1126	1126
query71	502	294	263	263
query72	5272	3579	3754	3579
query73	799	747	350	350
query74	9202	9315	9127	9127
query75	3803	3151	2724	2724
query76	3632	1161	739	739
query77	757	368	284	284
query78	10193	10042	9250	9250
query79	3586	834	572	572
query80	725	535	470	470
query81	499	259	221	221
query82	738	120	93	93
query83	210	166	161	161
query84	284	98	69	69
query85	803	354	303	303
query86	401	293	278	278
query87	4418	4640	4470	4470
query88	3603	2218	2182	2182
query89	418	313	288	288
query90	1841	203	198	198
query91	144	133	109	109
query92	79	59	61	59
query93	2802	1035	568	568
query94	668	425	293	293
query95	355	273	264	264
query96	472	550	265	265
query97	3374	3384	3342	3342
query98	225	273	196	196
query99	1463	1405	1284	1284
Total cold run time: 280967 ms
Total hot run time: 186628 ms

[doris-robot]

ClickBench: Total hot run time: 31.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8bc1ae63337a1898648841b09775cdffcfa62383, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.60	0.10	0.10
query5	0.57	0.54	0.54
query6	1.17	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.54
query10	0.57	0.57	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.59
query14	2.82	2.67	2.68
query15	0.92	0.84	0.85
query16	0.37	0.38	0.36
query17	1.03	1.05	1.06
query18	0.21	0.20	0.19
query19	1.92	1.83	2.00
query20	0.01	0.01	0.01
query21	15.38	0.90	0.54
query22	0.75	1.13	0.72
query23	14.91	1.41	0.63
query24	7.26	1.07	1.03
query25	0.51	0.31	0.07
query26	0.62	0.17	0.13
query27	0.05	0.04	0.04
query28	9.42	0.89	0.43
query29	12.55	3.97	3.34
query30	0.25	0.08	0.06
query31	2.81	0.60	0.38
query32	3.23	0.55	0.47
query33	2.99	3.06	3.00
query34	15.66	5.13	4.51
query35	4.59	4.58	4.58
query36	0.67	0.49	0.49
query37	0.09	0.06	0.06
query38	0.06	0.05	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.25 s
Total hot run time: 31.14 s

[airborne12]

run buildall

[airborne12]

run buildall

[doris-robot]

TPC-H: Total hot run time: 32350 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c202c4a01dd209e21b54d9f0e38c0a78b35dccfc, data reload: false

------ Round 1 ----------------------------------
q1	17598	5232	5040	5040
q2	2057	295	170	170
q3	10410	1227	733	733
q4	10207	1025	529	529
q5	7512	2314	2349	2314
q6	186	162	133	133
q7	919	769	603	603
q8	9292	1240	1119	1119
q9	4867	4618	4788	4618
q10	6809	2311	1879	1879
q11	479	275	263	263
q12	351	347	220	220
q13	17752	3656	3105	3105
q14	238	224	212	212
q15	530	496	478	478
q16	647	612	572	572
q17	567	862	344	344
q18	6778	6376	6406	6376
q19	1211	943	540	540
q20	311	324	195	195
q21	2845	2125	1941	1941
q22	1050	1003	966	966
Total cold run time: 102616 ms
Total hot run time: 32350 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5292	5136	5158	5136
q2	240	329	239	239
q3	2122	2687	2302	2302
q4	1447	1799	1374	1374
q5	4247	4126	4154	4126
q6	219	169	126	126
q7	1909	1846	1727	1727
q8	2623	2580	2525	2525
q9	7252	7293	7152	7152
q10	3027	3259	2797	2797
q11	589	517	497	497
q12	695	766	656	656
q13	3361	3887	3339	3339
q14	277	295	274	274
q15	543	485	480	480
q16	670	694	649	649
q17	1162	1645	1304	1304
q18	7761	7554	7478	7478
q19	904	866	872	866
q20	1962	2010	1868	1868
q21	5395	4911	4805	4805
q22	1066	1016	1012	1012
Total cold run time: 52763 ms
Total hot run time: 50732 ms

[doris-robot]

TPC-DS: Total hot run time: 191833 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c202c4a01dd209e21b54d9f0e38c0a78b35dccfc, data reload: false

query1	1409	1063	1031	1031
query2	6294	2004	1974	1974
query3	11003	4362	4567	4362
query4	53244	25779	22975	22975
query5	5210	622	507	507
query6	345	201	190	190
query7	4904	491	295	295
query8	313	250	239	239
query9	5823	2529	2513	2513
query10	437	306	265	265
query11	15146	14978	15030	14978
query12	166	111	117	111
query13	1057	501	401	401
query14	10534	6362	6424	6362
query15	210	193	179	179
query16	7063	657	541	541
query17	1062	708	552	552
query18	1520	407	309	309
query19	211	189	166	166
query20	137	127	129	127
query21	220	125	102	102
query22	4555	4470	4611	4470
query23	34090	33277	33377	33277
query24	5816	2419	2466	2419
query25	460	464	395	395
query26	694	279	167	167
query27	1883	483	327	327
query28	2808	2431	2402	2402
query29	569	557	443	443
query30	274	223	195	195
query31	873	856	788	788
query32	77	65	61	61
query33	452	348	304	304
query34	742	854	503	503
query35	827	837	755	755
query36	981	1015	902	902
query37	117	96	79	79
query38	4184	4327	4092	4092
query39	1528	1447	1464	1447
query40	215	123	109	109
query41	67	57	55	55
query42	132	114	116	114
query43	511	518	506	506
query44	1319	844	826	826
query45	185	179	164	164
query46	842	1030	656	656
query47	1867	1891	1852	1852
query48	378	413	309	309
query49	673	497	425	425
query50	726	755	422	422
query51	4222	4240	4213	4213
query52	111	106	102	102
query53	236	265	198	198
query54	482	489	427	427
query55	91	88	78	78
query56	260	280	245	245
query57	1199	1210	1128	1128
query58	242	230	245	230
query59	2702	2800	2876	2800
query60	288	270	262	262
query61	126	117	117	117
query62	765	714	682	682
query63	233	192	191	191
query64	1500	1017	686	686
query65	4529	4454	4429	4429
query66	779	437	294	294
query67	15785	15609	15663	15609
query68	6690	821	502	502
query69	536	305	269	269
query70	1216	1080	1127	1080
query71	441	301	266	266
query72	5750	3626	3793	3626
query73	1247	711	349	349
query74	8960	8887	9111	8887
query75	3502	3194	2703	2703
query76	4010	1176	743	743
query77	611	372	278	278
query78	10299	9916	9295	9295
query79	2879	830	585	585
query80	717	515	449	449
query81	499	269	274	269
query82	688	128	95	95
query83	180	171	150	150
query84	290	101	73	73
query85	755	360	306	306
query86	361	311	292	292
query87	4512	4434	4321	4321
query88	3421	2252	2182	2182
query89	412	325	283	283
query90	1706	189	193	189
query91	139	133	109	109
query92	73	60	54	54
query93	2583	1055	572	572
query94	713	412	305	305
query95	344	265	258	258
query96	474	560	271	271
query97	3353	3443	3278	3278
query98	227	207	205	205
query99	1446	1427	1255	1255
Total cold run time: 296737 ms
Total hot run time: 191833 ms

[doris-robot]

ClickBench: Total hot run time: 30.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c202c4a01dd209e21b54d9f0e38c0a78b35dccfc, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.25	0.06	0.07
query4	1.61	0.11	0.10
query5	0.56	0.53	0.55
query6	1.19	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.03	0.03
query9	0.58	0.53	0.52
query10	0.56	0.59	0.59
query11	0.14	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.59	0.61
query14	2.82	2.71	2.84
query15	0.93	0.86	0.87
query16	0.38	0.38	0.38
query17	1.02	1.03	1.04
query18	0.21	0.20	0.19
query19	1.91	1.79	2.00
query20	0.01	0.01	0.01
query21	15.35	0.90	0.53
query22	0.76	1.12	0.68
query23	14.95	1.38	0.63
query24	6.76	0.70	1.27
query25	0.56	0.24	0.24
query26	0.56	0.15	0.14
query27	0.05	0.06	0.05
query28	10.08	0.91	0.44
query29	12.54	3.91	3.25
query30	0.24	0.09	0.06
query31	2.83	0.58	0.38
query32	3.23	0.55	0.47
query33	3.03	3.07	3.07
query34	15.74	5.09	4.48
query35	4.56	4.56	4.51
query36	0.65	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.51 s
Total hot run time: 30.83 s

[qidaye]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.